It was a pleasure to deliver a talk last week at The Future Generation Enterprise Architecture (FGEA) Symposium hosted by ACS (Australian Computer Society) and organised by Asif Gill.
I focused on data trust at scale, challenging some conventional views on data quality in distributed, at-scale, AI-amenable environments. Some lessons come from CSIRO’s Data61 technology powering data.gov.au, the early years of hosting the Data Standards Body for Australia’s Consumer Data Right (e.g. open banking), and cross-organisation/border/supply chain data flow projects, not to mention the science and tech development for data/AI safety. Here are a few key points and selected slides:
𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐃𝐚𝐭𝐚 – 𝐁𝐞𝐲𝐨𝐧𝐝 𝐁𝐢𝐚𝐬 𝐚𝐧𝐝 𝐑𝐞𝐩𝐫𝐞𝐬𝐞𝐧𝐭𝐚𝐭𝐢𝐨𝐧
* Distributed Trust: Ensuring data integrity across various organisations or on-device data without direct oversight.
* Zero Trust: Unsupervised learning from online wild data is susceptible to data poisoning attacks and can’t be easily cleaned up.
* Trusted “License”: It’s about data rights and value redistribution, not just ownership or copyright.
* Trusted Artificial: Synthetic data can be useful, but when should we trust it?
𝐓𝐞𝐬𝐭𝐢𝐧𝐠/𝐕𝐚𝐥𝐢𝐝𝐚𝐭𝐢𝐨𝐧 𝐃𝐚𝐭𝐚 – 𝐁𝐞𝐲𝐨𝐧𝐝 𝐇𝐮𝐦𝐚𝐧 𝐅𝐞𝐞𝐝𝐛𝐚𝐜𝐤 𝐚𝐬 𝐭𝐡𝐞 𝐆𝐨𝐥𝐝 𝐒𝐭𝐚𝐧𝐝𝐚𝐫𝐝
* Trust in Evaluation Data: Accidental data leaks can invalidate testing outcomes, so tread carefully.
* Trust in Human Feedback: Often unreliable, using human feedback necessitates nuanced evaluation.
𝐒𝐲𝐬𝐭𝐞𝐦-𝐋𝐞𝐯𝐞𝐥 𝐃𝐚𝐭𝐚 – 𝐁𝐞𝐲𝐨𝐧𝐝 𝐌𝐨𝐝𝐞𝐥 𝐃𝐚𝐭𝐚
* Trusted Knowledge: It’s not just about training data. Addressing inconsistencies within your inference-time data sources and between external knowledge and AI knowledge is crucial.
* Trusted Trade-off: It’s never about single-dimension optimisation. Balancing privacy, fairness, and accuracy requires stakeholder involvement before and after deployment in context.
* Trusted Provenance: Ensuring data provenance throughout its lifecycle is essential to combat low-quality decision and misinformation.
Slides – see LinkedIn post: https://www.linkedin.com/feed/update/urn:li:activity:7214394551518519296/
