🧠 “𝗠𝗲𝗮𝗻𝗶𝗻𝗴𝗳𝘂𝗹 𝗵𝘂𝗺𝗮𝗻 𝗼𝘃𝗲𝗿𝘀𝗶𝗴𝗵𝘁” is everyone’s magic bullet—but nobody seems to know the magic sauce.
I opened Day 2 of the AI in Government Conference with a keynote on some of the hardest questions in AI-enabled decision-making:

• 𝘋𝘰 𝘩𝘶𝘮𝘢𝘯𝘴 𝘳𝘦𝘢𝘭𝘭𝘺 𝘩𝘢𝘷𝘦 𝘢𝘨𝘦𝘯𝘤𝘺—𝘰𝘳 𝘫𝘶𝘴𝘵 𝘵𝘩𝘦 𝘪𝘭𝘭𝘶𝘴𝘪𝘰𝘯 𝘰𝘧 𝘪𝘵, 𝘢𝘧𝘵𝘦𝘳 𝘵𝘩𝘦 𝘈𝘐 𝘩𝘢𝘴 𝘢𝘭𝘳𝘦𝘢𝘥𝘺 𝘴𝘩𝘢𝘱𝘦𝘥 𝘮𝘰𝘴𝘵 𝘰𝘧 𝘵𝘩𝘦 𝘥𝘦𝘤𝘪𝘴𝘪𝘰𝘯?
• 𝘈𝘳𝘦 𝘸𝘦 𝘮𝘪𝘴𝘵𝘢𝘬𝘪𝘯𝘨 𝘵𝘩𝘦 𝘱𝘳𝘦𝘴𝘦𝘯𝘤𝘦 𝘰𝘧 𝘢 𝘩𝘶𝘮𝘢𝘯 𝘧𝘰𝘳 𝘦𝘧𝘧𝘦𝘤𝘵𝘪𝘷𝘦 𝘰𝘷𝘦𝘳𝘴𝘪𝘨𝘩𝘵?
• 𝘊𝘢𝘯 𝘸𝘦 𝘵𝘳𝘶𝘴𝘵 𝘈𝘐 𝘳𝘢𝘵𝘪𝘰𝘯𝘢𝘭𝘦𝘴 𝘪𝘧 𝘸𝘦 𝘤𝘢𝘯’𝘵 𝘵𝘳𝘢𝘤𝘦 𝘰𝘳 𝘵𝘦𝘴𝘵 𝘵𝘩𝘦𝘪𝘳 𝘳𝘦𝘢𝘴𝘰𝘯𝘪𝘯𝘨?

We need to shift the focus from symbolic and procedural fixes to measurable effectiveness. I introduced some of the recent innovations from CSIRO’s Data61:

1. Metrics, methods and evaluation infrastructure to measure how different forms of oversight actually influence outcomes
2. Marginal risk assessment techniques to evaluate AI performance and risk without requiring ground truth
3. Methods to evaluate the faithfulness of AI rationales against external criteria, not just the AI’s internal workings

What’s your view? 𝙃𝙤𝙬 𝙨𝙝𝙤𝙪𝙡𝙙 𝙬𝙚 𝙢𝙚𝙖𝙨𝙪𝙧𝙚 𝙖𝙣𝙙 𝙚𝙫𝙖𝙡𝙪𝙖𝙩𝙚 𝙬𝙝𝙚𝙩𝙝𝙚𝙧 𝙝𝙪𝙢𝙖𝙣 𝙤𝙫𝙚𝙧𝙨𝙞𝙜𝙝𝙩 𝙞𝙨 𝙢𝙤𝙧𝙚 𝙩𝙝𝙖𝙣 𝙟𝙪𝙨𝙩 𝙩𝙝𝙚𝙖𝙩𝙧𝙚?

Slides: https://www.linkedin.com/feed/update/urn:li:activity:7343742258036281344/