New paper alert (working draft)!
“When you really understand what AI is doing, it’s no longer AI — it’s just boring automation.”
That old AI community joke captures the paradox of human oversight. We trust calculators and complex business process engines with millions of if–else rules, running hundreds of steps, responding to environmental changes, and actuating effects — all unsupervised — because we understand them. Yet we hesitate when an LLM performs a simple multiplication or when an agent takes just three steps on its own — not because the tasks are harder, but because their inner working cannot be fully grasped. If we forced complete understanding, we would strip away what makes these AI “intelligent” in the first place, collapsing them into boring automation and, perhaps more controversially, into the limits of human intelligence itself.
This is the oversight dilemma: humans must supervise without redoing the work themselves, and they must remain accountable for systems whose inner workings they cannot, and should not, fully understand.
Our paper proposes a way forward with three contributions:
- Operative vs Evaluative Agency: AI retains operative agency to do the work, while humans exercise evaluative agency to judge, steer, and substitute.
- Catalogue of oversight mechanisms: structured rationales, provenance capture, abstain triggers, appeal bundles, drift monitors, and more.
- End-to-end oversight patterns: practical designs for recurring oversight problems in evaluation, conformance, synthesis, and prioritisation.
For me, this is one of the most exciting lines of work we’re pursuing. Oversight is not just about safety — it is about designing the future of jobs and the role of humanity in systems where AI operates with autonomy while humans retain meaningful oversight. How we design these interfaces will decide whether we remain active partners or become passive bystanders.
https://www.researchgate.net/publication/395540553_Designing_Meaningful_Human_Oversight_in_AI
