Yesterday, at the DTA AI Government Showcase, I joined a panel on agentic AI and human accountability. Our discussion kept returning to a question I’ve been thinking about: what happens when human oversight is no longer primarily about catching AI mistakes?

We’ve made significant progress towards verifiability-first AI engineering (https://lnkd.in/gQsaNyUV), designing systems whose outputs can be checked by other AI systems, external tools, simulations, mathematical models or humans.

But what comes after that?

Recent AI advances in mathematics and scientific discovery suggest a future where AI can produce and verify results beyond what humans can readily comprehend. The bottleneck may increasingly be human understanding, rather than AI correctness.

And why does understanding still matter when the answer is already correct?

Because humans still need to decide what questions to ask next, what goals to pursue, which trade-offs to accept and how to govern collective choices. And sometimes, understanding is valuable simply because we want to make sense of the world.

This motivates a significant research milestone for us.

Together with Qinghua Lu and Ming Ding at CSIRO, we’ve just released Understanding-First AI Engineering.

We propose making human understanding a first-class engineering objective, with a reference architecture and architectural patterns that support understanding at both system and individual-case levels, including learning and exploration alongside increasingly autonomous AI work.

I see this as a natural progression from verifiability-first AI, and perhaps a glimpse of a more enduring stage of AI engineering.

What if the ultimate challenge is no longer making AI do the job correctly, but making sure humans can still understand, shape and find meaning in what AI achieves?

Paper on arXiv: https://arxiv.org/abs/2611.13022