Congratulations to CSIRO‘s Qinghua Lu on launching the AI and Software Engineering column in IEEE Software, the leading magazine for software practitioners. It’s a daring thing to take on at what may be the most interesting moment in the history of the field and the profession.

It was a pleasure to co-author the inaugural article, “AI Engineering: Engineering Intelligence, Engineering Trustworthiness”, out now in the latest issue.

In the article, we challenge ourselves with a deliberately extreme thought experiment. Imagine a future in which AI does the work better than we do. It comes up with more creative solutions, exercises better technical judgement, and verifies its own work and catches mistakes more reliably than we can. Even then, expert understanding how these systems work and what their actions mean for organisations and society would matter in its own right.

We see two related directions for AI engineering. Engineering intelligence means choosing where capabilities, intelligence and decisions should sit. When a model’s reasoning is hard to inspect, we may choose to give up some efficiency in exchange for safeguards outside the model that we can understand, test and enforce.

Engineering trustworthiness means designing systems that produce evidence we can understand and that faithfully reflects what they do. Even if AI designs its own harness and tools, what happens in practice depends on how those pieces interact with the model and with the people and organisations around it. Understanding those interactions helps us judge what our controls actually cover and what evidence would justify relying on the system.

Understanding is a state of mind you can’t outsource. Organisations, too, need to develop their own understanding of the systems they rely on to be accountable.

We’ve often learnt how things work by building and fixing them ourselves. As we delegate more of that work, we can design systems that let us understand what they did and probe how they might behave under different conditions.

Curiosity alone is reason enough to want to understand. Understanding also helps us judge whether the results serve our purposes and decide what to ask for next. As AI increasingly acts on behalf of people and shapes the lives of many others, being able to understand and question what it does becomes the essence of our job

Read the preprint

The author-supplied preprint is available below. If the viewer is unavailable on your device, use the link to open or download the PDF.