In December 2024, Geoffrey Hinton, one of the pioneers of artificial intelligence and a recipient of the Nobel Prize in Physics, used his Nobel banquet speech to warn about the technology he had helped develop.

Hinton recognised the enormous benefits AI could bring, including improvements in productivity, healthcare and scientific discovery. He also warned about the risks of misuse, economic inequality and the possibility of developing artificial intelligence more capable than ourselves.

His greatest concern was whether humanity would be able to remain in control of increasingly intelligent systems.

That is a serious problem. Yet there is another danger that deserves equal attention.

What happens to human capability when AI increasingly performs the work through which people develop and maintain their knowledge, skills and judgement?

The gradual erosion of human capability

Consider how people develop expertise.

An engineer learns by solving engineering problems. A doctor develops judgement through education, clinical experience and decision-making. A manager becomes more capable by investigating problems, weighing evidence and accepting responsibility for decisions.

Now suppose AI increasingly undertakes these activities.

The immediate benefits are obvious. Work can be completed more quickly, information becomes easier to obtain and many routine tasks disappear.

However, the experience previously gained through doing that work may also disappear.

The problem is particularly serious for younger people entering professions. If AI performs much of the work previously undertaken by trainees and junior staff, how will the next generation develop the expertise required to supervise it?

Existing professionals face a similar problem. Capabilities that are rarely exercised can deteriorate.

This does not mean AI inevitably makes people less intelligent. Much depends on how it is used. AI can support learning, challenge assumptions and introduce people to unfamiliar ideas.

The danger arises when assistance becomes substitution and people cease exercising the abilities they need to retain.

Who will remain capable of taking command?

Imagine an organisation in which AI systems perform increasingly complex operations with very little human intervention.

Initially, the organisation retains experienced staff who understand the work. They can identify mistakes, challenge recommendations and intervene when necessary.

As the systems improve, human intervention becomes less frequent.

Eventually, the organisation may find itself relying on people who have limited practical experience of the very activities they are supposed to supervise.

The AI may be performing exceptionally well. The weakness is now in the human command structure.

This is not an entirely new problem. Aviation and other highly automated industries have long recognised the need to maintain human competence when machines perform much of the routine work.

AI extends that problem across a much wider range of professions and activities.

Controlling what AI does

Hinton’s warning about losing control raises an immediate practical question.

How can an organisation ensure that an AI system does not take an inappropriate or unauthorised action?

In my book Orchestration and Command in the Age of AI, I propose a Four-Layer Assurance approach.

The first layer is the working AI carrying out the task.

The second is an independent checking AI examining its work.

The third is a Runtime Assurance Controller monitoring operations and enforcing predetermined limits.

The fourth is Human Command, retaining authority over consequential decisions.

An execution boundary controls whether proposed actions are permitted to proceed. Operations can be paused, flagged or rerouted when assurance requirements are not satisfied.

A record of decisions and actions provides evidence for subsequent investigation.

This is a proposed governance framework rather than a proven solution to every form of AI risk. Its purpose is to make control part of the operating system rather than relying entirely on people to recognise problems after they occur.

Maintaining the people who exercise control

Technical safeguards are only part of the answer.

My second book, Succession: AI, Human Capability and the Future of Human Command, addresses the connected problem of maintaining the human capability needed to exercise that authority.

An organisation cannot assume that placing a person in command automatically means that person is capable of making an informed decision.

Competence must be developed, maintained and demonstrated.

That requires education, experience, regular assessment and opportunities to practise without AI assistance.

It also requires organisations to think about succession. Who will replace today’s experienced professionals when the normal routes through which expertise was acquired have largely disappeared?

We need to preserve those routes or deliberately create alternatives.

Education has a responsibility

The same issues begin in schools and universities.

Banning AI is unlikely to provide a satisfactory long-term answer. Students need to learn how to use it effectively.

They also need to develop capabilities independently of it.

A course of study could combine AI-supported activities with work completed without AI assistance.

Assessment should examine the student’s contribution, including how they specify a task, investigate alternatives, challenge evidence, verify results and explain decisions.

The standard required will vary according to the student’s age and level of education.

At higher levels, students should be expected to demonstrate both effective use of AI and an independent understanding of the subject.

The objective is not simply to produce better assignments. It is to develop people capable of exercising informed judgement throughout their working lives.

From warnings to practical action

Hinton’s warnings deserve serious attention. The possibility of advanced AI escaping human control cannot be dismissed.

Neither should we overlook the more immediate changes taking place as organisations transfer increasing amounts of intellectual work to AI systems.

We need to address two connected requirements.

First, AI systems must operate within effective safeguards that preserve human authority.

Second, humans must retain the knowledge, competence and judgement needed to exercise that authority.

Meeting the first requirement without the second could leave organisations with sophisticated technical controls supervised by people who no longer understand the work.

Meeting the second without the first could leave capable people unable to intervene before an AI system takes an inappropriate action.

Both are needed.

The challenge is not merely to ensure that AI remains under human command.

It is to ensure that humans remain capable of commanding it.

Sources

Further articles and information: ai-humancapability.com

About the author

R T Scanlon is the author of three books on AI and human capability. He has worked in education, management and other practical roles, and holds an MBA with a dissertation focused on quality assurance.

Meet the author → Books on Amazon → (opens in a new tab) LinkedIn → (opens in a new tab)