When AI Goes Wrong, Who Is Responsible?

When an AI system causes harm, the first explanation may be that the AI made a mistake or behaved unexpectedly. That may describe the immediate technical event, but it does not answer the central accountability question: who was responsible for putting the system in that position and for controlling its use?

An AI can malfunction, misinterpret an instruction, exceed its assigned authority or produce a harmful result. It cannot presently carry moral or legal responsibility in the way a person or organisation can. Responsibility remains with the humans and organisations that designed, supplied, selected, configured, authorised, supervised and operated it.

Trace the AI to understand the failure. Trace the humans to establish responsibility.

Two different investigations

After an incident, investigators need to answer two related but different questions.

First, what did the AI do? The technical record should reconstruct its inputs, permissions, tool use, outputs, warnings and actions. That evidence can reveal the failure mechanism, show why safeguards did not work and help prevent repetition.

Second, what did the humans do? The accountability record should show who established the purpose, selected the system, granted its permissions, approved its deployment, received its warnings, checked its work and authorised or failed to prevent the consequential action.

Trying to reconstruct every internal mathematical operation of a model may be impossible and may not establish responsibility. The displayed reasoning of an AI may be incomplete or produced after the underlying decision. Human authority and decisions, however, can and should be recorded.

The human command chain

A defensible traceability system should record:

• who commissioned the task and defined its intended outcome;
• who chose the model, tools and operating arrangement;
• who granted access to data, communications, money or machinery;
• who set or changed permissions and stopping conditions;
• who received each warning and what response followed;
• who checked the output and what evidence shows that the review was meaningful;
• who approved, rejected or overrode the final action;
• which person or organisation retained ultimate command responsibility.

This prevents a damaging but convenient defence: the AI did it and nobody knows why. Even when the system behaves unexpectedly, the record should reveal which human decisions created the conditions in which that behaviour could produce harm.

Responsibility may be shared

Accountability will not always rest with the person nearest the machine. Responsibility may be distributed among several parties:

• a developer may have created a foreseeable defect or inadequate safeguard;
• a supplier may have exaggerated reliability or concealed limitations;
• senior leaders may have authorised deployment without adequate assurance;
• system designers may have provided excessive permissions or weak controls;
• operators may have misused the system or ignored clear instructions;
• human commanders may have disregarded warnings or approved unsafe action;
• management may have failed to provide competent staff, sufficient time or fatigue protection.

Traceability must therefore follow the command chain upwards as well as the operational chain downwards. A junior operator should not become the automatic scapegoat for unsafe systems, unrealistic workloads or decisions imposed by senior management.

AI evidence is still essential

Prioritising human accountability does not make the AI record unimportant. We still need to know which agent acted, what authority it had, which data and tools it used, whether its permissions or memory changed, what checks were performed and how the outcome developed.

But the purpose of that evidence is to understand and correct the technical failure and to connect it to the responsible human decisions. It should not become a means of transferring blame from people and organisations onto a machine.

Responsibility must remain attached to authority

The safest principle is that responsibility follows authority. Anyone who authorises a system to act must remain accountable for the adequacy of its purpose, permissions, supervision and safeguards. Delegating a task to AI does not delegate away responsibility for the outcome.

An AI may be the immediate cause of an incident. Human beings and organisations remain responsible for putting it in a position to cause that incident and for the quality of the controls surrounding it.

AI should never become the gap into which human responsibility disappears.

This article develops ideas explored in Orchestration and Command in the Age of AI and Succession: AI, Human Capability and the Future of Human Command.

For further information: ai-humancapability.com