Throughout history, humanity has developed increasingly powerful machines, many capable of causing serious harm. We have not generally responded by abandoning technological progress. Instead, we have learned to introduce safeguards, independent controls and procedures designed to prevent accidents.

Sometimes those safeguards were incorporated before a technology entered service. Frequently, they were introduced only after serious accidents demonstrated the dangers.

With increasingly autonomous artificial intelligence, we should surely be applying these lessons before the consequences become irreversible.

From Circular Saws to Passenger Lifts

Early industrial machinery often operated with exposed moving parts. Circular saw guards, emergency stops and other protections became increasingly important as experience demonstrated the dangers.

But there are examples of safeguards being built in from the beginning of a particular application.

In 1854, Elisha Otis demonstrated a safety mechanism that prevented a lift from falling when its supporting rope was cut. His first commercial passenger lift followed in 1857.

The lesson was simple: it was possible to develop an important technology while also addressing a foreseeable danger before exposing passengers to it.

Machines That Make Their Own Decisions

The comparisons become even more relevant when we examine machines capable of making important decisions independently.

Modern driverless trains automatically accelerate, brake and stop. Their operation is constrained by automatic train protection systems that supervise movement and intervene when safety limits are exceeded.

Nuclear power stations employ automatic reactor protection systems capable of initiating emergency shutdowns without waiting for human approval.

These systems demonstrate an established engineering principle: the equipment performing an operation should not necessarily have the final authority over whether that operation is safe.

Autonomous Weapons: An Even Closer Comparison

For decades, military organisations have operated weapons capable of making potentially lethal decisions without modern artificial intelligence.

The American Phalanx close-in weapon system, introduced in 1980, can automatically detect, evaluate, track and engage incoming threats using radar and programmed control systems.

Human commanders authorise its operating modes, but once operating autonomously, the system can respond faster than a person could assess and approve each individual engagement.

This introduces a fundamental question: how much authority should a machine possess, and what safeguards must accompany that authority?

History also provides warnings.

During the 2003 Iraq War, Patriot air-defence systems were involved in fatal friendly-fire incidents. These demonstrated the dangers of incorrect threat identification, automated engagement and inadequate human oversight.

Autonomy is therefore not a new problem. Neither is the need for independent protection.

The Boeing 737 MAX: When Human Control Was Not Enough

The Boeing 737 MAX accidents of 2018 and 2019 provide another important lesson.

The aircraft's original MCAS system could issue repeated nose-down stabiliser commands based on incorrect information from a single sensor.

Although pilots retained formal responsibility for controlling the aircraft, the interaction between automation, faulty information and the available means of intervention contributed to catastrophic consequences.

Following the accidents, the system was redesigned to compare information from two sensors and restrict repeated activation.

The lesson is that having a human nominally in command does not necessarily mean that the human can exercise effective control.

What Does This Mean for Artificial Intelligence?

Artificial intelligence introduces new possibilities because systems may become increasingly capable, interconnected and autonomous.

A relatively simple AI performing an isolated task may require little additional protection.

But when AI systems are connected into multi-process chains, their decisions can influence other systems, create further actions and potentially amplify errors.

As autonomy increases, the opportunities for human intervention may decrease.

And where AI contributes to developing more capable successor systems, the problem becomes more serious still.

Can human expertise and independent checking develop quickly enough to keep pace?

We should distinguish between controlling the development of AI and controlling its operation. Both matter.

My books Orchestration and Command in the Age of AI and Succession: AI, Human Capability and the Future of Human Command explore practical arrangements for maintaining independent checking, runtime assurance and meaningful human command.

These principles draw on lessons already established in other safety-critical industries.

Progress With Protection

The objective should not be to prevent the development of increasingly capable AI.

Human progress has always involved developing new tools, extending our capabilities and learning to manage the risks.

But the safeguards must be proportionate to the capabilities, complexity, autonomy and potential consequences of failure.

An independent protection system must possess the authority to intervene. Human commanders must retain the competence needed to exercise meaningful judgement. And the systems responsible for checking AI must not simply reproduce the errors of the systems they supervise.

If engineers already recognise the need for independent protection in comparatively limited autonomous machines, what justification is there for deploying increasingly powerful autonomous AI without equivalent—or stronger—assurance?

We have learned these lessons in aviation, nuclear engineering, transport and military operations.

Surely we should apply them to artificial intelligence before an avoidable disaster forces us to.

Sources

For further information: ai-humancapability.com

About the author

R T Scanlon is the author of three books on AI and human capability. He has worked in education, management and other practical roles, and holds an MBA with a dissertation focused on quality assurance.

Meet the author → Books on Amazon → (opens in a new tab) LinkedIn → (opens in a new tab)