When a Human Is in the Loop, Are They Really in Command?

Military use of artificial intelligence provides an unusually stark test of a question that applies far beyond defence: is it enough to place a human somewhere in an AI decision process, or must that person retain the knowledge, judgement, time and authority required to exercise meaningful command?

From human control to meaningful human command

The distinction is becoming explicit in current research and policy. In May 2026, a perspective in Nature Machine Intelligence argued that human control is a central principle in military AI governance but that the term itself remains unclear. The authors argue that safeguarding control requires attention to human-machine interaction throughout the AI life cycle, including research, development, testing, evaluation, validation and verification.

An April 2026 research paper went further, proposing the term “meaningful human command” for military human-robot interaction. That wording is important. Control can describe a formal arrangement: a person has an approval button, occupies a supervisory position or is legally accountable. Command implies something stronger: the human must be capable of understanding the situation sufficiently to direct, question and, when necessary, overrule the machine.

Policy is trying to preserve the human role

The UK Ministry of Defence stated in July 2026 that human responsibility and accountability must always be retained in decisions about the use of force. It said weapons that identify, select and attack targets require context-appropriate human involvement, which may include real-time supervision or control exercised through operational parameters.

In the United States, lawmakers introduced the Human Authority over Autonomous Weapons Act in July 2026, seeking to ensure that intentionally lethal use of autonomous or AI-enabled weapons remains subject to human oversight or approval. These initiatives show that retaining a human role is already a live governance issue rather than a hypothetical future concern.

The Pentagon–Anthropic dispute exposes another problem

The 2026 dispute between the Pentagon and Anthropic highlighted a different aspect of command: who determines the boundaries of an AI system's use when governments depend on commercial technology providers?

The BBC Radio 4 programme The Artificial Human examined the dispute in its two-part “AI: War Machine?” investigation. Its second episode asked whether military AI governance was keeping pace or whether responsibility was effectively being left to technology makers. The programme description noted that commercial contracts were helping to define military AI use while detailed policy, regulation and legal frameworks were still developing.

The dispute later became a major legal confrontation. Reuters reported that Anthropic resisted uses involving autonomous weapons and domestic surveillance, and that a US federal judge subsequently blocked the Pentagon's attempt to blacklist the company. Whatever view is taken of the contractual dispute, it demonstrates that questions about AI capability, permitted use, reliability and human authority are no longer theoretical.

The technology is also moving from ‘in the loop’ to ‘on the loop’

The operational relationship between human and machine is changing as well. In July 2026, DARPA reported live-flight testing of an AI-controlled F-16 using the VENOM autonomy system. A pilot can switch between traditional human control and AI control, while the programme describes the testing as “human-on-the-loop”. DARPA says the work is intended ultimately to allow human pilots to command and orchestrate teams of autonomous uncrewed aircraft.

That is a useful illustration of the wider transition. The human does not necessarily perform each task. Increasingly, the human may command systems that perform tasks autonomously. The quality of human command therefore depends less on manual execution and more on understanding capabilities, limitations, objectives, boundaries and abnormal situations.

The problem of AI speed

A further difficulty is emerging around decision speed. A September 2026 open-access conference paper argues that AI-enabled military decision-support systems can marginalise human agency even when humans formally retain authority. Time pressure, cognitive limits and opaque systems can combine to make human approval less meaningful. The authors propose training, human-centred design and clearly distributed responsibility as ways of resisting this erosion.

This is particularly important because automation bias can turn human oversight into little more than confirmation. A person who lacks time, confidence or sufficient independent expertise may approve the AI's recommendation without genuinely evaluating it.

Human Capability Assurance

These developments suggest a principle that can be applied to organisations generally. Governance structures alone cannot guarantee meaningful human command. The organisation must also maintain the human capability required to make those structures work.

Human Capability Assurance can be defined as: the systematic identification, maintenance and testing of the human knowledge, skills and judgement an organisation needs in order to supervise, challenge and, where necessary, operate independently of its AI systems.

In military settings the stakes are exceptional, but the underlying management problem is familiar. A hospital, bank, engineering company, law firm or public authority can all retain a nominal human decision-maker while allowing the underlying human expertise to weaken.

This leads to a useful distinction: putting a human in the loop is a procedural safeguard. Ensuring that the human has the capability to exercise independent judgement is a command safeguard.

What organisations can learn from the military debate

• Define what decisions must remain genuinely human, not merely formally approved by a human.
• Identify the knowledge and judgement needed to challenge the AI rather than simply operate it.
• Train for exceptions, failures and situations in which AI recommendations should be rejected.
• Test whether people can still exercise independent judgement under realistic time pressure.
• Preserve clear authority to stop, override or constrain the system.
• Review whether automation is gradually removing the experience through which future supervisors acquire expertise.

Military AI makes the issue unusually visible because failures can have immediate and grave consequences. Elsewhere, the erosion may be slower and easier to overlook. But the underlying question is the same: when an organisation says that a human remains in command, does that person still possess the capability to command?

For further information: ai-humancapability.com