The central problem. When an AI system operates reliably, its human commander intervenes less often. That appears to be evidence of success. It also removes the everyday practice through which the commander would otherwise maintain judgement, diagnostic skill and confidence. The remaining human work becomes rarer, but it is likely to be more unusual, uncertain and consequential.
A reliable AI system does not remove the need for a capable commander. It removes much of the experience that would ordinarily keep that commander capable.
The command paradox
Automation is usually introduced to reduce workload, error and delay. As it improves, routine interventions decline. The human increasingly supervises rather than performs the work. Yet the moment at which the human is most needed is likely to be the moment when the automated system encounters something outside its training, produces a convincing but unsafe answer, conflicts with another system or fails in a way that was not anticipated.
The commander must then move rapidly from long periods of observation to active judgement under pressure. If practical capability has been allowed to decay, the formal presence of a human in the loop offers little protection. Authority remains on paper, while the ability to exercise it has weakened.
This is not a new or uniquely artificial-intelligence problem. Safety-critical sectors have already learned that reliable automation must be accompanied by deliberate arrangements to preserve human competence.
Aviation shows the pattern most clearly
Modern aircraft can operate for long periods under highly capable automated control. This improves safety and consistency, but it can also reduce manual flying practice and create overreliance on automation. The United States Federal Aviation Administration states that, even when an aircraft is on autopilot, the crew must remain aware of its flightpath and able to intervene. Its flightpath-management guidance is expressly intended to help pilots maintain manual skills and avoid excessive reliance on automation.
Aviation does not wait for pilots to encounter rare catastrophic events in normal service. It uses scenario-based and simulator training to expose crews to stalls, upsets, automation surprises, unusual weather and other events they may seldom or never meet in ordinary operations. The FAA describes this as training for events that are rare but potentially catastrophic.
The lesson for AI command is direct. If the system works well for months or years, that success cannot be treated as evidence that the human commander remains capable. The organisation must create artificial opportunities to practise intervention, challenge and recovery.
Nuclear power replaces missing experience with simulation
Nuclear plants are designed so that serious incidents are extremely uncommon. Real incidents therefore cannot provide the experience required to prepare operators. The International Atomic Energy Agency requires simulator-based training across normal operation, anticipated operational occurrences and accident conditions. It recommends continuing training, regular reassessment and annual simulator exercises emphasising situations that do not occur frequently.
This is especially relevant to AI systems operating in finance, infrastructure, defence, medicine or public administration. The better the safeguards work, the fewer real failures commanders will see. Competence must therefore be exercised against simulated abnormalities, including combinations of failures rather than simple textbook faults.
Maritime operations preserve intervention competence
Ships increasingly rely on integrated bridge systems, electronic charts, radar, automatic identification and automated control. Maritime training nevertheless requires officers to understand system capabilities, limitations and potential errors. UK simulator standards require realistic emergency, hazardous and unusual conditions, with exercises recorded for assessment and debriefing.
The developing framework for remotely operated vessels makes the connection even clearer. Operators are expected to understand levels of autonomy, maintain situational awareness, recognise when an automated system has exceeded safe limits and intervene. That is effectively a model of AI command: automation performs much of the operation, but the human must retain the capability to recognise when authority must be exercised.
Medicine demonstrates automation bias and deskilling
Clinical decision-support systems can improve accuracy and efficiency, but they can also encourage users to accept automated advice without sufficient independent examination. A systematic review of automation bias found that overreliance can reduce vigilance in information seeking and processing. A 2026 systematic review of generative AI in healthcare reported recurring concerns about automation bias, clinician deskilling and erosion of diagnostic reasoning, while also recognising substantial benefits from AI support.
This matters because an AI commander must be able to distinguish a genuinely strong recommendation from a confident-looking error. If independent reasoning is not practised, verification may gradually become confirmation. The human appears to check the system but lacks the active knowledge needed to challenge it.
Cybersecurity and emergency management rehearse the unlikely
Cybersecurity teams do not assume that automated detection will recognise every attack. NIST guidance treats testing, training and exercises as necessary preparation for adverse situations, so organisations can respond, manage and recover when normal controls fail. Similarly, the World Health Organization uses simulation exercises to reveal planning weaknesses, clarify command, practise coordination and develop emergency-response skills before a real emergency occurs.
AI command requires the same discipline. Waiting for a real high-consequence AI incident to train the commander would be indefensible. The organisation must rehearse the failure before it occurs.
What the comparisons establish
| Sector | Effect of reliable automation | How capability is preserved |
|---|---|---|
| Aviation | Less manual flying and fewer encounters with abnormal events | Recurrent simulator work, scenario training and proficiency assessment |
| Nuclear power | Long periods of normal automated operation | Annual simulator exercises, accident scenarios and reassessment |
| Maritime | Dependence on integrated navigation and automated control | Emergency simulations, debriefing and intervention competence |
| Medicine | Reduced independent diagnosis and risk of automation bias | Continuing education, active verification and clinical simulation |
| Cybersecurity | Routine threats handled automatically | Tabletop exercises, red teaming and incident-response rehearsals |
| Emergency management | Major incidents occur too rarely to provide experience | Multi-agency simulations, drills and command practice |
Across these sectors, three conclusions recur: reliability reduces opportunities for real practice; the remaining human work becomes less frequent but more difficult; and competence must therefore be preserved deliberately rather than assumed.
Human Capability Assurance for AI command
Education and training cannot be a peripheral support function added after deployment. It must form part of the assurance architecture. A high-consequence AI operation should be unable to claim meaningful human command unless it can demonstrate how commanders remain competent during long periods of successful automation.
A suitable regime should include:
- initial education in the purpose, limits, failure modes and authority structure of the AI system;
- regular simulations of rare, ambiguous and rapidly developing failures;
- practice operating when the Working AI, Checking AI or Runtime Assurance Controller is wrong;
- manual or reduced-automation exercises so the commander retains an independent model of the task;
- unannounced tests of warning recognition, escalation and intervention;
- rehearsal of pause, isolation, rerouting, recovery and transfer of command;
- assessment of judgement under uncertainty, not merely procedural compliance;
- learning from near misses and incidents in other organisations and sectors;
- periodic recertification supported by competence records and remedial training.
Exercises must change as systems and threats change. Repeating predictable scenarios risks teaching commanders how to pass an exercise rather than how to command amid uncertainty. Some exercises should contain misleading signals, incomplete information, conflicting recommendations and failures that do not match the expected pattern.
Technical success is not sufficient assurance
The Four-Layer Assurance model reduces the probability that unsafe AI behaviour will reach execution. That is its purpose. But successful technical assurance can unintentionally weaken the human layer if intervention becomes so unusual that commanders lose practical capability.
The answer is not to make AI less reliable or to manufacture unnecessary human interference. It is to recognise that operational experience is no longer sufficient and deliberately replace the missing experience with education, simulation, assessment and continuing development.
Operational success must not be allowed to conceal declining command capability.
If an organisation cannot show how its AI commanders remain capable during long periods of successful automation, it cannot show that meaningful human command still exists.
About the wider work
This article develops the Human Capability Assurance argument presented in R T Scanlon’s book Succession: AI, Human Capability and the Future of Human Command. The book examines how organisations can retain capable human command as AI performs more work and human operational experience declines.
Sources
- Federal Aviation Administration. Certification Reform Efforts: Accounting for Automation, Increasing Flying Skills. https://www.faa.gov/aircraft/air_cert/airworthiness_certification/certification_reform
- Federal Aviation Administration. Advanced Qualification Program. https://www.faa.gov/training_testing/training/aqp
- Federal Aviation Administration. Pilot Training Rule: training for rare but catastrophic events. https://www.faa.gov/speeches/pilot-training-rule
- International Atomic Energy Agency. Recruitment, Qualification and Training of Personnel for Nuclear Power Plants. https://nucleus.iaea.org/sites/nss-oui/
- United Kingdom Maritime and Coastguard Agency. Small Ships Navigation and Radar Course Criteria: simulator standards. https://www.gov.uk/government/publications/small-ships-navigation-and-radar-course-criteria-2nd-edition
- National Institute of Standards and Technology. Guide to Test, Training and Exercise Programs for IT Plans and Capabilities, SP 800-84. https://www.nist.gov/publications/guide-test-training-and-exercise-programs-it-plans-and-capabilities
- World Health Organization. Simulation Exercises. https://www.who.int/emergencies/operations/simulation-exercises
- Goddard K et al. Automation bias and verification complexity: a systematic review. Journal of the American Medical Informatics Association, 2017. PubMed 27516495
- Generative Artificial Intelligence in Healthcare: Automation Bias, Deskilling, and Cognitive Implications: A Systematic Review, 2026. PubMed 42445869
Books, articles and free practical guides: ai-humancapability.com