Why Monitoring AI Agents Is Not Enough

AI agents are beginning to act across websites, databases, communications and organisational systems. Unlike an ordinary chatbot, an agent does not merely produce an answer for a person to consider. It can plan, select tools, create sub-agents and take a sequence of actions in the world.

That makes the recent call from the Partnership on AI for real-time monitoring extremely important. Its research identifies serious gaps in current agent frameworks, including weak records of agent identity, permission changes, memory changes and human approvals. Organisations may be deploying agents without being able to reconstruct clearly what they did or who authorised them.

Improved observability is therefore essential. But it is only the beginning.

Seeing an AI agent go wrong is not the same as being able to stop it.

From observation to intervention

A monitoring system may identify unusual behaviour, a breached policy or an action outside the agent's authority. If that warning merely appears on a dashboard, however, the damage may continue while a human tries to understand it. Autonomous activity can proceed at machine speed and can spread through connected systems before an operator responds.

Monitoring must therefore be connected to defined intervention powers. Depending on the seriousness and reversibility of the action, the surrounding control system should be able to:

• flag questionable activity for immediate review;
• pause an action before an irreversible step is taken;
• stop an agent that exceeds its purpose or authority;
• reroute the task to another system or a qualified human;
• isolate the agent from tools, data or external communications.

These powers must sit outside the working agent's control. A safeguard that the agent can disable, reinterpret or evade is not an adequate safeguard.

Independent checking

A working agent should not be expected to detect all of its own mistakes. The same model, assumptions and incomplete information that produced an error may also prevent it from recognising that error. Consequential work therefore needs an independent checking function examining the agent's proposed actions, evidence, compliance with instructions and developing pattern of behaviour.

Independence matters. Repeating the same request to the same system is not necessarily an independent check. The checking arrangement should be sufficiently separate to challenge the working agent rather than merely endorse it.

Traceability and the evidence record

Real-time control also needs a reliable evidence record. Each consequential action should be linked to the agent involved, its assigned purpose, the permissions in force, the tools and information used, warnings generated, checks performed and human approvals given. Changes to memory, permissions or operating conditions should also be recorded.

This record is not simply for investigating an incident afterwards. It gives the monitoring and checking systems the context required to judge whether an action is authorised and sensible while the work is still under way.

Human command must be real

The phrase human in the loop can create false reassurance. A person may technically approve an action while lacking the time, expertise, information or authority needed to challenge it. Meaningful command requires a competent and alert person who understands the objective, recognises the limits of the system and has the power to intervene.

The human commander's decisions must also be recorded. If a warning was ignored, a safeguard overridden or an agent given wider authority, the evidence chain should show who made that decision and under what authority.

A complete operating architecture

Safe agent operation therefore requires more than a stream of telemetry. It requires an operating architecture in which:

• a working AI performs the assigned task;
• an independent checking AI examines its work;
• a runtime assurance controller monitors activity and can flag, pause, stop or reroute it;
• a competent human commander retains authority and responsibility;
• a protected evidence record connects purpose, actions, warnings, interventions and decisions.

Observability tells us what is happening. Checking assesses whether it should be happening. Intervention prevents unsafe activity from continuing. Human command determines who has authority and responsibility.

Monitoring is valuable only when somebody - or something - can act on what it reveals.

This article develops ideas explored in Orchestration and Command in the Age of AI.

For further information: ai-humancapability.com