The presence of a human decision-maker does not automatically guarantee meaningful human oversight. If an AI system supplies incomplete or inaccurate evidence, the human may unknowingly rely upon it.

The Home Office says that trained caseworkers—not AI—make decisions on asylum claims. AI tools are used only to summarise interviews and help staff search policy information.

That appears to place a human firmly in control. But is the presence of a human decision-maker sufficient?

Open Rights Group has raised concerns about two AI tools used in asylum casework. It reports that a Home Office pilot found 9% of AI-generated case summaries contained inaccuracies or missing information serious enough for them to be removed. It also says that only 0.3% of summaries generated between April and July were checked by subject specialists and that no independent audit is planned.

The Home Office maintains that the tools are monitored and supported by existing quality-assurance processes.

The problem with human oversight

A caseworker may make the final decision, but that decision can still be influenced by an inaccurate or incomplete AI-generated summary.

If the human does not know that important information has been omitted, there may be nothing obvious to question. The person remains formally responsible while relying on evidence shaped by the AI.

This is why having a human in the process is not necessarily the same as having a human meaningfully in command.

Real human command requires:

  • access to the original evidence;
  • clear warnings about uncertainty and possible omissions;
  • independent checking of important AI outputs;
  • the authority and time to challenge the system; and
  • a record showing how the final decision was reached.

A layered approach

For consequential decisions, one safeguard is not enough.

  1. The working AI, which produces the summary or recommendation.
  2. An independent checking system, which compares the output with the original evidence.
  3. A runtime assurance controller, which flags, pauses or rejects outputs that fail defined tests.
  4. A trained human commander, which sees the evidence, understands the limitations and retains final authority.

The human should not merely approve what appears on the screen. The system must give that person the evidence and support needed to exercise informed judgement.

This principle extends beyond asylum decisions. It applies to healthcare, finance, recruitment, policing and any organisation using AI to inform decisions that materially affect people’s lives.

The important question is not simply: “Was a human involved?”

It is: “Did the human have the information, competence, time and authority required to remain genuinely in command?”

Sources

For further information: ai-humancapability.com