← All insights ← →

The Danger of Automation Bias: Is Human-in-the-Loop an Illusion?

Published: 10 March 2026

The Danger of Automation Bias: Is Human-in-the-Loop an Illusion?

A human looks at it. So everything is fine? That assumption is dangerous. It is widespread. And, at least in complex systems, it is wrong.

The European Data Protection Supervisor (EDPS) has systematically taken apart the common assumptions about human oversight of automated decision-making (ADM) systems in its latest TechDispatch. The finding should make every board and every executive team sit up: the mere presence of a human in the decision process protects against neither errors nor liability. We often start from the false premise that systems do not influence people. That assumption is wrong.

The human-in-the-loop illusion

The underlying problem is structural. When a system issues a recommendation and a human confirms it, the impression of a human decision is created. In reality, the decision was made long before. The human merely rubber-stamps it.

Automation bias is not a fringe phenomenon. A study in radiology showed that radiologists make worse decisions when they receive flawed AI recommendations, regardless of their experience. The machine influences human judgement even when the human formally has the final word.

The EDPS puts it plainly: when an operator accepts a system recommendation primarily because it came from an automated system, the decision process becomes, de facto, an automated one. The human in the loop is then nothing more than decoration.

The real question: where does complexity outstrip human oversight?

Instead of relying on the false comfort of human-in-the-loop, we need to ask a more realistic question: for which decisions can we still actually grasp the complexity of the system? And wherever we cannot, we need to be very careful.

Modern ADM systems make decisions based on billions of parameters. No human can check in real time whether a large language model or a neural network is drawing the right conclusions. Lisanne Bainbridge described this contradiction as early as 1983 in her essay “Ironies of Automation”: the system was deployed because it does the task better than a human. Yet the human is meant to monitor whether it works correctly. That is an impossible mandate.

The consequence is uncomfortable, but clear: wherever the complexity of a decision outstrips human reviewability, no ADM system may decide autonomously. Instead, the human must make the decision consciously, without bias, and with enough time, as an independent judgement. They must not simply rubber-stamp a machine recommendation.

That requires a clear triage: which decisions can we sensibly automate? Which ones need genuine human judgement? And for which is the complexity so high that neither human nor machine alone can decide responsibly?

The examples are not new, but the pattern still is, even with AI

Some of the most prominent cases go back years. That does not make them less relevant. On the contrary: they show a systemic pattern that keeps repeating itself.

Boeing 737 MAX (2018/2019): The MCAS system (Manoeuvring Characteristics Augmentation System) automatically pushed the aircraft’s nose down. Pilots did not understand the behaviour, received inadequate training, and had no adequate tools to intervene. 346 people died. The assumption that human and machine complement each other seamlessly proved a fatal error.

Air France 447 (2009): After the airspeed sensors failed, the autopilot disengaged. Pilots had to take over manually, immediately, in a crisis. Research shows that this exact handoff scenario is precisely what humans handle worst. In this specific case, Air France subsequently overhauled pilot training: more hands-on flying experience, less “system operator” work.

Uber (2018): The safety driver of an autonomous vehicle was distracted. The system failed to recognise the pedestrian in time. Human oversight failed because the system design did not ensure that the human actually stayed alert.

These cases are ten, fifteen years old. The pattern is unchanged: excessive trust in automation, inadequate interfaces, and the assumption that a human will step in when it matters.

The liability risk boards underestimate

The regulatory landscape is increasingly shifting responsibility towards those who deploy ADM systems. The EU AI Act, the GDPR (Art. 22), NIS2, and sector-specific regulation create concrete duties for deployers and their governing bodies.

What does that mean for liability?

Anyone who approves an ADM system and relies on “human-in-the-loop” as a safeguard must be able to prove that this oversight is effective. Not symbolically. Not formalistically. Effectively. The EDPS is clear: sham oversight does not protect against liability. On the contrary, it can be judged negligent if it can be shown that the conditions for genuine oversight were never in place.

Three concrete risks follow for boards:

Organisational fault: if a company deploys ADM systems without giving its operators the necessary training, time, and authority to intervene, the organisation is liable. Australia’s Robodebt system is a cautionary example: staff theoretically had the power to override automated debt notices. Management rewarded throughput, not accuracy. The result was systemic injustice.

Failure of due diligence in system procurement: anyone who procures an ADM system without examining its limits, explainability, and override mechanisms is in breach of their duty of care. IBM Watson for Oncology recommended unsafe treatments in certain cases. The assumption that doctors would simply catch the errors proved wrong, because they lacked the time, the tools, and the system understanding to do so.

Reversal of the burden of proof for rights violations: under the EU AI Act and the GDPR, deployers bear the burden of proving that their systems operate in line with fundamental rights. Anyone relying on human oversight must demonstrate that this oversight actually works. Audits, documentation, and empirical tests of oversight quality become mandatory, not optional.

Possible measures

The EDPS recommends a package of measures that goes beyond technical fixes:

  • Organisational: fair working conditions for operators. Enough time for review. Treat human oversight as a genuine safety measure, not a liability buffer. Training that also shows system errors, because direct exposure to failure has been shown to reduce automation bias.
  • Technical: where possible, use interpretable models instead of black boxes. Interfaces must encourage critical thinking, not quick confirmation. Even the colour scheme of an interface can nudge an operator towards premature agreement.
  • Structural: four-eyes principle. Regular external auditing. Awareness checkpoints. And a concept the EDPS calls “institutionalised distrust”: a duty to justify every ADM system, rotation of oversight mandates, collective decision-making, and full transparency about oversight practice.

The central insight

Human oversight cannot compensate for systemic flaws in ADM systems. It is not a safety net for faulty algorithms.

The question for boards and the C-suite is not whether a human is in the process. The question has three parts:

  • Can we still actually grasp the complexity of the decision? If not, no system may decide autonomously.
  • Does the human in the process have the competence, authority, time, and the right incentives to actually intervene? If not, the oversight is ineffective.
  • Can we demonstrate the effectiveness of the oversight? If not, we are standing on thin ice, both regulatory and in terms of liability.

Without this clarity, human-in-the-loop is nothing more than an inadequate formality. And a formality protects neither fundamental rights nor a board’s liability. Two things are essential. The systems must be understood in their functions and decisions. And the corresponding safeguards and guardrails must be in place, and their function must be reviewed regularly, so that control can be handed back to the human once the system reaches its limits.

References and further sources

Main article: TechDispatch – Human Oversight of Automated Decision-Making. European Data Protection Supervisor, 2025