The Duty-of-Care Trap: When the Board Must Supervise What It Cannot Inspect

Published: 7 April 2026

Article series: Limits of delegating to AI systems (Part 2)

In my previous article (Autonomous AI in Operations: The Accountability Questions the Board Must Resolve), I described the responsibility gap that arises once tasks are delegated to learning systems: future behaviour can no longer be predicted in the same sense as with classical software.

This article goes a step further: can the board supervise execution when a system’s decision logic cannot be directly and fully audited?

What law and ethics quietly assume

Art. 716a para. 1 no. 5 OR requires the board to supervise management. The EU AI Act requires human oversight for high-risk systems, including the ability to override and shut them down. And Floridi (2021) describes meta-autonomy as ‘decide-to-delegate’: humans should decide what they delegate, and must in principle be able to reclaim any delegation.

All three perspectives rest on the same silent assumption: whoever supervises must understand, or at least reliably follow, what the system does.

Human oversight: ‘being able to explain’ is part of the mandate

When a company delegates to a CEO or an executive team, oversight is possible in principle because the decision process can be questioned:

Which options were considered? What assumptions were made?
Why was this decision reached?

This traceability is no guarantee of quality. But it is possible in principle. Classical delegation logic in company law rests on exactly this.

Why this logic breaks down for many AI systems

Learning systems, particularly neural networks and reinforcement learning, often lack precisely this kind of inspectability. Matthias (2004) puts it precisely: such systems have no ‘catalogue’ of learned rules that can be read out and checked the way symbolic software can. The information they hold can only be inferred from behaviour.

The consequence for the board matters: outputs can be tested. Deviations can be measured. Within limits, explanation methods can be used, such as feature attribution (which shows which input factors most strongly influenced a model’s output) or surrogate models (a simplified ‘stand-in model’ that approximates the behaviour of the complex model to make it easier to understand). But the decision path cannot be fully audited the way it can with humans or rule-based systems. On top of this, complex learning systems can display behaviour that is not predictable at the component level.

In practice, a further factor comes into play: even where more insight would be theoretically possible, proprietary models, closed APIs and ongoing vendor updates often prevent genuine transparency across the whole supply chain. For many boards, the main reason for the lack of insight is not the mathematics. It is the supply chain.

The paradox

This creates the oversight paradox: the board has a duty of supervision, but in many cases the object of supervision can only be checked indirectly.

This is not a question of competence or budget. It is a combination of technology architecture and supply chain reality. It affects AI that decides on its own. It equally affects AI that prepares, prioritises or justifies decisions. The distinction between ‘delegating decision-making authority’ and ‘using a tool in decision preparation’ changes little about the oversight problem: in both cases, a process that cannot be fully traced feeds into the basis for the decision.

Not a new problem

Other industries know this pattern: pilots supervise systems they cannot follow in full detail. The difference does not lie in the problem. It lies in the answer. Safety-critical industries have spent years building standardised substitute mechanisms: certification chains, defined operating envelopes (safety cases), redundancy requirements and independent audit bodies.

In corporate governance, such standard mechanisms for AI are still missing in many companies. Laws and regulations offer no ready-made toolkit here and lag well behind technical reality. The board must therefore build this out concretely within its governance framework, to protect the organisation and itself.

What the board must actually do: proxy oversight instead of an illusion of transparency

The paradox cannot be resolved. But it can be made manageable if oversight is designed as a system.

Classify before delegating: not all AI is equally opaque. The board should require every system to be classified, for example: rule-based systems and decision trees with high traceability, statistical and ML models with medium traceability, deep learning and autonomously acting systems with low traceability. This classification is the basis for any meaningful oversight decision.
Make proxy mechanisms binding: where direct insight is not possible, documented substitute mechanisms are needed: output monitoring with clear KRIs such as error rates, drift indicators and escalation thresholds; statistical validation and robustness testing; regular red-teaming (targeted attack testing to expose vulnerabilities, misbehaviour and misuse scenarios); independent model reviews by internally separated or external bodies.
Enforce effective engagement, not just ‘human-in-the-loop’ on paper: evidence shows that opacity increases uncertainty and that specialists often respond by blindly accepting or ignoring outputs rather than integrating them effectively. Effective use only emerges once organisations establish dedicated routines to actively review and assess AI outputs. For the board, this means human oversight is a design problem spanning roles, competence, time and authority, not merely a process checkbox.
Establish demonstrability: oversight must be provable. Who checked what? Which deviation triggered which measure? When was a model frozen, replaced or newly approved? Without documentation, there is no oversight, only a claim.

The duty of supervision transforms

The oversight paradox does not mean the duty of supervision disappears. It means that, for AI, supervision becomes proxy oversight. Anyone using AI without adapting oversight instruments to the system’s particular characteristics moves, from a board perspective, into a duty-of-care trap: formally supervised, but not effectively so. Anyone who fails to address material AI risks despite their recognisable relevance risks liability under Art. 754 OR.

Aviation has shown that effective oversight is possible even without full transparency, but only with clear standards, metrics and escalation mechanisms. Corporate governance now needs exactly this discipline.

Sources

Floridi, L. (2021). Ethics, Governance, and Policies in Artificial Intelligence. Springer.
Matthias, A. (2004). The Responsibility Gap: Ascribing Responsibility for the Actions of Learning Automata. Ethics and Information Technology, 6, 175–183.
Art. 716a, 716b, 717, 754 OR (Swiss Code of Obligations).
Art. 14 EU AI Act (Regulation (EU) 2024/1689).