AI Environments: Human in the Loop Does Not Mean Human in Command

A year ago, I wrote on LinkedIn about the security of AI agents. That piece covered data leakage, shadow IT and the careless use of freely available tools. It was correct at the time, but it captured only part of a much larger governance challenge. I relearned this properly in my own lab, when I rebuilt and simulated an attack in an isolated environment.

The test setup: I embedded an instruction inside an entirely ordinary piece of text, the kind of message that lands in any inbox every day. The email agent read the text, treated the hidden instruction as a task, and sent me every spreadsheet from the files it could access. To be clear about what “hidden instruction” means here: a normal text that a human would read, at most, as marketing copy or a question, but that an AI interprets differently because it lacks the context. Nothing was visible or recognisable to me as an observer, and yet I had no control over what had been triggered.

The human in the process stopped nothing

A human was involved the entire time: me. That is what made me stop and think. Human in the loop, the formula that no AI discussion today can do without, was formally satisfied. It stopped nothing. The hidden instruction never surfaced as an output I could have reviewed. It acted out of sight. The failure did not lie in me checking the result too little. It lay in how the agent process itself was built. The same agent was allowed to read untrusted text, access confidential files and send data externally, all at once. Three capabilities in one hand, with no barrier between them. I chose this setup deliberately, because it matches many configurations being deployed and marketed right now. The focus tends to sit on what agents can do, rarely on how secure these assistants are. Yet security should already sit at the top of the list during architecture and design.

The security of agents and AI systems must already sit at the top of the list during the architecture and design phase.

A year ago, the risk was that a human carelessly revealed something. Today, the risk is that a system acts on its own, triggered by content nobody intended as a command. Classic safeguards work only partially here, because a filter built to catch sensitive data leaving the system sees a regular action by an authorised agent. The boundary between data and instruction, on which existing security architectures rest, is largely erased by an agent, because it reads both as language. The common security approach still largely rests on predictable actions inside a rule-based environment. AI systems and agents, however, respond dynamically to input and data, combined with their access rights.

Language is, once again, part of the problem

How we talk about this problem often obscures our view of it. Similar to the term “artificial intelligence,” which we often confuse with human intelligence, human in the loop depends on what we understand it to mean and what its precise task is. Human in the loop sounds reassuring at first, but it only says that a human sits somewhere, not what role that person has. Research recognises finer gradations:

• With human in the loop, the human approves every action.
• With human on the loop, the agent acts on its own and the human monitors.
• With human out of the loop, the agent acts largely alone. This gradation, though, only describes how close the human sits to events, not whether the human controls them.

The term that actually matters sits across this scale. The EU’s 2019 guidelines for trustworthy AI call it human in command (HIC). A related but distinct concept from the debate on autonomous weapons systems is meaningful human control (MHC). A human’s presence alone is not what decides the matter. What decides it is that person’s role and actual authority over the system, including the ability to intervene in time. This insight is uncomfortable. It challenges the idea, or rather the wish, that our AI assistant can act as our proxy even in our absence. We need to accept that a human’s presence alone, human in the loop, is not enough to keep control over the system.

A human’s mere presence in the process is not enough to keep control over the system.

Where a human only rubber-stamps decisions, that person’s control is decoration on paper. Anyone who approves outputs they do not understand, or reviews at the machine’s pace, satisfies the form and not the substance. Automation bias and oversight fatigue make this worse. It is a well-established fact that a system which rarely errs earns blind trust. In the cases documented so far, where attackers used agents as their own tool, the AI carried out most of the operation at a speed that reduced the human to a bystander. These cases rest only on vendor disclosures and should be read with corresponding caution; the direction, however, is clear.

Three levers

Three levers follow from this.

• Limit data access to the absolute minimum. What an agent cannot reach, it cannot leak.
• Limit its functions. An agent allowed to read need not also be allowed to send. Consistently separating these capabilities would have stopped my test attack in its tracks.
• Genuine human command, where the first two levers fall short. What is needed is not a human who merely watches. What is needed is a human who structurally controls the process, and who can intervene and stop it at any time. Where this exists, agents may act independently within clearly defined limits. Where it is missing, the human in the process is decoration, and it suggests a false sense of security.

Where only a human’s presence exists, rather than real control, that produces a false sense of security.

The question for the board

This shifts the question the board needs to ask. The old question was whether the tool was secure. That question can be delegated, to IT, to the vendor, or to the manufacturer. Then as now, it must be clear that responsibility for using these tools rests with the company.

The new question has to be broader. What authority have we granted a non-human actor, who controls the process, and who is responsible if the agent acts against us? This responsibility, too, cannot be delegated, because the vendor is highly unlikely to accept it, and the board’s duty of care applies regardless of whether a human or a machine carried out the action.

The board does not need to review architectures itself. It does need precise knowledge of which agents operate in the company, with what rights, and whether someone genuinely controls the process or merely accompanies it formally. Put differently: the board needs to know its risk exposure in detail, so it can make well-informed decisions about deploying such agents at the strategic level. One such decision, for example, might be that internal data and automated response processes may not be linked. The duty of care under Art. 754 OR does not ask whether a human or a machine acted. It asks who answers for it. Technology can be delegated. Responsibility cannot.