The Misattributed Phrase in the European AI Act
Everybody attributes the phrase “human in the loop” to the European AI Act. Procurement documents cite it. Vendors promise it. Compliance decks put it on a slide with the article number next to it.
However, Article 14 does not contain the phrase — not once. Neither does it mention “human on the loop.”
Instead, Article 14 lays out a list of capabilities required of whoever oversees the AI system. This overseer must thoroughly understand the system’s capabilities and limitations, remain aware of automation bias (the well-documented tendency to over-trust automated outputs), and correctly interpret the system’s results. They must be able to decide not to use the system’s output, override or disregard it, and interrupt the system’s operation, including through a stop button. For certain biometric identification uses, two separate natural persons must confirm the output.
Read that list carefully and notice what is missing: every item describes a competent person, but none specify when the system is allowed to act. A person can fulfill all these requirements while only reviewing exceptions after the fact, and the Act does not prohibit this.
Where the Phrase Actually Comes From
The phrase “human in the loop” originates not from AI regulation but from a 2012 report by Human Rights Watch and Harvard’s International Human Rights Clinic titled Losing Humanity. This report called for a pre-emptive ban on fully autonomous weapons and introduced a taxonomy that has since permeated AI marketing and policy discussions.
The report defines three terms:
- Human in the loop: A machine that selects targets and delivers force only upon human command.
- Human on the loop: A machine that selects and delivers force under the oversight of a human who can override it.
- Human out of the loop: A machine that performs both functions with no human involvement.
While “in” and “out of the loop” were already established control-system terms, Human Rights Watch attached profound moral weight to these distinctions. The report also warned that “on the loop” systems with only nominal supervision are, in practice, effectively autonomous.
Fourteen years later, two of these three phrases are often used interchangeably in AI marketing — usually by people unaware that these terms originally described conditions under which a machine might kill someone.
What Changes Between “In” and “On”
The difference between “in” and “on” the loop is fundamental.
In the loop: The default is inaction. The system waits for a human decision before acting. The human acts as a gatekeeper, and throughput is limited by human reaction time.
On the loop: The default is action. The system proceeds autonomously, and humans intervene only if necessary. The human functions as a brake, and the system’s throughput is unlimited.
Being in the loop means the system waits for you. Being on the loop means the system has already gone, and you are the reason it might come back.
This distinction explains the commercial appeal of “on the loop” systems but also why their oversight often degrades. A gate’s opening is recorded and auditable; a brake only needs to be available, making supervision difficult to verify. Nominal and real supervision can look identical on an organizational chart.
Why the Omission in Article 14 Is Probably Deliberate
The omission of an explicit “human in the loop” requirement in Article 14 is likely intentional. Mandating human approval for every high-risk output would render many AI systems impractical and unused, thus evading regulation entirely.
Article 14’s proportional requirements to risk and autonomy reflect sensible engineering and allow organizations to maintain “on the loop” oversight while still complying.
Similarly, the US Department of Defense’s Directive 3000.09 rejects the “loop” framing altogether, requiring instead appropriate levels of human judgment over the use of force. This may acknowledge that the binary “in” or “on” loop classification is too simplistic or serve as a way to avoid specifying which side an operation falls on.
The Keynote Where This Became Concrete
My interest in this topic was sparked by a keynote at the European Health Psychology Society conference by Alex Gillespie from the London School of Economics, titled “Using AI to analyze patient voice and hospital listening: insights into patient safety.”
The foundational work predates AI and involves no AI at all. Gillespie and Tom Reader developed the Healthcare Complaints Analysis Tool, published in BMJ Quality and Safety in 2016, to code what patients and families actually say when they complain.
Later studies across 1,110 complaints from 56 NHS trusts showed that patient complaints reveal problems missed by incident reporting. Another analysis covering 59 trusts found that the severity of clinical problems described in complaints was the only patient-generated measure associated with hospital-level mortality.
This association is cross-sectional and hospital-level — one step short of proving complaints predict death. However, it signals a critical safety signal buried in complaint letters that no one reads at scale.
That’s where a language model seems like the obvious answer — and where the preposition “in” or “on” the loop becomes more than just vocabulary.
The Finding That Should Slow Everyone Down
The strongest evidence comes from Gillespie’s own group. A team including Hannah Bunt, Alex Goddard, Reader, and Gillespie validated GPT-4o against human coders on three classification tasks, each with 1,500 items drawn from NHS complaint data.
On identifying reported speech (whether a passage quotes something someone said), the model achieved an F1 score of 0.91 — near-human agreement. On identifying repair, it achieved the same score.
However, on classifying harm (whether a complaint describes patient harm), weighted kappa ranged from 0.49 to 0.57 depending on the prompt. The model didn’t simply disagree with human coders — it systematically overestimated harm.
This moderate performance is unremarkable in exploratory research but far from acceptable for triage decisions where patient safety is on the line.
Independent replication studies have reported similar results: good agreement on complaint domain classification but poor agreement on severity and harm. Though such replication should be verified, it aligns with the original validation.
The pattern is clear: the model excels where language is explicit and struggles where clinical judgment is essential. It performs best on the least critical parts and worst where it matters most.
Therefore, no one should claim that AI reads patient complaints as well as humans do — the evidence says the opposite where it counts.
This finding highlights why the missing phrase matters. If a system’s weakest classification is harm, the human must act as a gate — not merely as a brake. Yet Article 14 permits the latter.
The Second Risk Nobody Regulates
Another risk raised at the keynote relates to research culture rather than oversight architecture.
Research incentives reward publishable metrics rather than real-world outcomes. An F1 score of 0.91 on reported-speech classification is a strong, publishable number. A weighted kappa of 0.57 on harm classification is a minor footnote.
Both appear in the same paper, but likely only the former survives into procurement decks and marketing materials.
This exemplifies the paper-versus-impact problem: honest but incomplete metrics are selectively highlighted due to incentive structures blind to patient safety, without fraud or hype.
Where the Preposition Ends Up
It is worth asking, of any system advertising human oversight: does the system wait for human approval, or does it act autonomously? If it acts autonomously, who watches, how many decisions can that person realistically review per hour, and what happens if they intervene too frequently?
Article 14 won’t ask these questions for you. It specifies a competent overseer but leaves the nature of human involvement up to the buyer.
What lingers is that a distinction originally developed to govern lethal autonomous weapons now silently governs how complaint letters from bereaved families are triaged. The 2012 warning about nominal supervision has traveled with that phrase word for word. And the phrase everybody quotes to prove a person is still in charge turns out not to appear anywhere in the law they cite.
Source: Here
