Someone asks whether the automation is safe. The answer comes quickly: don’t worry, it’s human in the loop AI. Question answered, the meeting moves on. But nothing has actually been said.
Human in the loop means there is a person somewhere in the process. It does not say where, what that person gets to see, how much time they have, or whether they are allowed to say no. Four completely different set-ups carry the same reassuring name, and in most situations three of them are the wrong one.
This article is about that choice: where you put the person, and what that costs.
Why human in the loop AI is suddenly everywhere
Search volume for the term in the Netherlands more than doubled in twelve months: from about 210 searches a month in September 2025 to 480 in August 2026. That is not a linguistic fashion. It is the sound of organisations that have put something into production and are wondering who is still watching.
What the research says about why projects fail
40%+
of agentic AI projects will be cancelled before the end of 2027. Gartner names three causes: rising costs, unclear business value and inadequate risk controls.
~130
of the thousands of vendors calling themselves ‘agentic’ actually are, according to Gartner. The rest are doing agent washing: existing chatbots and RPA relabelled.
Source: Gartner, press release, 25 June 2025.
Look at what is in that list. Not ‘the models are not good enough’. Instead: nobody had set down what the system was allowed to do on its own. That is not a technical problem. It is a design decision that was never made.
Position 1: approval beforehand, on everything
Every action waits for a person. The system prepares, someone approves, and only then does it happen.
This feels safest, and at the start it is. The problem comes with volume. At ten proposals a day, someone really reads them. At a hundred, a rhythm develops: open, scan, approve. At three hundred, the approve button has become a reflex.
This is called automation bias: the more often a system turns out to be right, the less attention people pay when they check it. The effect is well documented in aviation and in medical decision-making, and it applies in full to an employee who approves draft replies.
The painful conclusion: approval beforehand on everything automatically becomes less reliable the more volume you run. Exactly when you need it most.
When position 1 is the right one after all
- Irreversible actions. Transferring money, sending a contract, deleting something. If it cannot be undone, you check beforehand. Every time.
- The first six weeks. Start here, whatever the volume. You need those weeks to see what the system actually does, as opposed to what you think it does.
- Low volume, high value. Five quotes a week of fifty thousand euros: have a person look at those. That is not inefficiency, it is common sense.
Position 2: stepping in when in doubt
The system acts on its own, except when it is unsure. Then it refers the case to a person. In by far most SME situations, this is the most usable set-up.
Its strength is scalability: the amount of work that reaches a person grows with your exceptions, not with your volume. If your invoice flow doubles, your checking work does not.
The difficulty is in the word doubt. Software does not doubt; you have to define doubt. Usually in thresholds:
- Amount. Above a limit, it always goes to a person.
- New relationship. First invoice from an unknown supplier: always look.
- Deviation from the pattern. This supplier normally sends €400, now €4,000.
- Missing field. No purchase order number, no match, so no automatic decision.
- Low confidence score. The model indicates it cannot place the text well.
FLOW 1
Purchase invoice with a threshold check
Who it is for: Bookkeeping firms and SMEs from about fifty purchase invoices a month.
What it gives you: The routine work disappears, the judgement calls stay. The employee only sees the cases where their judgement adds something.
Watch out: Deliberately set the threshold too low at the start. Raising it later on the basis of measurements is easy; reversing a wrongly posted quarter is not.
That last item, the confidence score, deserves a warning. Language models are poorly calibrated: they sometimes give high confidence for a wrong answer. Use such a score as one signal alongside hard rules, never as the only gatekeeper.
Position 3: sampling afterwards
The system acts on its own. A person checks part of what happened afterwards. This is the cheapest form and the most underrated.
The error has already been made by the time you see it. That sounds unacceptable, but for cheap actions that are quick to correct, it is not. A wrongly labelled email takes ten seconds to fix. A draft text that is not good enough, you rewrite.
What makes a sample unique: it is the only position that tells you how good your system really is. With approval beforehand, you never see what would have happened without the correction. With a sample, you measure the real error rate, and you need that figure to justify the thresholds of position 2.
Example: email triage at a service provider
EXAMPLE 1
The situation. A consultancy receives sixty to eighty emails a day at its general address. Sorting them takes an employee about an hour a day, spread across the day, with the interruptions to other work that come with it.
The approach. A flow reads each email, determines the category and the label, and puts it in the right folder. The system is not allowed to delete and not allowed to reply. Every Friday the employee is shown twenty randomly chosen emails from that week, with the question: is this classification right?
The result. After eight weeks, the classification turned out to be right in nineteen of the twenty cases. The exception was always the same type: emails from existing clients with a new question, labelled ‘existing file’ instead of ‘new enquiry’. One change to the rules fixed that.
What was disappointing. For the first two weeks the sample was useless, because nobody had agreed what ‘right’ meant. Two employees labelled the same email differently. Define your categories before you start measuring, or you will be measuring disagreement instead of quality.
Illustrative example based on typical SME volumes, not a client case. In the free assessment we work through your own numbers.
Position 4: stepping in only when an alarm goes off
There is no person in the chain. There is a notification when something goes wrong.
This is not human in the loop. It is an automated system with a beep, sold under the same name because it sounds reassuring.
It can be legitimate: for purely technical steps that judge nothing, such as moving a file or syncing a record, an error notification is enough. It becomes dangerous as soon as the system judges something, because then the error you want to catch is exactly the error that triggers no alarm: an answer that is wrong but sounds plausible.
There is one more reason this position rarely works, and it is a human one. An alarm that has not gone off for months is ignored the moment it does. If you never hear anything, you stop listening.
The question you answer beforehand
All four positions can be defended. Which one fits follows from two questions you have to answer before you build anything.
| Error is visible quickly | Error is visible late | |
|---|---|---|
| Error is cheap | Position 3: sampling afterwards | Position 3 with weekly measurement |
| Error is expensive | Position 2: threshold check | Position 1: approval beforehand |
That second question, how quickly do you notice?, is almost always skipped, while it weighs more than the first. An expensive error you see within an hour is manageable. A cheap error you only discover after six months has repeated itself a hundred thousand times by then.
Four rules I work by
- Irreversible always means beforehand. Money, contracts, deletions. No exceptions.
- Start stricter than necessary. Relaxing on the basis of measurements is easy. Tightening up after an incident is expensive and painful.
- Always measure, even with approval beforehand. Without figures on your error rate, you can never relax things on solid grounds.
- Set down who may intervene, and how. An emergency stop that nobody can operate, because the only person who knows how it works is on holiday, does not really exist.
What this costs, and why that is intended
Human oversight makes automation less profitable. That is not a design flaw. It is the price of staying in control, and you should pay it openly instead of calculating it away.
A chain with approval beforehand might save sixty percent of the manual work instead of ninety. Anyone who promises you ninety percent is leaving the person out or not counting their time. Ask questions when you hear that promise.