Human in the Loop Is the Wrong Question

In 2005 I sat across a plastic table from a woman who was deciding whether to believe her own blood. The result was positive. My job that afternoon, in a counselling room that smelled of disinfectant and heat, was not to deliver a number — the number was trivial, a machine had produced it. My job was everything the machine could not do: to hold the silence, to judge from her face how much truth she could take in one sitting, to decide whether the man waiting outside was safe to tell, to know which clinic would still be open on Thursday. There was no system in that room. There was her, the result, and me. If you had asked me that day where the "human in the loop" was, I would not have understood the question. I was not in the loop. I was the loop.

I have been thinking about that room a great deal lately, because I have gone back to school to study artificial intelligence, and the phrase I keep meeting — in the papers, in the policy drafts, in the confident talks — is that one. Human in the loop. We must, everyone agrees, keep a human in the loop. And then the cleverer people, the ones who have read a little further, say: no, that is the wrong question. Do not ask where to insert a human as a fail-safe for imperfect automation. Ask instead what new problems the machine now lets us solve. Sangeet Paul Choudary made the point plainly in a recent conversation for FARSIGHT: treat AI as a productivity boost and you get a faster version of the same organisation; treat it as a redesign question and you get something new.

He is right. And I still do not accept the question. I have spent twenty years being suspicious of clean answers, and both of these — the safety-net question and the redesign question — are very clean. They are clean in the specific way answers become clean when they are written by people who will be fine either way.

The loop was a person

Let me say what the phrase assumes, because no one says it aloud. "Keep a human in the loop" assumes there is a loop — an automated process, humming along, into which we thoughtfully place a person as a check. The human is the exception; the machine is the default. That is a description of a world with so much automation that human judgement has become the scarce, optional garnish on top of it. It is a real world — the hospital with an electronic record for every patient, the bank with a fraud model on every transaction, the warehouse that routes itself. But it is not most of the world. In the counselling room, in the district health office with one working laptop, in the market where credit is a handshake and a memory, the arrangement is inverted. There is no automated default. The human is not the garnish; the human is the entire dish. The "loop" is a person, holding a system together with attention and relationship and no slack at all.

Build on top of what?

So when I am told the enlightened move is to stop treating the human as a backup and start asking what we can newly build, I want to ask the prior question the enlightened version skips: build on top of what? The redesign story imagines AI arriving into an organisation with a rich, coordinated substrate — layers of process it can rewire into something better. That substrate is exactly what the counselling room did not have. Drop a coordination-rewiring technology into a place where the coordination was one exhausted person, and you are not redesigning the system. You are discovering, often for the first time, that she was the system — usually at the moment you remove her.

Drop a coordination-rewiring technology into a place where the coordination was one exhausted person, and you do not redesign the system. You discover she was the system — at the moment you remove her.

Here I will concede the strong part of the argument, because it is strong. Choudary is right that AI is not mainly automating tasks; it is rewiring how work is coordinated. That is the real thing, and most people miss it because they are still counting which jobs the robot takes. The important change is not that a machine does your task. It is that the machine reorganises who must talk to whom, who decides, who waits for whom, where the information pools. I know this from my other life, the one I have spent in development finance, building the models that tell an investor whether a thing will hold. Coordination is where the value lives. Anyone who has run a feasibility study knows the technology is rarely the hard part; the coordination is the hard part, and whoever controls the coordination controls the returns.

Coordination moves the value

Which is exactly why I cannot let the reframe end where it wants to end — at "what new problems can we solve." Because rewiring coordination does not simply create value. It moves it. Every time you reorganise who must go through whom, you draw a new map of who collects the toll. A reframe that stops at the size of the new pie, and never asks who is now standing at the door of the bakery, is not a neutral observation about technology. It is a position — taken by people who expect to be inside the bakery.

I am a statistician by first training, and statisticians are professionally paranoid, so let me bring the paranoia. The models we are discussing are trained on the exhaust of the automated world — the dense, logged, coordinated world where every interaction already left a machine-readable trace. They learn the shape of that world extremely well. Then we deploy them into the other world, the counselling-room world, and act surprised when they misfire. It is not surprising; it is arithmetic. A model cannot represent a coordination layer that was never digitised, because to the model it does not exist. The community health worker who was the system leaves no trace in the training data, so the system built to replace her is optimised against a reality from which she has already been deleted. And when it fails — routes the drug to the wrong clinic, denies the loan to the reliable trader, flags the honest patient — the failure does not land on the people who built it. It lands on the people who were already holding everything together by hand. I have cleaned enough datasets to know the rule: the tidiest column is usually the one someone invented, and the people missing from the file are usually the ones doing the work.

Now put the two lives together, the health one and the finance one, because that is where this gets uncomfortable. When AI rewires coordination across an African health system, or an African market, the productivity is real — I am not a romantic, and I will not pretend the old way was kind. But ask where the productivity goes. Coordination that used to run through a nurse, a loan officer, a local broker — people embedded in the place, who kept a share of the value and spent it there — is rerouted through a platform. The platform is usually not in the place. It bills in a currency the place does not print. The efficiency is extracted upward and outward; the person who was the loop is thanked, and dropped. This is not a story about machines taking jobs. It is a story about the coordination layer being quietly repossessed — and the repossession described, in the confident talks, as a redesign.

Whose loop is it?

So, no. I do not want to argue about where to keep a human in the loop, and I do not even want to accept the more sophisticated invitation to ask what we can now build. Both questions take the loop for granted and argue about its furniture. The question worth the trouble is harder and less flattering: whose loop is it, who captures it when it is rewired, and who — already doing the coordinating with their own hands and no slack — gets written out of the redesign? That is not a question a technology answers. It is answered by a politics, and a contract, and a choice about who is in the room when the map is redrawn.

Whose loop is it, who captures it when it is rewired, and who — already doing the coordinating by hand, with no slack — gets written out of the redesign?

I do not have the resolution, and I distrust anyone offering one this early. That, I am told, is the point of this Institute — to name the futures still being fought over rather than to sell one. Good. This is a fight. On one side is a redesign that treats the human coordination layer of the poorer world as friction to be optimised away, and calls the extraction progress. On the other is a redesign that treats that layer as infrastructure — as the actual system, valued and paid and kept in the loop it already occupies. The technology does not choose between them. We do — deal by deal, model by model, clinic by clinic. I have sat in the room where a person was the whole system. I would like the people drawing the new maps to sit there once, before they tell me she was a backup.

---

Kelechi Ezekwe is a statistician and public-health and development-finance practitioner, and a graduate student in artificial intelligence at Luleå University of Technology. He writes from Copenhagen.