Every generation of software has had a bottleneck. Forty years ago the problem was computation: machines were slow, memory was scarce, and producing a result at all was often the hard part. Today computation is close to free. Agentic systems write code, language models summarize in seconds what once took an afternoon, and a scientific analysis that used to consume a week can now run before the coffee finishes brewing.
The bottleneck has not disappeared. It has moved, and it has moved somewhere we're less comfortable talking about. The question worth asking is not what software can produce. It is how institutions continue to exercise judgment once producing an answer stops being scarce. That is not, in the end, a question about artificial intelligence or even about software. It is an epistemological question about organizations: how a group of people who are supposed to know things together keeps knowing them, once any one of them, or any one system, can generate an answer faster than the group can agree on whether that answer deserves to be trusted.
I learned an early version of this years before large language models existed. The other lesson came much more recently, in a setting that had almost nothing to do with the first. It took a long time to notice they were teaching the same one.
What a Clustering Algorithm Cannot Tell You
The first software I supported professionally was flow cytometry software, used by immunologists and hematologists to make sense of blood and tissue samples. Most people imagine the difficulty lies in the mathematics. In practice, the mathematics was rarely what people struggled with. A multidimensional embedding of a high-parameter dataset took minutes to generate; thousands of cells resolved into apparently distinct populations on screen. The hard part started the moment the plot appeared.
During many a training session, or in the discussions that followed, a scientist would be reading two shapes on screen as two genuinely distinct cell populations, distinct enough to support a finding they wanted to write up. More often than I care to admit, changing one parameter (a compensation setting, a variable transformation, a threshold in feature selection) was enough to merge the two into a single continuous gradient. It went the other way too: what looked like one homogeneous population would split cleanly into two.
Nothing about the software was misleading in either direction. The embedding was mathematically correct given the inputs chosen. What confused all these scientists, again and again, was a choice they had made without registering that they were making one. Software does not discover biology; it visualizes assumptions, precisely and quickly, which is not the same thing as visualizing truth. I told myself for years I had been teaching people to use an instrument. I was actually teaching something closer to judgment: training people to ask what would have to be true, and what would convince them otherwise, before a shape on a screen was allowed to become a finding.
Three Ways of Answering the Same Question
Science answered the trust problem one way: if another laboratory follows exactly the same procedure, it should get the same result. That is reproducibility, and it has organized experimental science for a couple of centuries. Even reproducibility is less straightforward than it sounds. Many of the dimensionality-reduction methods researchers stake conclusions on, such as t-SNE and UMAP, are stochastic. Re-running the same analysis does not necessarily reproduce the same visualization, even when the underlying assumptions remain unchanged.
Regulated industry reached a related but different answer decades before anyone worried about it in AI. Nobody trusts a pharmaceutical batch record because it exists. They trust it because it can answer a narrower question: who entered this value, on what instrument, and what changed afterward. That is provenance: not "can this be reproduced" but "can this be reconstructed." A record built to survive an inspection months or years after the fact. A batch record can be perfectly provenanced and still describe a batch that should never have shipped; reconstructability tells you where a result came from, not whether it deserved to influence a decision.
Reproducibility and provenance solved genuinely different problems, in genuinely different worlds. Neither was built for a world where producing an explanation would become easier than evaluating one. That is exactly the world explanation now has to work in. Three domains, converging on the same underlying question in progressively harder forms: can this be reproduced? Can this be reconstructed? Can this be understood? None of the three fully answers it. Volume is what makes the gap impossible to ignore now. When an answer took a week to produce, there was time to interrogate it before acting on it. When answers arrive a hundred at a time before lunch, that time is the thing that has actually disappeared.
The Assistant That Only Answered Easy Questions
A German health insurance fund had built a knowledge assistant so that the case handlers who process claims and benefit decisions every day could ask a question about internal policy and get an answer drawn from years of accumulated circulars, guidelines, and procedural updates. Technically it worked well. A case handler typed a question and got a relevant, well-formed answer in seconds.
The problem only became visible once people actually used it. The assistant was reliably good at the easy questions: the routine claims, the standard entitlements, the cases every experienced case handler already knew how to resolve without looking anything up. Those were exactly the questions nobody needed help with. The cases that actually consumed time and caused hesitation were the edge cases: a request for continued home nursing care that fell between two eligibility categories, after a circular updated one threshold but left an older guideline sitting, unreconciled, in the same document store. On questions like that, the ones that mattered, the assistant either produced a confident answer that was wrong, or produced nothing useful at all.
That is a dangerous failure mode in this setting specifically, because a case handler acting on a wrong answer is not a productivity loss. It is a real decision about someone's coverage, made on behalf of a person who is not in the room to catch the mistake. An assistant that is only trustworthy on the questions nobody needed help with is worse than no assistant, because it teaches people to trust it well before the moment it actually needs to be trusted. The danger was never that the assistant made mistakes. People have always made mistakes, and so has every earlier generation of software. The danger was how easily confidence became detached from justification, once something was always sitting there with an answer ready, sounding exactly as certain on the hard question as it had on the easy one.
Here is why the pattern was so consistent. Easy questions are easy because the organization has already converged on an answer: the policy, the precedent, and the case handler's intuition all point the same direction, and any retrieval system will find that agreement and hand it back. Hard questions are hard for the opposite reason. The organization has not converged. Two circulars say different things. A rule was updated in one place and never reconciled with guidance sitting two folders over. No retrieval system can retrieve a consensus that does not exist. At best it retrieves competing fragments of institutional memory and hands someone the job of choosing between them: the same job the case handler already had, now with an extra, confidently worded voice in the room.
What the case eventually clarified was that the assistant had been built to answer a different question than the one that mattered. Retrieving an answer was never the hard part. What the hard cases actually required was something closer to a second opinion the case handler could interrogate: not just an answer, but the reasoning behind it, presented clearly enough that a person who remained accountable for the decision could judge whether the reasoning held up before acting on it.
It also opened a possibility nobody had designed for. Every disagreement between the assistant and the case handler was information, not noise. Sometimes the case handler was right and the documentation was outdated. Sometimes the documentation was right and the case handler had missed an update. But often it was neither, cleanly. A circular said yes. An older guideline, never formally retired, said no. The experienced case handler said it depends. A manager had overridden the same situation differently twice in the past year, and an appeal board had overturned it once. In cases like that the assistant hadn't failed. It had located the exact place where the institution had never actually decided what it believed.
An Instrument Aimed at the Institution
This is the point where the assistant stops being one kind of software and becomes another. A retrieval system answers questions. A system that surfaces disagreement, rather than quietly resolving it by handing the win to whichever document loaded first, exposes something no single document could show on its own: where an organization's practice and its policy have stopped agreeing, and how long the gap has been sitting there unnoticed. Organizations could always, in principle, have found this by reading every circular against every other circular, and against how their own staff actually behaved. Almost none did, because that comparison was expensive and reserved for audits, appeals, and the occasional scandal. What's new is not that the disagreement exists. It's that finding it no longer requires a special effort. It happens automatically now, as a byproduct of a tool people were already using to do something else. That is a different instrument, aimed at a different target: not the claim in front of the case handler, but the coherence of the institution behind them.
Most organizations currently treat disagreement as noise to be resolved quietly and as fast as possible. Whichever document, reviewer, or model output arrives first wins, and the friction disappears without a trace. A system that instead treats its own divergence from a case handler's judgment as a standing signal, rather than an error to be silently corrected, produces something most organizations have no instrument for today: a running map of where their own knowledge has quietly split into two versions of itself. Not an audit, which looks backward for what already went wrong, but something closer to a continuous measurement of where the organization no longer agrees with its own documentation, available as it happens rather than reconstructed painfully after the fact.
The Question Underneath Both Stories
Which is the actual question underneath both stories, once the specifics are stripped away. Not what AI can produce, but how an institution keeps exercising judgment once producing an answer, any answer, stops being the scarce resource that used to force someone to check it before acting on it. Computation being cheap is a technology story. An organization no longer being forced to check its own answers before acting on them is a story about institutional judgment, and it was already true of the insurance case before any model was involved. A case handler was capable of judgment long before the assistant existed. What the assistant changed was how often that judgment got tested.
Throughout my career the technology kept changing underneath this same problem. Flow cytometry became high-dimensional single-cell analysis. Laboratory software became cloud platforms. Retrieval systems became language models. What barely changed was the difficult part.
The people I trusted most never changed. They were never the ones who produced the fastest answer. They were the ones who still knew which answer deserved another question.
Software has become extraordinarily good at producing answers. Judgment remains stubbornly human, not because humans think faster, but because they still decide what is worth believing.