I Published a Number That Nothing Had Measured

A statistic appeared in my published research logbook. It read, roughly: the effect is +0.531, at 11.2 sigma.

It was not a measurement. The delta was real. The sigma was a string — literal text sitting inside a print statement, with nothing anywhere in the code computing it.

I did not notice. It had been published for a day.

How a number that isn't a number gets published

I had been reproducing an ICML 2026 paper — the one whose entire argument is that you can win the standard benchmark by returning empty answers most of the time. (I wrote about that separately.)

Part of that work involved checking a specific claim: does the paper's method make prediction intervals shorter? I ran the experiment, the script printed its summary line, and I copied that line into the logbook as a finding.

Here is the shape of the failure. The printed sentence contained two numbers side by side. One was computed. One was typed. And because they appeared together, in monospace, in terminal output, at the end of a script that had genuinely just run for a while — the typed one inherited the credibility of the computed one.

That is the whole mechanism. Not a hallucination in the dramatic sense. No system claimed something false about the world. A number was simply placed where a measurement goes, and the placement did the lying.

Terminal output is not evidence. It is a claim about a computation. Those are different things, and I had spent years not needing to distinguish them, because historically the two came from the same place.

What happened when I actually ran it

The obvious next move was to assume the finding was junk and pull it. That would also have been wrong.

The authors' script hardcodes five random seeds. I widened it to two hundred and — the part that mattered — dumped the per-seed arrays the script otherwise averages away, so there would be a file on disk rather than a line on a screen.

The result:

	printed	re-run, n=200
effect (α=0.05)	+0.531	+0.531454
sigma	11.2	19.11
standard deviation	—	0.393338

The delta reproduced to three decimal places. The original run had genuinely happened. Its artifact was simply never saved, so there had been no way to tell a real result from a typed one — which is precisely why it needed re-running rather than deleting.

The sigma was wrong. But it was wrong downward: the real separation is 19.1, not 11.2. The published claim had been more conservative than the evidence supported.

That is not a comfort. A number nobody measured landing on the favourable side is luck, not method. Had it gone the other way, I would have published an overstatement.

For completeness, since it is the actual finding: at α=0.05 the method lengthens intervals (+0.5315, 19.1σ), while at 0.10, 0.15 and 0.20 it shortens them (−0.367, −0.526, −0.617). Coverage holds at nominal throughout — 0.9493 against a target of 0.95 — so the paper's theorem is untouched. The paper never claimed universal shortening. The benchmark's auto-generated summary of the paper did.

Two lessons that point in opposite directions

The first is the obvious one. A printed number is not a measured number. Before a figure counts as a finding, something must have written it to disk. I now treat "what artifact did I open?" as the question, rather than "does this look right?" — because looking right is exactly the property the bad number had.

The second is less comfortable, and I nearly missed it.

When I found the hardcoded string, my immediate conclusion was this was fabricated, the finding is worthless. Confident, fast, and also not evidence. It was a hypothesis that felt like a verdict, produced by the same machinery that had produced the original error — asserting something because it was plausible rather than because it had been checked.

If I had acted on it, I would have retracted a correct finding.

Only the re-run settled it. Not the first instinct, not the second. The artifact.

What this costs, and why that matters

92 seconds of wall-clock on a laptop, 105 seconds of CPU. No GPU, no cloud — numpy and scikit-learn on a synthetic dataset small enough that the whole thing fits in memory.

That is the number I keep returning to. The verification was not expensive, or slow, or technically demanding. It just had not been done.

This is the actual shape of quality control in AI-assisted work, and it is not what people expect. The failure mode is not a system inventing something outrageous that a careful reader would catch. It is a small, plausible, well-formatted value in the right place — surrounded by correct ones, indistinguishable by inspection, and cheap to check if anyone thinks to.

Generating a hundred plausible lines is now nearly free. Checking a hundred lines costs exactly what it always did. The gap between those two curves is where every error of this kind lives, and no amount of reading more carefully closes it. What closes it is a rule: a number does not become a finding until you can name the file it came from.

I fixed this by adding a new entry to the logbook showing the re-run and the corrected figure, and leaving the original visible. Editing published output to make a past state look tidier is a bigger problem than the error it hides.

The original line is still there. So is the correction. That seems like the right way round.

---

The paper is Questioning the Coverage-Length Metric in Conformal Prediction (Min, Lu, Li, Zhang, Teng). The reproduction was submitted to the ICML 2026 agent reproduction challenge. Every figure in this post comes from a file on disk — claim1_seeds200.csv and the per-seed arrays behind it — and was recomputed from the raw arrays before publishing, which felt like the minimum given the subject.