An ICML Paper Won the Benchmark by Returning Nothing

There is a paper at ICML 2026 whose entire argument is a prank that works.

The authors built a method called the Prejudicial Trick. Here is what it does: most of the time, it returns nothing. An empty answer. Occasionally — a few percent of the time — it returns a padded one.

On the standard metric the field uses to evaluate this kind of system, it wins. Not by cheating the measurement. By satisfying it, exactly as written, while being completely useless.

That is the whole point. The authors weren't proposing a method. They were building a weapon to prove the metric is broken.

I spent an afternoon checking whether they were right. They were. Then I found something they didn't mention.

What the metric measures, and what it doesn't

The systems in question produce prediction intervals — instead of "this house is worth €340,000", they say "this house is worth between €310,000 and €370,000, and I'm right 90% of the time."

Two numbers grade that. Coverage: is the true answer inside your interval as often as you promised? Length: how wide is the interval? You want high coverage and short intervals, because a system that says "between €0 and €10 million, 100% correct" is technically honest and practically worthless.

Reasonable. Everyone reports it. And here is the hole the authors drove through it.

Return an empty interval 4% of the time. You lose a little coverage. Compensate by making the other 96% slightly wider — you buy the coverage back. Net effect on the scoreboard: coverage holds at 90%, and your average length drops.

You just won. Your method is better on paper than the method it wraps.

What it actually does: give the same house to it twice and it says "€310,000 to €370,000" the first time and nothing at all the second. The metric cannot see this, because the metric measures an average and correctness — and a coin flip has a perfectly good average.

I re-ran it

The paper reports 22.894 for the honest method and 22.614 for the trick. Slightly shorter intervals, same coverage. That's the headline.

I pulled their code and ran it. 22.8938. Their number, five significant figures, from a laptop CPU in about eight seconds. No GPU, no cloud — numpy, scikit-learn, and a linear regression on a small synthetic dataset.

The trick reproduces too, and so does the reason it's poison. The paper proves that the trick's instability — how much the answer for a fixed input jumps between runs — is mathematically guaranteed to be greater than zero, while the honest method's is exactly zero. I reproduced that on real datasets: 1.14 for the trick, 0.00 for the honest method. The number that exposes the fraud isn't subtle. It's just not the number anybody reports.

The part that isn't in the paper

Their code doesn't reproduce their paper.

Run the script exactly as published and you get 43.566 where the paper says 22.894. Nearly double. It looks like the paper is fabricated.

It isn't. The script hardcodes one constant — a bias value — at 20. The table was made with 10. The correct value is sitting in the same file, in a list the script never uses. Change one number and 15 of the table's 16 values come back.

Then the same disease in a second place: a list of per-dataset constants, three of which don't match the published table. And a third: the dependency file allows a version of pandas that the code cannot actually run on — install it fresh today and it crashes outright.

Three defects. Two of them fail silently, handing you plausible, wrong numbers.

I could only be confident about any of this because of a control. One dataset in their table reproduces perfectly using the constant they shipped. Same code, same machine, same afternoon. That's what separates "their constant is wrong" from "I broke something" — without it I'd have an accusation instead of a finding. If you take one methodological thing from this post, take that one.

I've filed the three fixes on their repo, and published the full reproduction with every command and output, as part of the Hugging Face and alphaXiv ICML 2026 reproduction challenge. The work is good. The packaging is quietly making it look like fraud.

Why any of this matters to you

You are probably not buying conformal prediction. You are being quoted numbers.

93% accuracy. 40% faster. 87% cost reduction.

Three things travel from that paper straight to that sentence.

One: a score is a proxy, and someone is always optimising against the proxy. Not necessarily dishonestly. Point any optimiser — a team, a training loop, a sales deck — at a number, and it will find the cheapest path to moving that number. Sometimes the cheapest path goes through doing the job well. Sometimes it goes through the gap between what's measured and what matters. The Prejudicial Trick is that gap, made visible on purpose.

Two: ask what was measured, not what was scored. "93% accurate" is not evidence until you know accurate at what, on whose data, against what baseline. The trick's coverage was a genuine 90%. Every number it reported was true. The number nobody asked for was the one that mattered.

Three — the one I'd actually use: ask for the variance, not the average. The trick is invisible in the average and obvious the instant you run the same input twice. That is not a hard test. It's the cheapest question in the room, and almost no benchmark answers it. When a vendor demos an AI system, run the same input three times. If the answers differ, the average they quoted you is hiding a distribution you haven't seen.

And underneath all three: published code frequently does not reproduce published results. Not through malice — through a constant left at the wrong value, a dependency range that rotted. If that happens in a peer-reviewed ICML paper whose authors clearly cared, assume it happens in the vendor demo that was assembled last Tuesday.

The honest limits

I checked three of this paper's five claims. Two need a medical dataset requiring an institutional agreement I don't have. I didn't verify the mathematical proofs — only that the experiments do what the paper says. One of my findings about a dataset I couldn't obtain is flagged in the logbook as an untested guess rather than a result, because it is one.

I also got things wrong along the way. I built an argument for the paper's typo that turned out to be invalid — the conclusion survived, but only because an adversarial check demolished the reasoning and found better evidence: the authors' own committed file contains the correct value. Their artifact, contradicting their table, sitting in the repo the whole time. I'd cloned it hours earlier and never opened it.

Which is the same lesson as the metric. I had a plausible chain of reasoning and treated it as proof. The trick has a plausible score and treats it as quality. Both are the same mistake: mistaking the thing you can easily measure for the thing you actually want.

The fix isn't cleverer metrics. It's remembering which of your numbers are proxies — and being willing to go and open the file.