A number of LLM-based recommendation rerankers are evaluated under an oracle protocol: the ground-truth item is guaranteed to be present in the set being scored, either by injecting it into the candidate list or by scoring it against sampled negatives.
It is worth being precise about what that measures. In deployment, a retriever decides what the reranker is allowed to see. If the item is not in the candidate set, no amount of reranking recovers it. The oracle protocol removes exactly that constraint — so it measures a reranker on a problem it will never be asked to solve.
The bound is combinatorial, not empirical¶
This is not a claim that rerankers help less than reported. It is a claim about the shape of the ceiling:
Two-stage performance is bounded by first-stage recall.
That is provable rather than observable. The reranker's achievable metric is capped by the candidate set it was handed, and the cap does not move when the reranker gets better. It moves when retrieval gets better.
The uncomfortable corollary: a great deal of reranking work is measured in the region where the bound is not binding, because the oracle protocol lifted it.
Falsification, not demonstration¶
A bound like this is easy to "confirm" — pick a setting, show the ceiling, publish. So the claim was set up to be attacked instead:
17 optimisation strategies
12 retrieval methods
10 reranker types
8 datasets
3 domains
The paper ships a falsification ledger: what was tried against the bound, and what happened. Not a results table — a record of the attempts, including the ones that looked like they might break it.
Lesson: if a claim is structural, the interesting artifact is not the experiment that agrees with it. It is the list of experiments that should have disagreed and didn't.
Every float resolves to a file¶
The public artifact carries a complete float → script → JSON map. Any number in the paper can be traced to the script that produced it and the file it was read out of. That is cheap to build while the experiments are running and near-impossible to reconstruct afterwards.
Two audits on a finished paper, two different defects¶
Both ran on what already looked done, and neither found what the other did:
- A structural audit caught an orphaned table cross-reference — a section pointed at the wrong label, a casualty of the ten-page squeeze.
- A bibliography audit caught an entry with a fabricated author list: the first author's surname was wrong and five of six co-authors are not authors of that paper. Replaced with Crossref's canonical record, along with an uncited duplicate of the same DOI.
The second one is why bibguard exists, and it found this in a camera-ready. A reference that resolves is not the same as a reference that is correct.
Lesson: run both kinds of audit before anything irreversible. They do not overlap, and a paper that passes one will still fail the other.
The Recall Ceiling of LLM Recommendation Reranking — CIKM 2026, main track, oral. Code and the falsification ledger: recall-ceiling-cikm2026.