1 Why a frontier benchmark must contain errors#

This is arithmetic and not an accusation, and stating it first prevents the rest reading as one.

A benchmark of some 2,500 research-grade questions is authored at the pointy end of every field it covers, which is exactly where its reviewers stand at the edge of their own knowledge. At any plausible post-review error rate the expected number of wrong keys exceeds one. The instrument built to test reasoning is bound by the same arithmetic of error rates as everything else, and a benchmark with no wrong keys would be the surprising object, not this.

What follows is therefore not a complaint about the benchmark, which is a serious piece of work whose difficulty is the reason it is useful here. It is an argument that a certain kind of answering system is the right instrument for finding the errors any such benchmark must have.

2 Certification turns answering into auditing#

The system, and the standing of what is claimed. The system is the substrate described in Graphic Equalisation, run with its general faculties: no per-question code, no answers baked in, and nothing added to it for this benchmark. The derivations, the certificates, and the running dossier they are recorded in are this programme’s own pre-registered implementation record, which is likewise not circulated with these papers, and it is a claim a reader may decline to credit on exactly the terms Graphic Equalisation’s Section 1 sets out. What is different here, and it is the reason this paper can be short, is that the five diagnoses below are stated fully enough to be re-derived from this page: a kernel rank, a sudden-approximation argument, a basis-function count, the parity of a theorem, and a grammar with its enumeration. A reader who declines to credit the record entirely can still check every conviction drawn from it.

What was audited. The public split of Humanity’s Last Exam (Phan et al., 2025), which the benchmark’s own paper gives as 2,500 questions and which is accompanied by a private held-out set not touched here. The run covered the text-only portion of that split, which numbered 2,370 questions in the snapshot audited. That count is this programme’s own count of the snapshot and not a figure the benchmark publishes: the benchmark states that around 14% of its questions require an image, which would put the text-only portion lower. The dataset revision is not recorded in the dossier, and the public dataset repository has been modified since the run, so nothing here establishes that these five keys stand in the current revision — a conviction against a since-corrected key is a different finding from one against a standing key, and this paper cannot tell a reader which of the two it holds. Each conviction below carries the benchmark’s own public identifier for the question, so the question and its key can be located directly.

The arrangement. The system answers only where it can derive the answer under an independent certificate — a second derivation by a structurally different route, a held-exact computation, or a conservation or limiting case forced by the stem. Where it cannot, it returns a miss. The key is read only afterwards, and only to compare.

That ordering is what makes the arrangement an audit. Every certificate for which the audited snapshot and key state are retained puts one key on trial and returns one of two verdicts: it confirms the key, or it convicts it. The two certified-but-unclassified answers reported in the abstract are not counted as either verdict. A conviction is banked only when the derivation is certified sound, the key’s error is demonstrable, and no reading of the stem rescues the key — the last checked adversarially, by trying to rescue it.

3 The five convictions#

Each is stated as: what the key says, what was certified, and where the key’s reasoning goes wrong. The diagnosis matters more than the disagreement — a conviction that cannot say why the key is wrong is only a second opinion.

Mathematics — rank of a Hurewicz kernel (66f407a066abc59ffb9a82ae; key 28, certified 95). For Y = X₁#X₂#X₃ with π₁(Y ) = ℤ∕5 ∗ℤ∕8 ∗ℤ∕2, the kernel of abelianisation is free by Kurosh, and its rank is 95 by three independent routes: an orbifold Euler characteristic (χ(K) = 80 ⋅ (−47∕40) = −94, rank = 1 − χ(K)), a Schreier deficiency count, and an explicit degree-80 cover (V = 80, E = 240, F = 66). The key computed ∏ (n_i − 1) = 28, which is the correct kernel rank only for two free factors — verified against [5,8] → 28, and against a divergent control [2,2,2] where the construction gives the correct 5 and the key’s formula gives 1.

Physics — sudden expansion of an infinite well (671cb3074d6192736975115b; key 4.03%, certified 0%). The ground state of a well of width L with the right wall moved instantaneously to 3L. Under a sudden change the state vector does not have time to evolve, so the expectation of the energy is unchanged. A decrease of 4.03% requires ⟨E⟩ to change, which the word “instantaneous” in the stem forbids.

Chemistry — contracted basis functions for toluene, 6-31G (67213cb9043b1e724244a1c6; key 54, certified 79). The item is filed in the benchmark under Physics. Counting per atom under 6-31G gives 9 for carbon and 2 for hydrogen, so 7 ⋅ 9 + 8 ⋅ 2 = 79. The key’s 54 corresponds to no coherent count — it matches none of the standard countings of the same basis.

Mathematics — domino tilings of an Aztec diamond (6724eed8b60e229bf3eecbbf; key 6, certified 8). The tiling count of an Aztec diamond of order n is 2^n(n+1)∕2 (Elkies et al., 1992), so it is a power of two for every n: 2, 8, 64, 1024, and so on. Six is not a power of two. The certified answer of 8 is the order-two count, obtained by two independent routes. This is the cleanest of the five, because refuting the key requires only the parity of a theorem and no recomputation of the question at all.

Computer science — true boolean expressions of length five (66ebb1e8709fbb756c1f7b74; key 44, certified 47). Alone among the five this one is not checkable without its grammar, so the grammar is given. Terminals are quoted and ∣ separates alternatives; E is the start symbol.

That is: & binds tighter than |, ! is prefix, and both binary operators associate to the left — the standard convention, and the only structural freedom is which of the two binary operators binds tighter. Two structurally independent routes over it — filtering all 7⁵ strings through a recursive-descent parser, and generating bottom-up from the productions — both yield 90 valid expressions of length five, of which 47 are true, on the identical string set. The key is self-refuting: the setter’s own enumeration of its 44 includes two expressions that evaluate to false while filed as true, and omits five genuinely true expressions of equal standing.

so that 44 = 47 − 5 + 2, which is a hand-enumeration error and not a different convention. The enumeration retains both redundant parentheses — ((T)) — and repeated negation — !!!!T — so no consistent grammar excludes what it excludes, and no precedence convention yields 44: the standard one gives 47, and the opposite one — | binding tighter than & — gives 43 over the same 90 valid strings.

4 What was not convicted, and why that section matters more#

Six further questions produced a sound internal value disagreeing with the key, and none was banked, because in each case a reading of the stem rescues the key. These are the cases that make the five above worth anything.

They include an underdetermined projectile problem where the explosion energy is unspecified, a liquid-mirror exponent that is regime-dependent on an unstated rest depth, an α-range question reproducible under one range law but not the textbook one, and two convention- or rounding-dependent values.

The sharpest is a graph-theoretic fixed-point count where an exhaustive search over the standard atlas finds seven against the key’s six, and the sole difference is the single-vertex graph K₁ — which is connected, and is a fixed point, and which the setter’s own code excludes by filtering degree-zero vertices. That exclusion is defensible, so no conviction was recorded, even though the literal reading of the stem gives seven. An audit willing to accuse here would be worth less everywhere.

5 What this is not#

Coverage is 43 questions of 2,370, which is 1.8%. The system declines the overwhelming majority of the benchmark, and nothing here should be read as a claim about how much of it can be answered. A benchmark score and a certified audit are different measurements and the second is not a weak version of the first.

Nor is any of it a claim about the reasoning of systems in general. The faculties are general rather than per-question — no hand-coding, no answer-baking, no reverse-engineering from keys — but the sample that reaches certification is selected precisely by being certifiable, which is a strong and non-random condition. The right reading is that certified coverage doubles as audit at no additional cost, and the audit is what is reported.

6 Falsifiability commitments#

Each commitment below is marked [shape] or [detail], in the sense of Graphic Equalisation’s Section 19. A [detail] failing is a revision: the account survives with that mechanism replaced. A [shape] failing costs range rather than a mechanism, because what fails is something the substrate is claimed to require or to permit — so the account does not hold where it claimed to, which is the more serious of the two and is still a boundary rather than an annihilation.

1.

[shape] Zero false accusations. Fails on the exhibition of a rescuing reading for any one of the five, in which case that conviction is withdrawn and the count is reported lower. Each is stated with its diagnosis specifically so that this is attemptable.

2.

[detail] The certificate is independent of the key. Fails if any certified value can be shown to depend on the key having been read, which the ordering is designed to prevent and which the record is kept to make checkable.

3.

[detail] The diagnoses locate the error. Fails where a key is wrong for a different reason than the one given — the conviction survives and the explanation does not, which is a distinct and weaker failure and is reported as such.

4.

[detail] Tier-B is not a holding pen. Fails if any of the six items named by kind in Section 4 is later convicted on evidence available at the time it was set aside, which would mean the bar was applied inconsistently rather than conservatively.

5.

[detail] The finding is about a discipline and not about this benchmark. Section 1 predicts that any frontier benchmark contains wrong keys. Fails if the same certified-answering run, over a second benchmark of comparable size and review standard, convicts nothing — in which case what is reported here is a fact about one benchmark rather than about what certified answering makes visible. This is the one commitment on the list for which the test has not been run.

Scope note (not a falsifier). Coverage is not the claim. This paper commits to the negative: no statement about benchmark performance is made or implied, and a reader who extracts one has found an error rather than a result.

References