Checks That Cannot Fail

Have you ever written a test, watched it pass, and only realised much later that it would have passed no matter what you did to the code underneath it? I don't think there's a general answer to "is this check capable of failing" - asking it in full generality is asking something halting-problem-shaped, and I'm not going to pretend otherwise. But the narrow version is answerable and cheap: for one specific check, can I construct one input it must reject, and does it? That question takes about five minutes, and every time I've bothered to ask it I've learned something I did not want to know.

This piece comes out of auditing two things that have nothing to do with each other. One is an empirical bandit benchmark for request routing, where a self-audit turned up twelve documented ways the study could have published a plausible but wrong conclusion. The other is a pair of cryptographic key-handling labs, PKCS#11 and envelope encryption, built to back the key engine and asset encryption case studies. Different domains, different failure surfaces, no shared code. The thing worth writing down is that not one of those failures was a math error, and they sort into the same four shapes in both places.

Here is the whole piece in one table. The rest is evidence for it:

None of these announce themselves. In all four the program runs, the output looks reasonable, and the write-up reads well - which is precisely why they need a mechanical check rather than a careful reader.

Shape 1: the check whose pass is unconditional

The clearest example is a metric, not a test. In the benchmark, "adaptation speed" was defined as: after the environment breaks at $t = 5000$, how many rounds until instantaneous regret returns to normal - where normal meant $1.1\times$ that policy's own average regret over the last 500 rounds.

Two students take a make-up exam after being ill, and the rule for "recovered" is back to your own usual average. The strong student, who normally scores 9, has to reach 9. The weak student, who normally scores 3, has to reach 3. The weak student recovers faster every single time, and it has nothing to do with health.

That is what the metric was measuring. Running it on Round Robin - a policy that cannot adapt to anything, by construction, because it ignores every observation it ever receives - gave $\tau_\text{adapt} = 1.95$ rounds, the fastest score in the table. UCB scored 314. Thompson Sampling scored 1367. Recomputed from the committed results, the rank correlation between this "adaptation speed" and actual final regret is $\rho = -0.836$ across all eleven policy configurations, and across the seven tuned configurations - the set anyone would actually compare - it is $\rho = -1.000$. A perfect inversion. The metric ranked policies in exactly reverse order of how well they performed, and it did so smoothly, with no error, no warning, and a table that looked entirely reasonable.

The same shape, in the crypto labs, showed up twice as an actual test rather than a metric. The JWKS key-rotation lab had an experiment asserting that a gateway which had cached a key during its grace period would still accept a token signed with it - and it re-verified that token against the same gateway instance that had just cached it moments earlier. It passed. It would also have passed if grace-period handling had been deleted entirely, because the assertion never touched a code path that could distinguish the two. The fix was one line of setup: construct a cold gateway that has never seen the key, and separately revoke a key and confirm the cold gateway rejects it.

The second one is worse, because the check that stopped working was itself a security check. The asset-encryption lab has a NetworkBoundary that inspects every payload crossing a simulated network hop against a watch-list of values that must never appear in plaintext, and the demo registers the test's own DEK on that list before running the protocol. It reported clean. It was clean - but it was also clean for the wrong reason: uploadFile() had been handed the test's buffer directly rather than a copy, and since the function correctly zeroizes the key material it works with when it's done, it was zeroizing the caller's buffer too. forbid() was registering thirty-two zero bytes. The whole leak detector was watching for a value that no longer existed anywhere.

The check. Run every verification against something that cannot possibly have the property being verified. If Round Robin scores well on adaptation, the metric is broken, not Round Robin. If the leak detector reports clean on a path deliberately built to leak, the detector is broken. This costs one extra assertion and it is the only one of the four antidotes here that catches its class reliably.

The asset-encryption lab now permanently ships that absurd control: a downloadFileInsecure() that unwraps the DEK and sends the raw bytes back, existing for no reason other than to be caught, every run, so the detector has to prove it still works before its clean report on the real path means anything.

Shape 2: the measurement that runs perfectly and measures something else

This one produces no anomaly at all. The code executes, the numbers are stable, they're reproducible to three decimal places, and they are about a different quantity than the one in the caption.

In the benchmark, two genuinely different things were being used interchangeably. Non-optimal rate is the share of rounds spent on an arm other than the environment's true optimum - it requires ground truth, and it measures how often you were wrong. Non-greedy rate is the share of rounds spent on an arm other than the one the policy itself currently believes is best - it needs no ground truth, and it measures how much you explored. A policy can be 0% non-greedy and 100% wrong, diligently exploiting a belief that happens to be mistaken. The results table defined the first; simulator.py computed the second, never once touching env.optimal_arms. A figure of 53.74% was then used to argue that a policy "must sustain a very high non-optimal selection rate, causing regret loss" - but under the definition actually computed, that number only says it explored a lot, and exploring a near-optimal arm costs almost nothing.

The crypto version was a benchmark rather than a metric. directHardwareSign() was supposed to measure the cost of one signing operation through the HSM boundary, and it did measure something very precisely and very reproducibly: it generated a fresh RSA-2048 keypair on every call before signing. The reported figure was dominated by key generation, an operation the real system performs approximately never. The fix - provision the key once, then sign - changed the number by more than an order of magnitude and changed which of the two architectures being compared looked faster.

The check. The sentence defining a quantity and the line of code computing it have to match word for word, and someone has to actually put them side by side. "Exploration" and "error" are different words. "Signing" and "keygen plus signing" are different words. Nothing in either system will ever tell you they've drifted apart, because from the inside both are working correctly.

Shape 3: the number that entered prose without a source

Three separate entries in the bandit catalogue are the same failure in different costumes, which is what makes it a class rather than an incident.

A "verification" section reported the Jensen gap between $\mathbb{E}[1/L]$ and $1/\mathbb{E}[L]$ as "varying from 12.5% to 14.8% depending on the skew." Both closed forms were already printed in the same document: $\mathbb{E}[L] = k\theta$ and $\mathbb{E}[1/L] = 1/((k-1)\theta)$, and multiplying them cancels $\theta$ entirely, giving $k/(k-1) = 1.25$ for $k=5$ - a constant, identical for every arm and every timestep. The paragraph also managed to say the gap "varies with skew" and "is constant across arms" a few lines apart, which cannot both be true, and which nobody noticed because both sentences read fine.

A derivation rested on a "normalised gap ≈ 0.35–0.4" with no formula, script, or reference behind it. Three different quantities in that document were all being called gap; the one the formula actually needed - the gap in online-normalised reward space - measures 0.0823, and substituting it changes the derived window requirement from $\tau \gtrsim 1200$ to $\tau \approx 30{,}481$. The qualitative conclusion survived and in fact got stronger, which is the uncomfortable part: a wrong constant had been sitting inside a correct argument, and the argument's correctness gave no signal at all about the constant.

And two headline sentences quoted ranges that contradicted the tables directly above them - "reduces regret by 30–50% across all scenarios" when three of six cells fell outside, one as low as 17.9%.

The fix for all three was mechanical: a script that regenerates every summary number directly from the raw results, so no aggregate in the report is typed by hand. It worked. It also did not stop the same failure from happening a fourth time, which I'll come back to.

The crypto version of this one is less interesting to describe and more embarrassing to admit: the first version of the key-engine lab's README quoted throughput figures that simply did not match what the code produced when run. Not off by a rounding error - describing a different run, on a different version, from memory.

The check. If a number appears in prose, a script should have produced it. If the quantity has a closed form, check the measurement against the formula before putting it in a section called "verification." And "approximately 0.35" with no provenance is a guess wearing the costume of a constant.

Shape 4: the comparison that was never fair

The benchmark's headline finding is this class, so it gets the short version here and the full treatment elsewhere: ε-greedy was run at c = 0.1, which yields $\sum_t \epsilon_t \approx 9.8$ exploration rounds across ten thousand requests with ten arms - roughly one random look per server for the entire run - while the policies it was being compared against each got a five-point grid search. The comparison then drew a conclusion about mechanisms, directed versus random exploration, from a table where one algorithm had been configured into uselessness and the others had been tuned. Under an equal budget the ranking inverts.

The tuning grids had a related problem with a distinctive tell: both non-stationary policies' optima sat on the last point of the grid with the trend still monotone through it. An optimum at the boundary is not a result, it is a report about where you stopped looking - turning a dial to its stop while the signal is still improving and calling the stop the best frequency. Extending the grid moved both optima outward until they hit the only place they could stop: the degenerate limits, where SW-UCB with $\tau = T$ is UCB and D-UCB with $\gamma = 1$ is UCB. The confirmation is bit-level rather than approximate - tuned SW-UCB's per-instance regret differs from UCB's by exactly $+0.00$ on 5 of 5 instances at every drift density tested.

The crypto labs hit this class from the opposite direction, where making the comparison more honest is what exposed it. Once key unwrapping went through a real PKCS#11 call instead of a simulated one, the "fast" envelope-encryption path started calling into the HSM on every single operation - which is precisely the bottleneck envelope encryption exists to avoid. The architecture hadn't changed; the benchmark had stopped flattering it. Both labs now cache the unwrapped key with an explicit TTL, and both write down what that cache costs in security terms rather than only what it saves in latency.

The check. A comparative claim needs its budget stated in the same sentence. "A beats B" is not a result until it says under what configuration, tuned how, with how much effort spent on each side - and when the loser's constants came from a previous paper's defaults, the finding is about the defaults.

The part that makes this worth writing down

The catalogue of eleven pitfalls was published in July. In August, writing up the benchmark for this site, I recomputed every number in its top-level README from the committed results instead of trusting the README - and three of its summary sentences did not survive.

A "median improvement of 8.1×" is not the median of anything; the actual median is 5.2×, and 8.10× is the value of one cell in the table directly above, lifted and relabelled as an aggregate. A claim that Thompson Sampling beats UCB "consistently, $p = 0.007$" holds on two of three scenarios and fails on the third at $p = 0.206$ - one scenario's $p$-value promoted to a general statement. And a caveat that a policy was "statistically indistinguishable" from a baseline was wrong in the specific way that word choice matters: paired across instances it is distinguishable in all three scenarios ($p = 0.0087$, $0.0088$, $0.0367$), because the differences are small but consistent in sign, which is exactly what a paired test is built to detect. The intended word was practically.

All three are shape 3. All three were written after the fix for shape 3 already existed and was working.

The reason is worth more than the errors. The fix - a script that generates every aggregate - was scoped to the artefact where the bug was found, which was the report. The README is a different published surface, written by hand, outside that pipeline, and the failure walked back in through the one door nobody had thought to cover. Someone will point out that this is just insufficient discipline, and that the answer is to be more careful next time, which is exactly the answer I'd distrust: the whole reason these twelve are worth cataloguing is that every one of them survived a careful author reading their own work several times. Care is what produced the plausible-looking numbers in the first place.

It is also not a coincidence which sentences were wrong. The headline ratio, the significance claim, the honest-sounding caveat - summary prose is where a number does the most rhetorical work and receives the least scrutiny, because by the time you're writing it you are describing something you already believe.

What actually generalises

Four checks, in the order I'd apply them, and none of them require knowing the domain:

Run every verification against something that cannot pass it. This is the only one that reliably catches its own class, and it is the cheapest.

Put the defining sentence next to the computing line. Not "does the code work" - does it compute the thing the caption says.

Re-derive rather than re-read when quoting your own prior work. Reading your own summary checks whether you remember it, not whether it is true.

Fix the failure mode, not the artefact. Every fix should end in a question: what else does this apply to that I haven't checked yet?

The bandit catalogue now has a twelfth entry, and the twelfth is a repeat of the tenth. I'd rather have that than a list that stopped growing, because a list that stopped growing would only mean I'd stopped re-deriving my own numbers - and this piece exists mostly because writing it out in full sentences is the check I trust most. Two of the findings in the research write-up it came from didn't exist until I had to explain the data to someone who wasn't me.