Reading the run report: how to referee our conformance run without trusting us

2026-10-25 · RTTP · IQA

The Conformance Run · No. 3 · Sources cited in this piece · the report as posted: PR #180 comment, byte-identical to the sealed copy · archive of record: iqa.org/conformance/lra-runner/ · the family: aps-conformance-suite PR #180 · the naming: message 0023, quoted verbatim · our declaration: letter 0015 · resolvers: rttp.com/open · iqa.org/open · Sister pillars: RTTP (rttp://, CRI 27) · IQA (iqa://, CRI 28)

The report is up. It is posted inline on the pull request, sealed on our archive of record, and reproducible from a public repository with one command — so the honest question is not whether to trust our summary of it, but how to read it yourself. This third part is a referee’s guide: what the report contains, which numbers carry weight, which ones are decoration, and where the interesting honesty lives — in the empty boxes, the pinned mutants, and a pair of resolvers that visibly disagree about the same artifact.

Figure 1: six parts, in reading order. Start at the claims declaration, not the green rows — the declaration tells you what a pass would even mean.

Read the declaration before the results

The report opens the way the lab’s announcement asked everyone to: by declaring claims. Our letter 0015 put two boxes in the table — evidence freshness claimed, authority validity claimed — and left the third empty on purpose: action-time authorization is not claimed, by design, because authorization belongs to the calling layer. The run report inherits that table exactly, and the family’s own scoring rule makes it load-bearing. Verbatim from the announcement: “An axis a mechanism does not claim is recorded as not claimed and is not scored.” So when you scan our per-vector table and see a column that ends in “not claimed” rather than pass or fail, you are not looking at a weakness we hid — you are looking at a boundary we published in advance, and the lab’s own grammar for keeping it visible. The announcement says the same of us specifically: “Nothing in the family turns a refusal into an action decision for your schemes.”

Read the modes second: one experiment, two verdicts

Every run declares its mode, and the two modes produce deliberately different stories. In cached-lease mode, our scheme answers from the lease carried in-band: the verifier checks the statement against its own clock, offline, and the T=119 call passes — including the variant where the revocation source is unreachable, because a cached lease never needs the network to prove freshness. In live-standing-read mode, the scheme must reach its organ; when the organ is unreachable, every call is refused — fail-closed, at all three timestamps. Neither story is the flattering one and the other embarrassing; they are two different products, priced honestly. The lease says: here is how fresh this evidence can be, and how old it may get before the verifier stops trusting it. The standing read says: here is what happens when the answer must come from the authority and cannot. A mechanism that only published one of the two stories would be easier to market and harder to compare. We published both, because letter 0015 pre-registered both.

Figure 2: the anatomy of a row. Mode first, three axes separately, the unclaimed column marked out of scope rather than failed — and the raw output printed after the table so our summary is never the only text in evidence.

Read the mutants first: green boards are cheap

The forward path is the least interesting page of the report: 48 runs, 176 calls, 528 per-axis comparisons, 176 composite verdicts, all matching the expected outcome. A board that is entirely green proves one of two things — either the implementation is correct, or the implementation is incapable of failing. The report’s real calibration page is the mutant battery: twelve mutated variants of the family’s logic, each of which must fail, each pinned to its exact intended failure set and nothing else. The second-worker-stale mutant trips only the second worker’s checks. The fabricated-live-read mutant is caught by tracing the read to its source. Twelve out of twelve, no collateral failures, no missed detections. That is what separates a measurement instrument from a rubber stamp: not that it agrees with the expected answers, but that it disagrees in precisely the places where reality has been bent. Add the negative path — a tampered copy of the vectors is rejected at load, refused the chance to produce any result at all — and the report has made its strongest claim without a single green row: this instrument cannot be fed lies quietly.

The amber and the green: what the resolvers demonstrate

One display in this story is not in the report but next to it. The sealed artifact is one thing at two addresses — both derived from the artifact’s own digest — and opening them shows our two resolvers behaving differently. rttp://94bbaf3c.conformance.iqa, the question side, parses the intent cleanly, then marks the pillar — conformance — as outside the seventeen-pillar table and the root as not the DNS-anchored one, in amber, and still resolves: the pillar space is open by grammar, operator-chosen labels are legal, and the resolver’s job is to parse and annotate, never to pretend it recognizes what it does not. iqa://94bbaf3c292abf69595bf5782d1db8cb.forge.iqa, the answer side, shows every check green: the organ forge is inside a closed set, the root is its own sovereign anchor. Two behaviors, both correct, one lesson — the same distinction the report quantifies. An open action space cannot be scored by a closed checklist, and a closed verification set must never wave through an unknown. The report’s “not claimed, not scored” row and the resolver’s amber badge are the same honesty wearing two uniforms.

The boundary lines

Four lines bound what this report claims, and each is printed in the report itself. First, the family is a revision-2 candidate under review; our run pins the digest of the vectors it consumed, so it certifies behavior against those frozen bytes, not against whatever the family becomes next. Second, this is one family, not the suite: we ran lease-revocation-axes because that is the seat we were named to; other families remain open. Third, the report measures our policy layer — revision-2’s vectors contain no cryptographic material by design — while our packages’ own self-test measures the crypto layer separately, 80 checks passing with 3 skipped on the default install of rttp 1.3.1, all 83 with the optional Ed25519 backend; neither number is presented as the other. Fourth, the determinism claim — three runs, byte-identical — extends only to the recorded environment, which the report prints in full. None of these lines weakens the result. They are what makes the result a measurement instead of a victory lap.

Since posting: re-verified in the lab’s own house

After this guide was drafted, the same report was filed as an interop record into the lab’s own repository (pull request #187), where the lab’s CI — not our claims — runs over the bytes. Three submissions, briefly: the maintainer reproduced our run at the filed head and got our numbers exactly — byte-identical output, exit 0, tampered inputs refused at the digest gate; the repository’s own checks then surfaced three process findings (a vendored manifest listing files the record does not carry, six CRLF files, one claim worded wider than its guard); and the third submission resolved all three without moving a number — the rerun at the new head is byte-identical to the recorded output. For a referee, the quiet part is the shape of that story: every finding was about packaging and wording, none about the measurement, and every fix landed as a new head on top of the same sealed result. The report has now survived a reproduction and a review, and neither changed its arithmetic.

One line: a run report is not a trophy cabinet — it is a calibration certificate, and the parts that impress a referee most are the empty box, the pinned mutants, and the amber badge that proves the instrument would tell you when it does not know.