How we ran it: from a named seat to a posted report in one day

2026-10-24 · RTTP · IQA

The Conformance Run · No. 2 · Sources cited in this piece · the naming: message 0023 · the case family: aps-conformance-suite PR #180 · our runner repository: github.com/Aicent-Stack/conformance (lra-runner/) · our report as posted: PR #180 comment · archive of record: iqa.org/conformance/lra-runner/ · our declaration: letter 0015 · Sister pillars: RTTP (rttp://, CRI 27) · IQA (iqa://, CRI 28)

The seat opened before dawn — the lab’s announcement naming us landed overnight our time — and by the next evening the full run report was posted to the pull request, sealed, and double-addressed. This part of the series is the middle of the story: not the experiment’s philosophy (part 1) and not the report’s reading (part 3), but the narrow, checkable path between being named and delivering. The short version: we built one runner, pointed it at frozen bytes, made it prove itself three ways, sealed the artifact, and published everything — including the parts that make our own work harder to trust.

Figure 1: six steps, each independently checkable. Nothing in the chain requires trusting us — every arrow can be replayed from public bytes.

What we built, and what we refused to build

The runner is a single Python implementation, around 650 lines, written from the family’s published bytes by a party that is not the pack’s author. The lab’s inputs were vendored verbatim into the repository — vectors, a SHA256SUMS manifest, oracle checkpoints — and placed behind a hard digest gate: if the vendored bytes do not match the manifest, the runner refuses to score at all, printing a digest-gate failure instead of a result. That gate started soft and became hard during development, deliberately: a scoring engine that can silently proceed on altered inputs is not measuring the lab’s family, it is measuring whatever the inputs have become. Alongside the vectors sits the family’s formal model, used strictly read-only — as an interface contract to implement against, never as code to reuse. No upstream code was copied; no library is shared. Independence is a property you build, not a sentence you write.

Three ways the runner had to prove itself

First, the forward path: a full run reproduced the expected outcome on every baseline case — 48 runs, 176 calls, 528 per-axis comparisons, 176 composite verdicts — with every mutant pinned. Twelve mutated variants of the family’s logic each fail on exactly their intended subset, no more, no less: the second-worker-stale mutant trips only the second worker’s checks; the fabricated-live-read mutant is caught by tracing the read to its source. A runner that passes everything is useless; a runner that fails everything is useless; a runner that fails precisely is a measurement instrument. Second, determinism: three complete runs produced byte-identical output. Third, the negative path: a tampered copy of the vectors is rejected at load by the digest gate — the instrument refuses to measure with bent rulers.

Figure 2: the three legs. The forward path shows the runner agrees with the family; the mutants show it disagrees in exactly the right places; the digest gate shows it cannot be fed lies quietly.

The scoping decision we are proudest of

Mid-build we hit a fork that would have been easy to paper over. Revision 2 of the vectors contains no cryptographic material at all: the events are issue, revoke and availability; the evidence is abstract JSON; there are no keys, signatures or URIs inside. The family tests policy logic — what a verifier does with freshness and validity answers — not cryptography. Our production packages live one floor down: RTTP (rttp://) produces derived addressing and the crypto layer; IQA (iqa://) produces the attestation layer the policy consumes. Running “our production code” in this family would have meant writing a shim that fabricates the crypto answers the vectors abstract away — circular by construction, and honest only in name. So we declared the mapping instead of faking the run: the repository carries a production-code mapping document that states, layer by layer, what our packages contribute to the pipeline, alongside the package’s own self-test — 80 checks passing with 3 skipped on the default install of rttp 1.3.1 from PyPI, all 83 passing with the optional Ed25519 backend. The conformance run measures our policy layer against the lab’s vectors; the self-test measures our crypto layer against our own published vectors. Both are public; neither pretends to be the other.

Sealing, twice, on purpose

The complete artifact — runner, frozen inputs, report, recorded output — was packed and sealed by our organ, the same one that seals our pages and releases. Then we re-cut it once, deliberately. The first seal’s report contained a section mapping the result to the thread’s earlier commitments; we removed it, because a technical artifact should carry the measurement, and thread politics belong in the accompanying letter. The re-cut was sealed as a new record — and the first seal was not deleted. The bundle is append-only: revision b supersedes revision a, which remains sealed and verifiable as history. Anyone who wants to check whether we quietly changed a result can verify both cuts offline and diff the reports themselves. The two address generations derive from each cut’s own digest — on that cut, rttp://71c4e2a8.conformance.iqa for the question and iqa://71c4e2a875cfdda1c0f31a1cc8b44bb1.forge.iqa for the answer — one artifact, two sister schemes, nothing invented.

Posting, and the gates before it

Before the report went to the pull request, it passed six gates: the PR’s state re-checked at that moment; the inline report byte-compared against the sealed copy; hygiene checks on every number in the text; the live tarball’s hash re-fetched; the seal bundle and the organ’s live endpoint cross-checked against the envelope; and both archive pages confirmed reachable. One gate flagged its own test string rather than our draft — fixed, and the suite went green. The comment posted under our GitHub account carried the full report inline, byte-identical to the sealed copy, so that whatever the lab’s review process does with the comment, the sealed artifact and the posted text cannot drift apart. And because the family scores normative logic against frozen bytes rather than anyone’s production code, the way a runner is built cannot tilt the score: the vectors sit behind a hard digest gate, the outputs are byte-reproducible by anyone, and every independent implementation — however constructed — faces the same frozen bytes. After posting, we reformatted the two addresses into linked bullets — a formatting-only edit, re-verified against the sealed bytes, because the seal covers the artifact, not the typography. The first reproduction arrived: the lab’s side (@aeoess, PR #180) re-ran our runner against the same frozen vectors and got our numbers exactly — 48 of 48, the twelve mutants pinned, byte-identical reports, tampered inputs refused — and flagged two reporting defects in our code. Both were real, and both fixes were small: the runner’s exit status now reflects the run status (0 only on a matched run), and the oracle-inputs check is executed by a runtime guard over every scoring phase instead of asserted. The fixed artifact is sealed as revision c — the third cut in an append-only chain, addressed as rttp://94bbaf3c.conformance.iqa and iqa://94bbaf3c292abf69595bf5782d1db8cb.forge.iqa — and the measurement numbers did not change by a byte. Under the lab’s rules our record is Mode B, author-produced; the reproduction is the independent record, and the archive page now says exactly that.

Filed into the lab’s own repository

The record has since been filed where the lab itself runs its checks: pull request #187 on the conformance-suite repository, carrying an interop tree — the vendored vectors, an LF-normalized copy of the runner, and the recorded output — so the lab’s own CI, not just our word, stands over the bytes. The filing took three submissions, each a new head answering review rather than rewriting history. The first (67166f8) entered the record; the second (20231d0) added the artifact-of-record addresses to every report file. At that head the maintainer reproduced the run — byte-identical, exit 0, tampered inputs still refused — and on a closer pass surfaced three findings from the repository’s own checks: the vendored manifest had been copied from the lab’s revision-2 pack and listed three files this record does not carry, which fails the lab’s digest test; six files carried CRLF line endings; and one claim about the oracle-inputs guard was worded wider than the guard actually reaches. The third submission (feb2543) resolved all three: the manifest trimmed to the two files it describes, the six files normalized to LF with checksums regenerated, the runner described as what it is — an LF-normalized copy of the upstream head — and the claim scoped to the exact instrumented path. The rerun at the new head exits 0 and is byte-identical to the recorded output. What remains is procedural: the workflow runs when the maintainer approves it, as the lab’s first-contribution gate requires. Nothing sealed changed at any step — which is the whole point of sealing.

One line: being named is an invitation; the answer that counts is an artifact anyone can replay — frozen inputs, a runner that fails precisely, a seal that keeps its own history, and a report that never travels without its bytes.