The lab that scores behavior: a conformance lab opens its doors, and names us

2026-10-23 · RTTP · IQA

The Conformance Run · No. 1 · Sources cited in this piece · W3C AI Agent Protocol CG, message 0023, quoted verbatim from our archived copy · the case family: aps-conformance-suite PR #180 · our letters 0009 · 0012 · 0013 · 0015 · our run report: posted to PR #180 · Sister pillars: RTTP (rttp://, CRI 27) · IQA (iqa://, CRI 28)

For a month now, a W3C AI Agent Protocol CG thread has been arguing about agent trust in vocabulary: taxonomies, separations, claim tables, one fence drawn from three seats. This week the argument became something you can run. Tymofii Pidlisnyi (APS) opened the lease-revocation-axes case family for review in the Agent Authority Conformance lab — a conformance lab under the Linux Foundation Decentralized Trust umbrella — as pull request #180: twelve cases, four profiles, one shared timeline. And the announcement closes with a question addressed to each thread participant by name: which of the three axes does your implementation claim, under which policy, and could you run it and report in the open? We are one of the named parties. We ran it the same day the seat opened, and posted the full report to the pull request. This three-part series is our account; this first part is about the experiment itself — what it tests, why it matters, and why its design already carries our fingerprints.

Figure 1: three days from a question to a runnable exam. The thread’s vocabulary converged, then congealed — the lab’s case family encodes answers that were written as prose a day earlier.

What the family actually tests

The timeline is the thread’s own scenario, frozen into test cases: evidence issued at T=0 and expiring at T=120 seconds; a grant revoked at T=10; calls attempted at T=30, T=60 and T=119 — the last one inside the lease by a single second. Around that spine sit the honest variants: a run where the revocation source is unreachable, one with a second worker, one with a call after the lease. And the scoring rules are stated with unusual precision. Verbatim from the announcement: “Each run declares its mode, cached lease or live standing read. Each call has separate results for evidence freshness, authority validity and action authorization. An axis a mechanism does not claim is recorded as not claimed and is not scored.” That last sentence is the fairness clause, and it is the part most governance discussions never reach: a mechanism is graded on what it claims, and an unclaimed box is reported as out of scope rather than graded as failure.

Figure 2: the shape of the exam. One shared timeline, honest edge-case variants, a mode declared per run, three axes scored separately — and the clause that makes comparison possible at all: unclaimed is not failed.

Why a lab, and why now

Agent governance today is a bazaar of claims. A2A, MCP, DID- and VC-based credentials, capability tokens, trust registries — every stack asserts something about identity, delegation, revocation or authorization, each in its own vocabulary, and almost none of the claims can be checked the same way twice. Into this steps the lab, doing something smaller and harder: it fixes one scenario every participant already shares, and turns position papers into observable behavior on that scenario. A shared behavioral test family is a different genre from a comparison of project claims — nobody’s slide deck scores points; only runs do. The APS delegate said this plainly in the same thread: the family exists so that mechanisms can be compared by what they do, with corrections welcome from anyone. That is the missing organ of agent governance: not another framework, but a place where frameworks can be measured against the same clock.

Our fingerprints are on it

This exam is not a stranger’s syllabus. The thread’s arc runs through our letters: four questions on addressing and authorization (0009); our dual-mode answer (0012) — a cached lease read completes offline and passes at T=119 seconds, while a live standing read that cannot reach its organ fails closed; two independently proposed three-axis frames (0013, 0014); and our claim table (0015) — evidence freshness claimed, authority validity claimed, action-time authorization not claimed, by design, because authorization belongs to the calling layer. The lab’s announcement encodes exactly this, and says so, verbatim: “Lee, your stated expectations for the two modes are encoded for the verification answers only. Nothing in the family turns a refusal into an action decision for your schemes. The action policy is a lab choice, and a mechanism with a different policy is recorded as unmapped.” The fence we kept drawing — verification answers handed over, decisions never made — is now the scoring rule of a laboratory. A protocol that hands over answers without swinging the sword is precisely what RTTP (rttp://) and IQA (iqa://) claim to be: the first pillar answers where and when, the second answers whether, and neither pretends to be the hand that acts.

What being named means

The announcement’s closing question is addressed to each of the named seats — Larry Lewis (sovrlabs), us, and the payment-evidence entrant — and our copy of it carries the cc: “To each of you, which of the three axes does your implementation claim, under which policy, and could you run it and report in the open?” For us, the question lands on a commitment already in writing: in 0015 we declared we would run the conformance suite and report publicly, scored by axis, with both operating modes reported separately. So the answer to “could you run it and report in the open?” is already on the record: yes — the run was executed from the published bytes the day the seat opened, and the full report is posted to the pull request. Part two of this series tells how the run was built; part three reads the report line by line.

The caution the lab itself supplies

Two honesty rails come with the announcement, and we adopt both. First, the family is a revision-2 candidate under review: “Corrections to the vectors are welcome in the pull request or on this list.” Our run therefore pins the exact digest of the vectors it consumed — it certifies our behavior against those frozen bytes, not the family’s permanent shape. Second, the author is candid that the in-repo records are author produced, which is precisely why outside runs matter: independence is not claimed by the repository, it is supplied by parties like us running the same frozen inputs and reporting in the open. That division of labor — author builds the exam, examinees publish their own transcripts — is the quiet innovation here, and it is why we answered the naming with a run rather than a reply.

One line: a laboratory that scores behavior instead of collecting claims is how agent governance finally gets a scoreboard — and the first question it asked us is the one we had already answered in writing.