# When there's no official test suite, make the parsers vote > transactions.dev · by visibility.cloud — 2026-07-31 > Written for the supply-chain ISV / traceability-vendor engineering org — the VP Engineering / CTO weighing build-vs-buy and pricing the maintenance tax. One thesis: **X12 publishes no conformance test suite and runs no certification program, so "correct" cannot be outsourced to an authority — this engine defines ground truth as what independent open-source parsers agree on, routes their disagreements to a journaled adjudication queue, and publishes the agreement rate as a number you can watch.** ## The absence, verified Before trusting any vendor's correctness story — including ours — it is worth knowing what the standards body actually sells. Checked against x12.org on 2026-07-31: the product list is Glass (view-only browser access to the standard), the EDI Standard itself, code directories, technical reports, and a tiered licensing program. There is no downloadable test corpus, no official conformance suite, no certification program you can pass. What X12 publishes free is a small library of illustrative scenario examples — useful, not systematic. The conformance-testing ecosystem that does exist (Edifecs tooling, CAQH CORE certification, the SNIP validation levels) is commercial and healthcare-centered, and SNIP is a taxonomy of validation types, not a published dataset. That is a search result, dated, and stated as ours: we found no official suite, and we looked with intent. Every EDI vendor's correctness claim therefore rests on something private. The difference worth paying for is whether the something is auditable. ## Four oracles instead of one authority The harness this engine climbs against is built from four mutually reinforcing checks, none of which requires trusting this project: 1. **A license-tagged corpus.** Fixtures from permissively licensed open parser repositories plus a majority of synthetic self-authored documents, every file pinned with source, license, and retrieval date in `CORPUS-PINS.json`. 2. **Cross-parser agreement.** Every corpus document runs through independent open-source X12 parsers as subprocesses — pyx12 (BSD-style, maintained since the early 2000s), Walmart's gozer (Apache-2.0), and node-x12 (MIT) — with outputs normalized to a comparison form of segment stream plus element values, deliberately not any parser's object model. 3. **Byte-exact round-trip.** `emit(parse(x)) === x` on every document, [reported per job](/blog/the-round-trip-law/). 4. **Property-based fuzzing.** Generated envelopes across every legal delimiter choice, fixed-width ISA padding violations, control-number mismatches, truncations — with the invariant set: never crash, never emit invalid canonical JSON, every rejection typed. The same differential idiom covers the acknowledgment layer: the engine's 997s are cross-checked against pyx12's independent 997 generation on a shared corpus, so even [the acks](/blog/three-acknowledgments-one-question/) rest on agreement rather than self-report. The parsers-agree subset is ground truth. Where the independent parsers disagree with each other — and they do, in instructive places — the document goes to an adjudication queue, and a human ruling decides which reading is correct, in writing, in an append-only journal. The ruling references land in the affected [grammar file's provenance block](/blog/grammars-are-data-grammar-pins/), so the chain from "the harness says this is correct" back to "because three parsers agreed" or "because ruling RJ-0012 decided it" is walkable. ## What an adjudication looks like ``` DISAGREEMENT dq-0041 856/004010 fixture: gozer/856-multi-tare.edi pyx12: HL 5 parented to HL 3 (accepts forward reference) gozer: HL 5 parented to HL 3 node-x12: rejects — HL02 references an HL not yet seen RULING RJ-0012 (2026-07, journaled): forward HL02 references are structurally legal; the tree is resolved after the loop completes. node-x12's rejection recorded as a known divergence, excluded from its vote on HL-ordering cases. ``` Two things about that record matter more than the verdict. The disagreement is preserved, not overwritten — the harness remembers that independent implementations read this case differently, which is exactly the information a partner-file dispute needs later. And the excluded vote is scoped: node-x12 still votes on everything else. No parser is an oracle; the population is. ## Why you can't overfit to this The anti-Goodhart argument, spelled out, because it is the actual reason a differential harness beats a private test suite. An engine tested only against its own fixtures converges on its own bugs. An engine tested against one reference parser converges on *that parser's* bugs. An engine that must simultaneously match the agreement set of three independent implementations, round-trip every byte, and survive a hundred thousand fuzzed envelopes has nowhere to hide a convenient misreading — any shortcut that games one oracle trips another. The harness specs themselves are digest-pinned with the same [journaled-ruling governance as the grammars](/blog/grammars-are-data-grammar-pins/), so the target cannot be quietly moved toward the implementation. The agreement rate — the fraction of the corpus on which this engine matches the parsers-agree subset, alongside the open adjudication count — is a published number, per transaction set and version. It is the honest form of a correctness claim: not "conformant," which no one can certify, but "agrees with the independent population here, diverges there, rulings attached." ## The build-vs-buy read If your team builds this in-house, the harness is the part nobody budgets: assembling a license-clean corpus, wiring three parsers as subprocess oracles, staffing the adjudication queue, and keeping all of it honest as X12 versions accrete — a permanent tax, paid in your engineers, against a layer that is not your product. The pitch of this layer is that the tax is ours and the evidence is public: pinned grammars, pinned corpus, published agreement rates, and a [CLI contract](/blog/the-verb-contract/) that lets your team — or your agents — re-run the verdicts on your own traffic. The access flow is at [/get-access/](/get-access/); tell it which sets and versions your traffic runs, because that is literally the prioritization input for the harness's next column. --- Access: https://transactions.dev/get-access/ · All posts: https://transactions.dev/blog/ · Machine face: https://transactions.dev/llms.txt