Conformance

How Flametrench SDKs are verified

The specification is the product. For that to mean something, every SDK must produce byte-identical behavior for byte-identical inputs on every spec-codified operation. The Flametrench conformance suite is the machinery that proves it.

Language-agnostic fixtures

Test vectors live as JSON files in the spec repo — not as code in each SDK. Every SDK loads the same files.

RFC 2119 levels

Every fixture file declares MUST, SHOULD, or MAY. A MUST failure means an implementation cannot claim conformance.

Spec-linked

Each fixture references the normative paragraph that authorizes it. When the spec moves, a grep finds what to update.

Drift-checked in CI

Each SDK vendors the fixtures and runs them; CI fails if the vendored copy diverges from upstream. Drift is impossible.

How a fixture looks

A fixture file declares one capability × one operation, then lists one or more test vectors. Each vector has an input and an expected — either a result value or an error type.

{
  "$schema": "../../fixture.schema.json",
  "spec_version": "0.1.0",
  "capability": "ids",
  "operation": "decode",
  "conformance_level": "MUST",
  "spec_section": "docs/ids.md#decoding-rules",
  "description": "Malformed wire-format IDs MUST be rejected.",
  "tests": [
    {
      "id": "decode.reject.max-uuid",
      "description": "Max UUID (version nibble f) is rejected.",
      "input":    { "id": "usr_ffffffffffffffffffffffffffffffff" },
      "expected": { "error": "InvalidIdError" }
    }
  ]
}

What's covered in v0.1

Capability	Fixture files	Runnable today
IDs 48 tests. Both @flametrench/ids (Node) and flametrench/ids (PHP) pass the complete suite.	encode · decode · decode-reject · is-valid · type-of	Yes
Identity Placeholder. Fixtures land when the first SDK ships an identity layer.	argon2id verification · hash-floor · OIDC normalization · session-rotation	Planned
Tenancy Placeholder. Requires SDK state; lands with the tenancy SDK layer.	invitation-accept · sole-owner-transfer · admin-remove hierarchy · role-change chain · removed_by attribution	Planned
Authorization Placeholder. Requires a tuple store; lands with the authz SDK layer.	check exact-match · set-form · no-derivation · uniqueness · cascade · pagination	Planned

How a third-party implementation claims conformance

1. Vendor a snapshot of spec/conformance/fixtures/ into your SDK's test tree.
2. Write a harness that loads index.json, iterates each fixture file, and dispatches each test to your implementation's operation.
3. Add a CI job that re-checks out flametrench/spec at main and diffs your vendored copy against upstream. Any drift fails the build.
4. Publish a CONFORMANCE.md in your repo declaring the spec version you target, which fixture files you pass, and which you skip (with reasons).

The reference harnesses in PHP and Node are each under 150 lines — harness-writing is a small investment for the behavioral guarantee the suite provides.

Why this is better than writing tests per SDK

Single source of truth. One fixture row instead of six duplicated inline constants across SDKs.
Zero porting cost for new languages. A Go, Rust, or Python SDK in the future consumes the same fixtures. New language, same tests.
Spec-grepable. Every fixture file cites its spec section. If docs/tenancy.md changes, finding the fixtures that need review is a grep spec_section away.
Machine-graded. Harnesses emit per-fixture pass/fail output. A conformance dashboard or badge system can consume it without parsing language-specific test runners.
Designed for third-party validation. Any external implementation can prove conformance against an unbiased external corpus — there's no "but their tests are lenient" argument to be made.

Full framework documentation, the meta-schema, and all fixtures live at github.com/flametrench/spec/tree/main/conformance.