Conformance vectors

What conformance gives youCopy Anchor Link

Two verifiers that agree on 302 exports, including hundreds of tampered ones, agree on what the specification means. That is the point of the kit: it lets the open Go verifier, the engine's own verifier and any verifier you write in another language be held to the same answers. The open verifier runs every vector, target ones included, in conformance_test.go. A differential test also compares it with the reference implementation's verifier over every vector and a deterministic mutation corpus.

Kit layoutCopy Anchor Link

The keys directory also holds a few deliberately odd key files (runner-uppercase.pub, runner-nbsp.pub, runner-oversize.pub, runner-short.pub). They test how a key file is read, and a harness must read key files as the shipped command does: a file over 1024 bytes is an error, never a truncated read.

• exports/<name>/ are golden export directories, generated deterministically by the Numinance evidence code over in-memory adapters with fixed keys, fixed identifiers and a fixed clock. The base exports are verifying Closed changes, an Abandoned, a RolledBack and an open (no terminal record) change, and a Closed change with an attestation gap. Every other export is one of them tampered in one stated way. The generator is private; a vector's meaning is its description and its expect.
• keys/ holds public keys only, one file each, 64 hex characters. No private key is stored and none protects anything.
• The log origin of every vector is numinance.test/log. runner.pub is the runner key and log.pub the log key. other-runner.pub signs only the attacker's export closed-basic-other-runner-key. third-runner.pub and other-log.pub sign nothing and stand for keys a caller pins by mistake.

One vector: expected.jsonCopy Anchor Link

The shape is defined by schemas/evidence/v1/vector-expected.schema.json. This is a real one:

{
  "vector": "closed-allow-unanchored",
  "description": "--allow-unanchored with no pinned log.",
  "status": "shipped",
  "export": "closed-basic",
  "flags": { "runner_keys": ["keys/runner.pub"], "allow_unanchored": true },
  "keys": ["keys/runner.pub"],
  "expect": {
    "level": "verified, unanchored",
    "exit_code": 0,
    "finding_codes": [],
    "target": { "level_enum": "verified", "weakened_by_flags": ["unanchored"] }
  },
  "covers": ["level:verified, unanchored", "exit:0"]
}

Statuses: shipped and targetCopy Anchor Link

A verifier conforms when it matches every shipped vector. It conforms to the newer parts of the specification when it also matches the target vectors and prints the expect.target values.

Materialize operationsCopy Anchor Link

Some conditions cannot be stored portably as data, so a vector asks the harness to build them on a fresh copy of the export:

Ten vectors use a materialize step. A harness on a system without symbolic links may skip those that need them, and says so.

Run one vector yourselfCopy Anchor Link

From the spec checkout, with the open numi built or on go run:

1. Make a fresh temporary directory. Copy exports/<export> into it. Apply materialize in order.
2. Build the command line from flags: --runner-key <root>/<file> for each runner key, the mode and allow flags, --log-key and --log-origin, then --expect-change and --expect-head for target vectors, then --json, then the directory.
3. Run it. Compare the exit code, the level, and the set of findings[].code with expect. ok is true exactly when the exit code is 0.

For a vector without a materialize step, steps 1 to 3 collapse to one command. The vector above is this:

numi evidence verify --runner-key keys/runner.pub --allow-unanchored --json exports/closed-basic

level='verified, unanchored'  level_enum='verified'  weakened_by_flags=['unanchored']  findings=[]

A vector with a materialize step needs one preparation. directory-refused-link replaces records.ndjson with a symbolic link to summary.json. Done by hand on a copy of closed-basic, the verifier refuses the link and never reads through it:

level=failed  findings=[directory_refused]

A small harnessCopy Anchor Link

This is a complete harness for the vectors without a materialize step. Run it from conformance/evidence/v1, passing the path to numi.

#!/usr/bin/env python3
"""Run conformance vectors against `numi evidence verify`. Run from conformance/evidence/v1.
Usage: run_vector.py NUMI VECTOR [VECTOR ...]   (vectors with a materialize step are skipped)"""
import json, subprocess, sys

numi, names = sys.argv[1], sys.argv[2:]
for name in names:
    e = json.load(open(f"vectors/{name}/expected.json"))
    if e.get("materialize"):
        print(f"{name}: skipped (needs materialize)")
        continue
    f = e["flags"]
    args = [numi, "evidence", "verify"]
    for k in f.get("runner_keys", []):
        args += ["--runner-key", k]
    for key, flag in [("trust_bundle_keys", "--trust-bundle-keys"), ("skip_signatures", "--skip-signatures"),
                      ("allow_unanchored", "--allow-unanchored"), ("allow_incomplete", "--allow-incomplete"),
                      ("allow_unsigned", "--allow-unsigned"), ("complete", "--complete")]:
        if f.get(key):
            args.append(flag)
    if f.get("log_key"):
        args += ["--log-key", f["log_key"]]
    if f.get("log_origin"):
        args += ["--log-origin", f["log_origin"]]
    if f.get("expect_change"):
        args += ["--expect-change", f["expect_change"]]
    if f.get("expect_head"):
        args += ["--expect-head", f["expect_head"]]
    args += ["--json", "exports/" + e["export"]]
    p = subprocess.run(args, capture_output=True, text=True)
    try:
        rep = json.loads(p.stdout)
    except ValueError:
        rep = {}
    codes = sorted({x["code"] for x in rep.get("findings", [])})
    x = e["expect"]
    ok = (p.returncode, rep.get("level"), codes) == (x["exit_code"], x["level"], x["finding_codes"])
    print(f"{name}: exit {p.returncode} level {rep.get('level')!r} findings {codes} -> {'match' if ok else 'MISMATCH'}")

closed-allow-unanchored: exit 0 level 'verified, unanchored' findings [] -> match
forged-record: exit 3 level 'failed' findings ['chain_broken'] -> match
directory-missing: skipped (needs materialize)
key-mode-none-is-usage-error: exit 2 level None findings [] -> match

Real runs against expected outcomesCopy Anchor Link

The same harness logic was run against the open numi (built from the checked-out revision) for the vectors below. Each row is the actual result compared with expect. Every row matched.

Beyond these 18, the harness was run over every vector that has no materialize step, 292 of the 302. All 292 matched on exit code, level and finding codes. The ten with a materialize step were not run through the harness; one of them, directory-refused-link, was reproduced by hand above. The open verifier's own test, go test ./sdk/go/evidenceverify, passed in the same checkout and runs all 302 together with the cases no vector can hold.

What has no vectorCopy Anchor Link

VECTOR-COVERAGE.md is generated and lists, for every finding code, level and exit code, which vectors cover it, and the cases that have none.

Writing a verifier in another languageCopy Anchor Link

1. Implement the strict decoder and canonical form first. The canon-* vectors (key order, case-variant keys, number forms, escapes, timestamps, non-printable characters in executor_id) exist because these are where independent implementations drift. The printable-character set is the Unicode table of the Go release that builds the reference verifier, and the vectors name the code points that must be refused.
2. Add the chain walk and signatures, then the index and manifest, then the anchor and attestation.
3. Run every shipped vector. Compare finding codes as sets.
4. Add the report keys level_enum, weakened_by_flags and anchored_head_record_hash, then the target vectors.