Verify a run

When to use thisCopy Anchor Link

Run verification whenever a reviewer, an auditor or a release gate must decide whether a change's record can be believed, without access to the system that produced it. The verifier is offline, read-only, bounded in what it reads, and open source, so the party doing the checking does not have to trust the party that made the export.

PrerequisitesCopy Anchor Link

• The open CLI built from a spec checkout: go build -o ./bin/numi ./cli/numi, or go run ./cli/numi .... See Install.
• An export directory. In this guide, the fixtures under conformance/evidence/v1/exports/.
• The pinned public keys: the runner key (64 hex characters), the transparency log key and the log origin. You obtain these from your own records, not from the export.

All commands below were run from conformance/evidence/v1 inside the spec checkout, using the prebuilt numi at revision 3c74d59.

What verified meansCopy Anchor Link

The verifier reads the records of one change, recomputes each hash, checks each signature against the keys you pinned, checks that the chain head is proved under the log you pinned, and re-derives the attestation from the chain. The result is one of several levels. verified is the top one, and it is the only one that rests entirely on keys you chose.

What verified does not say: that the code is correct, that the tests were thorough, that the model reasoned well, or that a compromised host logged every event. It says the provenance and chain integrity hold under the keys you pinned. Read Threat model and limits before relying on it.

change: chg_01010101010101010101010101010101
records: 12, head sha256:8dec4e3d…
state: ended Closed
level: verified
chain: records link, hash and end as stated
signatures: verified under the pinned runner keys
anchor: head proved under the pinned log
attestation: signed, re-derived from the chain
result: verified

Verify a good exportCopy Anchor Link

numi evidence verify \
  --runner-key keys/runner.pub \
  --log-key keys/log.pub --log-origin numinance.test/log \
  --complete exports/closed-author

change: chg_01010101010101010101010101010101
records: 12, head sha256:8dec4e3d5cf4bd7650ef6010671298793c05493cb20c3dd946bc3ef805689e48
state: ended Closed
level: verified
chain: records link, hash and end as stated
signatures: verified under the pinned runner keys
anchor: head proved under the pinned log
attestation: signed, re-derived from the chain
credential_scopes: unsourced: no credential broker statement is in the attestation
network_policy: unsourced: no egress proxy statement is in the attestation
not applicable: deployment.json (not applicable at PR level: the change closed from Merging (design section 6) and was never released, deployed or observed)
not applicable: observation.json (not applicable at PR level: ... same reason)
result: verified

Two lines are not green and should not be skimmed. credential_scopes and network_policy are unsourced: the attestation carries no statement from a credential broker or an egress proxy, so the export proves nothing about those. And deployment.json and observation.json are marked not applicable because this change closed at PR level, and the chain bears that out. A part declared not applicable that the chain contradicts is a failure (part_contradiction).

Four ways it failsCopy Anchor Link

The conformance suite ships a failing vector for each tampering case. Each of these runs below used the same pinned keys as the success above, against an export that the vector names. The absent: lines that every failed report also prints are omitted for brevity.

1. A record edited in place: chain_brokenCopy Anchor Link

The vector forged-record edits a record and leaves its record_hash, so the hash no longer matches the content.

change: chg_01010101010101010101010101010101
records: 0, head 
state: open (no terminal record)
level: failed
chain: failed
signatures: not verified
anchor: not proved
attestation: none
FAIL chain_broken: a record does not match its record_hash
result: failed

Notice records: 0 and no head: a report for a broken chain does not claim a head, signatures or an anchor.

2. A signature changed: signature_invalidCopy Anchor Link

The vector signature-edited-in-place changes one base64 character of a record's signature. The key is trusted and the signature fails.

level: failed
chain: failed
signatures: not verified
FAIL signature_invalid: a record signature does not verify
result: failed

3. A signer you did not pin: signer_untrustedCopy Anchor Link

An attacker who builds a complete, self-consistent chain with their own key gets a chain that is internally valid. It is rejected because their key is not one you pinned. This run points --runner-key at the wrong key for the closed-author export:

level: failed
FAIL signer_untrusted: a record names a key that is not trusted
FAIL part_contradiction: a part is declared not applicable and the chain does not bear it out
FAIL part_contradiction: a part is declared not applicable and the chain does not bear it out
result: failed

(The two extra findings follow from the signer failure: the not-applicable declarations cannot be borne out by a chain whose signatures are not trusted.)

4. Another genuine chain: expectation_mismatchCopy Anchor Link

A different but internally consistent export of the same change, signed by the pinned keys and anchored, verifies. A verifier cannot tell it from the original on its own. That is what --expect-head and --expect-change are for: you record the head hash when you approve a change, and the check later fails if the export presented is not that chain.

numi evidence verify \
  --runner-key keys/runner.pub \
  --log-key keys/log.pub --log-origin numinance.test/log \
  --expect-head sha256:cfe3d79b4663e5a178e3720a0f44c5503a79c7fe5c2230192fc3bdadb8e89393 \
  exports/closed-basic

change: chg_01010101010101010101010101010101
records: 4, head sha256:f660cbc6ee23d051bd40f405bc60f2947b10577324e15172e5f1464b624b0d8c
state: ended Closed
level: failed
chain: records link, hash and end as stated
signatures: verified under the pinned runner keys
anchor: head proved under the pinned log
attestation: signed, re-derived from the chain
FAIL expectation_mismatch: the head is not the one expected
result: failed

Here the chain and signatures are fine, which is the point: only the caller's own expectation catches a swap for another valid chain. --expect-change takes a chg_ id and exits 2 with a usage error for anything else.

The full set of vectors is under conformance/evidence/v1/vectors/*/expected.json, each naming its export, flags, exit code and finding codes. See Conformance vectors.

Exit codesCopy Anchor Link

Exit 0 is not the same as verified. An allowance flag such as --allow-unanchored makes a weaker pass exit 0 and suffixes the level (verified, unanchored). That is why a CI gate must read the JSON, not the exit code alone.

Wire it into CICopy Anchor Link

The rule from the CLI documentation: require level_enum == "verified" and an empty weakened_by_flags, never a prefix of the level string. A verified result can still carry declared-absent parts, so also read the absent list, or pass --complete.

#!/bin/sh
set -u
report=$(numi evidence verify --json \
  --runner-key keys/runner.pub --log-key keys/log.pub --log-origin numinance.test/log \
  --complete "$1")
code=$?
echo "$report" | jq -r '"level_enum=\(.level_enum) weakened_by_flags=\(.weakened_by_flags|length) absent=\((.absent // [])|length)"'
[ "$code" -eq 0 ] || { echo "verify exit $code"; exit 1; }
echo "$report" | jq -e '.level_enum == "verified" and (.weakened_by_flags | length == 0)' >/dev/null || { echo "weaker than verified"; exit 1; }
echo "gate passed"

level_enum=verified weakened_by_flags=0 absent=0
gate passed
level_enum=failed weakened_by_flags=0 absent=5
verify exit 3

Read the absent listCopy Anchor Link

--complete turns a declared-absent part into a failure. Without it, the same closed-basic export is verified, with five parts listed in absent:

{"level_enum":"verified","absent":["change.json","verification/","review/","deployment.json","observation.json"]}

With --complete the same export exits 3 and prints FAIL parts_absent: a part is absent and --complete was given. Decide per pipeline: a gate that must know a verifier report and a reviewer decision exist should demand --complete, or inspect absent for the parts it needs. Treat each absent part as a statement that this export does not say anything about that area.

Pin keys independentlyCopy Anchor Link

The flags give you three modes, and exactly one is required:

• --runner-key FILE (repeatable), with --log-key and --log-origin: the only mode that can reach verified. Any pinned runner key may sign any record, the attestation or the manifest.
• --trust-bundle-keys: trusts the keys the export states. The best level is self-consistent.
• --skip-signatures: checks no signature. The best level is unauthenticated.

level: self-consistent
signatures: verified under keys the export states only
anchor: not checked (no log pinned)
result: self-consistent

A key that arrives inside the export can only ever tell you the export agrees with itself. The attacker who forges a whole export brings their own key and their own bundle. The pin must therefore come through a channel the exporter does not control: a file in your own repository, a secrets store, a record your security team signed. There is no built-in trust root in the verifier and none in the spec. Rotating a key means updating your pin and, for old exports, keeping the old pin alongside the new one, since --runner-key is repeatable.

Failure modes and troubleshootingCopy Anchor Link

Next stepsCopy Anchor Link