EvalSet and ContextBundle
The alpha EvalSet kind for reference-only replay cases, and the produced ContextBundle record, which you read and validate but never author.
Both schemas are public. EvalSet is an alpha kind (numinance.dev/v1alpha1) and the open build reads it only in the example validator. ContextBundle is produced by the proprietary runtime's knowledge context, which is not generally available. The replay and the bundle builder are not in the open build.
These two kinds are different in kind. An EvalSet is something you write: a versioned list of replay cases that name a recorded outcome by digest. A ContextBundle is something Numinance writes: the knowledge selected for one stage, with provenance. The loader rejects a ContextBundle in your configuration tree.
EvalSet
An EvalSet is a versioned, reference-only set of replay cases. A case names a recorded replay outcome by digest and states what that outcome must show. It holds no free text, ticket, code or person, only tokens, enums, counts and digests. It takes no part in profile resolution and is not an input to the profile digest.
It is how an org checks that a change to an executor, a prompt or a policy did not change measured behaviour on situations it cares about, including adversarial ones. See evaluations for the operating practice and qualify autonomy for how evidence of behaviour feeds a human decision to raise a level.
Alpha kind
apiVersion for EvalSet is numinance.dev/v1alpha1, not numinance.dev/v1, and a loader reads it only when alpha kinds are switched on. Open numi validate --files switches them on so examples are checked. Open numi validate on a repository does not, and reports alpha kind is not enabled: EvalSet (shown below). Expect the shape to change before a stable release.
Fields
Prop
Type
A seeded case has extra rules: a case of category planted security bug or migration hazard must be seeded, and a seeded case must expect the change to be not accepted, not merged and not severe, with its caught_by layer among the expected failures. In other words, a planted defect that the checks catch is a pass of the system, and a planted defect that gets through shows up as a mismatch.
apiVersion: numinance.dev/v1alpha1
kind: EvalSet
metadata: {name: core-replay, scope: SCOPE_REPO}
spec:
version: 1.0.0
cases:
- name: bugfix-clean
outcome: {value: "sha256:0b2958497ee0c936c59fa8dfb74b43077ad626e5dafd62df6521c4dd1e581242"}
expect: {acceptance: EVAL_ACCEPTANCE_ACCEPTED, merged: true, severe: false, verification_failures: []}
- name: bugfix-lint-retry
outcome: {value: "sha256:70fd102e81fe47531de6d7921cbb496665bfa88fcc83f211129b4e76fe53b8ed"}
expect:
acceptance: EVAL_ACCEPTANCE_ACCEPTED
merged: true
severe: false
verification_failures:
- {layer: lint, count: 1}
- name: planted-sql-injection
outcome: {value: "sha256:88761b4780b189a6b983d61a91302970697577cad92842f3b1c8c44256d5ce4d"}
expect:
acceptance: EVAL_ACCEPTANCE_REJECTED
merged: false
severe: false
verification_failures:
- {layer: security-scan, count: 1}
- {layer: tests, count: 2}
category: EVAL_CATEGORY_PLANTED_SECURITY_BUG
seeded: {defect: sql-injection, caught_by: security-scan}The digests point at recorded outcomes. The set stores references, never the content of a replay, so an EvalSet cannot carry a ticket body or a person's name by construction. The case planted-sql-injection represents a recorded run on a change that contains the defect; the defect itself is not in the set.
ContextBundle
A ContextBundle is the knowledge selected for one stage of one work unit. It is produced by the knowledge context, stored by digest and never mutated: a refresh opens a new context_generation. It is never configuration, so it has no metadata and no extensions, and the configuration loader refuses it.
Why you would read one: it is the record of what the context resolver gave a stage, why, from which source version, under whose authority and at what cost. It makes context an auditable input rather than a hidden prompt. For the resolver itself, see the context resolver and the context-first concept.
Key fields
Prop
Type
apiVersion: numinance.dev/v1
kind: ContextBundle
spec:
change_id: chg_01JB2Q8W3N5M7K9P1R3T5V7X9Z
work_unit_id: wu_01JB2Q8Z4P6R8T0V2X4Z6B8D0F
stage: implement
context_generation: 1
authz_context_digest: {value: "sha256:5c2e8a1f3b7d9e0c4a6f8b2d1e3c5a7f9b0d2e4c6a8f1b3d5e7c9a0f2b4d6e8c"}
resolver: {model_id: eu.anthropic.claude-haiku-v1:0, prompt_version: resolver-3}
query_plan:
- {tool: searchConfluenceUsingCql, args_digest: {value: "sha256:7a9c1e3b5d7f9a2c4e6b8d0f1a3c5e7b9d2f4a6c8e0b1d3f5a7c9e2b4d6f8a0c"}, results: 6, truncated: false}
sources:
- {ref: confluence-eng/ENG/pages/4211, system: confluence-eng, title: Payments retry policy, source_version: "17", fetched_at: "2026-10-18T09:12:44Z",
digest: {value: "sha256:1e3c5a7f9b2d4e6c8a0f1b3d5e7c9a2f4b6d8e0c1a3f5b7d9e2c4a6f8b0d1e3c"}, rank: 1, selection_reason: Names the retry limits the task changes.,
classification: internal, redactions: 0, access_basis: KNOWLEDGE_AUTH_USER_DELEGATED, staleness: SOURCE_STALENESS_FRESH, trust: TRUST_CLASS_UNTRUSTED_RETRIEVED}
authored_constraints:
- {origin: ArchitecturePack/layered-service, id: no-domain-io, statement: The domain layer performs no I/O.,
origin_digest: {value: "sha256:9d1f3b5e7a9c2e4f6b8d0a1c3e5f7b9d2a4c6e8f0b1d3a5c7e9f2b4d6a8c0e1f"}, trust: TRUST_CLASS_SIGNED_POLICY}
cited_guidance:
- {ref: confluence-eng/ENG/pages/4211, section: retry-limits}
unresolved_questions:
- {question: "Does the new limit apply to card refunds?", blocking: false}
budget_used: {tokens: 5400, calls: 4, ms: 8200}
pinned_toolsets:
confluence-eng: {value: "sha256:3f1a9c0e5b7d2a4f6e8c1b3d5a7f9e2c4b6d8a0f1e3c5b7d9a2f4e6c8b0d1a3f"}
receipts:
- {value: "sha256:6b8d0f2a4c6e8b1d3f5a7c9e0b2d4f6a8c1e3b5d7f9a0c2e4b6d8f1a3c5e7b9d"}
conflicts:
- {topic: refund retry limit, claims: [{ref: confluence-eng/ENG/pages/4211, section: retry-limits}, {ref: confluence-eng/ENG/pages/4211, section: refunds}]}Trust is part of the record
Notice the two different trust classes in one document. authored_constraints came from signed policy, so they are authority-bearing in the sense that a prompt may state them as constraints. But the bundle says plainly that enforcement comes from the grant, not from the text. sources are always TRUST_CLASS_UNTRUSTED_RETRIEVED, and the decoder refuses a source that claims otherwise. Retrieved content can inform an author agent; it cannot change policy, tool lists or approvals. Every text field in the bundle is untrusted, labelled as such in a prompt, one line of at most 1024 bytes without control or bidi characters.
Validating a produced kind
numi validate --files accepts a ContextBundle because it checks it against its own strict schema rather than loading it as configuration: the apiVersion, a spec, the required fields of the spec and of each source, digests, enums and the source trust pin. The spec CLI notes that other constraints (authored-constraint trust and origin digest, CitedGuidance.ref pointing at a bundle source, unique source refs, at least two claims per conflict) are left to the bundle's producer. A file-level pass is therefore weaker than a producer-side check.
Security implications
- An EvalSet is reference-only on purpose. There is no field for free text, so a replay set cannot be a channel for ticket content, code or personal data.
- An EvalSet measures; it does not grant. Passing one does not raise an autonomy level. A human accepts promotion, and an agent cannot promote itself.
- A ContextBundle never holds credentials. Refs are redacted, since a ref can hold a credential, and tool arguments are stored as digests, not values.
- Do not commit a ContextBundle under
.numinance/. It is refused there, and a hand-edited bundle would be unverifiable evidence anyway.
Real failures
$ numi validate --files bad/doc.yaml # EvalSet with apiVersion numinance.dev/v1
invalid file "doc.yaml": kind: invalid: apiVersion is not numinance.dev/v1alpha1
(exit 3)
$ numi validate --files bad/doc.yaml # EvalSet first case renamed so cases are not sorted
invalid file "doc.yaml": kind: invalid: EvalSet cases are sorted by name without duplicates
(exit 3)
$ numi validate --files bad/doc.yaml # seeded case with its category removed
invalid file "doc.yaml": kind: invalid: an EvalSet case that plants a defect names a category
(exit 3)
$ numi validate --files bad/doc.yaml # ContextBundle with a metadata block added
invalid file "doc.yaml": kind: does not match the kind schema: schema: value does not match (violations: 1): \"/metadata\": additionalProperties
(exit 3)
$ numi validate --files bad/doc.yaml # a source with trust: TRUST_CLASS_SIGNED_POLICY
invalid file "doc.yaml": kind: invalid: a value of numinance.kinds.v1.BundleSource.trust fails its check
(exit 3)
$ numi validate . # ContextBundle copied to .numinance/policy/ in a repository
invalid repo ".numinance/policy/implement-stage.yaml": kind: produced kind: ContextBundle is not configuration
(exit 3)
$ numi validate . # EvalSet copied to .numinance/policy/ in a repository
invalid repo ".numinance/policy/core-replay.yaml": kind: alpha kind is not enabled: EvalSet
(exit 3)Next steps
Governance kinds
ControlPack, ResidencyPolicy and BudgetPolicy, the org-owned kinds that set floors and limits a repository and an agent cannot loosen.
Validation and errors
What the open numi validate checks, what it deliberately leaves to the engine, the exit codes, and the real error messages you will see.