Qualify autonomy

When to use thisCopy Anchor Link

Use this guide when a repository has been running at AUTONOMY_LEVEL_PR and a team wants to know what it would take to let one kind of change go further, for example dependency upgrades with LOW risk reaching AUTONOMY_LEVEL_MERGE. The answer is never "after N green runs the system promotes itself". The answer is: you write down the evidence you require, the system proposes a promotion only when its local record meets it, and a person accepts or rejects the proposal.

PrerequisitesCopy Anchor Link

• A repository onboarded with numi init (Onboard a repository) and a chosen starting level (Choose an autonomy level).
• Understanding of the autonomy ladder and of floors, which an org sets and a repository cannot lower.
• To collect real outcomes you need the engine. The open build only checks your YAML.

The shape of the processCopy Anchor Link

1. A playbook states a qualification: thresholds for volume and for bad outcomes.
2. The evals context records an outcome for each closed change, bound to a playbook and a component. Together they are a stratum.
3. From those outcomes it computes the stratum's qualification and publishes QualificationUpdated.
4. When the stratum meets the thresholds, the runner records a PromotionProposed event: the playbook, the component, the level it moves from and to, and the digest of the evidence it rests on.
5. A human approver decides it. PromotionDecided records the verdict and the digest of the decision record that the evidence chain holds.
6. Only a separate, gated policy path can change the level, and it re-reads the decision itself. The events are notifications. The event contract says a consumer never changes an autonomy level, widens a grant or resumes a change on one of these events alone.

The proposal is data. It changes no level, and only a human approver decides it.

Playbook.qualificationCopy Anchor Link

The generated bugfix-bounded playbook from numi init already carries a qualification block. This is the real file from numi init --ci github-actions:

spec:
  change_type: bugfix
  max_repair_attempts: 2
  qualification: {min_runs: 20, max_rework_rate: 1000, max_regression_rate: 0, max_rollback_rate: 50, zero_security_escapes: true}

Read the init values with the unit in mind. max_rework_rate: 1000 is 10 percent, max_rollback_rate: 50 is half of one percent, and max_regression_rate: 0 allows none. These are the values the template writes; they are a starting point to review, not a recommendation for your codebase and not a benchmark.

Three rules keep the thresholds honest:

• Tighten only. A narrower scope may only tighten a threshold: min_runs up, a maximum rate down, zero_security_escapes never to false. The reference implementation refuses a loosening.
• Absent is not permissive. An absent qualification means the playbook is not promotable unless the AutonomyPolicy floor qualifies it.
• Rates are bounded. The profile context refuses a rate above 10000. The open numi validate --files is a schema check only: a Playbook with max_rework_rate: 20000 still passes it (verified below), so do not read a green file check as proof the value is acceptable.

revision: "files"
ok       file "pb.yaml" Playbook "bugfix-bounded"

Qualification is per playbook, and further per component. A strong record for dependency upgrades says nothing about schema migrations. There is no global "autonomy score".

Stopping rulesCopy Anchor Link

The design includes several ways qualification stops or resets. The ones visible in the public contracts:

The rule of three is a statistical rule of thumb: with zero failures in N comparable runs, the 95 percent upper bound on the unseen failure rate is about 3/N. It is why a small min_runs proves little. It is also why the system reports an upper bound rather than the observed rate.

After a reset, promotion starts again from the base level. Lowering a level that is already active on a severity event is owned by the policy context and by the circuit breaker, not by the evals events. See Circuit breaker and Rollback and recovery.

EvalSet: reference-only replay casesCopy Anchor Link

An EvalSet is a versioned set of replay cases. A case names a recorded replay outcome by digest and states what it must show. It holds no free text, ticket, code or person, on purpose: free text could carry poisoned material into the set, so categories and defects are tokens.

Its apiVersion is numinance.dev/v1alpha1. The loader reads it only when the caller turns alpha kinds on, it takes no part in profile resolution, and it is not an input to the profile digest. The RFC calls the alpha gate an explicit loader option, off by default. The open numi validate --files accepts it, as the real run below shows.

apiVersion: numinance.dev/v1alpha1
kind: EvalSet
metadata: {name: core-replay, scope: SCOPE_REPO}
spec:
  version: 1.0.0
  cases:
    - name: bugfix-clean
      outcome: {value: "sha256:0b2958497ee0c936c59fa8dfb74b43077ad626e5dafd62df6521c4dd1e581242"}
      expect: {acceptance: EVAL_ACCEPTANCE_ACCEPTED, merged: true, severe: false, verification_failures: []}
    - name: bugfix-lint-retry
      outcome: {value: "sha256:70fd102e81fe47531de6d7921cbb496665bfa88fcc83f211129b4e76fe53b8ed"}
      expect:
        acceptance: EVAL_ACCEPTANCE_ACCEPTED
        merged: true
        severe: false
        verification_failures:
          - {layer: lint, count: 1}
    - name: planted-sql-injection
      outcome: {value: "sha256:88761b4780b189a6b983d61a91302970697577cad92842f3b1c8c44256d5ce4d"}
      expect:
        acceptance: EVAL_ACCEPTANCE_REJECTED
        merged: false
        severe: false
        verification_failures:
          - {layer: security-scan, count: 1}
          - {layer: tests, count: 2}
      category: EVAL_CATEGORY_PLANTED_SECURITY_BUG
      seeded: {defect: sql-injection, caught_by: security-scan}

revision: "files"
ok       file "core-replay.yaml" EvalSet "core-replay"

The digests are example values. In practice each outcome is the digest of the replay outcome record that a recording produces under the set and the case.

Fields and loader checksCopy Anchor Link

The loader enforces the ordering. A copy of the example with b-case before a-case fails:

revision: "files"
invalid  file "ev.yaml": kind: invalid: EvalSet cases are sorted by name without duplicates

Per the RFC, a planted security bug or migration hazard must be seeded, and a seeded case must expect not merged, not severe and not accepted. Replaying a fixture is also how you notice that a verifier layer stopped catching something. Outcomes recorded under an old bundle digest are not catch evidence for a new version of the set; they must be replayed again before a proposal uses them.

A worked path to MERGE for one playbookCopy Anchor Link

This is a walk-through of how the pieces fit, not a captured run.

1. The org AutonomyPolicy example holds defaults: {level: AUTONOMY_LEVEL_PR}, a rule granting AUTONOMY_LEVEL_MERGE to dependency-upgrade at RISK_CLASS_LOW, and floors that cap RISK_CLASS_HIGH at AUTONOMY_LEVEL_SUGGEST. The ladder is OBSERVE, SUGGEST, IMPLEMENT, PR, MERGE, STAGE, RELEASE, OPERATE.
2. A team starts a stratum (dependency-upgrade on one component) at PR. Each closed change adds an outcome.
3. When runs reaches min_runs with rates inside the playbook's thresholds and no severity reset, the runner records a proposal naming from_level PR and to_level MERGE with a basis_digest.
4. A human approver reviews the basis, including the evidence behind it (verified under keys they pinned: see Verify a run), and accepts or rejects.
5. If accepted, the gated path records the level change. The floors still apply: a promotion can never move a stratum above a floor, and the circuit breaker can still cap effective autonomy.
6. A later severity event resets the stratum to zero runs and the level is lowered through the policy path.

Failure modes and safeguardsCopy Anchor Link

Next stepsCopy Anchor Link