Bugfix with verification
A designed walkthrough of a bug fix that ends in a pull request, showing independent verification, bounded repair, escalation and the evidence each stage emits.
This is a storyboard of the Numinance engine, which is proprietary and not generally available. The configuration shown is exactly what numi init generates and validates with the open CLI. There is no runnable engine command in this guide.
- proto/numinance/kinds/v1/playbook.proto
- proto/numinance/kinds/v1/stage_registry.proto
- proto/numinance/kinds/v1/language_toolchain.proto
When to use this
Read this guide when you want to understand how a bug fix is expected to be shaped, verified and stopped, before you let any agent open pull requests against your code. It builds on Onboard a repository, whose generated files are the configuration discussed here.
The configuration that drives it
The generated playbook has one step for each identity kind that touches a bug fix:
apiVersion: numinance.dev/v1
kind: Playbook
metadata: {name: bugfix-bounded, scope: SCOPE_REPO}
spec:
change_type: bugfix
steps:
- {name: fix, capability_set: implement, risk: RISK_CLASS_MEDIUM, evidence: [diff, pr-ref], verifiers: [unit-tests, lint, typecheck]}
- {name: review, capability_set: review, risk: RISK_CLASS_LOW}
- {name: pr-status, capability_set: release, risk: RISK_CLASS_LOW}
max_repair_attempts: 2
qualification: {min_runs: 20, max_rework_rate: 1000, max_regression_rate: 0, max_rollback_rate: 50, zero_security_escapes: true}The verifier names resolve through the LanguageToolchain, which says which provider and check each name runs:
verifiers:
- {name: unit-tests, provider: dev.numinance.verifier-python, config: {check: pytest, timeout_seconds: 300}}
- {name: lint, provider: dev.numinance.verifier-python, config: {check: ruff, timeout_seconds: 60}}
- {name: typecheck, provider: dev.numinance.verifier-python, config: {check: mypy, timeout_seconds: 120}}And the stage registry maps those stages onto the fixed lifecycle, with a plan gate decided by policy and a final human gate:
- {name: plan-review, maps_to: CHANGE_STATE_PLAN_GATE, gate: {kind: GATE_KIND_PLAN}, requires: [triage]}
- {name: implement, maps_to: CHANGE_STATE_IMPLEMENTING, requires: [plan-review]}
- {name: verify-static, maps_to: CHANGE_STATE_VERIFYING, requires: [implement]}
- {name: ci-github-actions, maps_to: CHANGE_STATE_VERIFYING, requires: [verify-static]}Honest limits of the live verifier
unit-tests runs candidate code, which the Python verifier provider refuses to do outside a nested sandbox that is not built yet. Today only the replaying build serves it, so a live run of unit-tests is unavailable. lint and typecheck are static and run.
How verification and repair fit together
Verification and repair
When a check fails, what exactly can happen next, and what can never happen?
- 1● current
Author
author-agent
- 2pending
Deterministic checks
tests, lint, typecheck
- 3pending
Independent review
reviewer-agent
- 4pending
Rehearsal
disposable environment
- 5pending
Canary
release-pipeline
- 6pending
Release
release-pipeline only
Author implements the change
- Who acts
- author-agent
- Evidence recorded
- candidate and diff recorded by the runner
- Repair attempts
- attempt 0/2
What can happen next
- Check fails: bounded repair, then re-check
- Attempts or budget used up: pause and escalate; never widen the grant
- Reviewer disagrees: escalate to a human
- Canary regresses: rollback under the release identity
◆ Never, at any step
Repair cannot widen the grant, change protected paths, or approve itself. The author never reviews or deploys its own change; a human or policy accepts, not the model.
Step 1 of 8. Author implements the change. Acting identity: author-agent. Repair attempts 0 of 2.
Text description
Illustrative and simulated: no change is running. The generated Playbook sets max_repair_attempts: 2; the field accepts 0 to 10 and the control here shows 0 to 3.
- The author-agent implements. Deterministic checks (tests, lint, typecheck) run next.
- A reviewer-agent, a separate identity from the author, reviews independently and returns findings.
- The change is rehearsed in a disposable environment, then canaried and released by the release-pipeline, the only identity that deploys.
- If a check fails, the change goes back to the author for a bounded repair, with the findings reference as the only new input. After
max_repair_attemptsit is escalated to a human. - If a budget is used up, the change pauses and escalates; it never gets a wider grant. If the reviewer disagrees, it escalates. If the canary regresses, it rolls back.
Invariant in every case: repair cannot widen the grant, change protected paths, or approve itself.
Three properties carry the design:
- The verifier is not the author. The author-agent produces a candidate. The verifier provider and your CI run the checks, and the reviewer-agent holds a separate identity from the author. A check passing is a recorded fact, not the agent's claim.
- Repair is bounded.
max_repair_attemptsis 2 in the generated playbook, between 0 and 10 in the contract. The repair state loops back to verification at most that many times. - Exhaustion escalates. When attempts run out, the change moves to the escalated state and waits on a human signal. It does not widen its own grant, skip a verifier or retry indefinitely.
The storyboard
Understand and classify
The engine resolves repository context for the bug report and fixes the playbook (bugfix-bounded) and risk class. The fix step declares MEDIUM, so the change is at least MEDIUM. With the generated autonomy policy the level is PR.
Plan gate
The plan names the work units, the risk of each and the evidence required. Policy evaluates it before any side effect. A plan that touches a protected path, such as .numinance/policy/** or a workflow file, is denied and the denial is recorded.
Implement under an author grant
The author-agent works in a per-change sandbox. The implement capability set allows the branch push and the pull request. It does not allow merging or deploying.
Evidence emitted: diff.
Verify, then repair or pass
verify-static runs lint and typecheck through the verifier provider. ci-github-actions then runs the workflow your repository already defines, through the CI executor port. On a failure, the finding goes back to the author for one repair attempt, and the full verifier set runs again.
Review and the human gate
The reviewer-agent reads the candidate under the review set, which grants verifier.run and nothing that writes. At pr-gate a human maintainer approves, and policy rejects an approver that is the author or reviewer identity.
Evidence emitted: pr-ref. The change ends at the pull request; no stage merges.
An illustrative stage log
plan-review allow decision recorded
implement candidate ready evidence: diff
verify-static lint ok, typecheck FAIL (finding recorded)
repair attempt 1 of 2 candidate updated
verify-static lint ok, typecheck ok
ci-github-actions workflow passed run reference recorded
review reviewer-agent no blocking findings
pr-gate awaiting human approver_role=maintainer, not author or reviewerAnd the path where repair does not work:
verify-static typecheck FAIL
repair attempt 2 of 2 candidate updated
verify-static typecheck FAIL
escalated attempts exhausted waiting on a human signalWhat a human approves
- The pull request, at the human gate, with a distinct identity from the agents.
- A decision after escalation: a human may supply a signal, ask for more work within the existing grant, or abandon the change.
- Any later promotion of this playbook, based on its qualification record.
Failure modes and safeguards
| Failure | Safeguard |
|---|---|
| The agent claims a test passed | The pass is recorded from the verifier or CI run, not from the claim. |
| Declared evidence is missing | The step is PARTIAL, not success. |
| Repair keeps failing | The loop ends after max_repair_attempts and the change escalates. |
| The fix edits policy or CI files | Protected paths are denied by mount and by branch protection. |
| A flaky or unavailable verifier | unit-tests is currently unavailable live; do not treat a skipped check as a pass. Evidence records what ran. |
Evidence from such a change can be checked offline later. A verified result proves the chain and signatures under keys you pin; it does not prove the fix is correct or the tests thorough. "Don't trust the agent. Check the evidence." See Verify a run.
Check the configuration today
go run ./cli/numi validate --files .numinanceThis is the schema and loader check from Onboard a repository, exit 0 on the generated tree.
Next steps
Dependency upgrade
A designed walkthrough of a low-risk dependency upgrade change, from triage to a pull request, tied to the real Playbook and AutonomyPolicy examples.
Release and incident response
A designed walkthrough of a governed release with a canary window and rollback, event-driven incident response, the four breaker positions, and human-only break-glass.