Rollback and recovery
Bounded repair, rehearsal, canary and rollback for changes, and how to recover and prove recovery after an incident.
Numinance does not heal itself. It repairs a bounded number of times, checks independently, rolls back, and escalates to a human.
Bounded repair
When a verifier fails, the runtime may let the agent try again, but only within limits:
- The attempts are finite per playbook.
max_repair_attemptsis 0 to 10, and the generated default playbook uses 2. - Each attempt runs under the same grant, budgets and policy as the first.
- Independent verifiers, not the author agent, decide whether a repair worked. Deterministic checks go first and the reviewer is one input among several.
- When attempts are exhausted the change escalates to a human or ends. It does not loop.
Verification and repair
When a check fails, what exactly can happen next, and what can never happen?
- 1● current
Author
author-agent
- 2pending
Deterministic checks
tests, lint, typecheck
- 3pending
Independent review
reviewer-agent
- 4pending
Rehearsal
disposable environment
- 5pending
Canary
release-pipeline
- 6pending
Release
release-pipeline only
Author implements the change
- Who acts
- author-agent
- Evidence recorded
- candidate and diff recorded by the runner
- Repair attempts
- attempt 0/2
What can happen next
- Check fails: bounded repair, then re-check
- Attempts or budget used up: pause and escalate; never widen the grant
- Reviewer disagrees: escalate to a human
- Canary regresses: rollback under the release identity
◆ Never, at any step
Repair cannot widen the grant, change protected paths, or approve itself. The author never reviews or deploys its own change; a human or policy accepts, not the model.
Step 1 of 8. Author implements the change. Acting identity: author-agent. Repair attempts 0 of 2.
Text description
Illustrative and simulated: no change is running. The generated Playbook sets max_repair_attempts: 2; the field accepts 0 to 10 and the control here shows 0 to 3.
- The author-agent implements. Deterministic checks (tests, lint, typecheck) run next.
- A reviewer-agent, a separate identity from the author, reviews independently and returns findings.
- The change is rehearsed in a disposable environment, then canaried and released by the release-pipeline, the only identity that deploys.
- If a check fails, the change goes back to the author for a bounded repair, with the findings reference as the only new input. After
max_repair_attemptsit is escalated to a human. - If a budget is used up, the change pauses and escalates; it never gets a wider grant. If the reviewer disagrees, it escalates. If the canary regresses, it rolls back.
Invariant in every case: repair cannot widen the grant, change protected paths, or approve itself.
Rehearsal and canary
A playbook step can set rehearse, which means the step runs first in a disposable environment. Rehearsal catches what unit tests miss, such as a migration that fails against realistic data, before it touches a real target.
For releases the designed flow adds a canary window. A small portion is released, health signals are watched against thresholds, and the release either proceeds or rolls back. Canary and rollback outcomes feed qualification, so a playbook with frequent rollbacks cannot promote.
Rollback
A rollback is a change, not a side channel. It goes through the same policy, grants, approvals and evidence as any other, and it is performed by the release pipeline identity, the only identity allowed to deploy. The author agent cannot roll back its own release by reaching around the pipeline.
Where the playbook includes a rollback plan, the runtime records the plan before the release and the result after. A rollback scenario is also an adversarial case in an EvalSet, so you can test it before you need it.
Escalation
Escalation is the planned end of every bounded path. The runtime escalates when repair attempts are exhausted, a budget is spent, a policy returns require-approval, verifier results conflict, or confidence is low. An escalation is a handoff to a named human with the plan, the evidence so far and the reasons. It is not a failure state to hide.
Recovery after an incident
The designed incident workflow, triggered by an abnormal action or a tripped breaker, runs in this order:
- Freeze the run at the next checkpoint.
- Revoke credential handles.
- Downgrade the component to PR-only or read-only.
- Snapshot the sandbox.
- Seal the evidence sequence.
- Record the executor, model, extension and policy versions in force.
- Open an incident change and notify your paging and security systems.
- Require explicit human recovery, with a reason.
Runs resume from the last content-addressed snapshot, not mid-turn. Budgets count across restarts.
Contain. Move the circuit breaker to the position the incident needs.
Understand. Export the evidence for the affected changes and verify it under keys you pinned.
Fix cause. Correct policy, the playbook, the extension or the eval set. Qualification for the affected playbook restarts if the event was severe.
Recover. A human resets the breaker with a reason. Work resumes at the lower level and climbs only through normal qualification.
Evidence of recovery
Recovery should leave a record that others can check without trusting you. The breaker trip, the reset, the human principal and the reason, any approval, any rollback and the incident change are all written to the evidence chain by the runner identity. The open verifier can check the chain offline:
numi evidence verify --runner-key runner.pub --log-key log.pub --log-origin <origin> --complete <export-dir>A verified result shows provenance and chain integrity under the keys you pinned. It does not show that the fix was correct or that every event on a compromised host was logged. See Verify a run and Threat model and limits.
Upgrades and disaster recovery
Runners are designed to pull a new release during your window, verify signatures, drain work at a checkpoint, switch, and revert automatically if health checks fail. Database state is backed up continuously, evidence storage is write-once, and the data plane can be re-provisioned from infrastructure-as-code against restored backups. Loss of the vendor control plane does not lose evidence or stop in-flight authorised work.
Failure modes
- Repair that hides a defect. A change that passes only after several repairs is a signal. Repair counts feed qualification.
- Rollback that was never rehearsed. Include rollback cases in your EvalSet.
- Resuming too quickly. Reset the breaker only after the cause is addressed, or the same trigger recurs.