# Reliable Automation Operations Runbook

> Classification: **SYNTHETIC DEMO**. This is a neutral work sample, not a client or production claim.

## Invariants

- A job has one durable ID, one business key and an append-only state history.
- Side effects require an idempotency key and a persisted attempt record.
- A lease expires; a worker identity and fencing token prevent stale completion.
- Human-gated actions never transition to approved without an attributable user event.

## Triage

1. Determine impact: one job, one tenant, one dependency or the whole queue.
2. Freeze dangerous consumers; do not delete queue items.
3. Compare queue depth, oldest age, lease age, failure class and dependency health.
4. Choose retry, replay, compensate, quarantine or rollback.
5. Record the decision, operator, evidence and affected identifiers.

## Retry matrix

| Failure | Default | Why |
|---|---|---|
| Validation | Quarantine | Same input will fail again |
| Rate limit | Scheduled retry | Respect server delay and budget |
| Network before send | Retry | No side effect was attempted |
| Timeout after send | Reconcile, then retry idempotently | Outcome is unknown |
| Authentication | Stop dependency | Repetition can lock or amplify |
| Internal invariant | Stop consumer | Continuing risks corrupt state |

## Recovery gate

Bulk replay requires a dry-run count, exact filter, sample inspection, maximum batch, rate cap, rollback/compensation plan and explicit human confirmation. The replay uses the original business key but a new attempt ID.

## Completion

Close only after queue age returns to target, new failures remain stable for two observation windows, reconciled totals match, customer-visible status is correct and the incident timeline names the preventive follow-up.