> ## Documentation Index
> Fetch the complete documentation index at: https://docs.wednesdayai.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Restart recovery

# Restart recovery

On gateway start the restart-recovery control plane claims a single-owner epoch, inspects first-party and plugin adapters for interrupted work, classifies evidence, applies safe dispositions, then opens root-work admission. Fire-and-forget startup orchestrators (`src/gateway/startup-recovery.ts`) are gone; recovery is this awaited sequence.

There is no config block for recovery budgets or journal paths. The journal is `${stateDir}/restart-recovery.sqlite` (`stateDir` from `OPENCLAW_STATE_DIR` or `~/.openclaw`). The owner lock is `${stateDir}/restart-recovery-owner.sock`.

Recovery uses one 30-second startup deadline and schedules at most 256 adapter mutations per boot. The deadline covers streamed inspection and every `reattach`, `resume`, `complete`, and `cancel` mutation. An inspection timeout keeps the boot not ready. An apply timeout records `unknown_settlement` and also keeps the boot not ready because the timed-out mutation may still be running.

## Startup phases

Snapshot `phase` walks:

`listen` → `claim_owner` → `discover` → `classify` → `reconcile` → `prepare_runtime` → `ready`

`blocked` is the failure phase (journal unwritable/corrupt, owner lock held, producer prepare failed). Admission stays closed.

A subsystem safety hold is different from a durable blocked item. Session storage no longer holds
root work: the fail-fast lifecycle opens storage before this control plane runs, and a storage
start failure exits the process instead of degrading it (see
[Session storage recovery](https://docs.wednesdayai.dev/admin/gateway/session-storage-recovery)).

`ready` may still be `degraded: true` when items remain `blocked` (unknown settlement). Root work is admitted; operators repair those items with Doctor.

A blocked item clears on its own once the subsystem that owns its source finalizes it. While items are blocked, the gateway re-checks them every 15 seconds. An item whose source is no longer recoverable is recorded `completed`, its incident closes, and `degraded` returns to `false`. For example, a task pass interrupted by a crash is held while its lease may still be live. Once the task dispatcher reclaims the expired lease (the run becomes `preempted` with `recovered: true`), the item clears.

## Probes

| Probe | While recovering | When ready |
| - | - | - |
| `/healthz` (and `/health`) | `200` `{ ok: true, status: "live" }` | same |
| `/readyz` (and `/ready`) | `503` with a recovery snapshot, or `{ recovery: null, reasonCodes: ["recovery_pending"] }` if no runtime is installed | `200`; body includes `recovery.ready`, `recovery.degraded`, counts, reason codes |

Probes are unauthenticated. `Cache-Control: no-store`. Only `GET` and `HEAD` are allowed (`405` otherwise). Dockerfile HEALTHCHECK uses `/healthz`.

Chat `send` / `chat.send` / `agent` RPCs reject with `RECOVERY_NOT_READY` (`recovery_pending`) until admission opens.

## Restart window

Between shutdown and the next listener, connections are refused (`ECONNREFUSED`). The window is
bounded by the restart drain (`gateway.restartResilience.drainTimeoutMs`, default 30 s, for active
tasks and ACP turns), then a shutdown budget of up to 25 s (15 s task dispatcher drain, 5 s
interactive work, 5 s storage close), then boot. An idle gateway restarts in a few seconds.

`listening on …` means the process is bound. `gateway ready: accepting work (startup Nms)` means
root work is admitted; when restart recovery runs, `/readyz` returns 200 at the same point.
Measure the window from the old process's `signal SIGTERM received` line to the new
`gateway ready` line.

`GatewayClient` reconnects with a 1 s→30 s backoff, and CLI sessions park requests until the
next hello. Other callers retry after reconnect.

Idle `agent_idle` triggers and cron schedule windows wait while admission is closed and fire on
the first tick after it opens, so a boot no longer produces one `RECOVERY_NOT_READY` enqueue
failure per configured agent. A `HH:MM-HH:MM` schedule window that elapses entirely during the
wait is skipped rather than fired late, the same as any other tick the monitor misses. A cron
schedule window whose occurrences pass during the wait fires once, not once per missed occurrence.

## Storage-backed RPCs during a restart

Session storage opens during boot through the fail-fast lifecycle — before the restart-recovery
fence — so a storage start failure exits the process and nothing serves. While the open is still
in flight, storage-backed methods — run history, `chat.history`, the session-run and task reads,
and the session writers (`sessions.create`, `sessions.patch`, `sessions.reset`, `sessions.delete`,
`sessions.compact`, `chat.inject`) — are held at dispatch for up to 10 seconds so a client that
reconnects during a restart never sees an empty list or an error burst: it gets the real transcript
once storage is ready. If the gateway is still starting after 10 seconds, they answer retryable
`RECOVERY_NOT_READY` (`storage_starting`), and the log carries a single warning from the
`gateway/storage-fence` subsystem — `session storage still starting after 10000ms; storage-backed
methods answer RECOVERY_NOT_READY` — once per boot, not one line per request. The full method
classification is in [Protocol](https://docs.wednesdayai.dev/admin/gateway/protocol.md#session-storage-readiness).

`UNAVAILABLE` with `details.code = "storage_start_failed"` from these methods means the storage
gate never reached ready in this process while the gateway kept serving: restart recovery failed
non-fatally after storage was expected to open (a journal or owner-fence failure, or a failure of
the storage-dependent startup work itself). `details.reasonCodes` names the cause —
`owner_lock_held`, `prior_owner_unproven` or `owner_fence_unavailable` (the restart-recovery owner
fence could not be established),
`journal_unavailable` / `journal_unwritable` / `journal_corrupt` (the state dir's
restart-recovery journal cannot be opened, written, or parsed),
`storage_supervisor_start_failed` (the storage-dependent startup work failed after the open), or
`assembly_failed` (recovery assembly threw); a failure that carries no specific code answers
`recovery_start_failed`. This is rare by construction: storage itself exits the process on
failure. Clients may latch on the answer; it clears only after the restart. Diagnose with the
gateway log lines `restart-recovery journal unavailable` / `restart-recovery owner fence
unavailable`, `storage.v1` (`wednesdayai status` or `wednesdayai doctor --deep`), and the
restart-recovery troubleshooting below.

During shutdown the hold applies in reverse: the storage state turns `stopping` at shutdown entry, so requests arriving between admission close and socket close answer retryable `RECOVERY_NOT_READY` (`storage_stopping`, "gateway is stopping") instead of a storage error, held for up to the same 10 seconds. The socket can close before the hold expires, though, so a client may instead see the connection drop. From the client's perspective both answers are retryable — reconnect and send the request again; it lands on the next process.

## What recovers

On **every** install:

* outbound delivery (`core:outbound-delivery`)
* session runs (`core:session-runs`)
* agent turns (`core:agent-turns`)
* Pi embedded (`core:pi-embedded`)
* ACP runtime (`core:acp-runtime`)
* plugin-owned work without an adapter (`core:plugin-owned-work` → `unsupported_runtime` block). A plugin's task passes are not included: the task pipeline recovers them.

Task-run (`core:task-runs`) and cron/heartbeat (`core:cron-heartbeat`) adapters assemble only when
the SQL session runtime publishes `taskRuns` / `cronJobs`.

Session-run and embedded-Pi resume require a persistent agent-turn work queue. On the default loopback runtime, automatic resume and `sessions.runs.continue` fail closed before changing run state; configure durable session storage or cancel the run instead.

### Interrupted user runs

Turns started through the Gateway `agent` RPC (including `wednesdayai agent` without `--local`)
are recorded as session runs of kind `chat` with recovery mode `off`, so the idle trigger sees
them. Recovery never resumes an `off` run. When a crash leaves one `running`, the next start
finalizes it as `abandoned`: its `recoveryReason` starts with `recovered on restart:` and a
`session_run.recovered` event is appended. The item is cancelled, not blocked, so it does not set
`degraded`.

A `running` `off` run that belongs to the current Gateway instance is live, not a crash leftover.
Ingress that is not admission-gated (the OpenAI-compatible and OpenResponses HTTP endpoints, and
node `voice.transcript` / `agent.request`) can start one before recovery finishes. Recovery
neither inspects nor changes it, and the run finalizes its own row.

### ACP turns

ACP turns are recorded as session runs of kind `acp` with recovery mode `off`, so they participate
in the same restart machinery as other interactive work:

* The restart gate counts active ACP turns, and the drain waits for them alongside queued commands
  within `gateway.restartResilience.drainTimeoutMs` (default 30000). Draining closes root-work
  admission, so new work is refused before it reaches ACP: gateway RPC callers (`chat.send`,
  `agent`) receive the retryable `RECOVERY_NOT_READY`, and inbound channel messages are refused at
  ingress. A message admitted just before the drain began, but not yet sent to the agent, is
  rejected with `ACP_GATEWAY_DRAINING`. In every case nothing ran, so the message is resent, not
  continued by replying.
* On both restart and stop, a 5 second cooperative cancel runs before the server closes. Each
  cancelled turn finalizes its row as `cancelled` with the shutdown reason (`gateway restart` or
  `gateway stop`), and its dispatcher delivers one interruption notice to the bound thread while
  channels are still up. A turn the drain catches mid-setup ends `cancelled` with reason
  `gateway draining: not sent`, so it is distinguishable from a turn cancelled mid-run.
* At boot, a prior instance's non-terminal `acp` rows are finalized as `abandoned` with a
  `session_run.recovered` event, and one notice is delivered per affected turn (at-least-once,
  marker-guarded; 24 hour lookback, so older rows are not re-notified). The item is cancelled, not
  blocked, so it never sets `degraded`. A `cancelled` row whose shutdown-time notice did not reach
  the thread gets the boot notice too. Sessions whose `recoveryMode` is `decide` and that a crash
  left `interrupted`, `needs_recovery_decision`, or still `running` get the same at-least-once
  notice, with one gap: a restart
  that stayed in-process (the fresh-process respawn failed, or `OPENCLAW_NO_RESPAWN` is set) keeps
  the same instance id, so decide rows that same process marked are treated as current and get no
  notice.
* At boot the gateway also settles each affected session's harness state (bounded to 5 seconds per
  session): after the runtime backend reports healthy, it asks the runtime for status and
  cooperatively cancels a still-running prior owner (`reason: "gateway-restart-recovery"`), then
  resets the session meta to idle. It never spawns a harness process to do this; if the backend
  does not become healthy inside the budget, the settle is skipped with a warning and the surviving
  owner is left to the next boot.

The user-facing behavior, including the notice text and how to continue, is in
[ACP agents](https://docs.wednesdayai.dev/users/tools/acp-agents.md#gateway-restarts).

### Background child processes

Background commands started through the exec and process tools run under the process supervisor,
and so do agent turns on CLI backends (for example a `claude-cli` provider): those turns are
supervisor children too. On restart and stop the supervisor cancels those children, waits up to
\~3.5 s for them to exit (enough for the SIGTERM → SIGKILL escalation), and flushes its ledger of
live children — with a final bounded sweep right before the process exits, for children spawned
while the server was closing. The ledger is `${stateDir}/process-supervisor/gateway-children.json`,
written with mode 0600; a file it cannot parse is renamed aside as
`gateway-children.json.corrupt-<timestamp>`, never deleted. Children cancelled this way carry
reason `gateway-shutdown`, and model fallback treats that cancel as an abort instead of re-running
the turn on a fallback model.

Crash-orphan reaping has prerequisites: it runs at boot only when restart recovery is running and
unblocked (skipped or blocked boots never touch the ledger), and only on POSIX. On Linux the
identity probe reads `/proc` directly; on macOS it uses `ps`; Windows gets only the
stop-at-shutdown guarantee. If the gateway itself crashed and a child outlived it, the next
eligible boot reaps the orphan by matching the recorded start time against the child's real
elapsed process time; only children proven to belong to a prior instance are killed, and entries
a probe could not verify are kept for a later boot (capped at 7 days) instead of being erased.

## Doctor

`wednesdayai doctor` (`openclaw` alias) calls `noteRecoveryHealth`, which drives generation-bound RPCs:

* `doctor.recovery.inspect`
* `doctor.recovery.plan` (requested action, typically `resume`)
* `doctor.recovery.apply` (plan + authority token)

Plans bind to the current owner generation. A restart invalidates an unused plan. `status` and `health` project the same `recovery.v1` snapshot (owner generation and startedAt only; bootId/pid are redacted).

`doctor.recovery.inspect` separates `items`, the current epoch's actionable inventory, from `historyItems`, retained audit rows from prior epochs and the legacy journal table. Plans can target only `items`; a historical item ID returns `doctor_item_missing`.

Each inspected item carries `itemId`, a `title`, a `reasonText` sentence and an `actions` list projected by the gateway, so the CLI and the Control UI show the same text. Adapters record a `subject` (agent, session, run, cron job, channel, trigger, timestamps and a preview of at most 200 characters) and the `operatorActions` their `apply` accepts for the item; an empty list offers no action. A rejected plan or apply carries `reasonText` next to `reasonCode`.

**Run again** is `doctor.recovery.apply` with `replay: true` on a `cancel` plan. After the cancel applies, the gateway starts the work again from the target the adapter recorded in the journal (never from the request): `cron.run` with `force` for a cron job, or a nudge into the same session for a nudge turn the gateway itself created. It starts the work at most once per item, and only when the cancel really discarded the item. The outcome is recorded as a `replay` action row (`replay_started`, `replay_not_started` or `replay_failed`); the apply response returns `replay: { result: "requested" }` without waiting. Session runs, nudges from other agents or plugins, one-shot (`at`) cron jobs, cron wake or alert turns, and a cron schedule with no later occurrence offer no Run again. Every action row records the adapter, source id and acting client (`<client id>@<device id>`).

Doctor publishes only evidence observed after a mutation: a fresh matching adapter observation first, then the observation returned by `apply`. It never reuses pre-apply evidence. A mutation that returns without either form of evidence is rejected with `doctor_evidence_changed`.

If an adapter throws after its mutation may have started, automatic recovery records the item as `unknown_settlement` and does not retry it again in the same epoch. A successor epoch may retry only after fresh inspection proves `not_started`; the operation ID and attempt count remain stable across that succession.

## Troubleshooting

* **`/readyz` stays 503 with `recovery_pending`:** no runtime installed, or admission still recovering. Check gateway logs for journal/lock errors.
* **`/readyz` 503, `phase: blocked`, `journal_unwritable` / `journal_corrupt`:** the sqlite file at `${stateDir}/restart-recovery.sqlite` is preserved; do not delete it to “fix” corruption.
* **`/readyz` 200 but `degraded: true`:** blocked items remain. An interrupted task pass clears on its own after its lease is reclaimed. Inspect the others with Doctor, and do not restart in a loop expecting unknown settlement to resume on its own.
* **Two live gateways on one state dir:** the second fails session-storage ownership and exits (the database is owned by the first gateway; see [Session storage recovery](https://docs.wednesdayai.dev/admin/gateway/session-storage-recovery)). It may instead fail restart-recovery owner CAS / kernel lock (`owner_lock_held`). Stop the prior process before starting the second.
* **`storage_start_failed` from run history or `chat.history` with the gateway otherwise healthy:** the storage gate never reached ready while this process kept serving — a restart-recovery failure, not a storage failure (storage start failures exit the process). `details.reasonCodes` names the hold (`owner_lock_held`, `prior_owner_unproven`, `owner_fence_unavailable`, `journal_unavailable`, `journal_unwritable`, `journal_corrupt`, `storage_supervisor_start_failed`, `assembly_failed`). Remove the cause — stop the other process, or repair the state dir's journal — then restart the gateway. Clients may latch on this answer; it clears only after the restart.
* **A user reports `ACP error (ACP_TURN_INTERRUPTED)` or `ACP_GATEWAY_DRAINING` around a restart:** expected behavior, not a fault. The first is a turn the shutdown cancelled; the second is a message that arrived while the gateway was draining and was never sent to the agent. Inspect the rows with `sessions.runs.list` (filter `kind: "acp"`); what the user should do is in [ACP agents](https://docs.wednesdayai.dev/users/tools/acp-agents.md#gateway-restarts).

## Related

* [Session storage recovery](https://docs.wednesdayai.dev/admin/gateway/session-storage-recovery)
* [Doctor](https://docs.wednesdayai.dev/admin/gateway/doctor.md)
* [Outbound delivery](https://docs.wednesdayai.dev/admin/gateway/outbound-delivery.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.