Session storage recovery
Session storage (SQLite by default, Postgres optional) is the only store for sessions and conversation history. The gateway starts it on every boot — configured or not — and treats it as fail-fast: if storage cannot open, the gateway exits with the cause and the service manager restarts it. There is no degraded mode, no JSONL fallback, and no serving without storage.Startup sequence
startSessionStorage (src/config/sessions/storage-lifecycle.ts) runs this sequence, bounded
end-to-end by session.storage.recovery.startupProbeTimeoutMs (default 600000 ms):
- Acquire storage ownership for the database as role
gateway, waiting up to the same budget for acliholder to finish. - Open the runtime: create the backend candidate, run
session.storage.migration.validateOnStartvalidation when configured, prove connectivity with a first realsessions.list()round-trip, import every legacysessions.jsonindex, then commit the activation. - Start the background import sweep (below) and publish storage.
FatalConfiguredStorageStartError carrying the underlying cause
(connectivity, credentials, schema locks). Nothing ever serves JSONL instead.
A late-resolving start never publishes: once the deadline aborts the open, the candidate is closed
and never committed, even if the backend answers afterwards.
Single ownership
One process owns a session store at a time (ADR 0097 D4):- SQLite: a kernel-held lock on
<sessions.db>.owner.lockplus a private holder sidecar (pid, role, hostname, start time). A process crash releases the kernel lock automatically. Do not delete a lock file to bypass a live holder. - Postgres: a dedicated owner connection holds
pg_try_advisory_lock, scoped to the database and schema;ctx_storage_ownerrecords the holder.
EPERM, ENOTSUP or EROFS, storage
logs a warning and continues; verify the mount’s effective access controls. Other permission
hardening errors still fail the open.
While a gateway owns the database, CLI commands never open it: every storage read and maintenance
write goes to the gateway over RPC. With no gateway running, a CLI command takes ownership as
cli for the duration of the command (no retention, no import sweep). Errors you may see:
openclaw (pid N) is using session storage; retry when it finishes.— another CLI command holds it.Session storage is owned by a gateway on <hostname> (pid N); run this command where that gateway runs.
agent --local refuses while a gateway owns the database. Only
session-storage restore requires the gateway to be stopped.
Lifecycle states and storage.v1
The lifecycle exposes four phases:starting, ready, failed, stopping. Status and doctor
project the snapshot as {version:"storage.v1", state, configuredBackend, liveBackend, since, lastError}
where configuredBackend is the configured backend, liveBackend is sqlite, postgres, or
null before publish, and lastError carries the start failure cause. Inspect with
wednesdayai status, wednesdayai doctor --deep, or
openclaw session-storage status (openclaw is the alias).
Legacy import done and total describe work in the current sweep, excluding completed-receipt
no-ops. A fully completed pass has done === total; repeating an already completed import
reports zero new work. Pending, failed, and attempt-capped source counts describe receipts
separately.
Storage-backed RPCs (run history, chat.history, the session-run and task reads, the session
writers) are held at dispatch for up to 10 seconds while the start completes; after that they
answer retryable RECOVERY_NOT_READY (storage_starting). During shutdown they answer
storage_stopping until the socket closes. Because a start failure exits the process, a serving
gateway normally reports ready. See
Protocol for the method classification.
Configuration
backend defaults to sqlite (the database lives at <stateDir>/sessions.db). The only
surviving session.storage.recovery keys are:
session.storage.mode, session.storage.fallbackToJsonlOnError, and the retired
session.storage.recovery.*, session.maintenance.*, and prune-only
session.storage.consumerClaims.progressWindowMs / maxRetentionBackstopMs keys are removed
automatically when a legacy config loads; the gateway logs each change at start.
Legacy JSONL import
Installs upgraded from the retired file backend keep their history:sessions.json indices and
per-session *.jsonl transcripts are import sources only, never served or written again.
- Index import runs before storage publishes: every agent’s
sessions.jsonbecomes index rows. The database always wins — a key that already has a row or tombstone is left untouched, so import never changes a current session winner and never resurrects a deleted session. - Transcript import runs on first access to a legacy session (including from inside an active
turn, such as
/compactor a voice inject) and again in a throttled background sweep, newest first. All valid entries are imported, including branches; imported rows are markedmetadata.legacyImportand keep their original timestamps. - Receipts live in the database (
ctx_legacy_imports), so the import is resumable: failed sources retry on the next owner open, imported-but-unarchived sources finish archiving, and only completed sources are skipped. A failing source never fails startup.wednesdayai session-storage importruns the sweep to completion on demand and retries sources past their attempt cap. When a sweep pass finishes (not aborted) with actual work, the gateway logs onelegacy import completedline summarizing the pass:sources(sources processed),failed,rejected(rejected legacy lines), andpendingArchive(imported sources whose archive move is still pending). A pass with nothing to import logs no summary line. Containment compares real (symlink-resolved) paths, so a state dir reached through a symlink (a symlinked~/.openclaw, or the macOS/var->/private/varlink) no longer drops transcripts named by an index entry’ssessionFile. Discovered transcripts are still recorded under the configured path spelling, so each file keeps a single import receipt; a sessions-dir symlink that points outside the dir is still rejected.
<stateDir>/legacy-session-archive/<agentId>/ (files 0600, dirs 0700). Reset, deleted, and
replayed transcript archives, session-runs.jsonl, and everything under <stateDir>/recovery/**
are archived without import and kept indefinitely: they are the only copy. A fully imported
archive copy with zero rejected lines is purged after 30 days. Before the first import on SQLite
the owner writes a snapshot to <stateDir>/backups/.
Transcripts whose session key cannot be resolved are imported under
agent:<agentId>:legacy:<sessionId>. The operator override map
<stateDir>/recovery/session-key-overrides.json ({ "<sessionId>": "<full sessionKey>" }) is
still honoured for import key resolution; add mappings there before running
wednesdayai session-storage import.
Backups
wednesdayai session-storage backup writes a consistent SQLite copy with VACUUM INTO from a
separate read-only worker connection (appends never wait on it). Gateway backup paths stay
under <stateDir>/backups/; a local-owner CLI can use another --out path. The backup
connection waits up to 5 seconds for a lock, so two backups, or a backup and the pre-import
snapshot, no longer fail immediately with database is locked when they open a cold WAL database
together.
wednesdayai session-storage restore <file> replaces the database offline — stop the gateway
first; restore refuses while a gateway owns storage. The source is recovered through SQLite
(committed -wal frames included) from a private temporary copy, never from the backup file
itself, into a staged 0600 file that is integrity-checked and fsynced before anything
moves. Existing live -wal/-shm files are written to their 0600 fsynced
.pre-restore-<timestamp> archive names first, then the current database is checkpointed so
its committed WAL state lands in the main file: a blocked checkpoint aborts the restore
before anything changes, a corrupt or missing database keeps the sidecar archives, and a
successful checkpoint drops them (their content is folded into the archived main). The main file is
hardlinked — never removed — into its .pre-restore-<timestamp> archive, leftover live
sidecars are removed, and the staged file is swapped onto the live path with a single atomic
rename. A failed restore never deletes a pre-restore archive — the error names the paths it
kept — while a failure after the rename leaves the restored database published. For Postgres
use pg_dump / managed backups.
wednesdayai session-storage validate also runs SQLite integrity and context checks on a separate
read-only worker connection, not the gateway event loop or live writer’s queue. With WAL enabled,
the checks share one consistent read snapshot. Without WAL, each statement releases its read
lock before the next scan so validation does not hold up owner commits for the entire pass.
The validation connection waits up to 5 seconds for an owner commit before reporting a lock
error. On non-WAL mounts, a single long integrity_check or context scan still delays owner
writes while its read lock is held; if it outlasts the writer’s busy timeout, the write can
fail with database is locked. Run validation (including doctor’s checks) during a quiet
period on large databases or slow non-WAL mounts. Validation retains the full integrity
check rather than substituting a less thorough check.
Legacy user-row cleanup
User rows written before the__openclaw.ingress carrier shipped can carry baked-in channel framing blocks (Conversation info (untrusted metadata): …), a Current time: … line, and a leading message stamp in content_text, clean_user_message and the message text content. Core never rewrites these rows; the render layer guards against double-stamping, but the artifacts stay in storage until you clean them.
scripts/sessions/clean-legacy-user-rows.ts removes them, per row, with an undo log. It selects only role='user' rows whose message JSON has no __openclaw.ingress (post-upgrade rows are never candidates), never touches raw_entry, raw_line, raw_sha256, content_blocks, provider_payload or metadata, skips rows where a text field would change from non-empty to empty (reported as ambiguous), and is idempotent — a second run changes nothing. SQLite ctx_fts resyncs via its UPDATE trigger; Postgres content_text_tsv is a generated column. JSONL transcripts are out of scope: they are the raw log and render-time guards handle them.
-
Back up first. The store runs with WAL enabled, so use SQLite’s backup API, not a plain
cp:For Postgres, take a dump of thectx_*tables before proceeding. -
Dry-run (default; prints
scanned/changed/ambiguous/skippedNoopoverall and per field): -
Apply with an undo log (
--undo-logis required with--apply; the path must not exist — the run refuses to truncate an existing log): -
Verify. Re-run the dry-run: it should report
changed=0. Spot-check cleaned rows and confirm full-text search follows the cleanedcontent_text. -
Restore if needed (conditional and atomic; rows that moved on after the cleanup are never overwritten):
--agent <id> (exact agent id) or --limit <n>. Full procedure, verification queries and restore semantics: dev-docs/runbooks/legacy-user-row-cleanup.md in the repository.
Postgres notes
Schema setup (tables, indexes, idempotent column adds) runs inside one transaction under a 5-secondlock_timeout (a separate advisory lock serializes concurrent schema passes and is not
bounded by that timeout). If another transaction holds a conflicting lock on an existing table for
longer than that — typically a long-running or idle-in-transaction session, such as a stuck
client, a backup, or a manual psql session — the schema pass fails, storage start fails with it,
and the gateway exits with the cause. Nothing is corrupted: find the blocking transaction
(pg_stat_activity, pg_locks), let it finish or cancel it, then start the gateway again.
Idle pool-client failures do not crash the gateway. The pool (4 connections) and every client
it hands out carry error listeners that log warnings and replace failed idle clients.
Sustained warnings mean a connectivity problem.
The dedicated owner connection is different: each probe has a 5-second deadline, with
30 seconds between probes. Three consecutive probe misses signal loss; a successful probe
resets the miss count. An explicit connection error or close signals loss immediately.
Loss permits one bounded reattach attempt; failure or another loss before 10 stable minutes
terminates the process, since serving without ownership could admit a second writer.
Troubleshooting
- Gateway exits at startup with
FatalConfiguredStorageStartError: the configured backend did not open withinstartupProbeTimeoutMs, or the open failed. The fatal error carries the cause — connectivity, credentials, or a Postgres schema lock — and the CLI prints it as one redacted line:Gateway failed to start: <full cause chain>(nested causes included, e.g.connect ECONNREFUSED 127.0.0.1:5432for an unreachable Postgres). Fix the cause and let the service manager restart the gateway. No JSONL was served or written. session storage is owned by...: another process owns the database. For acliholder, retry when it finishes; for a gateway on another host or config, run the command where that gateway runs so it can proxy over RPC.- Legacy import stuck or a source failing:
wednesdayai session-storage statusreports per-state counts;wednesdayai session-storage importretries failed sources immediately. Repeated failures name the source path and cause in the gateway log — never edit transcripts in place while the gateway runs. session storage unavailable: ... flagged for SQL reconciliation(binding_needs_reconcile): a storage write failed mid-batch and the reconciliation read that followed failed too, leaving the agent’s store binding flagged. While that binding’s retained snapshot is empty, session reads report this unavailable error instead of serving an empty list; a flagged non-empty snapshot keeps serving its last contents until the next locked write reconciles it from SQL. Fix the underlying storage failure (connectivity, credentials, permissions, locks); the next locked write clears the flag. This is a retryable unavailability, not data loss: the rows are still in the database.- State stays
startinginstatus: a large legacy index import runs before publish. Watch the log; the 10-second RPC hold answers retryablestorage_startingmeanwhile.