Gateway troubleshooting
This page is the deep runbook. Start at /help/troubleshooting if you want the fast triage flow first.Command ladder
Run these first, in this order:wednesdayai gateway statusshowsRuntime: runningandRPC probe: ok.wednesdayai doctorreports no blocking config/service issues.wednesdayai channels status --probeshows connected/ready channels.
Anthropic 429 extra usage required for long context
Use this when logs/errors include:HTTP 429: rate_limit_error: Extra usage is required for long context requests.
- Selected Anthropic Opus/Sonnet model has
params.context1m: true. - Current Anthropic credential is not eligible for long-context usage.
- Requests fail only on long sessions/model runs that need the 1M beta path.
- Disable
context1mfor that model to fall back to the normal context window. - Use an Anthropic API key with billing, or enable Anthropic Extra Usage on the subscription account.
- Configure fallback models so runs continue when Anthropic long-context requests are rejected.
- /providers/anthropic
- /reference/token-use
- /help/faq#why-am-i-seeing-http-429-ratelimiterror-from-anthropic
No replies
If channels are up but nothing answers, check routing and policy before reconnecting anything.- Pairing pending for DM senders.
- Group mention gating (
requireMention,mentionPatterns). - Channel/group allowlist mismatches.
drop guild message (mention required→ group message ignored until mention.pairing request→ sender needs approval.blocked/allowlist→ sender/channel was filtered by policy.
Bundled plugin warning at startup
Use this when gateway logs includeConfig warnings: followed by
plugin not found: <id> for a plugin you expect to be bundled, especially when
plugins info or normal channel behavior suggests the plugin is actually
available.
plugins info <id>reportsOrigin: bundledandStatus: loaded.plugins.installs.<id>.sourceisbundled, or an older copied bundled record can be reconciled tobundled.- The stored
sourcePathstill points at the current bundled extension path. - The id appears only in stale config (
plugins.entries,allow,deny, orslots.memory) for a plugin you no longer use.
- Restart the gateway so startup reconciliation can refresh bundled install
records: Linux
systemctl --user restart wednesdayai-gateway; macOSwednesdayai gateway restart --deep. - Run
wednesdayai plugins update <id>to reconcile plugin install metadata before npm update selection runs. - Remove stale plugin references from
plugins.entries,plugins.allow,plugins.deny, orplugins.slots.memoryif the plugin is no longer wanted.
plugins.load.paths. Bundled plugins are
discovered from the installed WednesdayAI package, and a valid source: "bundled"
record is enough for validation and config-schema checks.
Plugin or skill install refuses
Use this whenwednesdayai plugins install <id> or wednesdayai skills install <id>
exits with a refusal and nothing was installed. Catalog installs are verified, not
trusted: compatibility first (no network), then Sigstore provenance, then digest. The
full message reference is Compatibility.
plugin "<id>" <version> is not compatible with this host: <reason>. Run `wednesdayai update` to upgrade WednesdayAI, then retry.(or the same shape withskill) — the catalog entry needs a newer host than you run.refusing to install "<id>" <version>: provenance verification failed: <detail>— the npm artifact has no SLSA attestation, was signed by a different workflow or ref, or its digest differs from the catalog pin.could not load the Sigstore trusted root (needs https://tuf-repo-cdn.sigstore.dev): ...— the Sigstore trusted-root refresh could not reach the TUF mirror. Check network egress tohttps://tuf-repo-cdn.sigstore.dev(corporate proxies and airgapped hosts are the usual cause), then retry. Nothing was installed.- A leftover
.openclaw-install-stage-*directory next to the install target (~/.openclaw/extensions/or~/.openclaw/skills/) — the install failed and the stage cleanup also failed. Stage directories are dot-prefixed and skipped by plugin and skill discovery, so they are inert; it is safe to delete the leftover directory manually, and the refusal error includes acleanup also failed: ...note naming it.
- For a compatibility refusal: run
wednesdayai update, then retry the install. - For a provenance refusal: do not retry from the bare npm spec. The artifact must be re-published through the release workflow so a correct attestation exists.
- For a trusted-root or network failure: restore egress to
https://tuf-repo-cdn.sigstore.devand npmjs, then retry. - For a leftover stage directory: confirm no install is running, then remove
.openclaw-install-stage-*under the state extensions/skills roots.
Host plugin SDK warning at startup
Use this when gateway logs includeplugins: host plugin SDK unavailable; plugins load it through jiti: ....
At startup the gateway prepares its own copy of the plugin SDK and hands it to
plugins, so they do not reload the SDK through jiti. This warning means that
handoff failed; the text after the colon is the underlying error. Plugins still
load through the jiti fallback and keep working, but startup is slower and
plugins run against their own SDK copy instead of the host one.
Fix options:
- Restart the gateway. The handoff is retried on every startup, so a transient failure clears on its own.
- If the warning repeats, read the error at the end of the log line. A missing or damaged plugin-sdk build in the installed package is the usual cause; update or reinstall WednesdayAI.
Plugin hook never fires
Use this when a plugin hook handler never runs and the gateway startup log shows aPLUGIN_HOOK_NOT_DISPATCHED warning, for example:
[plugins] startup: summary line. The warning means the plugin registered a
hook name that this gateway does not dispatch, so the handler is ignored while
the plugin keeps loading. Run wednesdayai plugins status to see the warning
and its remediation, then check the hook parity map for
the WednesdayAI hook name to register instead. The plugin shows as degraded
in wednesdayai plugins list.
ACP startup identity reconcile fails at startup
Use this when gateway logs includestartup identity reconcile failed for <session-key>: code=ACP_BACKEND_UNAVAILABLE from the acp/manager subsystem,
or the summary line
acp startup identity reconcile (renderer=...): checked=N resolved=N failed=N
with a nonzero failed count.
At gateway startup the gateway re-resolves ACP sessions whose harness identity
is still pending. If a session’s backend is not healthy yet, the gateway waits
up to 30 seconds (polling every 250 ms) for it to report healthy, then retries
each deferred session once. Sessions whose backend is still unhealthy when the
wait expires count as failed and log this warning. They are not lost: each
session re-resolves on its next use.
acpx runtime backend registered (command: ..., expectedVersion: ..., pluginLocalInstall: ...)followed within about 30 seconds byacpx runtime backend ready.- No
acpx local binary unavailable or mismatched (...); running plugin-local installline. That line means the plugin-localacpxbinary does not match the pin and an npm install ran during startup. - Run the
commandpath from theacpx runtime backend registeredline with--version; it must print the same version as theexpectedVersionon that line, which is the exactacpxversion declared in the plugin’spackage.json.
- If
acpx runtime backend readynever appears, read the earlieracpx runtime setup failed: ...oracpx runtime backend probe failed after local installlines for the root cause, fix it (missing or brokenacpxbinary, adapter download, harness authentication), and restart the gateway. - If the plugin-local binary version is wrong, restart the gateway and let the plugin-local install re-pin it to the declared version, or reinstall the acpx plugin.
- Affected sessions need no manual repair: they re-resolve on their next use.
Dashboard control ui connectivity
When dashboard/control UI will not connect, validate URL, auth mode, and secure context assumptions.- Correct probe URL and dashboard URL.
- Auth mode/token mismatch between client and gateway.
- HTTP usage where device identity is required.
device identity required→ non-secure context or missing device auth.device nonce required/device nonce mismatch→ client is not completing the challenge-based device auth flow (connect.challenge+device.nonce).device signature invalid/device signature expired→ client signed the wrong payload (or stale timestamp) for the current handshake.unauthorized/ reconnect loop → token/password mismatch.gateway connect failed:→ wrong host/port/url target.
- waits for
connect.challenge - signs the challenge-bound payload
- sends
connect.params.device.noncewith the same challenge nonce
Gateway service not running
Use this when service is installed but process does not stay up.Runtime: stoppedwith exit hints.- Service config mismatch (
Config (cli)vsConfig (service)). - Port/listener conflicts.
Gateway start blocked: set gateway.mode=local→ local gateway mode is not enabled. Fix: setgateway.mode="local"in your config (or runwednesdayai configure). If you are running WednesdayAI via Podman using the dedicatedopenclawuser, the config lives at~openclaw/.openclaw/openclaw.json.refusing to bind gateway ... without auth→ non-loopback bind without token/password.another gateway instance is already listening/EADDRINUSE→ port conflict.RECOVERY_NOT_READY(recovery_pending) on chat oragentcalls → the gateway is listening but restart recovery has not opened admission yet. Wait for thegateway ready: accepting work (startup Nms)log line;/readyzreturns200at the same point. Idle and cron triggers deferred across this window fire once admission opens. See Restart recovery.Gateway failed to start: ...→ the gateway process exited during startup. The fatal line prints the full error cause chain (nestedcauses walked to the root, redacted), so the underlying reason is in the message — for exampleGateway failed to start: ... configured session storage startup failed ... connect ECONNREFUSED 127.0.0.1:5432when the configured Postgres is unreachable. Fix the named cause and let the service supervisor restart the gateway. Storage start causes are covered in Session storage recovery.
Gateway restart refused on invalid config
Symptom:wednesdayai gateway restart exits non-zero with Config invalid, names the config file and the exact issue path, and the gateway keeps running.
This is the pre-flight restart gate working as intended: the gateway never stops when the config it would boot into is already known-invalid. Before this gate, a bad config edit turned a restart into a crash loop under systemd or launchd supervision.
Recovery:
- Read the issue path in the refusal output, for example
plugins.entries.<id>.config: invalid config: must NOT have additional properties. - Repair with
wednesdayai doctor --fix, or fix the named key manually. - Verify with
wednesdayai config validate(printsConfig valid). - Retry
wednesdayai gateway restart.
gateway stopand shutdown signals (SIGTERM,SIGINT) are never gated;SIGUSR1watchdog restarts remain gated.- Watchdog restarts from config reload (
gateway.reload.mode="restart"or"hybrid") are gated the same way: an invalid edit is skipped, the gateway keeps running the previous valid config, and the refusal warning appears in the gateway log. The warning names the config file, the exact issue path, and the same repair commands (wednesdayai doctor --fix,wednesdayai config validate). - A first start of a stopped gateway is not gated; invalid config there surfaces as a normal startup error. When launchd or systemd runs that start, the gateway waits for the config file to change (or 5 minutes) before exiting, so the service does not restart-loop; fix the config and the service manager restarts it.
Channel connected messages not flowing
If channel state is connected but message flow is dead, focus on policy, permissions, and channel specific delivery rules.- DM policy (
pairing,allowlist,open,disabled). - Group allowlist and mention requirements.
- Missing channel API permissions/scopes.
mention required→ message ignored by group mention policy.pairing/ pending approval traces → sender is not approved.missing_scope,not_in_channel,Forbidden,401/403→ channel auth/permissions issue.
Telegram fleet polling or webhook conflicts
Use this when Telegram logs repeat messages likeNetwork request for 'getUpdates' failed!, account health shows conflicted,
or webhook mode receives requests but does not dispatch updates.
- Per-account
mode,state,owner,retryAt,poll.lastPollAt, andwebhook.lastRequestOutcomein the Channels UI. conflictedstate ortelegram.poll.conflictdiagnostics, which usually means two local accounts, two gateway processes, or a webhook and polling process are trying to own the same bot token.- Repeated
getUpdatesnetwork errors, which usually point to DNS, IPv6, TLS, proxy, firewall, or host egress problems toapi.telegram.org. - A
Polling stall detectedlog line, which means the built-in watchdog force-restarted a wedgedgetUpdatescycle; frequent stalls point to a stuck socket or proxy between the gateway andapi.telegram.org. - Webhook
rejected,malformed,timeout, or non-2xx outcomes, which point to a bad public URL, wrong secret token, reverse-proxy mismatch, or a slow webhook handler.
- Give each enabled
channels.telegram.accounts.<id>entry its ownbotTokenortokenFile;TELEGRAM_BOT_TOKENis only a fallback for the default account. - Ensure one bot token has exactly one owner: one configured account, one gateway process, and either polling or webhook mode.
- If polling, let WednesdayAI clear webhook ownership non-destructively at
startup; if a webhook owner reappears later, polling re-runs that cleanup
automatically on
409instead of conflict-looping. Treat any manualdrop_pending_updatesoperation as destructive unless you intentionally want to discard pending Telegram updates. - If direct Bot API egress is unstable, configure
channels.telegram.proxyor the Telegramnetworksettings described in the Telegram channel guide.
- /channels/telegram#multi-bot-fleet
- /channels/telegram#polling-webhook-and-ownership
- /diagnostics/otel-langfuse#telegram-fleet-diagnostics
Cron and heartbeat delivery
If cron or heartbeat did not run or did not deliver, verify scheduler state first, then delivery target.- Cron enabled and next wake present.
- Job run history status (
ok,skipped,error). - Heartbeat skip reasons (
quiet-hours,requests-in-flight,alerts-disabled).
cron: scheduler disabled; jobs will not run automatically→ cron disabled.cron: timer tick failed→ scheduler tick failed; check file/log/runtime errors.main job requires payload.kind="systemEvent"→ stored job type and payload are incompatible. Recreate or edit it as a Timeline message, Check first, then nudge, or isolated Assistant task job.wake-gate-empty→ a wake-gate cron check found no work, so the skipped run is expected.heartbeat skippedwithreason=quiet-hours→ outside active hours window.heartbeat: delivery target unavailable; delivered to session chat→ the configured orlasttarget couldn’t be resolved, wasn’t ready, or failed to send, so the output went to the session chat.fallbackReasonnames the cause (no-external-route,target-unavailable,unknown-account,channel-not-ready,send-failed);staleChannel/staleAccountIdname the lost route only when it survived resolution. A route whose account was removed from config is cleared before the fallback, so ano-external-routewithout those fields, on a session that used to deliver externally, usually means its account was removed; check the session’s stored route againstchannels.<channel>.accounts.heartbeat skippedwithreason=dm-blocked→ heartbeat target resolved to a DM-style destination whileagents.defaults.heartbeat.directPolicy(or per-agent override) is set toblock.
Session run history is empty or stale
Main-session turns (chat, cron, heartbeat) are recorded as session-run rows in the configured session storage backend, powering run history and crash recovery. Recording is best effort: a failed store write never fails the turn itself, so a broken store stays invisible unless you look for the warnings. Ifsessions.runs.list returns no rows or stale timestamps:
session-run store write failedwarns from subsystemsessions/session-runs/recorder. The warn is rate limited to one per minute per operation and carries asuppressedcount of failures held back in that window, so a persistent failure shows as a steady drip instead of log flooding.- Schema drift after an upgrade. With
session.storage.backend: "postgres"or"sqlite"the gateway adds any missing session tables and columns automatically on boot, so a gateway restart repairs drift without manual SQL. That schema pass waits at most 5 seconds for Postgres locks on existing tables: a long-running or idle-in-transaction session (stuck client, backup, manualpsql) fails storage startup and the gateway exits with the cause. Resolve the blocker (pg_stat_activity,pg_locks) and start the gateway again; there is no file fallback.
error field from one
warning line and check database connectivity, credentials, and permissions.
Related:
Node paired tool fails
If a node is paired but tools fail, isolate foreground, permission, and approval state.- Node online with expected capabilities.
- OS permission grants for camera/mic/location/screen.
- Exec approvals and allowlist state.
NODE_BACKGROUND_UNAVAILABLE→ node app must be in foreground.*_PERMISSION_REQUIRED/LOCATION_PERMISSION_REQUIRED→ missing OS permission.SYSTEM_RUN_DENIED: approval required→ exec approval pending.SYSTEM_RUN_DENIED: allowlist miss→ command blocked by allowlist.
Browser tool fails
Use this when browser tool actions fail even though the gateway itself is healthy.- Valid browser executable path.
- CDP profile reachability.
- Extension relay tab attachment for
profile="chrome".
Failed to start Chrome CDP on port→ browser process failed to launch.browser.executablePath not found→ configured path is invalid.Chrome extension relay is running, but no tab is connected→ extension relay not attached.Browser attachOnly is enabled ... not reachable→ attach-only profile has no reachable target.
Agent tool calls fail
Use this when a tool call fails even though the gateway is healthy, or an agent keeps falling back fromedit to exec or a whole-file write.
[tools] <tool> failed: <message>(error level): the tool executed and its execute stage failed, for example a file that does not exist or an edit that found no match. The message is the tool’s own error text.[tools] <tool> rejected: <category>(warn level): the agent loop rejected the call before execution. Categories:schema validation failed (N issues)(the arguments did not match the tool schema),unknown tool, andblocked(a policy denied the call). A tool name that is not a valid identifier shows as<invalid tool name>.[tools] <tool> failed: details redacted(warn level): the call ended in an error that is not a rejection, most often a cancelled run aborting the tool.
sessions_history for that session, or read the
session JSONL file under ~/.openclaw/agents/<agentId>/sessions/.
The session reply carries a one-line warning for the same failure, for example
⚠️ 📝 Edit: in notes.md failed (not found). The suffix is a bounded failure
class, not the raw error:
(invalid tool arguments)- the model called the tool wrong, for example an edit whoseoldTextdoes not match the file.(not found)- the target file or directory does not exist.(permission denied)- the call was denied: errno codes (EACCES,EPERM,EROFS) or a workspace/sandbox guard (workspaceOnly, symlink and hardlink guards). Only the errno codes mean the filesystem refused access; a guard denial means changingworkspaceOnlyor the sandbox policy, not file permissions.(blocked by policy)- a tool policy or loop guard blocked the call, or a plugin hook blocked it without giving a reason. A hook block with a reason shows a barefailed.- A bare
failedwith no suffix means the error fit no known class.
exec/bash failures stay out
of the chat at default verbosity; their full output is in the journal and the
session transcript.
The file tools accept the argument shapes models commonly send: edit,
write, read and delete take path and also accept file_path; delete
removes one file and refuses directories, the workspace root, and paths outside
the workspace; edit additionally
accepts legacy top-level oldText/newText or old_string/new_string, which
are folded into the edits list, and edits may arrive as a JSON string;
write accepts empty content (it creates or truncates the file). A recurring
edit rejected: schema validation failed on those shapes means the gateway
predates the alias-handling fix, or the agent was instructed to use an argument
shape the current schema does not define.
Related:
Agent keeps re-running BOOTSTRAP onboarding
Use this when the agent re-injectsBOOTSTRAP.md on every session and
onboarding never completes. Onboarding is stamped complete
(onboardingCompletedAt) only once BOOTSTRAP.md is gone from the workspace,
and the ritual ends with “Delete this file”, which needs filesystem tools.
Check:
-
Confirm
BOOTSTRAP.mdstill exists in the agent workspace. Then check the tool policy:Thedeletetool ships with thegroup:fsgroup and thecodingprofile. Amessagingprofile (the onboarding default on older installs) has no fs tools at all, so the agent cannot writeIDENTITY.mdor deleteBOOTSTRAP.md. -
If
tools.exec.security: "deny"blocks shell access, the agent has no exec fallback either;deleteis the only way to finish the ritual.
tools.allow is empty or unset,
add the fs group alongside your profile, keeping any existing alsoAllow
entries:
tools.allow already lists tools, add "group:fs" to that list instead.
Config validation rejects allow and alsoAllow in the same scope.
New installs get this from onboarding automatically when no explicit
tools.profile is set. tools.fs.workspaceOnly: true scopes read/write/
edit/apply_patch to the workspace; delete is always workspace-contained.
If the workspace uses lanes, the gateway must be on a version where lane seed
files are copied only at lane creation: earlier versions re-copied
BOOTSTRAP.md from the persona root on every turn, so a lane could never
finish onboarding. Upgrading fixes this; the persona-root BOOTSTRAP.md stays
as the lane seed template by design, so completion is per lane.
Tool output caps and context pruning log lines
Use this when the gateway logstool result exceeded source cap,
tool-result context guard reduced tool output, or
prompt ledger unavailable; recomputing projections, or when you want to
confirm that context pruning and the tool-output cap are working.
tool result exceeded source cap→ a single tool result exceeded the in-memory source cap (400,000text chars). The agent keeps a head-and-tail excerpt; the full raw bytes are stored out-of-band when database write-through is on. Expected once per oversized result, not per call.tool-result context guard reduced tool output→ one line per eviction batch, carryingtoolNames. Expected after a cache-TTL pruning round or an aggregate eviction, not on every call.prompt ledger unavailable; recomputing projections→ the stored pruning ledger row for a session could not be read (corrupt, timed out, or written by a future build). The session recomputes projections from stored history and logs this warning once. A corrupt or unknown-version row is overwritten; a timed-out read leaves the row for the next run. Repeated warnings on the same session point at storage corruption.
- Large
tool result exceeded source capvolume: cap output at the tool level or return less per call; the in-memory context is bounded by design. - No
tool-result context guard reduced tool outputlines and largecacheWriteafter idle gaps: checkagents.defaults.contextPruning(modecache-ttl) and that the session actually went idle pastttl. - Repeated
prompt ledger unavailablewarnings on the same session: check session storage health withwednesdayai doctor --deepand Session storage recovery.
If you upgraded and something suddenly broke
Most post-upgrade breakage is config drift or stricter defaults now being enforced.1) Auth and URL override behavior changed
- If
gateway.mode=remote, CLI calls may be targeting remote while your local service is fine. - Explicit
--urlcalls do not fall back to stored credentials.
gateway connect failed:→ wrong URL target.unauthorized→ endpoint reachable but wrong auth.
2) Bind and auth guardrails are stricter
- Non-loopback binds (
lan,tailnet,custom) need auth configured. - Old keys like
gateway.tokendo not replacegateway.auth.token.
refusing to bind gateway ... without auth→ bind+auth mismatch.RPC probe: failedwhile runtime is running → gateway alive but inaccessible with current auth/url.
3) Pairing and device identity state changed
- Pending device approvals for dashboard/nodes.
- Pending DM pairing approvals after policy or identity changes.
device identity required→ device auth not satisfied.pairing required→ sender/device must be approved.