Task operations
The task pipeline boots with the Gateway when enabled and runs registered background passes through the work queue with budgets and durable run records. Concepts: Task pipeline. CLI:wednesdayai task.
Configuration
All keys live undertasks in the Gateway config. Defaults shown.
- Budget enforcement is units-based and checked at admission (
perRun,perLane, andglobal). A run that would exceed a window cap is deferred (not failed) and retried after the window; the run’sbudget_denialscounter increments. Attempts count toward the cap, not only successful completions.costUsdcaps are not checked in this release. - The default lane key is
lane:<passId>when the enqueue does not setpayload.scopeKey. - The global monthly ceiling is a hard stop: once accrued units reach it, dispatching defers until an operator resets the budget or the month rolls over.
tasks.restartLoopis honored at Gateway boot (defaultsmaxRestarts: 5,windowMinutes: 10). Changing it requires a restart (Linux:systemctl --user restart wednesdayai-gateway; macOS:wednesdayai gateway restart --deep).- Restart-loop detection is durable. Each Gateway start writes one row to the task
budget ledger (scope
governor:restart), so starts are counted across real process restarts and across every Gateway sharing the same database. The start aftermaxRestartsinsidewindowMinutes(the sixth by default) defers everyllm-class pass, includingheartbeat-tick, in every running Gateway until the oldest counted start leaves the window (retryAt= oldest start +windowMinutes). Deterministic passes keep running. Earlier releases kept these timestamps in memory, so the detector never fired across real restarts. - Until a Gateway’s startup row is written, its
llm-class passes are deferred asgovernor_error; the write is retried with backoff (1 s doubling to 5 minutes). If the ledger cannot be written,llmwork stays deferred (fail closed) rather than running uncounted. A metering-store outage still defers every pass, deterministic or not. - Dispatch is priority-ordered with bounded fairness: after 8 consecutive claims of
higher-priority passes while a lower-priority pass has ready work, the next claim goes to
the longest-waiting lower-priority pass. A permanently busy high-priority pass (for
example
heartbeat-tick) is delayed by at most one lower-priority pass per 8 claims, and the order is unchanged when no lower-priority work is waiting. - A
dry_runexecution (tasks.runDryRun) skips budget admission entirely, including the restart-loop gate, so it runs even when a window or the monthly ceiling is exhausted. It writes no budget entry and keeps its audited run, outcome, and measurement rows. tasks.enabled: falsedisables boot entirely (atask.pipeline_disableddiagnostic is emitted once).- The idle monitor only starts when the task pipeline itself is running, so
tasks.enabled: trueis a prerequisite: settingtasks.idle.enabled: trueor a non-emptytasks.schedule.windowswhiletasks.enabledisfalsedoes nothing (the pipeline never boots, so there is no monitor to start). With the pipeline enabled, either of those settings starts the idle monitor, which fires theagent_idleplugin hook when an agent crosses the idle threshold or a schedule window opens. Plugins such as Dream Cycle consume it to run background work. If durable run storage is unavailable the monitor does not start and atask.idle_monitor_disableddiagnostic is emitted instead of failing silently. - Idleness is judged from recorded session runs. Turns started through the Gateway
agentRPC, includingwednesdayai agentturns sent through the Gateway, are recorded as session runs (kindchat, recoveryoff), so the idle trigger fires after them. Awednesdayai agent --localturn is not visible to the Gateway (see agent). A crash leaves such a runrunning; the next Gateway start finalizes it asabandoned(see Restart recovery). - Run records carry
stageandtrigger_sourcecolumns:trigger_sourceisbootfor the gateway boot enqueue,task-dispatcherfor ordinary enqueues,cronforcron-run,heartbeatforheartbeat-tick, or the trigger named by the enqueuing plugin (the idle monitor path producesidle/schedule). Both appear inwednesdayai task status --json. Cron job definitions are not stored intask_runs; they live incron_jobs. See Where cron jobs are stored. tasks.enabled: falsekeeps unmigrated cron jobs and heartbeat on their pre-pipeline paths (legacy timer and intervalrunOnce). Jobs already flipped tomigrated = truestay on the pipeline claim path and will not fire until the pipeline is enabled again. Publiccron.*methods and heartbeat skip reasons stay the same.
Gateway methods
Plugins can register additional read-only methods alongside these: the
Dream Cycle plugin adds
dream.status, dream.weights, and
dream.candidates.
Status filters are validated: an unknown status value returns an invalid-request error
rather than an empty list. An out-of-range or fractional limit is clamped into 1..1000
instead of rejected.
tasks.status reports each source separately. runsAvailable is false (with runsError)
when the run store is missing or fails, and budgetWindow.available is false (with
budgetWindow.error) when the budget store is missing or fails. One failing source never hides
the other. Besides its existing totals, budgetWindow carries the running governor’s
windowHours, cap (the per-run, per-lane, or global cap for that scope; null for tenant),
ceilingCap (global scope only), windowStartTs (the later of now - windowHours and the
last reset), and windowUsageUnits/windowUsageCostUsd summed from windowStartTs, so usage
compares directly with cap. Without a running pipeline the window fields are null.
tasks.runEvents { taskRunIds } returns events keyed by run id, plus retentionDays (events
older than this are pruned). Each entry has a kind: metrics (the pass.metrics row),
phase (one pass.phase row), outcome (lifecycle rows, including dry_run, with only the
denial, preemption reason, and recovered fields), or unsupported, malformed, oversize, and
unknown, which carry no payload. Rows over 16 KiB are reported as oversize without being
read. At most 200 rows per run, 2,000 overall, and 1 MiB are returned; truncation keeps the
newest rows and is reported per run in truncated. Ids must be 1..128 characters with no
control characters; duplicates are ignored. The jsonl-fs backend answers UNAVAILABLE.
tasks.runDryRun { taskId, requestId? } returns { ok, taskId, workId }. workId is the
database id of the enqueued work item, and it equals the workId of the run in tasks.status
that executes it, so a caller can follow its own dry run without guessing from timestamps.
Without requestId, every call enqueues a new work item. With requestId, the call is
idempotent per task: the id is trimmed and NFC-normalized, and the work item’s fingerprint is
derived from the task id plus that normalized id, so a retry with the same requestId for the
same task returns the same workId and enqueues nothing new. A requestId that is not a string,
is empty after trimming, is longer than 128 characters, or contains a control character (C0,
DEL, or C1) is rejected with INVALID_REQUEST. The CLI (wednesdayai task run --dry-run) sends
no requestId, so each invocation is a new dry run.
Read-scoped connections also receive a dreams.update broadcast event with payload
{ ts, kind: "task" | "dream" }. kind: "task" follows a committed task.completed,
task.failed, task.preempted, task.budget_deferred, task.budget_ceiling_reached, or
task.preemption_ceiling_reached diagnostic; kind: "dream" follows any dream.* diagnostic.
It carries no run data (clients re-fetch with tasks.status and tasks.runEvents) and is
dropped for slow consumers. The Dreams view refreshes on it.
Rollback appends its audit event before the status write; once
that durable marker lands, a retry is a no-op. A persist failure after a successful rollback
returns a distinct rollback_persist_failed error — and if the marker itself failed to
persist, a retry re-invokes the handler, which is why rollback handlers must be idempotent
(see Task pipeline).
Run measurement
Each run can carry measurement rows, written in the same transaction that completes the run, so a run never shows an outcome without the measurement that belongs to it:pass.metrics(at most one per run, schema version 1):mode(normalordry_run),measurementState(complete, orunavailablewithmeasurementReason: "handler_unsettled"when a pass ignored its abort signal past the timeout), token counts,costUsd,costSource, model call counts,elapsedMs, and optionalqualityScore,authoring,skipped, and pass-specificitems. Each row is at most 16 KiB.pass.phase(up to 190 per run): a named phase with itsordinal,offsetMsfrom the run start,durationMs, andstatus(ok,error, oraborted).- Outcome rows carry the structured budget denial and the preemption reason.
costSource says how to read costUsd:
Pricing rule. Cost is computed by the Gateway from the model entry under
models.providers. A model with no config entry, and a model declared without a cost
block, both report costSource: "unpriced" with costUsd: null — an absent price is never
treated as free. A model whose cost block is all zeros is a genuinely free model: it reports
costSource: "priced" with costUsd: 0. Set cost (input, output,
cacheRead, cacheWrite) on a model to get real spend. Budget window units are governor
admission units, not model spend: budgetWindow.windowUsageUnits and the caps never include
costUsd.
Budget denials. A deferred run’s outcome row names the gate that stopped it:
{ reason: "window", scope, usage, cap } (a run, lane, or global window cap),
{ reason: "ceiling", scope: "global", usage, cap } (the monthly ceiling), or
{ reason: "restart_loop" } (the restart-loop guard). A denied run makes no model call and
writes no pass.metrics row.
Preemption. A preempted outcome row carries preemption.reason: lease (the dispatcher
lost the work lease or preempted the pass for live work), reclaim (the run was left running
by a crashed process, and the next claim of its expired work finalized it as preempted with
recovered: true and no pass.metrics row), or shutdown (a cooperative Gateway stop). The run
record’s preemptionCount counts lease and reclaim preemptions; shutdown never advances it
or the preemption ceiling.
Dry runs. A dry_run run writes a dry_run outcome row and a pass.metrics row with
mode: "dry_run". Built-in passes preview in dry-run mode: they make no model call and change
no lessons, proposals, dream state, or run logs, so any counts in items are what a normal run
would do. A dry run returns no successors and does not feed Dream Cycle adaptation.
Verifying a deployment
bash scripts/verify-substrate-webui-surface.sh is the evidence command for this surface. By
default it builds a local fixture: a temporary state directory and config (SQLite session
storage, tasks, Learning Core, and Dream Cycle enabled, no model providers or credentials), and
it starts this checkout’s Gateway from dist/ on a free loopback port, then stops it and removes
the fixture. Set OPENCLAW_VERIFY_GATEWAY_URL and OPENCLAW_VERIFY_GATEWAY_TOKEN to run the
same checks against a running Gateway instead; the script never restarts it. Each check prints
one PASS or FAIL line:
- preflight:
hello.features.methodsadvertisestasks.runDryRun,tasks.status,tasks.runEvents, anddream.status. A missing method is reported as a disabled-plugin prerequisite (exit 3), not as an RPC failure. - dry-run:
tasks.runDryRunoflearning-core.consolidationwith arequestIdunique to this invocation returns{ ok, taskId, workId }. It needs no model credentials and never depends on the latest run. - run-status:
tasks.statusis polled until the run with thatworkIdcompletes. - run-events:
tasks.runEventsreturns a schema-version-1pass.metricsrow withmode: "dry_run"for exactly that run. - read-scope:
dream.statusanswers on a connection that holds onlyoperator.read, and the same connection is refusedtasks.runDryRun.
node --import tsx scripts/verify-substrate-webui-surface-live.mts verifies the live half,
SC5-SC11. It boots three throwaway gateway fixtures from this checkout’s dist/, drives real
agent sessions against a real configured model, and exercises the v2 operator console in
headless Chromium, so a run makes real model calls and costs real tokens. The model is selected
with OPENCLAW_VERIFY_MODEL (required, <provider>/<model>); its credential comes from the
operator config, which the verifier reads read-only from OPENCLAW_VERIFY_PROVIDER_CONFIG
(default ~/.openclaw/openclaw.json). Two more variables tune the run:
OPENCLAW_VERIFY_RUN_TIMEOUT_MS (default 240000) bounds each poll and contemplation wait, and
OPENCLAW_VERIFY_KEEP_TMP=1 keeps the fixture directories on disk instead of deleting them.
The verifier refuses to run against a stale checkout: dist/entry.js and
dist/control-ui-v2/index.html must exist, and the commit recorded in dist/build-info.json
must equal git rev-parse HEAD, so the evidence names the code under test. Run
pnpm build && pnpm control:build:packaged first. Preflight also launches headless Chromium
before any fixture boots, so a missing Playwright browser fails fast.
Each fixture is a temporary /tmp/vswl.* root with a generated openclaw.json written 0600
(it embeds the provider credential), a detached gateway process group on free loopback ports,
and a SIGTERM-then-SIGKILL teardown. The verifier never touches the operator’s gateway: the
checks create real sessions, lessons and skill proposals, which would mutate live learning
state. The three fixtures cover, one each:
- learning: real sessions plus a consolidation and distill pass — the learning checks (SC8, SC9, SC11) and the Dreams and Learning console views (SC5, SC6, SC7 run rows).
- dream-priced: a dream cycle under a stub-priced model — upgrade-day produce, a priced contemplation with cost assertions, and the live Dreams view update.
- dream-unpriced: the unconfigured-producer bucket and the
unpricedcontemplation cost row.
PASS [<id>] <detail> or
FAIL [<id>] <detail> line; a skipped check prints FAIL [<id>] skipped: <reason>, and checks
that depended on it are skipped too. When every check ran and passed, the run ends with
VERIFY PASS (16 checks) and exits 0. A failed check or internal error exits 1. A missing
prerequisite — no OPENCLAW_VERIFY_MODEL, an unreadable provider config, a missing or stale
build — prints FAIL [preflight] PREREQUISITE <detail> before any gateway is spawned and exits 3. SIGINT and SIGTERM run the same cleanup path and exit 130.
Warning: with OPENCLAW_VERIFY_KEEP_TMP=1 the fixture directories stay on disk, and each
one contains the operator’s provider credential in its openclaw.json. The run prints the paths
it kept (INFO kept <root> (contains the provider credential; delete it)); delete them after
inspecting.
Shutdown and recovery
- Gateway shutdown stops the pipeline before storage closes and awaits the dispatcher drain
so no pass writes to closed storage. In-flight passes are aborted through their signal and
get up to 15 seconds to settle; a pass that stops in time is recorded as
preemptedwith reasonshutdown, which does not count toward the preemption ceiling. A pass that ignores the signal past the drain is left as a crash would leave it: its work stays claimed until the lease expires, then the next start reclaims it (and that reclaim does count). The stop force-exit budget is 20 seconds, and the generated LaunchAgent (ExitTimeOut) and systemd unit (TimeoutStopSec) allow 30 seconds, so the service manager does not kill the process first. On existing installs,wednesdayai doctorflags the shorter stop timeout; reinstall the service (wednesdayai gateway install --force) to pick it up. The agent-turn producers (cron and heartbeat) are stopped and parked delayed-turn waiters are cancelled first; then the agent-turn runnerstop()is awaited the same way: it stops claiming and finishes in-flight claimed turns of every kind (cron, heartbeat, nudge) before storage closes. Restart confirmation counts those claimed turns together with embedded runs. A stop request (systemctl --user stop wednesdayai-gateway,wednesdayai gateway stop,SIGTERM, orSIGINT) wins over an in-progress restart evaluation or drain, so the process exits promptly instead of hanging until the supervisor sendsSIGKILL. Stop-wins does not cover the in-process lock reacquire window (F-2026-08-19-04). - On boot, recovery classifies each recoverable run. Same-instance
runningrows becomeretrying(status rewrite only, no attempt increment, no in-place checkpoint resume). Stale-instancerunningrows requeue and consume an attempt. There is no pre-classification interrupt rewrite of everyrunningrow. Deferred runs wait for their idle window. Terminal runs are untouched. See Task pipeline recovery. - Recovery ceiling: recoverable runs (
running/retrying/preempted/deferredfromlistRecoverableRuns) use that run’sretry_policywhen present, otherwise the default (maxAttempts: 5,maxMs: 300000). A run that has hit its attempt budget (exceptpreempted) or has been stuck inretryinglonger thanmaxAttempts * maxMs(default 25 minutes sinceupdatedAt/startedAt) moves todlqon the next boot recovery pass. Age is always UTC: SQLite space-format timestamps are not interpreted as the host timezone. For aretryingrun with attempts remaining, a missing or unparsable timestamp skips the age-stall arm and falls through torequeue-after-backoff;deferredstayswait-idleandpreemptedstaysrequeue-immediate. That exhaust transition emits atask.recovery_exhausteddiagnostic (visible in OTel/Langfuse as spanopenclaw.task.recovery_exhaustedwhendiagnostics-otelis enabled). Notasks.*config key controls this; the change takes effect on the next Gateway start (Linux:systemctl --user restart wednesdayai-gateway; macOS:wednesdayai gateway restart --deep). - Attempt counters only increase on attempt-consuming recovery requeues (
requeueandrequeue-after-backoff). Same-instance resume, preemption requeues, and budget deferrals do not burn attempts. - Run event logs are pruned after 30 days by the bundled
task-runlog-retentionpass. The 30-day period is fixed in this release (there is notasks.*retention key yet); use a dry run to preview (wednesdayai task run --dry-run task-runlog-retention).
Troubleshooting
- Runs stuck in
deferredwith risingbudget_denials: a window cap or the monthly ceiling is exhausted. Inspectwednesdayai task status, thenwednesdayai task budget reset --scope globalif intended. llm-class passes deferred right after restarts: the restart-loop detector tripped (restartLoopconfig); it clears once restarts leave the window.preempted_ceilingruns: the pass was repeatedly aborted (gateway restarts or manual preemption). This is terminal by design and distinct from failure; re-enqueue the task if the work is still needed.- Runs that used to stay in
retryingforever now appear asdlq: expected after the recovery ceiling. ChecklastError/ thetask.recovery_exhaustedspan reason (retry exhaustedvsstalled beyond age threshold). Re-enqueue the work if it is still needed; investigate why the pass never completed if age-stall is frequent. - A just-restarted Gateway immediately parks fresh interrupted runs as
dlqwithstalled beyond age threshold: that was a timestamp-parse bug on non-UTC hosts (SQLite UTC written without aZ, then read as local time). Current recovery ages those stamps as UTC. After you install a build that includes the fix, restart the Gateway. A genuine stall still parks the run after 25 minutes inretrying. - Missing task/dream spans in Langfuse or Tempo: confirm
diagnostics.otelis enabled and thediagnostics-otelplugin is loaded. Task and dream lifecycle events export asopenclaw.task.*/openclaw.dream.*spans — see OTel & Langfuse.