Task pipeline
The task pipeline runs registered background task passes through the Gateway work queue with durable run records, unit budgets, preemption, and crash recovery. It is the Tier 2 execution substrate: work item claims and run records commit in one transaction, so a crash can never strand a claim without a matching run record. Operator commands:wednesdayai task. Configuration and operations:
Task operations.
Architecture
- Task registry (
src/backbone/task-registry.ts): passes register a plain object (id,priority,scope,costClass,run(ctx), optionalrollback(ctx)andretryPolicy). Dispatch order is priority ascending, stable for ties. - Dispatcher (
src/backbone/task-dispatcher.ts): each tick claims ready work items per pass topic (task.<passId>), governs the claim against budgets, then runs the pass with anAbortSignal. Concurrency is shared across pass topics (round-robin slots). - Governor (
src/backbone/task-governor.ts): units-only budget enforcement at admission, per run (taskRunId), lane (tenantOrLaneKey, defaultlane:<passId>), and global (global) scopes over a rolling window, plus a hard global monthly ceiling read from the same scope key the dispatcher writes (global). Attempts count toward the cap, not only successful completions.costUsdcaps are not checked. A restart-loop detector defersllm-class passes when the gateway restarts too often. - Stores (
src/config/sessions/):task_runsandtask_run_eventstables plustask_budget_entriesandtask_budget_totals, implemented for SQLite and Postgres. The JSONL store is dev-only and refuses transactional claims. Run rows carrystage(the pass’s declared stage label, if any) andtrigger_source(task-dispatcherfor ordinary enqueues,cron/heartbeatfor the core cron and heartbeat passes, or the trigger named by the enqueuing plugin — the idle monitor path producesidle/schedule). Thistask_runs.trigger_sourceis a different column from the session-analyticstrigger_sourceonctx_session_entriesand has different values. - Plugin seam (
src/backbone/plugin-task-passes.ts): plugins register passes viaapi.registerTaskPassduringregister()(buffered, drained into the registry at Gateway boot, namespaced<pluginId>.<passId>) and enqueue governed work viaapi.enqueueTaskWorkonce the pipeline is running. See Background task passes.
Core cron and heartbeat passes
The Gateway registers two core passes at boot (src/gateway/task-pipeline-boot.ts):
Cron job definitions live in the sibling
cron_jobs table (migrated is the
cutover bit). claimCronJobForPipeline flips migrated and inserts the
task_runs row in one storage transaction. See Cron jobs
and ADR 0055.
Do not replace crontab cron.retry with DEFAULT_TASK_RETRY_POLICY. Do not put
heartbeat rows in cron_jobs.
What not to do:
Run lifecycle (FSM)
pending → running → completed | failed | retrying | dlq | preempted | deferred | preempted_ceiling | rolled_back | cancelled
- Co-commit:
startTaskRunFromWorkclaims the work item and inserts therunningrun row in one transaction, and mints a claim token for that attempt.completeTaskWorkGuardedcloses the claim and, in one transaction, writes the run status, the outcome event (with any budget denial and preemption reason), the pass measurement rows, and any successors. It is fenced on the claim token: it returnscommitted,already_committed(a replay of the same completion),stale(the claim was reclaimed; nothing is written), orconflict(another completion already finalized the run). A stale completion therefore never overwrites the attempt that replaced it.completeTaskWorkis deprecated and delegates to it. - Preemption consumes no attempt: an aborted pass requeues with
preemptionCount + 1. At the preemption ceiling the run is deferred for a cool-down; at twice the ceiling it terminates aspreempted_ceiling(distinct from failure, never DLQ). The outcome row records the reason:lease,reclaim, orshutdown. A cooperative shutdown preemption does not advancepreemptionCountor the ceiling. - Crash reclaim: when a new claim takes over expired work whose run is still
running(the process died mid-pass), the same transaction finalizes the old run aspreemptedwith reasonreclaimandrecovered: true, advances its preemption count, and starts the new run from that count. The old run gets nopass.metricsrow, because its in-memory measurement died with the process. - Retry policy is data: a JSON policy (
maxAttempts,backofffixed or exponential,baseMs,maxMs) rides on the work item and is validated with strict integer bounds; exhaustion sends the run todlq. - Budget denials defer the run (
deferred,budget_denials + 1) with a retry time; they never consume attempts.
Recovery and retention
On boot,recoverTaskRuns classifies every recoverable run. It does not pre-mark every
running row interrupted before classification:
dry_run mode is preserved across recovery. updateRunStatus accepts an optional attempts
field so attempt-consuming requeues can persist the counter across jsonl, SQLite, and Postgres
backends.
Recovery ceiling (age stall + attempt exhaustion)
Runs selected for recovery do not loop forever.listRecoverableRuns yields
running / retrying / preempted / deferred (not ordinary pending work). Before the
status-specific arm, recovery applies a ceiling using that run’s retry_policy when
present, otherwise the default (maxAttempts: 5, maxMs: 300_000):
- Attempt exhaustion — if
status !== "preempted"andattempts >= maxAttempts, the run goes todlqwith reasonretry exhausted (max attempts reached). - Age stall — if status is
retryingand age sinceupdatedAt(elsestartedAt) exceedsmaxAttempts * maxMs(default 25 minutes), the run goes todlqwith reasonstalled beyond age threshold. Age is computed withparseTaskRunTimeMs(UTC-safe). A missing or unparseable timestamp skips this arm and falls through torequeue-after-backoff— do not treat garbage dates as already-stalled.
task.recovery_exhausted diagnostic (taskId, taskRunId,
attempts, reason) so OTel/Langfuse surfaces the terminal transition. Operator view:
Task operations.
The bundled task-runlog-retention pass prunes task_run_events older than 30 days. In
dry_run mode it reports the would-prune count without deleting.
Timestamp conventions
TaskRunRecord timestamps (startedAt, updatedAt, completedAt) are ISO-8601 UTC with a
Z suffix at the record boundary. Every backend normalizes on read through
toTaskRunTimeIso / parseTaskRunTimeMs:
- Postgres stores ISO-8601 text and compares as text (never cast a TEXT column to
timestamp). - SQLite stores space-separated UTC (
YYYY-MM-DD HH:MM:SS.mmmviaformatSqliteTime). The worker still writes that on-disk form;rowToTaskRunRecordconverts it to ISO-Z before the classifier or any other consumer sees the row. - JSONL (dev-only) stores ISO-Z and re-normalizes on read.
markRunningRunsInterruptedwrites statusretrying.
parseTaskRunTimeMs. Do not call
Date.parse on a raw SQLite value: ECMAScript treats the space-format string as local
time, which inflates age by the host TZ offset and can send a just-interrupted run to
dlq. Offset-less ISO date-times (2026-08-12T11:31:13.402) are also unparseable —
Date.parse would treat them as local. Calendar-invalid values (2026-02-30, and
the same date in space-format or ISO-offset) return null so Date.parse cannot
overflow them to a later real day and false-DLQ the run.
What not to do:
Writing a pass
Core passes register directly on the registry; plugins use the identical shape throughapi.registerTaskPass (which namespaces the id as <pluginId>.<passId> and buffers the
registration until Gateway boot). The id you pass in must be bare: a . throws
PLUGIN_REGISTER_ERROR and the plugin does not load.
ctx.signal (preemption aborts long work), be idempotent across replay
(the same work item can run again after a crash), and respect ctx.mode === "dry_run" by
skipping side effects. The same idempotency contract applies to rollback: execution is
at-least-once — if the durable rolled_back marker fails to persist, an operator retry
re-invokes the handler.
A pass that needs follow-on work returns it: run resolves to { successors: [...] } (each
{ taskId, triggerFingerprint, payload?, mode?, availableAt?, triggerSource? }, taskId a bare
pass id of the same owner). The dispatcher inserts the successors under their normal
idempotency keys inside the same guarded transaction that completes the run, so a successor
exists only if its predecessor completed and a stale completion (after a reclaim) inserts none.
Dry-runs, failed runs, and preempted runs return no successors; a retry returns them again.
Recording pass measurement
Every run gets a measurement buffer on itsPassContext. The SDK helpers write into it; the
dispatcher commits it with the run’s outcome as pass.metrics and pass.phase rows (see
Run measurement for the row fields):
withPassPhase(ctx, name, fn)timesfnas a named phase (ok,error, oraborted) and rethrows its error unchanged. Ordinals follow call order, so a repeated name stays distinct. At most 190 phases are kept; extra phases are counted inphasesDropped.recordPassMetrics(ctx, input)buffers the run’s metrics; the last call wins. Invalid input is dropped with a warning. It never throws and does no I/O.itemsholds at most 64 keys; keys and phase names must be identifiers and neverObject.prototypemember names.createPassUsageAccumulator()rolls upAnalysisResults intocostSource, tokens,costUsd, and call counts. Anot_attemptedcall is not a model call; anunavailablecall makes tokens and cost unknown while keeping the known sums.- Once the dispatcher settles the run, the buffer is frozen and later records are ignored. A
pass that ignores
ctx.signalpast its timeout gets a synthetic row withmeasurementState: "unavailable"andmeasurementReason: "handler_unsettled"instead of whatever it recorded.
Emitting task diagnostics from a pass
PassContext.emitDiagnostic accepts DiagnosticEventInput (the closed input union). Prefer
the builders so the event type and action stay aligned and OTel can export the span:
as Parameters<typeof emitDiagnosticEvent>[0]. Builders
are input-shaped (no ts/seq); emitDiagnosticEvent stamps those fields. Live field names
matter: use passId (not passName) and attempts (not attempt).