Skip to main content

Task pipeline

The task pipeline runs registered background task passes through the Gateway work queue with durable run records, unit budgets, preemption, and crash recovery. It is the Tier 2 execution substrate: work item claims and run records commit in one transaction, so a crash can never strand a claim without a matching run record. Operator commands: wednesdayai task. Configuration and operations: Task operations.

Architecture

  • Task registry (src/backbone/task-registry.ts): passes register a plain object (id, priority, scope, costClass, run(ctx), optional rollback(ctx) and retryPolicy). Dispatch order is priority ascending, stable for ties.
  • Dispatcher (src/backbone/task-dispatcher.ts): each tick claims ready work items per pass topic (task.<passId>), governs the claim against budgets, then runs the pass with an AbortSignal. Concurrency is shared across pass topics (round-robin slots).
  • Governor (src/backbone/task-governor.ts): units-only budget enforcement at admission, per run (taskRunId), lane (tenantOrLaneKey, default lane:<passId>), and global (global) scopes over a rolling window, plus a hard global monthly ceiling read from the same scope key the dispatcher writes (global). Attempts count toward the cap, not only successful completions. costUsd caps are not checked. A restart-loop detector defers llm-class passes when the gateway restarts too often.
  • Stores (src/config/sessions/): task_runs and task_run_events tables plus task_budget_entries and task_budget_totals, implemented for SQLite and Postgres. The JSONL store is dev-only and refuses transactional claims. Run rows carry stage (the pass’s declared stage label, if any) and trigger_source (task-dispatcher for ordinary enqueues, cron / heartbeat for the core cron and heartbeat passes, or the trigger named by the enqueuing plugin — the idle monitor path produces idle / schedule). This task_runs.trigger_source is a different column from the session-analytics trigger_source on ctx_session_entries and has different values.
  • Plugin seam (src/backbone/plugin-task-passes.ts): plugins register passes via api.registerTaskPass during register() (buffered, drained into the registry at Gateway boot, namespaced <pluginId>.<passId>) and enqueue governed work via api.enqueueTaskWork once the pipeline is running. See Background task passes.

Core cron and heartbeat passes

The Gateway registers two core passes at boot (src/gateway/task-pipeline-boot.ts): Cron job definitions live in the sibling cron_jobs table (migrated is the cutover bit). claimCronJobForPipeline flips migrated and inserts the task_runs row in one storage transaction. See Cron jobs and ADR 0055. Do not replace crontab cron.retry with DEFAULT_TASK_RETRY_POLICY. Do not put heartbeat rows in cron_jobs. What not to do:
Do this instead:

Run lifecycle (FSM)

pending → running → completed | failed | retrying | dlq | preempted | deferred | preempted_ceiling | rolled_back | cancelled
  • Co-commit: startTaskRunFromWork claims the work item and inserts the running run row in one transaction, and mints a claim token for that attempt. completeTaskWorkGuarded closes the claim and, in one transaction, writes the run status, the outcome event (with any budget denial and preemption reason), the pass measurement rows, and any successors. It is fenced on the claim token: it returns committed, already_committed (a replay of the same completion), stale (the claim was reclaimed; nothing is written), or conflict (another completion already finalized the run). A stale completion therefore never overwrites the attempt that replaced it. completeTaskWork is deprecated and delegates to it.
  • Preemption consumes no attempt: an aborted pass requeues with preemptionCount + 1. At the preemption ceiling the run is deferred for a cool-down; at twice the ceiling it terminates as preempted_ceiling (distinct from failure, never DLQ). The outcome row records the reason: lease, reclaim, or shutdown. A cooperative shutdown preemption does not advance preemptionCount or the ceiling.
  • Crash reclaim: when a new claim takes over expired work whose run is still running (the process died mid-pass), the same transaction finalizes the old run as preempted with reason reclaim and recovered: true, advances its preemption count, and starts the new run from that count. The old run gets no pass.metrics row, because its in-memory measurement died with the process.
  • Retry policy is data: a JSON policy (maxAttempts, backoff fixed or exponential, baseMs, maxMs) rides on the work item and is validated with strict integer bounds; exhaustion sends the run to dlq.
  • Budget denials defer the run (deferred, budget_denials + 1) with a retry time; they never consume attempts.

Recovery and retention

On boot, recoverTaskRuns classifies every recoverable run. It does not pre-mark every running row interrupted before classification: dry_run mode is preserved across recovery. updateRunStatus accepts an optional attempts field so attempt-consuming requeues can persist the counter across jsonl, SQLite, and Postgres backends.

Recovery ceiling (age stall + attempt exhaustion)

Runs selected for recovery do not loop forever. listRecoverableRuns yields running / retrying / preempted / deferred (not ordinary pending work). Before the status-specific arm, recovery applies a ceiling using that run’s retry_policy when present, otherwise the default (maxAttempts: 5, maxMs: 300_000):
  1. Attempt exhaustion — if status !== "preempted" and attempts >= maxAttempts, the run goes to dlq with reason retry exhausted (max attempts reached).
  2. Age stall — if status is retrying and age since updatedAt (else startedAt) exceeds maxAttempts * maxMs (default 25 minutes), the run goes to dlq with reason stalled beyond age threshold. Age is computed with parseTaskRunTimeMs (UTC-safe). A missing or unparseable timestamp skips this arm and falls through to requeue-after-backoff — do not treat garbage dates as already-stalled.
Either exhaust path emits a task.recovery_exhausted diagnostic (taskId, taskRunId, attempts, reason) so OTel/Langfuse surfaces the terminal transition. Operator view: Task operations. The bundled task-runlog-retention pass prunes task_run_events older than 30 days. In dry_run mode it reports the would-prune count without deleting.

Timestamp conventions

TaskRunRecord timestamps (startedAt, updatedAt, completedAt) are ISO-8601 UTC with a Z suffix at the record boundary. Every backend normalizes on read through toTaskRunTimeIso / parseTaskRunTimeMs:
  • Postgres stores ISO-8601 text and compares as text (never cast a TEXT column to timestamp).
  • SQLite stores space-separated UTC (YYYY-MM-DD HH:MM:SS.mmm via formatSqliteTime). The worker still writes that on-disk form; rowToTaskRunRecord converts it to ISO-Z before the classifier or any other consumer sees the row.
  • JSONL (dev-only) stores ISO-Z and re-normalizes on read. markRunningRunsInterrupted writes status retrying.
Age-stall and any other duration math must use parseTaskRunTimeMs. Do not call Date.parse on a raw SQLite value: ECMAScript treats the space-format string as local time, which inflates age by the host TZ offset and can send a just-interrupted run to dlq. Offset-less ISO date-times (2026-08-12T11:31:13.402) are also unparseable — Date.parse would treat them as local. Calendar-invalid values (2026-02-30, and the same date in space-format or ISO-offset) return null so Date.parse cannot overflow them to a later real day and false-DLQ the run. What not to do:
Do this instead:

Writing a pass

Core passes register directly on the registry; plugins use the identical shape through api.registerTaskPass (which namespaces the id as <pluginId>.<passId> and buffers the registration until Gateway boot). The id you pass in must be bare: a . throws PLUGIN_REGISTER_ERROR and the plugin does not load.
Passes must honor ctx.signal (preemption aborts long work), be idempotent across replay (the same work item can run again after a crash), and respect ctx.mode === "dry_run" by skipping side effects. The same idempotency contract applies to rollback: execution is at-least-once — if the durable rolled_back marker fails to persist, an operator retry re-invokes the handler. A pass that needs follow-on work returns it: run resolves to { successors: [...] } (each { taskId, triggerFingerprint, payload?, mode?, availableAt?, triggerSource? }, taskId a bare pass id of the same owner). The dispatcher inserts the successors under their normal idempotency keys inside the same guarded transaction that completes the run, so a successor exists only if its predecessor completed and a stale completion (after a reclaim) inserts none. Dry-runs, failed runs, and preempted runs return no successors; a retry returns them again.

Recording pass measurement

Every run gets a measurement buffer on its PassContext. The SDK helpers write into it; the dispatcher commits it with the run’s outcome as pass.metrics and pass.phase rows (see Run measurement for the row fields):
  • withPassPhase(ctx, name, fn) times fn as a named phase (ok, error, or aborted) and rethrows its error unchanged. Ordinals follow call order, so a repeated name stays distinct. At most 190 phases are kept; extra phases are counted in phasesDropped.
  • recordPassMetrics(ctx, input) buffers the run’s metrics; the last call wins. Invalid input is dropped with a warning. It never throws and does no I/O. items holds at most 64 keys; keys and phase names must be identifiers and never Object.prototype member names.
  • createPassUsageAccumulator() rolls up AnalysisResults into costSource, tokens, costUsd, and call counts. A not_attempted call is not a model call; an unavailable call makes tokens and cost unknown while keeping the known sums.
  • Once the dispatcher settles the run, the buffer is frozen and later records are ignored. A pass that ignores ctx.signal past its timeout gets a synthetic row with measurementState: "unavailable" and measurementReason: "handler_unsettled" instead of whatever it recorded.

Emitting task diagnostics from a pass

PassContext.emitDiagnostic accepts DiagnosticEventInput (the closed input union). Prefer the builders so the event type and action stay aligned and OTel can export the span:
Do not cast ad-hoc objects through as Parameters<typeof emitDiagnosticEvent>[0]. Builders are input-shaped (no ts/seq); emitDiagnosticEvent stamps those fields. Live field names matter: use passId (not passName) and attempts (not attempt).