Skip to main content

Task pipeline

The task pipeline runs registered task passes through the Gateway work queue with durable run records, unit budgets, preemption, and crash recovery. A pass is a plain object that claims one work item per run; the claim and the run record commit in a single storage transaction, so a crash can never strand a claim without a matching run. The pipeline is the execution substrate for the core cron-run and heartbeat-tick passes and for any passes your plugin registers. It requires a SQLite or Postgres session store — the dev-only JSONL store refuses transactional claims and the pipeline stays disabled.

The TaskPass contract

run receives a PassContext:
The narrow storage type is deliberate: pass code must never touch the budget ledger or the raw bus. There is no escape hatch to a fuller runtime — do not cast ctx.storage to something wider.

Registering a pass from a plugin

Call api.registerTaskPass inside your plugin’s register(). Registrations are buffered during plugin load and drained into the registry at Gateway boot under the namespaced id <pluginId>.<passId>, so plugin ids can never collide with core passes.
Rules enforced at registration:
  • The pass id must not contain . — the boot drain namespaces it as <pluginId>.<passId>, and a dot in either half lets two registrations collide. A dotted id throws PLUGIN_REGISTER_ERROR and the plugin does not load.
  • A duplicate (pluginId, passId) pair throws at registration time.
  • registerTaskPass is synchronous and only buffers. A pass registered after Gateway boot is never dispatched — and if the pipeline is disabled at boot, the buffer is discarded so a later boot never picks up stale passes.

Enqueueing governed work

A registered pass does nothing until work is enqueued against its topic. Plugins enqueue through api.enqueueTaskWork:
  • taskId is the bare pass id. The backend namespaces it — never pass <pluginId>.<passId> yourself.
  • triggerFingerprint scopes idempotency: the work item id is taskId:triggerFingerprint:mode. Use a stable fingerprint (a date bucket, a cursor) to make re-enqueues collapse, or a unique one to force separate runs.
  • triggerSource appears in wednesdayai task status --json output and in task_outcome hooks.
  • The call rejects until the task pipeline has started (and when tasks.enabled: false). Catch and retry or schedule around Gateway boot — do not assume it resolves.

Run lifecycle

  • Co-commit: claiming the work item and inserting the running row happen in one transaction; completion closes the claim, finalizes the run, and applies budget entries in one transaction.
  • Preemption consumes no attempt. An aborted pass requeues with preemptionCount + 1. At preemptionCeiling (default 3) the run is deferred for a one-hour cool-down; at twice the ceiling it terminates as preempted_ceiling — a distinct terminal state, never DLQ.
  • Budget denials defer the run (budget_denials + 1) with a retry time. Deferrals never consume attempts.
  • Retry exhaustion sends the run to dlq.

Budgets

The governor meters units only, at admission time (each claimed run meters estimatedUnits: 1). Checks evaluate run → lane → global, then the monthly ceiling; the first failure wins.
  • Per-run scope keys on the run id; per-lane on lane:<passId>; global on global. A payload.scopeKey you supply overrides the derived key.
  • The rolling window is tasks.budget.windowHours (default 24); the hard global ceiling is monthly.
  • Attempts count toward caps, not only successful completions.
  • costUsd fields exist in the config schema but are not enforced — do not rely on them for spend control.
  • llm-costClass passes are additionally deferred when the Gateway restart-loops (default: more than 5 restarts in 10 minutes) so a crash-looping host cannot burn budget.
Budget denial is not an error: the run defers and retries later. Design passes to tolerate repeated deferral.

Preemption and timeouts

  • The dispatcher aborts in-flight runs through ctx.signal on Gateway stop and on preemption. Honor the signal: check ctx.signal.aborted in loops and pass it to fetch/child processes. A pass that ignores it still runs to completion, but its work is duplicated after requeue.
  • The default run timeout is 30 s. Set timeoutMs on the pass for longer work; when it exceeds the 35 s claim lease, the dispatcher renews the lease at half-lease cadence so a crashed gateway still frees the work within one lease window.
  • On timeout the dispatcher aborts the controller and records the run as failed (not preempted) — a timed-out run consumes a retry attempt.
What not to do:
Do this instead:

Retry policy

Retry policy is data on the work item, validated with strict integer bounds:
  • maxAttempts: 1–50 (default policy: 5)
  • backoff: "fixed" | "exponential" (default policy: exponential)
  • baseMs: 100–3,600,000 (default policy: 1000)
  • maxMs: 100–86,400,000 (default policy: 300,000)
When you omit retryPolicy, the default policy above applies. Exhaustion sends the run to dlq.
Do not invent a second retry language for cron-style work. The core cron-run pass deliberately uses { maxAttempts: 1 } so the public cron.retry config stays the operator-visible backoff — layering task retries on top would double-retry cron jobs.

Crash recovery and idempotency

On boot, every recoverable run is classified: Recovery applies a ceiling using the run’s own retry policy (default: 5 attempts, 300 s max backoff):
  • Attempt exhaustion — attempts >= maxAttempts (except preempted) → dlq, reason retry exhausted (max attempts reached).
  • Age stall — a retrying run older than maxAttempts × maxMs (25 minutes under the default policy) → dlq, reason stalled beyond age threshold.
Execution is at-least-once: the same work item can run again after a crash. Passes must be idempotent across replay, and rollback carries the same contract — if the durable rolled_back marker fails to persist, an operator retry re-invokes the handler. Respect ctx.mode === "dry_run" by skipping all side effects.

Core passes

The Gateway registers these at boot; your namespaced ids cannot collide with them.

Emitting diagnostics

ctx.emitDiagnostic accepts the closed DiagnosticEventInput union. Use the taskDiagnostic builder so the event type and action stay aligned and OTel can export the span:
Do not cast ad-hoc objects into the emit signature. Builders are input-shaped (no ts/seq — the emitter stamps those). Live field names matter: passId (not passName) and attempts (not attempt).

Async safety

Blocking the agent loop api.registerTaskPass is synchronous and only buffers an object — it must not perform I/O at all. If your plugin needs startup I/O before deciding to register a pass, make the plugin’s register function async and await it. Pass run() functions execute inside the Gateway’s event loop. They never block the dispatch tick (the dispatcher awaits them), but a pass doing synchronous blocking I/O (fs.readFileSync, execSync) stalls the entire Gateway — every channel, hook, and heartbeat — for the duration. Never use sync I/O APIs in a pass. A slow run() holds its concurrency slot and its work-lease until it finishes or times out; with the default dispatchConcurrency: 1, one slow pass delays every other pass’s next run. Do not do this:
Do this instead: