Task pipeline
The task pipeline runs registered task passes through the Gateway work queue with durable run records, unit budgets, preemption, and crash recovery. A pass is a plain object that claims one work item per run; the claim and the run record commit in a single storage transaction, so a crash can never strand a claim without a matching run. The pipeline is the execution substrate for the corecron-run and heartbeat-tick passes and for any passes your plugin registers. It requires a SQLite or Postgres session store — the dev-only JSONL store refuses transactional claims and the pipeline stays disabled.
The TaskPass contract
run receives a PassContext:
storage type is deliberate: pass code must never touch the budget ledger or the raw bus. There is no escape hatch to a fuller runtime — do not cast ctx.storage to something wider.
Registering a pass from a plugin
Callapi.registerTaskPass inside your plugin’s register(). Registrations are buffered during plugin load and drained into the registry at Gateway boot under the namespaced id <pluginId>.<passId>, so plugin ids can never collide with core passes.
- The pass id must not contain
.— the boot drain namespaces it as<pluginId>.<passId>, and a dot in either half lets two registrations collide. A dotted id throwsPLUGIN_REGISTER_ERRORand the plugin does not load. - A duplicate
(pluginId, passId)pair throws at registration time. registerTaskPassis synchronous and only buffers. A pass registered after Gateway boot is never dispatched — and if the pipeline is disabled at boot, the buffer is discarded so a later boot never picks up stale passes.
Enqueueing governed work
A registered pass does nothing until work is enqueued against its topic. Plugins enqueue throughapi.enqueueTaskWork:
taskIdis the bare pass id. The backend namespaces it — never pass<pluginId>.<passId>yourself.triggerFingerprintscopes idempotency: the work item id istaskId:triggerFingerprint:mode. Use a stable fingerprint (a date bucket, a cursor) to make re-enqueues collapse, or a unique one to force separate runs.triggerSourceappears inwednesdayai task status --jsonoutput and intask_outcomehooks.- The call rejects until the task pipeline has started (and when
tasks.enabled: false). Catch and retry or schedule around Gateway boot — do not assume it resolves.
Run lifecycle
- Co-commit: claiming the work item and inserting the
runningrow happen in one transaction; completion closes the claim, finalizes the run, and applies budget entries in one transaction. - Preemption consumes no attempt. An aborted pass requeues with
preemptionCount + 1. AtpreemptionCeiling(default 3) the run is deferred for a one-hour cool-down; at twice the ceiling it terminates aspreempted_ceiling— a distinct terminal state, never DLQ. - Budget denials defer the run (
budget_denials + 1) with a retry time. Deferrals never consume attempts. - Retry exhaustion sends the run to
dlq.
Budgets
The governor meters units only, at admission time (each claimed run metersestimatedUnits: 1). Checks evaluate run → lane → global, then the monthly ceiling; the first failure wins.
- Per-run scope keys on the run id; per-lane on
lane:<passId>; global onglobal. Apayload.scopeKeyyou supply overrides the derived key. - The rolling window is
tasks.budget.windowHours(default 24); the hard global ceiling is monthly. - Attempts count toward caps, not only successful completions.
costUsdfields exist in the config schema but are not enforced — do not rely on them for spend control.llm-costClass passes are additionally deferred when the Gateway restart-loops (default: more than 5 restarts in 10 minutes) so a crash-looping host cannot burn budget.
Preemption and timeouts
- The dispatcher aborts in-flight runs through
ctx.signalon Gateway stop and on preemption. Honor the signal: checkctx.signal.abortedin loops and pass it to fetch/child processes. A pass that ignores it still runs to completion, but its work is duplicated after requeue. - The default run timeout is 30 s. Set
timeoutMson the pass for longer work; when it exceeds the 35 s claim lease, the dispatcher renews the lease at half-lease cadence so a crashed gateway still frees the work within one lease window. - On timeout the dispatcher aborts the controller and records the run as
failed(not preempted) — a timed-out run consumes a retry attempt.
Retry policy
Retry policy is data on the work item, validated with strict integer bounds:maxAttempts: 1–50 (default policy: 5)backoff:"fixed"|"exponential"(default policy: exponential)baseMs: 100–3,600,000 (default policy: 1000)maxMs: 100–86,400,000 (default policy: 300,000)
retryPolicy, the default policy above applies. Exhaustion sends the run to dlq.
Crash recovery and idempotency
On boot, every recoverable run is classified:
Recovery applies a ceiling using the run’s own retry policy (default: 5 attempts, 300 s max backoff):
- Attempt exhaustion —
attempts >= maxAttempts(exceptpreempted) →dlq, reasonretry exhausted (max attempts reached). - Age stall — a
retryingrun older thanmaxAttempts × maxMs(25 minutes under the default policy) →dlq, reasonstalled beyond age threshold.
rollback carries the same contract — if the durable rolled_back marker fails to persist, an operator retry re-invokes the handler. Respect ctx.mode === "dry_run" by skipping all side effects.
Core passes
The Gateway registers these at boot; your namespaced ids cannot collide with them.Emitting diagnostics
ctx.emitDiagnostic accepts the closed DiagnosticEventInput union. Use the taskDiagnostic builder so the event type and action stay aligned and OTel can export the span:
ts/seq — the emitter stamps those). Live field names matter: passId (not passName) and attempts (not attempt).
Async safety
Blocking the agent loopapi.registerTaskPassis synchronous and only buffers an object — it must not perform I/O at all. If your plugin needs startup I/O before deciding to register a pass, make the plugin’sregisterfunction async and await it. Passrun()functions execute inside the Gateway’s event loop. They never block the dispatch tick (the dispatcher awaits them), but a pass doing synchronous blocking I/O (fs.readFileSync,execSync) stalls the entire Gateway — every channel, hook, and heartbeat — for the duration. Never use sync I/O APIs in a pass. A slowrun()holds its concurrency slot and its work-lease until it finishes or times out; with the defaultdispatchConcurrency: 1, one slow pass delays every other pass’s next run. Do not do this:Do this instead:
Related
- Task pipeline configuration — every
tasks.*config key, default, and effect - Cron jobs — the admin reference for the scheduler built on
cron-run - Plugin SDK — the full
apisurface, includingregisterTaskPassandenqueueTaskWork - Your first plugin — plugin structure, manifest, and loading