> ## Documentation Index
> Fetch the complete documentation index at: https://docs.wednesdayai.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Testing policy

> Testing policy: frameworks, coverage thresholds, which suite to run, and how to add regressions

# Testing policy

This guide owns the testing policy. For lanes, worker settings and environment variables, see [Tests](https://docs.wednesdayai.dev/next/developers/reference/test.md). For the live, Docker and provider test kit, see [Testing kit](https://docs.wednesdayai.dev/next/users/help/testing.md).

## Policy

* **Framework**: Vitest 4 with V8 coverage.
* **Naming**: `*.test.ts` colocated with the source. End-to-end tests use `*.e2e.test.ts`.
* **Thresholds**: `vitest.config.ts` sets global coverage thresholds of 70% for lines, functions and statements, and 55% for branches.
* **Coverage run**: `pnpm test:coverage` runs `vitest.unit.config.ts` with coverage, so only the unit suite counts toward the thresholds.
* **Stubs**: prefer per-instance stubs over `SomeClass.prototype.method = ...` unless you document why.
* **Environment**: use `vi.stubEnv()` scoped per test. `test/setup.ts` restores stubs after every test with `afterEach(vi.unstubAllEnvs())`, because Vitest applies `unstubEnvs` only before each test; without this the last test of a file leaks its stubs into the next file in the worker.
* **Synthetic plugin configs**: under Vitest, an explicit test plugin config (for example `plugins.load.paths`) no longer cold-loads the bundled-by-default plugins (`device-pair`, `learning-core`, `phone-control`, `talk-voice`) that it does not name. To test with one of them, name it: `entries: { "<id>": { enabled: true } }` or `allow: ["<id>"]`. Do not stub `OPENCLAW_BUNDLED_PLUGINS_DIR` or stamp `activationMigrationVersion` just to avoid the load cost; `applyTestPluginDefaults` pre-disables unnamed bundled defaults for test configs (`src/plugins/config-state.ts`).
* **Workers**: do not set the worker count above 16. It has been tried.
* **Symlinked temp directory**: the CI Node test lane sets `TMPDIR` to a symlink, matching macOS where `/var` links to `/private/var`. When a test compares a temp path with a path Node resolved (such as `import.meta.url` or `realpathSync`), resolve both sides first. Build fixture temp roots under `await fs.realpath(os.tmpdir())` when the code under test resolves paths. Gate Linux-only `/proc` fixtures with an explicit `platform: "linux"` and mock the proc read. Do not assume Linux-only fixtures such as `/dev/shm`; inject the error instead (for example `EXDEV` via `vi.spyOn(fs, "link")`).
* **Colored console output**: subsystem console lines carry ANSI color whenever the shell looks color-capable (`TERM=xterm-256color`, `COLORTERM` or `TERM_PROGRAM` set), even without a TTY. Before an anchored assertion such as `toMatch(/^…/)` on a captured console line, strip it with `stripAnsi` from `src/terminal/ansi.ts`; otherwise the test passes under `NO_COLOR` and fails in an ordinary terminal (#1524).
* **Reachability over isolation**: a test that mocks past the dispatch or routing seam, by calling the changed function directly, proves the unit. It does not prove the unit is reached on the real path. For any bugfix or behavior change, add at least one test that drives the real entry point, and assert the original symptom (for example, "N completions to 1 turn") rather than only an internal return value. An isolation-only green suite once hid a dead-code feature (ADR 0006).
* **Plugin loads**: a plugin that imports `wednesdayai/plugin-sdk` makes jiti load the SDK graph from TypeScript source unless the caller prepared the host SDK first: 20-40 s of synchronous work per worker, which times out under full-lane load (#1029). `loadGatewayPlugins` and the conformance runner call `preparePluginSdkHost()` (`src/plugins/sdk-alias.ts`) for you, so plugins get the host's own SDK instances through jiti `virtualModules` (ADR 1032). A test that calls `loadOpenClawPlugins` directly with an SDK-importing plugin must `await preparePluginSdkHost()` first. When a test exercises loader behavior rather than a specific plugin, write small fixture plugins and point `OPENCLAW_BUNDLED_PLUGINS_DIR` at their directory (`src/plugins/loader.test-helpers.ts`); leave real-plugin coverage to the plugin's own tests and the e2e lane. The legacy `openclaw/plugin-sdk` specifier still loads (ADR 2057).
* **Timing assertions**: do not bound how fast something happened with wall-clock time (`expect(Date.now() - startedAt).toBeLessThan(n)`); full-lane load stretches sub-second work past any fixed bound (#1868). Prove promptness causally: wedge the slow stage on a promise that never settles, so reaching the assertion proves nothing waited on it, and let the test timeout catch a hang. When a real timer is the behavior under test, assert lower bounds and size the window well above child-process or I/O startup latency (#1870). Complexity guards compare calibrated batch timings at input sizes n and 8n against a ratio threshold, never an absolute ms bound (#1873).
* **Temp fixtures**: create fixture files in a private `fs.mkdtempSync(path.join(os.tmpdir(), "<prefix>-"))` directory and remove it recursively, never bare `os.tmpdir()`: code that scans its own directory (log rotation, backups) pays for every entry there (#1812). Prefer the existing helpers: `makeTempWorkspace` from `src/test-helpers/workspace.ts` already removes its dirs after the test file (import it statically), and `withTempDir` from `src/test-utils/temp-dir.ts` scopes a dir to a callback (#1886, #1952). As a backstop, every Vitest config inherits the global setup `test/global-tmpdir.ts`, which points `TMPDIR`, `TEMP` and `TMP` at one run-scoped `wai-vt-*` directory and removes it when the run ends, so a forgotten fixture cannot outlive the run (#2035). The tsx, jiti and Node compile caches inside it are links to the host temp dir, so spawned CLIs stay warm. Keep cleaning up per file anyway: a run holds every file's leftovers until it ends. Fixed paths outside `os.tmpdir()` are not covered.
* **Live gateway paths**: the deploy gate runs on the gateway host. Under Vitest, the default file log goes to `$OPENCLAW_TEST_HOME/logs` (#1263). Never write to, rotate, or delete files under `/tmp/openclaw` (`resolvePreferredOpenClawTmpDir()`) from a test. Known remaining offender: TTS output (#2021).
* **Live gateway port**: under Vitest (or any child carrying `OPENCLAW_TEST_HOME`), `GatewayClient` refuses to connect to a loopback URL on the default port 18789 and fails with `live-port-guard:`, so a tokenless test client can never reach the host's live gateway. Start a test gateway on an ephemeral port (`src/gateway/test-helpers.e2e.ts`) and pass its URL, or mock `gateway/call`. Only `vitest.live.config.ts` sets the opt-in `OPENCLAW_TEST_ALLOW_LIVE_GATEWAY=1`. Outside tests, a source-checkout client dialing the default port with no token or password logs a one-time warning.
* **Suite-scoped gateway fixtures**: under `installGatewayTestHooks({ scope: "suite" })`, every case starts from a reset `testState`, so `testState` set in `beforeAll` (identity links, `dmScope`, store paths, auth) is gone by the first case unless the suite keeps it. Call `keepSuiteTestState()` (`src/gateway/test-helpers.server.ts`) at the end of the top-level `beforeAll`. Each case then starts from a fresh copy of that fixture, the config the server booted with. Never call it from a nested `describe`'s `beforeAll`: it would leak that fixture into sibling describes. Use the nested describe's `beforeEach` instead.
* **Host-dependent fixtures**: a fixture generated from host runtime data (ICU case mappings) must record the host's version and compare across versions by a documented stability rule, not demand an exact match. Example: `unicodeVersion` in `test/fixtures/exec-allowlist-glob-parity.json`. Regenerate it with `pnpm exec:glob-parity:gen` only for matcher or generator changes, never for a Node upgrade (#1983).

## Commands

| Command | What it runs |
| - | - |
| `pnpm test` | Script policy suites, then every default-on lane through `scripts/test-parallel.mjs` |
| `pnpm test:fast` | Unit suite only (`vitest.unit.config.ts`) |
| `pnpm test:coverage` | Unit suite with V8 coverage |
| `pnpm test:e2e` | Gateway end-to-end smoke (needs `pnpm build` first) |
| `pnpm test:live` | Live provider tests. The script sets `OPENCLAW_LIVE_TEST=1` and `CLAWDBOT_LIVE_TEST=1` itself |

On memory-constrained hosts, run `OPENCLAW_TEST_PROFILE=low OPENCLAW_TEST_SERIAL_GATEWAY=1 pnpm test`.

`pnpm test` includes the e2e lane and the gateway canvas tests, which run the built CLI (`dist/entry.js`) and the A2UI bundle. Run `pnpm build` first, or those tests fail with `missing dist/entry.(m)js` or a canvas `503`.

Docker runners: `pnpm test:docker:live-models`, `pnpm test:docker:live-gateway`, and the onboarding end-to-end run `pnpm test:docker:onboard`. See [Testing kit](https://docs.wednesdayai.dev/next/users/help/testing.md) for details.

### Real infrastructure in tests

* Default lanes (`pnpm test`, `pnpm test:fast`, and the per-area lanes) never start Docker and never reach a real Postgres or Redis. `test/test-env.ts` deletes `OPENCLAW_DB_URL`, `OPENCLAW_DATABASE_URL` and `DATABASE_URL` unless `OPENCLAW_LIVE_PG_TEST=1`, so a DSN exported in your shell does not reach them.
* Real-Postgres tests, including those in `scripts/`, are named `*.pg.integration.test.ts` and gate on `OPENCLAW_LIVE_PG_TEST=1` plus `OPENCLAW_DATABASE_URL`. Base `vitest.config.ts` excludes them; they run only through `pnpm test:pg` (`vitest.pg.config.ts`).
* `pnpm test:pg` with no URL starts its own `postgres:16` container. With `OPENCLAW_DB_URL` set it requires `--authorize-isolated-database` (a throwaway database on your server, no Docker) or `--authorize-disposable-url` (the whole database is disposable).
* Tests that need Docker itself additionally gate on `OPENCLAW_TEST_PG_LANE=container`.
* Fixtures shared by a live file and its fake or SQLite sibling go in `<base>.test-helpers.ts`.
* Redis-backed behavior is tested with fakes (`src/config/sessions/ctx-redis-cache.test.ts`). A future real-Redis test follows the same pattern with its own opt-in lane.

## Which suite should I run?

* Editing logic or tests: run `pnpm test`, and `pnpm test:coverage` if you changed a lot. That already includes core, channels, extensions, gateway, e2e and the workspace suites (`apps/web`, `apps/control`, `packages/gateway-browser-client`, `packages/host-bridge`) unless you opt a lane out.
* Debugging a single e2e file: run `pnpm test:e2e` or pass a path. Do not treat a missing e2e section in a failed `pnpm test` log as proof that e2e was skipped. The runner may have aborted after an earlier lane.
* Debugging "my bot is down", provider-specific failures, or tool calling: run a narrowed `pnpm test:live`.

If `pnpm test` flakes on a loaded host, rerun once before treating it as a regression, then isolate with `pnpm vitest run <path/to/test>`.

## GitHub Actions

The PR verification workflows are **parked** in `.github/workflows-disabled/` (ADR 0118, ADR 2399). GitHub cannot run them, and `pnpm check` fails if a workflow outside the allowlist appears in `.github/workflows/`. The local commands below are the proof; do not restore a workflow to validate a PR.

| Workflow (archived) | Local equivalent |
| - | - |
| CI (`ci.yml`) | `pnpm check`, `pnpm build`, `pnpm build:strict-smoke`, `pnpm web:build:packaged`, `pnpm test`, `pnpm test:docker:session-storage`, `pnpm check:docs` |
| Docs surface check | `node scripts/docs-surface-check.mjs --mode=source` |
| Install Smoke | `pnpm test:dockerfiles:smoke`; `pnpm test:web-image:smoke`; after a stable promotion `pnpm test:install:hosted` |
| Runtime Risk | `node scripts/ci/select-risk-tests.mjs --base origin/main --head HEAD` |
| Sandbox Common Smoke | `pnpm test:dockerfiles:smoke` |
| Workflow Sanity | `pnpm lint:workflows` |
| npm-publish verify-dev (removed job) | `pnpm release:verify-dev` after merges that publish to `dev` |
| Deterministic Security (PR trigger removed) | `git-hooks/pre-push` secret scan; full scans: `gitleaks dir .` and `uvx --with 'setuptools<81' semgrep==1.101.0 scan --config security/semgrep/wednesdayai.yml` |

The per-job command list for every parked workflow lives in `.github/workflows-disabled/README.md`.

These stay enabled:

* **Deterministic Security**: a weekly (Tuesday 04:17 UTC) and on-demand gitleaks and semgrep full-repo scan; PRs are scanned locally.
* **release-please**, **npm-publish**: releases need the npm OIDC trusted-publishing identity, which exists only in GitHub Actions; **npm-parity** is the daily registry watchdog that opens issues.
* **docs-site-links**: a weekly live-site link check against `docs.wednesdayai.dev`; it opens a `docs site link gaps` issue.
* **CodeRabbit**: a GitHub App that reviews PRs. It is configured in `.coderabbit.yaml` and uses no Actions minutes.

A PR gets only a CodeRabbit review in GitHub; run the local gates before opening it and list them under "Deterministic gate evidence". `pnpm web:build:packaged` is the web bundle-budget gate (`apps/web/bundle-budget.json`): neither `pnpm check` nor `pnpm build` runs it, so a PR that touches `apps/web`, `packages/gateway-browser-client`, or code the web client imports runs it before merge (#2501).

List the commit SHA the local gates ran on. A PR with no Actions runs, or a CodeRabbit review that was skipped or rate-limited, is expected — not a finding or a blocker; do not re-request it.

No CI catches breakage between PRs: after merging a batch, run `pnpm check` on `main` locally.

### Local hooks

* `pnpm install` points `core.hooksPath` at `git-hooks/`, except under `CI=true`.
* pre-commit fixes lint and format on staged files.
* pre-push runs the gitleaks scan of unpushed commits, then `pnpm check`, then the advisory local review.
* Install gitleaks with `brew install gitleaks` (or a release binary). Without it the push goes ahead with a warning.
* A false positive goes into `.gitleaksignore` by Fingerprint.
* `OPENCLAW_SKIP_PREPUSH=1` skips the hook for WIP pushes.

## Adding regressions

When you fix a provider or model issue discovered in live:

* Add a CI-safe regression if possible. Use a mock or stub provider, or capture the exact request-shape transformation.
* If the issue is inherently live-only (rate limits, auth policies), keep the live test narrow and opt-in through environment variables.
* Prefer the smallest layer that catches the bug:
  * Provider request conversion or replay bug: a direct models test.
  * Gateway session, history or tool pipeline bug: a gateway live smoke or a CI-safe gateway mock test.

## Related

* [Tests](https://docs.wednesdayai.dev/next/developers/reference/test.md)
* [Testing kit](https://docs.wednesdayai.dev/next/users/help/testing.md)
* [Coding style](https://docs.wednesdayai.dev/developers/contributing/coding-style.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.