Testing
WednesdayAI has a default-onpnpm test parallel runner, a live Vitest suite, and a small set of Docker runners.
This doc is the live, Docker and provider test kit:
- What each suite covers (and what it deliberately does not cover)
- How live tests discover credentials and select models/providers
Build prerequisite
pnpm test is not self-contained: a few suites exercise the shipped artifacts rather
than the TypeScript sources, so run pnpm build first (the archived CI lane ran
pnpm canvas:a2ui:bundle for the same reason; PR verification workflows are parked in
.github/workflows-disabled/ — ADR 0118, ADR 2399).
test/gateway.multi.e2e.test.tsandtest/gateway.restart-recovery.e2e.test.tsspawndist/index.js.test/cli-json-stdout.e2e.test.tsrunsopenclaw.mjs, which boots fromdist/entry.js.src/gateway/server.canvas-auth.test.tsfetches the A2UI assets, whichpnpm canvas:a2ui:bundle(the first step ofpnpm build) generates intosrc/canvas-host/a2ui/a2ui.bundle.js. That file is gitignored, so a fresh checkout has no copy and the request answers503 A2UI assets not found.
Quick start
Most days:- Full gate (expected before push):
pnpm build && pnpm check && pnpm test
- Coverage gate:
pnpm test:coverage - E2E suite:
pnpm test:e2e
- Live suite (models + gateway tool/image probes):
pnpm test:live
Test suites (what runs where)
Think of the suites as “increasing realism” (and increasing flakiness/cost):Unit / integration (default pnpm test)
- Command:
pnpm test. Lanes, worker and memory environment variables, thevmForksopt-in and abort behavior are documented in Tests. - Files:
src/**/*.test.ts,extensions/**/*.test.ts, plus e2e files fromvitest.e2e.config.ts, the web client suite underapps/web/src/**, the Control UI v2 suite underapps/control/src/**, the gateway-browser-client suite underpackages/gateway-browser-client/src/**, and the host-bridge suite underpackages/host-bridge/src/** - Scope:
- Pure unit tests
- In-process integration tests (gateway auth, routing, tooling, parsing, config)
- Channel, extension, core, gateway, and e2e smoke as part of the same command
- Deterministic regressions for known bugs
- Expectations:
- Runs locally by default (
pnpm test); no GitHub check runs it, because the PR verification workflows are archived (ADR 0118, ADR 2399) - No real keys required
- Runs locally by default (
Web client
- Command:
pnpm test:web(also runs as theweblane ofpnpm test) - Config:
apps/web/vitest.config.ts— a single jsdom project, not a browser-mode run - Files:
apps/web/src/**/*.test.ts,apps/web/src/**/*.test.tsx - Setup:
apps/web/vitest.setup.ts - Scope: the
openclaw-user-clientReact app — gateway connection and pairing/setup-code handling, the runs view (grouping/controller/model), the attachments/preferences/voice/runtime-config providers, the theme provider, the app shell, and the native component layer underapps/web/src/ui/ - Run one file on its own with
pnpm --dir apps/web exec vitest run --config vitest.config.ts <file>
Control UI v2 app
- Command:
pnpm test:control(also runs as thecontrollane ofpnpm test) - Config:
apps/control/vitest.config.ts— a single jsdom project - Files:
apps/control/src/**/*.test.ts,apps/control/src/**/*.test.tsx - Setup:
apps/control/vitest.setup.ts - Scope: the
openclaw-control-uiReact console (Control UI v2, the default operator console at the gateway root) — the runtime-config parser, the gateway, runs, fleet, approvals and config stores, the shared schema-form config engine (apps/control/src/config-form/), the shell (sign-in, sidebar, command palette, theme), and the views across the five navigation groups: Monitor (Overview, Sessions, Runs, Recovery, Usage, Logs, Debug), Configure (Agents, Automations, Channels, Skills, Settings), Fleet (Nodes, Instances, Storage), Learning (Learning, Dreams) and Plugins (pages contributed by plugins). Gateway connection behaviour comes frompackages/gateway-browser-client, faked with the fake connection and socket itstest-supportexport provides. - Type-check with
pnpm tsgo:control(part ofpnpm check); a coverage report withpnpm --dir apps/control exec vitest run --coverage(the repo-widepnpm test:coveragegate counts coresrc/only and excludesapps/**). - Run one file on its own with
pnpm --dir apps/control exec vitest run --config vitest.config.ts <file>
Gateway browser client
- Command:
pnpm test:gateway-browser-client(also runs as thegateway-browser-clientlane ofpnpm test) - Config:
packages/gateway-browser-client/vitest.config.ts— a single jsdom project - Files:
packages/gateway-browser-client/src/**/*.test.ts - Setup:
packages/gateway-browser-client/vitest.setup.ts - Scope: the private browser WebSocket client that
apps/webandapps/controlshare — the connect handshake and device-identity signing, the bounded device-credential store, and the runs controller, model and grouping. Itsgateway-browser-client/test-supportexport is the fake connection and socket both app suites import. - Run one file on its own with
pnpm --dir packages/gateway-browser-client exec vitest run --config vitest.config.ts <file>
Host bridge
- Command:
pnpm test:host-bridge(also runs as thehost-bridgelane ofpnpm test) - Config:
packages/host-bridge/vitest.config.ts— a single node project - Files:
packages/host-bridge/src/**/*.test.ts - Scope: the message contract between the web client and an app host (ADR 2242 D2) — envelope
and payload validators, snapshot revisions, host detection, posting and request timeouts. The
web client’s own suite (
apps/web/src/app/AppShell.host-bridge.test.tsx) drives the adapter. - Run one file on its own with
pnpm --dir packages/host-bridge exec vitest run --config vitest.config.ts <file>
Web client and brand e2e suites
Component tests in jsdom cannot see scrolling, fonts,color-scheme, or the frames the browser
actually sends. These suites drive the built web client or console in real Chromium against a real
gateway (harness: test/helpers/user-client-web-harness.ts; the packaging suite checks the
packaged bundle without a browser). They are part of the e2e lane and need
pnpm build first. Suites that check the RPC frames a client sends assert against
USER_CLIENT_RPC_ALLOWLIST (test/helpers/user-client-web-allowlist.ts), which derives from the
gateway’s narrow-session method tables; never hand-list methods (ADR 2428).
The unit-level guard
test/brand-tokens-contrast.test.ts (contrast ratios and color-scheme per
theme) runs in the normal pnpm test lane and needs no browser. See
Brand tokens for what each guard pins.
E2E (gateway smoke)
- Also runs as the serial e2e lane inside default
pnpm testunlessOPENCLAW_TEST_INCLUDE_E2E=0. - Command:
pnpm test:e2e - Config:
vitest.e2e.config.ts - Files:
test/**/*.e2e.test.ts,src/**/*.e2e.test.ts,extensions/**/*.e2e.test.ts - Requires
pnpm buildfirst (the gateway e2e suites spawndist/index.js). - Runtime defaults:
- Uses Vitest
forks, notvmForks: vmForks leaked thevi.mockmodule registry across files in the shared worker, so whole-lane runs failed on order-dependent pollution while every file passed in isolation. - 1 worker locally; in CI at most 2 (a quarter of the CPU count, floored at 1).
- Runs in silent mode by default to reduce console I/O overhead.
- Uses Vitest
- Useful overrides:
OPENCLAW_E2E_WORKERS=<n>to force worker count (capped at 16).OPENCLAW_E2E_VERBOSE=1to re-enable verbose console output.
- Scope:
- Multi-instance gateway end-to-end behavior
- WebSocket/HTTP surfaces, node pairing, and heavier networking
- Expectations:
- Runs locally only; the CI pipeline that ran it is archived (ADR 0118, ADR 2399)
- No real keys required
- More moving parts than unit tests (can be slower)
Live (real providers + real models)
- Command:
pnpm test:live - Config:
vitest.live.config.ts - Files:
src/**/*.live.test.ts - Default: enabled by
pnpm test:live(setsOPENCLAW_LIVE_TEST=1) - Scope:
- “Does this provider/model actually work today with real creds?”
- Catch provider format changes, tool-calling quirks, auth issues, and rate limit behavior
- Expectations:
- Not CI-stable by design (real networks, real provider policies, quotas, outages)
- Costs money / uses rate limits
- Prefer running narrowed subsets instead of “everything”
- Live runs will source
~/.profileto pick up missing API keys
- API key rotation (provider-specific): set
*_API_KEYSwith comma/semicolon format or*_API_KEY_1,*_API_KEY_2(for exampleOPENAI_API_KEYS,ANTHROPIC_API_KEYS,GEMINI_API_KEYS) or per-live override viaOPENCLAW_LIVE_*_KEY; tests retry on rate limit responses.
Live: Android node capability sweep
- Test:
src/gateway/android-node.capabilities.live.test.ts - Script:
pnpm android:test:integration - Goal: invoke every command currently advertised by a connected Android node and assert command contract behavior.
- Scope:
- Preconditioned/manual setup (the suite does not install/run/pair the app).
- Command-by-command gateway
node.invokevalidation for the selected Android node.
- Required pre-setup:
- Android app already connected + paired to the gateway.
- App kept in foreground.
- Permissions/capture consent granted for capabilities you expect to pass.
- Optional target overrides:
OPENCLAW_ANDROID_NODE_IDorOPENCLAW_ANDROID_NODE_NAME.OPENCLAW_ANDROID_GATEWAY_URL/OPENCLAW_ANDROID_GATEWAY_TOKEN/OPENCLAW_ANDROID_GATEWAY_PASSWORD.
- Full Android setup details: Android App
Live: web client deploy verification
- Script:
node scripts/verify-client-chat-v3-p2-live.mjs --live(network-free contract test:scripts/verify-client-chat-v3-p2-live.test.mjs) - Goal: verify a deployed gateway plus its web client against the client-chat-v3-p2 contract end to end: assistant picker and chats, model/Think/reasoning controls, queue and steer, rename and Suggest, reply quotes, history paging and search, the active-leaf guard, narrow-scope patch refusals, assistant colours, and the delivery-mirror dedupe.
- Run against a real, running gateway; it costs real model calls:
OPENCLAW_GATEWAY_TOKENis required (take it from the gateway unit env; the script never prints it).OPENCLAW_GATEWAY_URLoverrides the defaultws://127.0.0.1:18789.- A headless Chromium pairs a throwaway
/v2/clientdevice (PLAYWRIGHT_BROWSERS_PATHoverrides~/.cache/ms-playwright).
- Flags:
--only SC7,SC8runs a subset (pairing and throwaway chats still happen, unasserted).--fix-proof --fix-proof-session <key>sends three real probe messages through that routed direct chat (required; the script never picks one) and asserts noactive-leaf-changedrefusals, no duplicate mirror rows and a leaf that rests on each turn’s own last message.--keepskips cleanup.
- Safety and cleanup: throwaway sessions are labelled with the run id and deleted on exit, the paired device is removed, and the fix-proof target chat is written but never deleted. SIGINT/SIGTERM/SIGHUP run the same cleanup and exit 130. Operator direct chats other than the fix-proof target are never touched.
- Exit codes:
0all selected checks pass,1any check fails or cleanup could not remove a resource (CLEANUP FAIL),2usage error (missing--liveor token, unknown check id,--fix-proofwithout--fix-proof-session) or preflight failure. - Output: one
=== SC<id> ===evidence block per check ending inSC<id> PASS, thenVERIFY PASS (<n> checks)orVERIFY FAIL: <ids>.
Live: model smoke (profile keys)
Live tests are split into two layers so we can isolate failures:- “Direct model” tells us the provider/model can answer at all with the given key.
- “Gateway smoke” tells us the full gateway+agent pipeline works for that model (sessions, history, tools, sandbox policy, etc.).
Layer 1: Direct model completion (no gateway)
- Test:
src/agents/models.profiles.live.test.ts - Goal:
- Enumerate discovered models
- Use
getApiKeyForModelto select models you have creds for - Run a small completion per model (and targeted regressions where needed)
- How to enable:
pnpm test:live(orOPENCLAW_LIVE_TEST=1if invoking Vitest directly)
- Set
OPENCLAW_LIVE_MODELS=modern(orall, alias for modern) to actually run this suite; otherwise it skips to keeppnpm test:livefocused on gateway smoke - How to select models:
OPENCLAW_LIVE_MODELS=modernto run the modern allowlist (Opus/Sonnet/Haiku 4.5, GPT-5.x + Codex, Gemini 3, GLM 4.7, MiniMax M2.5, Grok 4)OPENCLAW_LIVE_MODELS=allis an alias for the modern allowlist- or
OPENCLAW_LIVE_MODELS="openai/gpt-5.2,anthropic/claude-opus-4-6,..."(comma allowlist)
- How to select providers:
OPENCLAW_LIVE_PROVIDERS="google,google-antigravity,google-gemini-cli"(comma allowlist)
- Where keys come from:
- By default: profile store and env fallbacks
- Set
OPENCLAW_LIVE_REQUIRE_PROFILE_KEYS=1to enforce profile store only
- Why this exists:
- Separates “provider API is broken / key is invalid” from “gateway agent pipeline is broken”
- Contains small, isolated regressions (example: OpenAI Responses/Codex Responses reasoning replay + tool-call flows)
Layer 2: Gateway + dev agent smoke (what “@openclaw” actually does)
- Test:
src/gateway/gateway-models.profiles.live.test.ts - Goal:
- Spin up an in-process gateway
- Create/patch a
agent:dev:*session (model override per run) - Iterate models-with-keys and assert:
- “meaningful” response (no tools)
- a real tool invocation works (read probe)
- optional extra tool probes (exec+read probe)
- OpenAI regression paths (tool-call-only → follow-up) keep working
- Probe details (so you can explain failures quickly):
readprobe: the test writes a nonce file in the workspace and asks the agent toreadit and echo the nonce back.exec+readprobe: the test asks the agent toexec-write a nonce into a temp file, thenreadit back.- image probe: the test attaches a generated PNG (cat + randomized code) and expects the model to return
cat <CODE>. - Implementation reference:
src/gateway/gateway-models.profiles.live.test.tsandsrc/gateway/live-image-probe.ts.
- How to enable:
pnpm test:live(orOPENCLAW_LIVE_TEST=1if invoking Vitest directly)
- How to select models:
- Default: modern allowlist (Opus/Sonnet/Haiku 4.5, GPT-5.x + Codex, Gemini 3, GLM 4.7, MiniMax M2.5, Grok 4)
OPENCLAW_LIVE_GATEWAY_MODELS=allis an alias for the modern allowlist- Or set
OPENCLAW_LIVE_GATEWAY_MODELS="provider/model"(or comma list) to narrow
- How to select providers (avoid “OpenRouter everything”):
OPENCLAW_LIVE_GATEWAY_PROVIDERS="google,google-antigravity,google-gemini-cli,openai,anthropic,zai,minimax"(comma allowlist)
- Tool + image probes are always on in this live test:
readprobe +exec+readprobe (tool stress)- image probe runs when the model advertises image input support
- Flow (high level):
- Test generates a tiny PNG with “CAT” + random code (
src/gateway/live-image-probe.ts) - Sends it via
agentattachments: [{ mimeType: "image/png", content: "<base64>" }] - Gateway parses attachments into
images[](src/gateway/server-methods/agent.ts+src/gateway/chat-attachments.ts) - Embedded agent forwards a multimodal user message to the model
- Assertion: reply contains
cat+ the code (OCR tolerance: minor mistakes allowed)
- Test generates a tiny PNG with “CAT” + random code (
provider/model ids), run:
Live: Anthropic setup-token smoke
- Test:
src/agents/anthropic.setup-token.live.test.ts - Goal: verify Claude Code CLI setup-token (or a pasted setup-token profile) can complete an Anthropic prompt.
- Enable:
pnpm test:live(orOPENCLAW_LIVE_TEST=1if invoking Vitest directly)OPENCLAW_LIVE_SETUP_TOKEN=1
- Token sources (pick one):
- Profile:
OPENCLAW_LIVE_SETUP_TOKEN_PROFILE=anthropic:setup-token-test - Raw token:
OPENCLAW_LIVE_SETUP_TOKEN_VALUE=sk-ant-oat01-...
- Profile:
- Model override (optional):
OPENCLAW_LIVE_SETUP_TOKEN_MODEL=anthropic/claude-opus-4-6
Live: CLI backend smoke (Claude Code CLI or other local CLIs)
- Test:
src/gateway/gateway-cli-backend.live.test.ts - Goal: validate the Gateway + agent pipeline using a local CLI backend, without touching your default config.
- Enable:
pnpm test:live(orOPENCLAW_LIVE_TEST=1if invoking Vitest directly)OPENCLAW_LIVE_CLI_BACKEND=1
- Defaults:
- Model:
claude-cli/claude-sonnet-4-6 - Command:
claude - Args:
["-p","--output-format","json","--dangerously-skip-permissions"]
- Model:
- Overrides (optional):
OPENCLAW_LIVE_CLI_BACKEND_MODEL="claude-cli/claude-opus-4-6"OPENCLAW_LIVE_CLI_BACKEND_MODEL="codex-cli/gpt-5.3-codex"OPENCLAW_LIVE_CLI_BACKEND_COMMAND="/full/path/to/claude"OPENCLAW_LIVE_CLI_BACKEND_ARGS='["-p","--output-format","json","--permission-mode","bypassPermissions"]'OPENCLAW_LIVE_CLI_BACKEND_CLEAR_ENV='["ANTHROPIC_API_KEY","ANTHROPIC_API_KEY_OLD"]'OPENCLAW_LIVE_CLI_BACKEND_IMAGE_PROBE=1to send a real image attachment (paths are injected into the prompt).OPENCLAW_LIVE_CLI_BACKEND_IMAGE_ARG="--image"to pass image file paths as CLI args instead of prompt injection.OPENCLAW_LIVE_CLI_BACKEND_IMAGE_MODE="repeat"(or"list") to control how image args are passed whenIMAGE_ARGis set.OPENCLAW_LIVE_CLI_BACKEND_RESUME_PROBE=1to send a second turn and validate resume flow.
OPENCLAW_LIVE_CLI_BACKEND_DISABLE_MCP_CONFIG=0to keep Claude Code CLI MCP config enabled (default disables MCP config with a temporary empty file).
Recommended live recipes
Narrow, explicit allowlists are fastest and least flaky:-
Single model, direct (no gateway):
OPENCLAW_LIVE_MODELS="openai/gpt-5.2" pnpm test:live src/agents/models.profiles.live.test.ts
-
Single model, gateway smoke:
OPENCLAW_LIVE_GATEWAY_MODELS="openai/gpt-5.2" pnpm test:live src/gateway/gateway-models.profiles.live.test.ts
-
Tool calling across several providers:
OPENCLAW_LIVE_GATEWAY_MODELS="openai/gpt-5.2,anthropic/claude-opus-4-6,google/gemini-3-flash-preview,zai/glm-4.7,minimax/minimax-m2.5" pnpm test:live src/gateway/gateway-models.profiles.live.test.ts
-
Google focus (Gemini API key + Antigravity):
- Gemini (API key):
OPENCLAW_LIVE_GATEWAY_MODELS="google/gemini-3-flash-preview" pnpm test:live src/gateway/gateway-models.profiles.live.test.ts - Antigravity (OAuth):
OPENCLAW_LIVE_GATEWAY_MODELS="google-antigravity/claude-opus-4-6-thinking,google-antigravity/gemini-3-pro-high" pnpm test:live src/gateway/gateway-models.profiles.live.test.ts
- Gemini (API key):
google/...uses the Gemini API (API key).google-antigravity/...uses the Antigravity OAuth bridge (Cloud Code Assist-style agent endpoint).google-gemini-cli/...uses the local Gemini CLI on your machine (separate auth + tooling quirks).- Gemini API vs Gemini CLI:
- API: WednesdayAI calls Google’s hosted Gemini API over HTTP (API key / profile auth); this is what most users mean by “Gemini”.
- CLI: WednesdayAI shells out to a local
geminibinary; it has its own auth and can behave differently (streaming/tool support/version skew).
Live: model matrix (what we cover)
There is no fixed “CI model list” (live is opt-in), but these are the recommended models to cover regularly on a dev machine with keys.Modern smoke set (tool calling + image)
This is the “common models” run we expect to keep working:- OpenAI (non-Codex):
openai/gpt-5.2(optional:openai/gpt-5.1) - OpenAI Codex:
openai-codex/gpt-5.3-codex(optional:openai-codex/gpt-5.3-codex-codex) - Anthropic:
anthropic/claude-opus-4-6(oranthropic/claude-sonnet-4-5) - Google (Gemini API):
google/gemini-3-pro-previewandgoogle/gemini-3-flash-preview(avoid older Gemini 2.x models) - Google (Antigravity):
google-antigravity/claude-opus-4-6-thinkingandgoogle-antigravity/gemini-3-flash - Z.AI (GLM):
zai/glm-4.7 - MiniMax:
minimax/minimax-m2.5
OPENCLAW_LIVE_GATEWAY_MODELS="openai/gpt-5.2,openai-codex/gpt-5.3-codex,anthropic/claude-opus-4-6,google/gemini-3-pro-preview,google/gemini-3-flash-preview,google-antigravity/claude-opus-4-6-thinking,google-antigravity/gemini-3-flash,zai/glm-4.7,minimax/minimax-m2.5" pnpm test:live src/gateway/gateway-models.profiles.live.test.ts
Baseline: tool calling (Read + optional Exec)
Pick at least one per provider family:- OpenAI:
openai/gpt-5.2(oropenai/gpt-5-mini) - Anthropic:
anthropic/claude-opus-4-6(oranthropic/claude-sonnet-4-5) - Google:
google/gemini-3-flash-preview(orgoogle/gemini-3-pro-preview) - Z.AI (GLM):
zai/glm-4.7 - MiniMax:
minimax/minimax-m2.5
- xAI:
xai/grok-4(or latest available) - Mistral:
mistral/… (pick one “tools” capable model you have enabled) - Cerebras:
cerebras/… (if you have access) - LM Studio:
lmstudio/… (local; tool calling depends on API mode)
Vision: image send (attachment → multimodal message)
Include at least one image-capable model inOPENCLAW_LIVE_GATEWAY_MODELS (Claude/Gemini/OpenAI vision-capable variants, etc.) to exercise the image probe.
Aggregators / alternate gateways
If you have keys enabled, we also support testing via:- OpenRouter:
openrouter/...(hundreds of models; usewednesdayai models scanto find tool+image capable candidates) - OpenCode Zen:
opencode/...(auth viaOPENCODE_API_KEY/OPENCODE_ZEN_API_KEY)
- Built-in:
openai,openai-codex,anthropic,google,google-vertex,google-antigravity,google-gemini-cli,zai,openrouter,opencode,xai,groq,cerebras,mistral,github-copilot - Via
models.providers(custom endpoints):minimax(cloud/API), plus any OpenAI/Anthropic-compatible proxy (LM Studio, vLLM, LiteLLM, etc.)
discoverModels(...) returns on your machine + whatever keys are available.
Credentials (never commit)
Live tests discover credentials the same way the CLI does. Practical implications:- If the CLI works, live tests should find the same keys.
-
If a live test says “no creds”, debug the same way you’d debug
wednesdayai models list/ model selection. -
Profile store:
~/.openclaw/credentials/(preferred; what “profile keys” means in the tests) -
Config:
~/.openclaw/openclaw.json(orOPENCLAW_CONFIG_PATH)
~/.profile), run local tests after source ~/.profile, or use the Docker runners below (they can mount ~/.profile into the container).
Deepgram live (audio transcription)
- Test:
extensions/deepgram/src/audio.live.test.ts - Enable:
DEEPGRAM_API_KEY=... DEEPGRAM_LIVE_TEST=1 pnpm test:live extensions/deepgram/src/audio.live.test.ts
BytePlus coding plan live
- Test:
src/agents/byteplus.live.test.ts - Enable:
BYTEPLUS_API_KEY=... BYTEPLUS_LIVE_TEST=1 pnpm test:live src/agents/byteplus.live.test.ts - Optional model override:
BYTEPLUS_CODING_MODEL=ark-code-latest
Live: voice loop (real speech)
- Test:
src/talk/voice-loop.live.test.tsand scriptscripts/voice-live-smoke.mjs(both drive the samesrc/talk/voice-loop-driver.tsagainst a deployed gateway; the script is the operator-invoked post-deploy gate). - Goal: stream real-speech turns (ElevenLabs-synthesized audio, one of them a mid-thought pause)
plus interruptions through real ElevenLabs STT/TTS, then assert the pinned
SC3/SC7/SC12/SC16talk.metricsbounds plusturnShape(each normal utterance starts exactly one turn and is never cancelled; the paused turn instead ends exactly once, with anyturn.cancelledreasonturn-resumed) andSC4(one assistant reply per turn inchat.history, no failure reply persisted, the continuation half of the paused utterance kept). - Enable (test):
OPENCLAW_GATEWAY_URL=... ELEVENLABS_API_KEY=... LIVE=1 pnpm test:live src/talk/voice-loop.live.test.ts(20 turns, 5 interruptions). The file runs only undervitest.live.config.ts, never in the default lanes.OPENCLAW_VOICE_LIVE_STT_PROVIDER/OPENCLAW_VOICE_LIVE_TTS_PROVIDERoverride the provider pair (defaultelevenlabson both). - Operator command:
PASS or FAIL plus a JSON summary whose checks object carries sc3, sc7,
sc12, sc16, and, in speech mode, turnShape and sc4. --stt/--tts select the provider
pair; --silent-audio streams silence instead and is only meaningful against a fake-provider
gateway. Speech synthesis runs client-side and needs ELEVENLABS_API_KEY (or XI_API_KEY);
ELEVENLABS_BASE_URL overrides the API origin for a region-locked key. The driver sends one
100 ms audio chunk every 100 ms for the whole session (silence while it waits on a reply), so run
it from a host under 100 ms RTT to the gateway.
Docker runners (optional “works in Linux” checks)
These runpnpm test:live inside the repo Docker image, mounting your local config dir and workspace (and sourcing ~/.profile if mounted):
- Direct models:
pnpm test:docker:live-models(script:scripts/test-live-models-docker.sh) - Gateway + dev agent:
pnpm test:docker:live-gateway(script:scripts/test-live-gateway-models-docker.sh) - Onboarding wizard (TTY, full scaffolding):
pnpm test:docker:onboard(script:scripts/e2e/onboard-docker.sh) - Gateway networking (two containers, WS auth + health):
pnpm test:docker:gateway-network(script:scripts/e2e/gateway-network-docker.sh) - Plugins (custom extension load + registry smoke):
pnpm test:docker:plugins(script:scripts/e2e/plugins-docker.sh)
bun scripts/dev/discord-acp-plain-language-smoke.ts --channel <discord-channel-id> ...- Keep this script for regression/debug workflows. It may be needed again for ACP thread routing validation, so do not delete it.
OPENCLAW_CONFIG_DIR=...(default:~/.openclaw) mounted to/home/node/.openclawOPENCLAW_WORKSPACE_DIR=...(default:~/.openclaw/workspace) mounted to/home/node/.openclaw/workspaceOPENCLAW_PROFILE_FILE=...(default:~/.profile) mounted to/home/node/.profileand sourced before running testsOPENCLAW_LIVE_GATEWAY_MODELS=.../OPENCLAW_LIVE_MODELS=...to narrow the runOPENCLAW_LIVE_REQUIRE_PROFILE_KEYS=1to ensure creds come from the profile store (not env)
Docs sanity
Run docs checks after doc edits:pnpm docs:list.
Offline regression (CI-safe)
These are “real pipeline” regressions without real providers:- Gateway tool calling (mock OpenAI, real gateway + agent loop):
src/gateway/gateway.test.ts(case: “runs a mock OpenAI tool call end-to-end via gateway agent loop”) - Gateway wizard (WS
wizard.start/wizard.next, writes config + auth enforced):src/gateway/gateway.test.ts(case: “runs wizard over ws and writes auth token config”)
Agent reliability evals (skills)
We already have a few CI-safe tests that behave like “agent reliability evals”:- Mock tool-calling through the real gateway + agent loop (
src/gateway/gateway.test.ts). - End-to-end wizard flows that validate session wiring and config effects (
src/gateway/gateway.test.ts).
- Decisioning: when skills are listed in the prompt, does the agent pick the right skill (or avoid irrelevant ones)?
- Compliance: does the agent read
SKILL.mdbefore use and follow required steps/args? - Workflow contracts: multi-turn scenarios that assert tool order, session history carryover, and sandbox boundaries.
- A scenario runner using mock providers to assert tool calls + order, skill file reads, and session wiring.
- A small suite of skill-focused scenarios (use vs avoid, gating, prompt injection).
- Optional live evals (opt-in, env-gated) only after the CI-safe suite is in place.