Measuring performance
This guide covers how to measure. Performance budgets are defined by code path name inROADMAP.md in the repository. Measure against those budgets and do not copy numbers into other documents.
Model latency bench (local keys)
Script:scripts/bench-model.ts
Usage:
source ~/.profile && pnpm tsx scripts/bench-model.ts --runs 10- Optional environment variables:
MINIMAX_API_KEY,MINIMAX_BASE_URL,MINIMAX_MODEL,ANTHROPIC_API_KEY - Default prompt: “Reply with a single word: ok. No punctuation or extra text.”
CLI startup bench
Script:scripts/bench-cli-startup.ts
Usage:
pnpm tsx scripts/bench-cli-startup.tspnpm tsx scripts/bench-cli-startup.ts --runs 12pnpm tsx scripts/bench-cli-startup.ts --entry dist/entry.js --timeout-ms 45000
--version--helphealth --jsonstatus --jsonstatus
pnpm build first when you benchmark dist/entry.js.
Runtime budgets
Config:config/perf-budgets.json · Runner: scripts/perf/ (ADR 0040, ADR 1967)
Usage:
--runs N to change the number of sequential boot samples (default 5) and --measure-baselines to write the measured values back into config/perf-budgets.json.
check:perf is a before-PR command, not part of pnpm check: it needs the prerequisite build above and boots the built gateway.
What it measures per run (--runs defaults to 5, sequential):
- Gateway cold start: spawn
dist/entry.js gateway runto the first HTTP 200 from/ready(not/health, which answers as soon as the listener opens). Reports p50 and p95. - Idle RSS: sampled 2 s after
/readyanswers. - Package size: tarball,
dist/andextensions/bytes fromnpm pack --dry-run --json --ignore-scripts.
HOME, OPENCLAW_HOME, OPENCLAW_STATE_DIR, OPENCLAW_CONFIG_PATH and its cwd, and OPENCLAW_NO_RESPAWN=1 so the measured PID is the gateway itself. Inherited credentials (*_TOKEN, *_SECRET, *_PASSWORD, *_API_KEY, channel identity variables) are stripped, so no live channel or provider of yours starts and baselines do not depend on the host’s shell. Channels and plugins load normally — their boot cost is part of the budget. Every exit path stops the child and removes the temp dir.
Exit codes:
0— noblock-mode budget with a measured baseline failed. Advisory outcomes still exit zero: awarn-mode budget that is over target or regressed beyond its baseline tolerance prints a::warning::annotation, and ablock-mode budget onbaseline: null(unmeasured is unproven) never fails regardless of status.1— only ablock-mode budget with a measured baseline failed (over target or regressed). New budgets start inwarn.2— a prerequisite is missing (dist/entry.js,dist/web-client,dist/control-ui-v2, or an unsupported platform), the arguments are invalid (unknown flag, or--runsthat is not a positive integer), or an adapter or the baseline-recording step errored.130/143— interrupted by SIGINT / SIGTERM; the measured gateway is stopped and the temp dir cleaned up before exit.
--measure-baselines rewrites config/perf-budgets.json with the measured values and writes a dated report under dev-docs/workstreams/platform-budgets-gates/diagnostics/. Targets and baselines live in the config — do not copy numbers into docs.
Related
- Tests
- Testing policy: complexity guards in tests use input-scaling ratios, not absolute time bounds
- Coding style