Deep Observability Developer Guide
Deep observability is split between app-side contracts and stack-side storage. Application code emits bounded events, metrics, spans, logs, and capture results. The/srv monitoring stack stores, dashboards, profiles, alerts, and reports on those signals.
Do not add a second metrics export path for this feature. App metrics flow through the existing OpenTelemetry meter in diagnostics-otel.
Boundary event contract
Use the diagnostic event bus for app boundary attribution. The public shape is defined byDiagnosticBoundaryEvent:
src/plugins/hooks.ts. The broader kinds are part of the shared event contract and should be instrumented in separate, source-specific slices.
Emit plugin boundary events
Boundary events should wrap the real production dispatch path, not only an isolated helper. Use low-cardinality fields for metric labels, and leave raw run, session, user, or chat identifiers to traces/logs where policy can drop or redact them. Correct pattern:component.id for the bounded plugin or component name only.
Runtime metrics emitted by diagnostics-otel
Whendiagnostics.otel.runtime.enabled is true, extensions/diagnostics-otel/src/runtime-metrics.ts registers these instruments:
The collector registers callbacks only while enabled and removes callbacks on stop. Keep new runtime metrics behind the same
diagnostics.otel.runtime.enabled gate unless they are already part of a cheaper always-on signal.
Attribute policy
extensions/diagnostics-otel/src/attribute-policy.ts bounds metric and trace/log attributes. The policy exists to protect Prometheus, Loki, Tempo, and Langfuse from cardinality blowups and sensitive values.
Use semantic convention names where they exist, such as nodejs.*, v8js.*, process.*, system.*, and gen_ai.*. Use wednesdayai.* or the existing openclaw.* namespace for WednesdayAI-specific concepts such as plugin hooks, routing, recovery, and queue lanes.
Never use these as metric labels:
runIdsessionKeyuserIdchatId- prompt text
- tool payloads
- file contents
- credentials
Gateway diagnostics methods
The gateway handlers live insrc/gateway/server-methods/diagnostics.ts and are registered in the core gateway method map.
Do not present
diagnostics.profiling.start or diagnostics.profiling.stop as active Pyroscope collection. Real Pyroscope SDK or eBPF scheduling is monitoring-stack follow-up work.
Release performance sweep
scripts/release-performance-sweep.ts supports dry and live modes. Dry mode returns fixture data. Live mode uses the gateway client path and requires diagnostics.runtime.snapshot; --capture also requires diagnostics.capture.cpu and diagnostics.capture.report.
The sweep writes release-performance-sweep.json and release-performance-sweep.md when --output-dir is set. Keep the output safe for operator review: no raw prompts, credentials, chat IDs, session keys, or user identifiers.
Async safety
Pluginregister functions may be synchronous or async, but they run during gateway startup. Do not perform synchronous blocking I/O in register().
Do not do this: