Skip to main content

Deep Observability Developer Guide

Deep observability is split between app-side contracts and stack-side storage. Application code emits bounded events, metrics, spans, logs, and capture results. The /srv monitoring stack stores, dashboards, profiles, alerts, and reports on those signals. Do not add a second metrics export path for this feature. App metrics flow through the existing OpenTelemetry meter in diagnostics-otel.

Boundary event contract

Use the diagnostic event bus for app boundary attribution. The public shape is defined by DiagnosticBoundaryEvent:
The current production instrumentation emits plugin hook boundary events from src/plugins/hooks.ts. The broader kinds are part of the shared event contract and should be instrumented in separate, source-specific slices.

Emit plugin boundary events

Boundary events should wrap the real production dispatch path, not only an isolated helper. Use low-cardinality fields for metric labels, and leave raw run, session, user, or chat identifiers to traces/logs where policy can drop or redact them. Correct pattern:
What not to do:
That turns high-cardinality identifiers into metric labels after export. Use component.id for the bounded plugin or component name only.

Runtime metrics emitted by diagnostics-otel

When diagnostics.otel.runtime.enabled is true, extensions/diagnostics-otel/src/runtime-metrics.ts registers these instruments: The collector registers callbacks only while enabled and removes callbacks on stop. Keep new runtime metrics behind the same diagnostics.otel.runtime.enabled gate unless they are already part of a cheaper always-on signal.

Attribute policy

extensions/diagnostics-otel/src/attribute-policy.ts bounds metric and trace/log attributes. The policy exists to protect Prometheus, Loki, Tempo, and Langfuse from cardinality blowups and sensitive values. Use semantic convention names where they exist, such as nodejs.*, v8js.*, process.*, system.*, and gen_ai.*. Use wednesdayai.* or the existing openclaw.* namespace for WednesdayAI-specific concepts such as plugin hooks, routing, recovery, and queue lanes. Never use these as metric labels:
  • runId
  • sessionKey
  • userId
  • chatId
  • prompt text
  • tool payloads
  • file contents
  • credentials

Gateway diagnostics methods

The gateway handlers live in src/gateway/server-methods/diagnostics.ts and are registered in the core gateway method map. Do not present diagnostics.profiling.start or diagnostics.profiling.stop as active Pyroscope collection. Real Pyroscope SDK or eBPF scheduling is monitoring-stack follow-up work.

Release performance sweep

scripts/release-performance-sweep.ts supports dry and live modes. Dry mode returns fixture data. Live mode uses the gateway client path and requires diagnostics.runtime.snapshot; --capture also requires diagnostics.capture.cpu and diagnostics.capture.report. The sweep writes release-performance-sweep.json and release-performance-sweep.md when --output-dir is set. Keep the output safe for operator review: no raw prompts, credentials, chat IDs, session keys, or user identifiers.

Async safety

Plugin register functions may be synchronous or async, but they run during gateway startup. Do not perform synchronous blocking I/O in register(). Do not do this:
Do this instead:
Hook and tool handlers run on the shared Node.js event loop. A slow hook delays the user-visible turn, so emit boundary timing around the actual hook work and move long-running work out of the synchronous startup path.