Deep Observability Stack Handoff
This page is the repository-owned handoff for the/srv/shared implementation work. It does not edit /srv/shared; it names the expected app signals and the stack changes needed to receive, store, query, and report on them.
For operator setup, see Deep Observability. For developer contracts, see Deep Observability Developer Guide. For dashboard, alert, recording-rule, and daily/weekly report design, see Deep Observability Dashboards and Reports.
WednesdayAI app emits
The application side now has three signal classes:- OTLP traces, metrics, and optional logs from
diagnostics-otel, including trace IDs on exported log records for trace-to-log correlation. - Low-overhead runtime metrics from the OTel meter when
diagnostics.otel.runtime.enabledis true: event loop utilization, event loop delay p99, GC duration, process memory, CPU time, uptime, V8 heap/space metrics, resource usage snapshots and per-window deltas, Linux file-descriptor count, and/proc/self/iocounters. - Boundary events for plugin hooks, exported through OTel metrics/traces with bounded labels such as
boundary.kind,openclaw.operation,openclaw.component.id,hook.name, and status. - The primary session write interceptor was audited and left behaviorally unchanged: injected tracers are used when provided, and the module-level OpenTelemetry fallback is intentionally inert when no provider is installed.
- Authenticated gateway capture methods:
diagnostics.runtime.snapshotfor read-only process memory, CPU, uptime, and resource usage.diagnostics.capture.cpufor bounded CPU deltas over a requested window.diagnostics.capture.heapfor V8 heap snapshots under the configured capture directory.diagnostics.capture.reportfor an allow-listed Node diagnostic report with environment variables, command line, stacks, module paths, shared object paths, prompt text, credentials, and config omitted.diagnostics.profiling.startanddiagnostics.profiling.stopas the future Pyroscope adapter seam.
runId, sessionKey, userId, chatId, prompt text, tool payloads, credentials, and file contents belong in traces/logs only when explicitly gated and redacted.
/srv/shared changes
/srv/shared/docker-compose.yml
Add or update services and mounts:
- Add a Pyroscope service on the shared network with persistent storage under
/data/docker-volumes/pyroscopeor another retained bind mount. - Add a Pyroscope Grafana datasource dependency so Grafana starts after the profile store is reachable.
- Add optional Grafana Alloy or Beyla for eBPF profiling and HTTP/gRPC RED discovery. Keep this optional because eBPF needs host privileges and should be deployable as a windowed diagnostic tool.
- Add OTel collector host mounts if using the collector
hostmetricsreceiver:/proc:/hostfs/proc:ro/sys:/hostfs/sys:ro/:/hostfs:ro
- Add Promtail journald access for
openclaw-gateway.service:/var/log/journal:/var/log/journal:ro/run/log/journal:/run/log/journal:ro/etc/machine-id:/etc/machine-id:ro
- Keep existing Docker/cAdvisor/node-exporter coverage. It is useful for host/container baseline, but it does not identify the user-level gateway process or child agent PIDs by itself.
/srv/shared/otelcol/config.yaml
Keep the existing OTLP receiver and Tempo/Prometheus/Loki exporters. Add a hostmetrics receiver for process-over-time visibility:
hostmetrics/wednesdayai to the metrics pipeline. If the collector version uses different process scraper names, keep the intent: process scraper coverage for openclaw-gateway.service, the main wednesdayai/openclaw Node process, and child agent processes.
Use resource labels such as service.namespace=wednesdayai and low-cardinality process labels. Do not label metrics by session or run.
/srv/shared/config/promtail-config.yml
Keep the Docker scrape. Add a journald scrape for the Linux user service:
/run/log/journal, mount that path and configure the matching journal path. The goal is Loki coverage for the real systemd user service, not only Docker containers.
/srv/shared/grafana/provisioning/datasources/*.yml
Add a Pyroscope datasource:
/srv/shared/config/prometheus.yml
If OTel remote write remains the only metrics path, no new scrape is needed for app runtime metrics. Add scrape entries only for new services that expose their own Prometheus endpoints, for example Pyroscope, Alloy, Beyla, or process-exporter if that path is chosen instead of OTel hostmetrics.
Recommended optional process-exporter fallback:
hostmetrics process scraper or process-exporter as the baseline process-over-time source. Do not run both unless label names are intentionally separated to avoid duplicate panels and alerts.
/srv/shared/config/alert_rules.yml
Add WednesdayAI-specific rules once the metrics land:
- Gateway process missing for more than two minutes.
- RSS growth slope sustained over 30-60 minutes.
- Heap used near V8 heap limit.
- Event loop utilization or delay sustained above the dashboard threshold.
- GC pause p99 sustained high.
- Process disk I/O write/read spikes.
- Capture enabled, or Pyroscope profiling configured by the stack, in
alwaysmode longer than the intended window if operators want guardrails. - OTel collector dropped spans, metrics, or logs.
- CPU rate by process name and host.
- RSS and heap by process name and host.
- Disk read/write bytes by process name and host.
- Plugin hook duration by
openclaw.component.id,openclaw.operation, and status. - Model token usage by provider/model and status.
Profiling operating model
Use the same mode names asdiagnostics.capture.mode for the current app-side capture gate. Until a Pyroscope SDK or eBPF adapter is added, diagnostics.profiling.start and diagnostics.profiling.stop are gateway control seams only; real continuous or scheduled profiling is owned by the /srv stack change.
For initial deployment,
always is reasonable while the system is being characterized. Once stable, switch to manual or schedule and keep runtime/process metrics on 24x7.
Verification
Run these from/srv/shared after implementing the stack changes:
- Prometheus has runtime metrics such as
process_memory_usage,process_cpu_time, andv8js_gc_duration. - Loki can query
{job="openclaw-gateway"}or{unit="openclaw-gateway.service"}. - Tempo traces link to Loki logs by
trace_idortraceId. - Pyroscope is queryable before enabling
alwaysmode.