Skip to main content

Deep observability runtime metrics

The diagnostics-otel extension ships a runtime metrics collector (extensions/diagnostics-otel/src/runtime-metrics.ts) that exports Node.js process health — event loop, garbage collection, V8 heap, CPU, and process counters — as standard OTel metrics alongside the application metrics. This page documents that metric surface so you can build dashboards, alerts, and recording rules against it without reading the collector source. The collector is config-driven, not part of the plugin SDK: operators enable it with diagnostics.otel.runtime (see Deep observability). As a plugin author you do not interact with it directly — you consume its output in your OTel backend, or you follow the same patterns if you ship your own collector.

Contract

The runtime config shape (RuntimeMetricsConfig) and the collected snapshot:
Collection semantics:
  • A setInterval timer (default 15 s) takes a fresh snapshot; every observable gauge reports the latest snapshot when scraped.
  • Per-window deltas (CPU time, page faults, fs reads/writes) are computed against the previous snapshot, not process start.
  • Event loop delay p99 is measured with monitorEventLoopDelay({ resolution: 20 }), reset each window.
  • GC pauses arrive via a PerformanceObserver on gc entries — they are event-driven, not windowed.
  • stop() fully unwinds everything it started; there is no background work left behind.

Metric surface

Prometheus converts dots to underscores: nodejs.eventloop.utilization → nodejs_eventloop_utilization. The /proc-backed metrics silently report nothing on non-Linux platforms — dashboards should not alert on their absence alone.

Alerting baselines

Start with warning-level alerts and tune during a burn-in period (from the deep observability stack guidance):
  • Event loop utilization or delay p99 sustained above target for 5 minutes.
  • GC pause p99 spikes or a rising major-GC rate.
  • RSS growth slope sustained over 30–60 minutes; heap used approaching v8js_memory_heap_limit.
  • File descriptor count trending upward without a corresponding session load change.
Keep recording-rule outputs bounded — precompute wednesdayai:event_loop_utilization:avg5m-style rules rather than repeating raw quantile expressions in every panel.

What not to do

Do not build runtime dashboards or alerts on high-cardinality labels. The extension’s metric attribute policy (extensions/diagnostics-otel/src/attribute-policy.ts) is an allowlist: raw IDs (runId, sessionKey, chatId, userId, accountId, messageId) are dropped before metrics are recorded, and only bounded names (provider, model, channel, status, tool.name, …) pass through. If you add your own metrics via the plugin SDK’s meter, apply the same rule:
Also avoid correlating these process metrics with per-conversation data in the same panel — use the trace tree (OTel and Langfuse diagnostics) for per-run attribution and keep runtime metrics at the process level.