OTel and Langfuse diagnostics
WednesdayAI exports traces, metrics, and logs via the OpenTelemetry Protocol (OTLP). Thediagnostics-otel extension wires core run, model-call, and tool events into a nested OTel span tree. Langfuse is supported as a first-class backend via a convenience config block that derives the ingest endpoint and Basic-auth header automatically.
All configuration lives under diagnostics.otel in ~/.openclaw/openclaw.json. Changes require a gateway restart (Linux: systemctl --user restart openclaw-gateway; macOS: openclaw gateway restart --deep).
For the broader logging surface (file logs, log levels, OTLP log export) see Logging. For Grafana/Prometheus dashboards and alerting see Grafana and Prometheus setup.
Quick start with Langfuse
OPENCLAW_OTEL_PRELOADED=1 skipped NodeSDK startup, and whether OTEL_SDK_DISABLED is set. Header values, URL userinfo, query strings, and fragments are intentionally redacted from logs.
Without raw
captureContent, traces still carry safe summaries and hashes, but raw prompt, completion, system prompt, tool argument, and tool schema payloads are omitted. See captureContent below.~/.openclaw/extensions/diagnostics-otel, keep that managed copy current — a globally updated WednesdayAI package does not automatically replace an already-installed managed extension.
Configuration reference
diagnostics.otel
captureContent
Object form with per-field control:captureContent: falseor unset — captures nothing.captureContent: true(bare boolean) — enables input messages, output messages, tool inputs, tool outputs, and tool definitions.systemPromptstays disabled.- Object form with
enabled: false— overrides all sub-keys; captures nothing. - Object form — each unset sub-key defaults to
false.
openclaw.model.*.summary and openclaw.model.*.hash attributes instead of raw content. Trace headers in Langfuse are still populated from safe message summaries (counts, roles, redacted hashes), or from the first/last captured raw model I/O when capture is on.
Langfuse convenience block
When
langfuse is set and no explicit endpoint is configured, the plugin derives the global OTLP endpoint as {host}/api/public/otel and generates the Basic-auth header Langfuse’s OTLP ingest requires. User-supplied Authorization in otel.headers always wins over the derived auth.
filterInfraSpans
Whentrue (the default), a span-level filter wraps the Langfuse OTLP exporter. Only spans carrying a Langfuse trace name, observation type, or gen_ai.tool.name attribute reach Langfuse; infrastructure spans (Telegram polling, webhook intake, account state) are dropped. All other OTLP backends (Tempo, a custom endpoint) never get the wrapper — infrastructure spans still flow there unfiltered.
Set filterInfraSpans: false when you want the full raw span stream in Langfuse, for example while debugging Telegram polling latency.
Endpoint precedence
For each signal, the plugin resolves the first available endpoint in this order:
Two environment switches change startup behavior:
OTEL_SDK_DISABLED=truedisables OpenTelemetry SDK behavior globally.OPENCLAW_OTEL_PRELOADED=1tells this extension not to start its own NodeSDK — use it only when another provider already exports the same signals. The startup log then reportsnodeSdk=skipped-preloaded.
Trace hierarchy
Spans form a nested tree matching the actual execution path. WhensessionSpans is enabled (the default), turns sharing a sessionKey are grouped under a parent openclaw.session trace root:
sessionSpans: false the openclaw.session layer is omitted and each openclaw.run is its own trace root. Tool spans that arrive after the model stream ends (common in streaming) are still parented to the model-call span, not the run span.
Task and dream spans
When the task pipeline or Dream Cycle emits diagnostics,diagnostics-otel exports a short span per event, named openclaw.<event.type> (for example openclaw.task.completed).
Task attributes (when present): openclaw.task.action, openclaw.task.id, openclaw.task.run_id, openclaw.task.stage, openclaw.task.attempts, openclaw.task.reason, openclaw.task.error, openclaw.task.duration_ms.
Dream attributes (when present): openclaw.dream.action, openclaw.dream.cycle_fingerprint, openclaw.dream.stage, openclaw.dream.pass_id, openclaw.dream.reason, openclaw.dream.cap.
Error span status is set for task.failed, task.recovery_exhausted, task.boundary_violation, dream.contemplation_failed, and dream.fetch_failed.
Session spans
Whenlangfuse.sessionSpans is true (default), the plugin maintains one openclaw.session trace-root span per active session key:
- One session span per session key — created on the first run and reused across subsequent turns.
langfuse.trace.nameis set tosession:<8hex>on the session span;openclaw.runchild spans omit it so they surface as observations under the session, not separate traces.- Idle timeout — the span closes after
sessionIdleTimeoutMswith no run starts or completions (default 30 minutes). The next run opens a new session span. - Gateway stop — open session spans are ended synchronously on shutdown; there is no graceful drain.
- Cost roll-up —
langfuse.trace.costis set on each run span to that run’s accumulated cost. Langfuse aggregates per-run costs in its session view.
gen_ai.thread.id (the run id), langfuse.session.id (the session key), langfuse.user.id, and a hashed openclaw.sessionKey for privacy-safe debugging.
Cost attribution
The plugin computes USD cost peropenclaw.run and sets langfuse.trace.cost on the run span. Cost accumulates across model calls within the run and is written once at run completion; zero-cost runs omit the attribute.
Built-in pricing table (USD per million tokens):
Date-suffix and region-prefix variants (for example
claude-sonnet-4-6-20250622, us.claude-opus-4-8) are normalised before lookup. When the provider reports token counts but no cost, fallback cost is computed from this table; when no entry matches, cost is not set.
Override or extend pricing for unlisted models with diagnostics.otel.modelPricing:
modelPricing entries win over the built-in table for the same provider/model key.
Telegram fleet diagnostics
Telegram fleet supervision emits channel-specific diagnostic events for account lifecycle, polling, webhook intake, Bot API calls, update handling, and delivery. The OTel exporter maps them to spans namedopenclaw.telegram.<event> — for example telegram.poll.started, telegram.webhook.received, telegram.update.deduped, telegram.delivery.flood_wait — and to the openclaw.telegram.* metrics listed below.
Privacy behavior:
- Raw bot tokens, raw chat/user/thread IDs, webhook secrets, webhook URLs, and message content are never exported. Identity is carried as stable hashes (
telegram.account.hash,telegram.chat.hash,telegram.user.hash, …). captureContentdoes not override Telegram redaction. Telegram-correlated trace I/O, model input/output, system prompts, tool definitions, tool spans, and status text are replaced withtelegram-content-redacted.- OTLP log records that look Telegram-shaped get a redacted body (
telegram-log-redacted) and drop aggregate argument fields that could contain IDs or message content.
Metrics
The plugin exports these OTel metrics (Prometheus surfaces them with underscores — see Grafana and Prometheus setup):
The
system_prompt_flip / system_prompt_stability pair tells you when something rewrites the system prompt between turns and busts provider prefix caches — see System prompt stability metrics.
Troubleshooting
No traces arrive in Langfuse
No traces arrive in Langfuse
- Confirm the startup summary contains
tracesEnabled=true, the expected sanitizedtraceUrl,traceHeaders=Authorization=present, andnodeSdk=started. - If it says
nodeSdk=skipped-preloaded, check whyOPENCLAW_OTEL_PRELOADED=1is set in the gateway environment — the preloaded provider must then be configured to export traces to Langfuse. - If
otelSdkDisabledis notunset, remove or overrideOTEL_SDK_DISABLED=truefor the gateway service. - Check per-signal env vars:
OTEL_EXPORTER_OTLP_TRACES_ENDPOINTcan route traces away from the Langfuse endpoint. - If
openclaw --versionis current but extension behavior is stale, check for a managed copy under~/.openclaw/extensions/diagnostics-oteland refresh it.
Empty input/output in Langfuse
Empty input/output in Langfuse
- Summaries present, raw content missing — expected when
captureContentis disabled; the trace header shows message-count and role summaries. EnablecaptureContentfor raw text. - Run completed with no model call (routing or command-only turn) — the trace header is synthesized from channel and session identity. Check
openclaw.runspan attributes foropenclaw.channelandopenclaw.sessionKeyHash. - Telegram-correlated runs — redaction markers instead of raw text are expected, not a capture failure.
Runs appear as separate traces instead of grouped under a session
Runs appear as separate traces instead of grouped under a session
Confirm
diagnostics.otel.langfuse.sessionSpans is not false and that langfuse.session.id appears on openclaw.run spans. If it is missing, the extension may not be receiving a sessionKey from the gateway — verify the extension is active with openclaw gateway status --deep.No langfuse.trace.cost on run spans
No langfuse.trace.cost on run spans
Cost requires a pricing-table entry or a
modelPricing override for the provider/model in use, and a non-zero accumulated cost. The Grafana cost panel additionally requires the openclaw_cost_usd_total series — confirm the metrics endpoint is reachable.Only infrastructure noise in Langfuse, or none at all
Only infrastructure noise in Langfuse, or none at all
filterInfraSpans: true (default) drops polling/webhook spans from Langfuse only. Set it to false to send everything to Langfuse, or point a custom endpoint (Tempo, collector) at the unfiltered stream.For users
If your administrator has tracing enabled, you can view your conversations in Langfuse. A healthy agent turn appears as one trace withopenclaw.run at the root; model and tool calls appear underneath it.
Background maintenance work also leaves short traces named like openclaw.task.completed or openclaw.dream.cycle_completed — those are not chat messages. If a background job gives up after too many retries you may see openclaw.task.recovery_exhausted marked as an error. That means the system stopped retrying and parked the job; ask your administrator if something you expected never finished. A gateway restart by itself does not park interrupted jobs — those are retried. recovery_exhausted means the job had already used its retries or had been stuck for about 25 minutes.
For developers
The implementation lives inextensions/diagnostics-otel/src/service.ts. Startup resolves shared and per-signal OTLP endpoints, derives Langfuse Basic auth only when the langfuse block owns the endpoint, starts the NodeSDK for traces and metrics, and starts the logs exporter separately.
hasPreloadedOtelSdk() returns true only when OPENCLAW_OTEL_PRELOADED === "1". It does not inspect OpenTelemetry globals, because importing @opentelemetry/api is not proof that another provider intentionally owns export. In preloaded mode the extension still subscribes to diagnostic events, but trace export belongs to the preloaded provider.
Model content is never reconstructed from saved conversation history. The run attempt builds a mutable model-call context, updated from the final input transform after projection and transform_llm_input hooks, and subscriber handlers emit model.call.started / model.call.completed diagnostic events from the assistant stream lifecycle. Keep new model-content fields optional and bounded at the diagnostic emitter before adding exporter mappings.
To emit task or dream diagnostics from a plugin, use the SDK builders so type and action stay aligned — do not cast loose objects:
ts / seq — emitDiagnosticEvent stamps them. Field names are part of the contract (passId, not passName; attempts, not attempt). The exporter handles every task.* and dream.* action in one path and starts a span named openclaw.${evt.type}; the payload switch is exhaustiveness-checked, so a new diagnostic variant fails compilation until an export case is added — unknown types are never silently dropped.
Related
- Observability — traced span overview and task recovery
- Logging — log levels, file logs, OTLP log export
- Grafana and Prometheus setup — dashboards and alert rules
- Deep observability — runtime snapshots and capture controls
- Prompt caching — system prompt stability and cache diagnostics