Skip to main content

OpenTelemetry & Langfuse Diagnostics

WednesdayAI exports traces, metrics, and logs via OpenTelemetry Protocol (OTLP). The diagnostics-otel extension wires core run, model-call, and tool events into a nested OTel span tree. Langfuse is supported as a first-class backend via a convenience derivation that handles endpoint and Basic-auth setup automatically. For runtime metrics, diagnostic capture, and release performance sweeps, see Deep Observability. For the shared Grafana/Prometheus/Loki/Tempo/Pyroscope stack handoff, see Deep Observability Stack Handoff.

For users

Use this page when you expect conversation traces in Langfuse. A healthy agent turn appears as a trace with openclaw.run as the root span. Model calls and tool calls appear underneath that run. Background maintenance and dream-cycle work (when your administrator has those features on) also leave short traces named like openclaw.task.completed or openclaw.dream.cycle_completed. They are not chat messages. If a background job gives up after too many retries, you may see openclaw.task.recovery_exhausted marked as an error span — that means the system stopped retrying and parked the job; ask an administrator if something you expected never finished. A Gateway restart by itself does not park a job that was just interrupted; those jobs are retried. recovery_exhausted means the job had already used its retries or had been stuck for about 25 minutes. Trace header input/output — Langfuse shows input and output on every trace, even without raw content capture:
  • When captureContent is disabled, the trace header is populated from safe message summaries (counts, roles, and redacted hashes — not raw text). The first captured model input summary populates langfuse.trace.input; the last output summary populates langfuse.trace.output.
  • When captureContent is enabled, the trace header uses the first captured raw model input and final captured model output.
  • For runs that complete with no model call (for example, a command-only or routing turn), the trace header is synthesized from the channel identity and session key hash, with the exit status as the output.
If the trace header is empty (no input/output at all), see Troubleshooting below. When model content capture is enabled, Langfuse shows two levels of I/O:
  • The trace header uses the first captured model input and final captured model output from the root openclaw.run span.
  • Each generation observation uses the model-call prompt and completion captured from the real provider stream boundary.
Telegram-correlated runs still show redaction markers instead of raw message text. That is expected and is not a capture failure. Infrastructure spans — Telegram polling, webhook intake, and account-state spans are filtered from Langfuse export by default. They still appear in Tempo and any other configured OTLP backend. See filterInfraSpans to opt out.

For administrators

Configure this in ~/.openclaw/openclaw.json, then restart the gateway service. On Linux, use:
After startup, look for the redacted exporter summary:
The line reports effective signal URLs, whether auth headers are present, whether OPENCLAW_OTEL_PRELOADED=1 skipped NodeSDK startup, and whether OTEL_SDK_DISABLED is set. Header values, URL userinfo, query strings, and fragments are intentionally redacted from logs. If the gateway loads a managed plugin from ~/.openclaw/extensions/diagnostics-otel, keep that managed copy current. A globally updated WednesdayAI package does not automatically replace an already installed managed extension. For agentic workflow operations, watch both traces and metrics:
  • openclaw.tool.loop spans show loop detector warnings and blocks under the active model-call or run context.
  • openclaw.session_run.* spans show durable session-run transitions and recovery decisions.
  • openclaw.task.* / openclaw.dream.* spans show Gateway task-pipeline and Dream Cycle lifecycle events (see Task and dream spans).
  • openclaw.task.recovery_exhausted (error status) means boot recovery moved a stuck run to dlq after attempt exhaustion or age stall — check Task operations.
  • openclaw.diagnostics_otel.content.truncated counts content values shortened before export.
  • openclaw.diagnostics_otel.content.redacted counts Telegram-sensitive values replaced before export.
  • openclaw.diagnostics_otel.handler.failures counts exporter handler failures caught without crashing the gateway.
  • openclaw.session.system_prompt_flip counts within-session system prompt hash changes. Each flip is a prefix-cache bust for the whole prompt, so a rising counter means something is rewriting the system prompt between turns. Runs that record the per-slot breakdown (ADR 2105) carry slot and a owner attribute normalized slot-aware (core or plugin:<id> for claim slots; custom for the additive custom.* slots and other free-form owners). See Prompt caching.
  • openclaw.session.system_prompt_stability reports cumulative hash comparisons and flips as a gauge with openclaw.stat="comparisons" and openclaw.stat="flips" labels, refreshed on each diagnostic heartbeat. A healthy fleet trends toward flips / comparisons near zero.

For developers

The implementation lives in extensions/diagnostics-otel/src/service.ts. Startup resolves shared and per-signal OTLP endpoints, derives Langfuse Basic auth only when the langfuse convenience block owns the endpoint, starts the NodeSDK for traces and metrics, and starts the logs exporter separately. hasPreloadedOtelSdk() intentionally returns true only when OPENCLAW_OTEL_PRELOADED === "1". It does not inspect OpenTelemetry globals, because importing @opentelemetry/api is not proof that another provider intentionally owns export. In preloaded mode, diagnostics-otel still subscribes to diagnostic events, but trace export belongs to the preloaded provider. Model content is not reconstructed from saved conversation history. The run attempt builds a mutable modelCallContext, updates it from the final input transform after projection and transform_llm_input hooks, and the subscriber handlers emit model.call.started / model.call.completed diagnostic events from the assistant stream lifecycle. Keep new model-content fields optional and bounded at the diagnostic emitter before adding exporter mappings.

Emitting task and dream diagnostics

Task and dream lifecycle events are members of the closed DiagnosticEventPayload union. Produce them with the SDK builders so type and action stay aligned; do not cast loose objects:
Builders are input-shaped (omit ts / seq — emitDiagnosticEvent stamps them). Live field names are part of the contract: use passId (not passName) and attempts (not attempt). PassContext.emitDiagnostic is typed as DiagnosticEventInput, so typos fail at compile time. The OTel exporter handles every task.* and dream.* action in one recordTaskDreamDiagnostic path and starts a span named openclaw.${evt.type}. Unknown future types must not be silently dropped: the payload switch is exhaustiveness-checked (fail closed at compile time when a new variant is added without a case). Also exported from wednesdayai/plugin-sdk for fetch facades: GuardedFetchOptions and GuardedFetchResult (alongside fetchWithSsrFGuard).

Quick start

Without raw captureContent, traces still carry safe summaries and hashes, but raw prompt, completion, system prompt, tool argument, and tool schema payloads are omitted. Langfuse OTLP ingest requires the /api/public/otel endpoint and Basic auth; the langfuse block derives both from host, publicKey, and secretKey.

Configuration reference

diagnostics.otel

captureContent (object form)

Set captureContent: true (bare boolean) to enable input messages, output messages, tool inputs, tool outputs, and tool definitions. It still leaves systemPrompt disabled; use object form when you need exact control. Content is character-capped at 128 KB per field; sensitive tokens are redacted via the configured redactPatterns. When capture is disabled, model spans export openclaw.model.*.summary and openclaw.model.*.hash attributes instead of raw content. Captured model content maps to both Langfuse and OTel semantic attributes: langfuse.observation.cost_details is emitted only when the runtime event includes real cost details from the provider. langfuse.trace.cost is the accumulated USD cost for the run, set from the built-in pricing table or a diagnostics.otel.modelPricing override; see Cost attribution. Response metadata and time-to-first-chunk attributes are exported when the model-call event carries those provider fields; the embedded message-stream path currently emits output content, usage, and any known cost details from the runtime data available at message_end. Retrieval and evaluation telemetry is not emitted yet. Those fields will be added only after real WednesdayAI retrieval or evaluation events exist.

langfuse convenience block

When langfuse is set and no explicit endpoint is configured, the plugin derives the global OTLP endpoint as {host}/api/public/otel and generates the required Basic-auth header. host must be a bare http or https URL without query string or fragment. Signal-specific endpoint overrides continue to work independently. If tracesEndpoint, metricsEndpoint, or logsEndpoint is set, that signal routes to the configured target and does not inherit Langfuse auth.

filterInfraSpans

When true (the default), a span-level filter wraps the Langfuse OTLP exporter. Any span carrying langfuse.trace.name in the telegram.* / openclaw.telegram.* namespace, or ATTR_LANGFUSE_OBSERVATION_TYPE matching an infrastructure pattern, is dropped before export. All other OTLP backends (Tempo, custom endpoint) are unaffected — infrastructure spans still flow there. Set filterInfraSpans: false when you want the full raw span stream in Langfuse, for example during a debugging session where Telegram polling latency matters.

Endpoint precedence

For each signal, the plugin resolves the first available endpoint in this order: OTEL_SDK_DISABLED=true can disable OpenTelemetry SDK behavior. OPENCLAW_OTEL_PRELOADED=1 tells this extension not to start its own NodeSDK; use it only when another provider has already been configured to export the same signals.

Trace hierarchy

Spans are structured as a nested tree matching the actual execution path. When langfuse.sessionSpans is enabled (the default), turns that share a sessionKey are grouped under a parent openclaw.session trace root:
When langfuse.sessionSpans: false, the openclaw.session layer is omitted and each openclaw.run is its own trace root (the behaviour before Phase 3). This nesting is built by the plugin using internal span context maps. Tool spans that arrive after message_end (common in streaming) are correctly parented to the model-call span, not the run span. If no model call is active, loop spans fall back to the retained run context.

Task and dream spans

When the task pipeline or Dream Cycle emits diagnostics, diagnostics-otel exports a short span per event. Span name is openclaw.<event.type> (for example openclaw.task.completed, openclaw.dream.cycle_started). Task attributes (when present on the event): Dream attributes (when present): Error span status is set for: task.failed, task.recovery_exhausted, task.boundary_violation, dream.contemplation_failed, dream.fetch_failed. Task actions exported: started, completed, failed, preempted, budget_deferred, budget_ceiling_reached, preemption_ceiling_reached, boundary_violation, pipeline_disabled, idle_monitor_disabled, plugin_pass_rejected, runlog_retention, recovery_exhausted. Dream actions exported: cycle_started, cycle_completed, pass_completed, stage_skipped, contemplation_unconfigured, contemplation_skipped, contemplation_failed, rate_capped, jsonl_parse_skipped, fetch_failed, crystallisation_candidate.

Session spans

When langfuse.sessionSpans is true (default), the plugin maintains one openclaw.session trace-root span per active sessionKey. All openclaw.run spans for that session are children of it.
  • One session span per session key — the span is created on the first run for a given session and reused across subsequent turns.
  • langfuse.trace.name is set to session:<8hex> on the session span (the Langfuse trace root); openclaw.run child spans omit it so they surface as observations under the session, not as separate traces.
  • Idle timeout — the session span closes automatically after sessionIdleTimeoutMs of inactivity (default 30 minutes). The timer resets on each run start and run completion. After the span closes, the next run for that session key opens a new one.
  • Gateway stop — all open session spans are ended synchronously on gateway shutdown; no graceful drain is performed.
  • Cost roll-up — langfuse.trace.cost is set on each run span to the accumulated cost for that run (summed across model calls). Langfuse aggregates these per-run costs in its session view. There is no cross-run cost attribute on the session span itself. See Cost attribution.
To disable session spans entirely: set diagnostics.otel.langfuse.sessionSpans: false.

Cost attribution

The plugin computes USD cost for each openclaw.run and sets langfuse.trace.cost on the run span when cost is attributable. Cost is accumulated additively across model calls; the final accumulated value is written once at run completion. Built-in pricing table (USD per million tokens): Date-suffix and region-prefix variants (e.g. claude-sonnet-4-6-20250622, us.claude-opus-4-8) are normalised before lookup. Fallback cost — when the provider event includes token counts but no cost details, the plugin computes fallback cost from the built-in table. When no pricing entry exists for the provider/model, cost is not set and the run span omits langfuse.trace.cost. Override / extend — add entries for unlisted models via diagnostics.otel.modelPricing:
modelPricing overrides win over the built-in table for the same provider/model key. langfuse.trace.cost is set only when the accumulated run cost is greater than zero. Zero-cost runs (no model calls, or all model calls with unknown pricing) omit the attribute.

Langfuse session grouping

Each run span carries: In Langfuse: each agent run appears as one trace; all runs within a conversation session are grouped under one session view.

Telegram fleet diagnostics

Telegram fleet supervision emits channel-specific diagnostic events for account lifecycle, polling, webhook intake, Bot API calls, update handling, and delivery. The OTel exporter maps these events to spans named openclaw.telegram.<event> and to Telegram-specific metrics. Event names:
  • telegram.account.state
  • telegram.api.request
  • telegram.poll.started
  • telegram.poll.succeeded
  • telegram.poll.failed
  • telegram.poll.backoff
  • telegram.poll.conflict
  • telegram.webhook.received
  • telegram.webhook.rejected
  • telegram.webhook.reconciled
  • telegram.update.received
  • telegram.update.deduped
  • telegram.update.queued
  • telegram.update.processed
  • telegram.delivery.queued
  • telegram.delivery.sent
  • telegram.delivery.failed
  • telegram.delivery.flood_wait
Representative attributes: Privacy behavior:
  • Raw bot tokens, raw chat IDs, raw user IDs, raw thread IDs, raw webhook secrets, raw webhook URLs, and Telegram message content are not exported.
  • captureContent does not override Telegram redaction for Telegram-correlated root trace I/O, model input/output, system prompts, tool definitions, tool spans, captured tool input, output, blocked reason, and status text. These fields are replaced with telegram-content-redacted.
  • OTLP log records that look Telegram-shaped use a redacted body and drop aggregate argument fields that could contain Telegram IDs or message content.

Splitting traces and metrics to different backends

To send traces to Langfuse and metrics/logs to a local OTel collector:
Traces flow to Langfuse (derived from langfuse.host); metrics and logs flow to the local collector via their signal-specific endpoints.

Troubleshooting

No traces arrive in Langfuse

  1. Confirm the startup summary contains tracesEnabled=true, the expected sanitized traceUrl, traceHeaders=Authorization=present for Langfuse, and nodeSdk=started.
  2. If it says nodeSdk=skipped-preloaded, check why OPENCLAW_OTEL_PRELOADED=1 is in the gateway environment. The preloaded provider must be configured to export traces to Langfuse.
  3. If otelSdkDisabled=true, remove or override OTEL_SDK_DISABLED=true for the gateway service.
  4. Check per-signal environment variables. OTEL_EXPORTER_OTLP_TRACES_ENDPOINT can route traces away from the configured Langfuse endpoint.
  5. If wednesdayai --version is current but the extension behavior is stale, check for a managed copy under ~/.openclaw/extensions/diagnostics-otel and reinstall or refresh it.

Empty input/output in Langfuse

Trace header input/output is now populated even when captureContent is off:
  • Summaries present, raw content missing — this is expected when captureContent is disabled. The trace header shows message-count and role summaries. Enable captureContent if you need raw prompt and completion text.
  • Both trace header and generation fields are empty — the run may have completed before any model call occurred (for example, a routing or command-only turn with no LLM invocation). In that case, the trace header is synthesized from the channel and session identity. Check the openclaw.run span attributes for openclaw.channel and openclaw.sessionKeyHash to confirm the run identity.
  • Trace header truly blank — confirm the extension version is ≥ 0.4.7 (Phase 3 / Langfuse trace gap fixes). Older builds did not promote summaries to the root span.

Traces arrive but no session grouping

Check that gen_ai.thread.id and langfuse.session.id appear on root run spans. These require the diagnostics-otel extension to be enabled and on version ≥ 0.4.4 (Phase 2). If attributes are missing, confirm diagnostics.otel.traces: true and the extension is active via wednesdayai gateway status --deep.

Tool spans appear as siblings, not children of model-call

This is fixed in ≥ 0.4.4 (Phase 2). In older versions, the activeModelCallContexts map was cleared before tool dispatch. Update the installed extension.

Runs appear as separate traces instead of nested under a session

Confirm diagnostics.otel.langfuse.sessionSpans is not set to false and the extension version is ≥ 0.4.7 (Phase 3). In older builds, the openclaw.session span layer did not exist; each run was its own trace root. Also verify langfuse.session.id appears on openclaw.run spans — if it is missing, the extension may not be receiving a sessionKey from the gateway.

No langfuse.trace.cost on run spans

Cost attribution requires a matching entry in the built-in pricing table or a diagnostics.otel.modelPricing override for the provider/model in use. If the model normalises (date-suffix stripped) to a key in the table but cost is still absent, check that accumulatedCost > 0 — zero-cost runs omit the attribute. The cost panel in the Grafana dashboard requires openclaw_cost_usd_total series; confirm the OTLP metrics endpoint is reachable and metrics export is enabled.

Zero token counts on spans

Fixed in ≥ 0.4.4. Older builds wrote 0 when token fields were absent; current builds omit the attribute instead.

Metrics

The plugin exports the following OTel metrics: