OpenTelemetry & Langfuse Diagnostics
WednesdayAI exports traces, metrics, and logs via OpenTelemetry Protocol (OTLP). Thediagnostics-otel extension wires core run, model-call, and tool events into a nested OTel span tree. Langfuse is supported as a first-class backend via a convenience derivation that handles endpoint and Basic-auth setup automatically.
For runtime metrics, diagnostic capture, and release performance sweeps, see Deep Observability. For the shared Grafana/Prometheus/Loki/Tempo/Pyroscope stack handoff, see Deep Observability Stack Handoff.
For users
Use this page when you expect conversation traces in Langfuse. A healthy agent turn appears as a trace withopenclaw.run as the root span. Model calls and tool calls appear underneath that run.
Background maintenance and dream-cycle work (when your administrator has those features on) also leave short traces named like openclaw.task.completed or openclaw.dream.cycle_completed. They are not chat messages. If a background job gives up after too many retries, you may see openclaw.task.recovery_exhausted marked as an error span — that means the system stopped retrying and parked the job; ask an administrator if something you expected never finished. A Gateway restart by itself does not park a job that was just interrupted; those jobs are retried. recovery_exhausted means the job had already used its retries or had been stuck for about 25 minutes.
Trace header input/output — Langfuse shows input and output on every trace, even without raw content capture:
- When
captureContentis disabled, the trace header is populated from safe message summaries (counts, roles, and redacted hashes — not raw text). The first captured model input summary populateslangfuse.trace.input; the last output summary populateslangfuse.trace.output. - When
captureContentis enabled, the trace header uses the first captured raw model input and final captured model output. - For runs that complete with no model call (for example, a command-only or routing turn), the trace header is synthesized from the channel identity and session key hash, with the exit status as the output.
- The trace header uses the first captured model input and final captured model output from the root
openclaw.runspan. - Each generation observation uses the model-call prompt and completion captured from the real provider stream boundary.
filterInfraSpans to opt out.
For administrators
Configure this in~/.openclaw/openclaw.json, then restart the gateway service. On Linux, use:
OPENCLAW_OTEL_PRELOADED=1 skipped NodeSDK startup, and whether OTEL_SDK_DISABLED is set. Header values, URL userinfo, query strings, and fragments are intentionally redacted from logs.
If the gateway loads a managed plugin from ~/.openclaw/extensions/diagnostics-otel, keep that managed copy current. A globally updated WednesdayAI package does not automatically replace an already installed managed extension.
For agentic workflow operations, watch both traces and metrics:
openclaw.tool.loopspans show loop detector warnings and blocks under the active model-call or run context.openclaw.session_run.*spans show durable session-run transitions and recovery decisions.openclaw.task.*/openclaw.dream.*spans show Gateway task-pipeline and Dream Cycle lifecycle events (see Task and dream spans).openclaw.task.recovery_exhausted(error status) means boot recovery moved a stuck run todlqafter attempt exhaustion or age stall — check Task operations.openclaw.diagnostics_otel.content.truncatedcounts content values shortened before export.openclaw.diagnostics_otel.content.redactedcounts Telegram-sensitive values replaced before export.openclaw.diagnostics_otel.handler.failurescounts exporter handler failures caught without crashing the gateway.openclaw.session.system_prompt_flipcounts within-session system prompt hash changes. Each flip is a prefix-cache bust for the whole prompt, so a rising counter means something is rewriting the system prompt between turns. Runs that record the per-slot breakdown (ADR 2105) carryslotand aownerattribute normalized slot-aware (coreorplugin:<id>for claim slots;customfor the additivecustom.*slots and other free-form owners). See Prompt caching.openclaw.session.system_prompt_stabilityreports cumulative hash comparisons and flips as a gauge withopenclaw.stat="comparisons"andopenclaw.stat="flips"labels, refreshed on each diagnostic heartbeat. A healthy fleet trends towardflips / comparisonsnear zero.
For developers
The implementation lives inextensions/diagnostics-otel/src/service.ts. Startup resolves shared and per-signal OTLP endpoints, derives Langfuse Basic auth only when the langfuse convenience block owns the endpoint, starts the NodeSDK for traces and metrics, and starts the logs exporter separately.
hasPreloadedOtelSdk() intentionally returns true only when OPENCLAW_OTEL_PRELOADED === "1". It does not inspect OpenTelemetry globals, because importing @opentelemetry/api is not proof that another provider intentionally owns export. In preloaded mode, diagnostics-otel still subscribes to diagnostic events, but trace export belongs to the preloaded provider.
Model content is not reconstructed from saved conversation history. The run attempt builds a mutable modelCallContext, updates it from the final input transform after projection and transform_llm_input hooks, and the subscriber handlers emit model.call.started / model.call.completed diagnostic events from the assistant stream lifecycle. Keep new model-content fields optional and bounded at the diagnostic emitter before adding exporter mappings.
Emitting task and dream diagnostics
Task and dream lifecycle events are members of the closedDiagnosticEventPayload union. Produce them with the SDK builders so type and action stay aligned; do not cast loose objects:
ts / seq — emitDiagnosticEvent stamps them). Live field names are part of the contract: use passId (not passName) and attempts (not attempt). PassContext.emitDiagnostic is typed as DiagnosticEventInput, so typos fail at compile time.
The OTel exporter handles every task.* and dream.* action in one recordTaskDreamDiagnostic path and starts a span named openclaw.${evt.type}. Unknown future types must not be silently dropped: the payload switch is exhaustiveness-checked (fail closed at compile time when a new variant is added without a case).
Also exported from wednesdayai/plugin-sdk for fetch facades: GuardedFetchOptions and GuardedFetchResult (alongside fetchWithSsrFGuard).
Quick start
captureContent, traces still carry safe summaries and hashes, but raw prompt,
completion, system prompt, tool argument, and tool schema payloads are omitted. Langfuse OTLP
ingest requires the /api/public/otel endpoint and Basic auth; the langfuse block derives
both from host, publicKey, and secretKey.
Configuration reference
diagnostics.otel
captureContent (object form)
captureContent: true (bare boolean) to enable input messages, output messages, tool inputs, tool outputs, and tool definitions. It still leaves systemPrompt disabled; use object form when you need exact control. Content is character-capped at 128 KB per field; sensitive tokens are redacted via the configured redactPatterns.
When capture is disabled, model spans export openclaw.model.*.summary and openclaw.model.*.hash
attributes instead of raw content.
Captured model content maps to both Langfuse and OTel semantic attributes:
langfuse.observation.cost_details is emitted only when the runtime event includes real cost
details from the provider. langfuse.trace.cost is the accumulated USD cost for the run, set from
the built-in pricing table or a diagnostics.otel.modelPricing override; see Cost attribution.
Response metadata
and time-to-first-chunk attributes are exported when the model-call event carries those provider
fields; the embedded message-stream path currently emits output content, usage, and any known cost
details from the runtime data available at message_end.
Retrieval and evaluation telemetry is not emitted yet. Those fields will be added only after
real WednesdayAI retrieval or evaluation events exist.
langfuse convenience block
When
langfuse is set and no explicit endpoint is configured, the plugin derives the global OTLP endpoint as {host}/api/public/otel and generates the required Basic-auth header. host must be a bare http or https URL without query string or fragment.
Signal-specific endpoint overrides continue to work independently. If tracesEndpoint, metricsEndpoint, or logsEndpoint is set, that signal routes to the configured target and does not inherit Langfuse auth.
filterInfraSpans
When true (the default), a span-level filter wraps the Langfuse OTLP exporter. Any span carrying langfuse.trace.name in the telegram.* / openclaw.telegram.* namespace, or ATTR_LANGFUSE_OBSERVATION_TYPE matching an infrastructure pattern, is dropped before export. All other OTLP backends (Tempo, custom endpoint) are unaffected — infrastructure spans still flow there.
Set filterInfraSpans: false when you want the full raw span stream in Langfuse, for example during a debugging session where Telegram polling latency matters.
Endpoint precedence
For each signal, the plugin resolves the first available endpoint in this order:OTEL_SDK_DISABLED=true can disable OpenTelemetry SDK behavior. OPENCLAW_OTEL_PRELOADED=1 tells this extension not to start its own NodeSDK; use it only when another provider has already been configured to export the same signals.
Trace hierarchy
Spans are structured as a nested tree matching the actual execution path. Whenlangfuse.sessionSpans is enabled (the default), turns that share a sessionKey are grouped under a parent openclaw.session trace root:
langfuse.sessionSpans: false, the openclaw.session layer is omitted and each openclaw.run is its own trace root (the behaviour before Phase 3).
This nesting is built by the plugin using internal span context maps. Tool spans that arrive after message_end (common in streaming) are correctly parented to the model-call span, not the run span. If no model call is active, loop spans fall back to the retained run context.
Task and dream spans
When the task pipeline or Dream Cycle emits diagnostics,diagnostics-otel exports a short span per event. Span name is openclaw.<event.type> (for example openclaw.task.completed, openclaw.dream.cycle_started).
Task attributes (when present on the event):
Dream attributes (when present):
Error span status is set for:
task.failed, task.recovery_exhausted, task.boundary_violation, dream.contemplation_failed, dream.fetch_failed.
Task actions exported: started, completed, failed, preempted, budget_deferred, budget_ceiling_reached, preemption_ceiling_reached, boundary_violation, pipeline_disabled, idle_monitor_disabled, plugin_pass_rejected, runlog_retention, recovery_exhausted.
Dream actions exported: cycle_started, cycle_completed, pass_completed, stage_skipped, contemplation_unconfigured, contemplation_skipped, contemplation_failed, rate_capped, jsonl_parse_skipped, fetch_failed, crystallisation_candidate.
Session spans
Whenlangfuse.sessionSpans is true (default), the plugin maintains one openclaw.session trace-root span per active sessionKey. All openclaw.run spans for that session are children of it.
- One session span per session key — the span is created on the first run for a given session and reused across subsequent turns.
langfuse.trace.nameis set tosession:<8hex>on the session span (the Langfuse trace root);openclaw.runchild spans omit it so they surface as observations under the session, not as separate traces.- Idle timeout — the session span closes automatically after
sessionIdleTimeoutMsof inactivity (default 30 minutes). The timer resets on each run start and run completion. After the span closes, the next run for that session key opens a new one. - Gateway stop — all open session spans are ended synchronously on gateway shutdown; no graceful drain is performed.
- Cost roll-up —
langfuse.trace.costis set on each run span to the accumulated cost for that run (summed across model calls). Langfuse aggregates these per-run costs in its session view. There is no cross-run cost attribute on the session span itself. See Cost attribution.
diagnostics.otel.langfuse.sessionSpans: false.
Cost attribution
The plugin computes USD cost for eachopenclaw.run and sets langfuse.trace.cost on the run span when cost is attributable. Cost is accumulated additively across model calls; the final accumulated value is written once at run completion.
Built-in pricing table (USD per million tokens):
Date-suffix and region-prefix variants (e.g.
claude-sonnet-4-6-20250622, us.claude-opus-4-8) are normalised before lookup.
Fallback cost — when the provider event includes token counts but no cost details, the plugin computes fallback cost from the built-in table. When no pricing entry exists for the provider/model, cost is not set and the run span omits langfuse.trace.cost.
Override / extend — add entries for unlisted models via diagnostics.otel.modelPricing:
modelPricing overrides win over the built-in table for the same provider/model key.
langfuse.trace.cost is set only when the accumulated run cost is greater than zero. Zero-cost runs (no model calls, or all model calls with unknown pricing) omit the attribute.
Langfuse session grouping
Each run span carries:
In Langfuse: each agent run appears as one trace; all runs within a conversation session are grouped under one session view.
Telegram fleet diagnostics
Telegram fleet supervision emits channel-specific diagnostic events for account lifecycle, polling, webhook intake, Bot API calls, update handling, and delivery. The OTel exporter maps these events to spans namedopenclaw.telegram.<event>
and to Telegram-specific metrics.
Event names:
telegram.account.statetelegram.api.requesttelegram.poll.startedtelegram.poll.succeededtelegram.poll.failedtelegram.poll.backofftelegram.poll.conflicttelegram.webhook.receivedtelegram.webhook.rejectedtelegram.webhook.reconciledtelegram.update.receivedtelegram.update.dedupedtelegram.update.queuedtelegram.update.processedtelegram.delivery.queuedtelegram.delivery.senttelegram.delivery.failedtelegram.delivery.flood_wait
Privacy behavior:
- Raw bot tokens, raw chat IDs, raw user IDs, raw thread IDs, raw webhook secrets, raw webhook URLs, and Telegram message content are not exported.
captureContentdoes not override Telegram redaction for Telegram-correlated root trace I/O, model input/output, system prompts, tool definitions, tool spans, captured tool input, output, blocked reason, and status text. These fields are replaced withtelegram-content-redacted.- OTLP log records that look Telegram-shaped use a redacted body and drop aggregate argument fields that could contain Telegram IDs or message content.
Splitting traces and metrics to different backends
To send traces to Langfuse and metrics/logs to a local OTel collector:langfuse.host); metrics and logs flow to the local collector via their signal-specific endpoints.
Troubleshooting
No traces arrive in Langfuse
- Confirm the startup summary contains
tracesEnabled=true, the expected sanitizedtraceUrl,traceHeaders=Authorization=presentfor Langfuse, andnodeSdk=started. - If it says
nodeSdk=skipped-preloaded, check whyOPENCLAW_OTEL_PRELOADED=1is in the gateway environment. The preloaded provider must be configured to export traces to Langfuse. - If
otelSdkDisabled=true, remove or overrideOTEL_SDK_DISABLED=truefor the gateway service. - Check per-signal environment variables.
OTEL_EXPORTER_OTLP_TRACES_ENDPOINTcan route traces away from the configured Langfuse endpoint. - If
wednesdayai --versionis current but the extension behavior is stale, check for a managed copy under~/.openclaw/extensions/diagnostics-oteland reinstall or refresh it.
Empty input/output in Langfuse
Trace header input/output is now populated even whencaptureContent is off:
- Summaries present, raw content missing — this is expected when
captureContentis disabled. The trace header shows message-count and role summaries. EnablecaptureContentif you need raw prompt and completion text. - Both trace header and generation fields are empty — the run may have completed before any model call occurred (for example, a routing or command-only turn with no LLM invocation). In that case, the trace header is synthesized from the channel and session identity. Check the
openclaw.runspan attributes foropenclaw.channelandopenclaw.sessionKeyHashto confirm the run identity. - Trace header truly blank — confirm the extension version is ≥ 0.4.7 (Phase 3 / Langfuse trace gap fixes). Older builds did not promote summaries to the root span.
Traces arrive but no session grouping
Check thatgen_ai.thread.id and langfuse.session.id appear on root run spans. These require the diagnostics-otel extension to be enabled and on version ≥ 0.4.4 (Phase 2). If attributes are missing, confirm diagnostics.otel.traces: true and the extension is active via wednesdayai gateway status --deep.
Tool spans appear as siblings, not children of model-call
This is fixed in ≥ 0.4.4 (Phase 2). In older versions, theactiveModelCallContexts map was cleared before tool dispatch. Update the installed extension.
Runs appear as separate traces instead of nested under a session
Confirmdiagnostics.otel.langfuse.sessionSpans is not set to false and the extension version is ≥ 0.4.7 (Phase 3). In older builds, the openclaw.session span layer did not exist; each run was its own trace root. Also verify langfuse.session.id appears on openclaw.run spans — if it is missing, the extension may not be receiving a sessionKey from the gateway.
No langfuse.trace.cost on run spans
Cost attribution requires a matching entry in the built-in pricing table or a diagnostics.otel.modelPricing override for the provider/model in use. If the model normalises (date-suffix stripped) to a key in the table but cost is still absent, check that accumulatedCost > 0 — zero-cost runs omit the attribute. The cost panel in the Grafana dashboard requires openclaw_cost_usd_total series; confirm the OTLP metrics endpoint is reachable and metrics export is enabled.
Zero token counts on spans
Fixed in ≥ 0.4.4. Older builds wrote0 when token fields were absent; current builds omit the attribute instead.