> ## Documentation Index
> Fetch the complete documentation index at: https://docs.wednesdayai.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# OTel & Langfuse Diagnostics

# OpenTelemetry & Langfuse Diagnostics

WednesdayAI exports traces, metrics, and logs via OpenTelemetry Protocol (OTLP). The `diagnostics-otel` extension wires core run, model-call, and tool events into a nested OTel span tree. Langfuse is supported as a first-class backend via a convenience derivation that handles endpoint and Basic-auth setup automatically.

For runtime metrics, diagnostic capture, and release performance sweeps, see [Deep Observability](https://docs.wednesdayai.dev/next/admin/gateway/deep-observability.md). For the shared Grafana/Prometheus/Loki/Tempo/Pyroscope stack handoff, see [Deep Observability Stack Handoff](https://docs.wednesdayai.dev/next/admin/diagnostics/deep-observability-stack-handoff).

## For users

Use this page when you expect conversation traces in Langfuse. A healthy agent turn appears as a trace with `openclaw.run` as the root span. Model calls and tool calls appear underneath that run.

**Background maintenance and dream-cycle work** (when your administrator has those features on) also leave short traces named like `openclaw.task.completed` or `openclaw.dream.cycle_completed`. They are not chat messages. If a background job gives up after too many retries, you may see `openclaw.task.recovery_exhausted` marked as an error span — that means the system stopped retrying and parked the job; ask an administrator if something you expected never finished. A Gateway restart by itself does not park a job that was just interrupted; those jobs are retried. `recovery_exhausted` means the job had already used its retries or had been stuck for about 25 minutes.

**Trace header input/output** — Langfuse shows input and output on every trace, even without raw content capture:

* When `captureContent` is disabled, the trace header is populated from safe **message summaries** (counts, roles, and redacted hashes — not raw text). The first captured model input summary populates `langfuse.trace.input`; the last output summary populates `langfuse.trace.output`.
* When `captureContent` is enabled, the trace header uses the first captured raw model input and final captured model output.
* For runs that complete with no model call (for example, a command-only or routing turn), the trace header is synthesized from the channel identity and session key hash, with the exit status as the output.

If the trace header is empty (no input/output at all), see [Troubleshooting](#troubleshooting) below.

When model content capture is enabled, Langfuse shows two levels of I/O:

* The trace header uses the first captured model input and final captured model output from the root `openclaw.run` span.
* Each generation observation uses the model-call prompt and completion captured from the real provider stream boundary.

Telegram-correlated runs still show redaction markers instead of raw message text. That is expected and is not a capture failure.

**Infrastructure spans** — Telegram polling, webhook intake, and account-state spans are filtered from Langfuse export by default. They still appear in Tempo and any other configured OTLP backend. See [`filterInfraSpans`](#filterinfraspans) to opt out.

## For administrators

Configure this in `~/.openclaw/openclaw.json`, then restart the gateway service. On Linux, use:

```bash theme={"dark"}
systemctl --user restart openclaw-gateway
journalctl --user -u openclaw-gateway -n 200 --no-pager
```

After startup, look for the redacted exporter summary:

```text theme={"dark"}
diagnostics-otel: exporters configured tracesEnabled=true metricsEnabled=true logsEnabled=true preloadedSdk=false otelSdkDisabled=unset traceUrl=https://langfuse.example.com/api/public/otel/v1/traces traceHeaders=Authorization=present nodeSdk=started
```

The line reports effective signal URLs, whether auth headers are present, whether `OPENCLAW_OTEL_PRELOADED=1` skipped NodeSDK startup, and whether `OTEL_SDK_DISABLED` is set. Header values, URL userinfo, query strings, and fragments are intentionally redacted from logs.

If the gateway loads a managed plugin from `~/.openclaw/extensions/diagnostics-otel`, keep that managed copy current. A globally updated WednesdayAI package does not automatically replace an already installed managed extension.

For agentic workflow operations, watch both traces and metrics:

* `openclaw.tool.loop` spans show loop detector warnings and blocks under the active model-call or run context.
* `openclaw.session_run.*` spans show durable session-run transitions and recovery decisions.
* `openclaw.task.*` / `openclaw.dream.*` spans show Gateway task-pipeline and Dream Cycle lifecycle events (see [Task and dream spans](#task-and-dream-spans)).
* `openclaw.task.recovery_exhausted` (error status) means boot recovery moved a stuck run to `dlq` after attempt exhaustion or age stall — check [Task operations](https://docs.wednesdayai.dev/next/admin/gateway/tasks.md#shutdown-and-recovery).
* `openclaw.diagnostics_otel.content.truncated` counts content values shortened before export.
* `openclaw.diagnostics_otel.content.redacted` counts Telegram-sensitive values replaced before export.
* `openclaw.diagnostics_otel.handler.failures` counts exporter handler failures caught without crashing the gateway.
* `openclaw.session.system_prompt_flip` counts within-session system prompt hash changes. Each flip is a prefix-cache bust for the whole prompt, so a rising counter means something is rewriting the system prompt between turns. Runs that record the per-slot breakdown (ADR 2105) carry `slot` and a `owner` attribute normalized slot-aware (`core` or `plugin:<id>` for claim slots; `custom` for the additive `custom.*` slots and other free-form owners). See [Prompt caching](https://docs.wednesdayai.dev/next/developers/reference/prompt-caching.md#system-prompt-stability-metrics).
* `openclaw.session.system_prompt_stability` reports cumulative hash comparisons and flips as a gauge with `openclaw.stat="comparisons"` and `openclaw.stat="flips"` labels, refreshed on each diagnostic heartbeat. A healthy fleet trends toward `flips / comparisons` near zero.

## For developers

The implementation lives in `extensions/diagnostics-otel/src/service.ts`. Startup resolves shared and per-signal OTLP endpoints, derives Langfuse Basic auth only when the `langfuse` convenience block owns the endpoint, starts the NodeSDK for traces and metrics, and starts the logs exporter separately.

`hasPreloadedOtelSdk()` intentionally returns true only when `OPENCLAW_OTEL_PRELOADED === "1"`. It does not inspect OpenTelemetry globals, because importing `@opentelemetry/api` is not proof that another provider intentionally owns export. In preloaded mode, `diagnostics-otel` still subscribes to diagnostic events, but trace export belongs to the preloaded provider.

Model content is not reconstructed from saved conversation history. The run attempt builds a mutable `modelCallContext`, updates it from the final input transform after projection and `transform_llm_input` hooks, and the subscriber handlers emit `model.call.started` / `model.call.completed` diagnostic events from the assistant stream lifecycle. Keep new model-content fields optional and bounded at the diagnostic emitter before adding exporter mappings.

### Emitting task and dream diagnostics

Task and dream lifecycle events are members of the closed `DiagnosticEventPayload` union. Produce them with the SDK builders so type and `action` stay aligned; do not cast loose objects:

```ts theme={"dark"}
import { emitDiagnosticEvent, taskDiagnostic, dreamDiagnostic } from "wednesdayai/plugin-sdk";

emitDiagnosticEvent(
  taskDiagnostic("completed", {
    taskId: "my-plugin.my-pass",
    taskRunId,
    passId: "my-pass",
    durationMs,
  }),
);

emitDiagnosticEvent(
  dreamDiagnostic("pass_completed", {
    cycleFingerprint,
    stage: "n1",
    passId: "consolidate",
  }),
);
```

Builders are **input-shaped** (omit `ts` / `seq` — `emitDiagnosticEvent` stamps them). Live field names are part of the contract: use `passId` (not `passName`) and `attempts` (not `attempt`). `PassContext.emitDiagnostic` is typed as `DiagnosticEventInput`, so typos fail at compile time.

The OTel exporter handles every `task.*` and `dream.*` action in one `recordTaskDreamDiagnostic` path and starts a span named `openclaw.${evt.type}`. Unknown future types must not be silently dropped: the payload switch is exhaustiveness-checked (fail closed at compile time when a new variant is added without a case).

Also exported from `wednesdayai/plugin-sdk` for fetch facades: `GuardedFetchOptions` and `GuardedFetchResult` (alongside `fetchWithSsrFGuard`).

## Quick start

```json theme={"dark"}
{
  "diagnostics": {
    "otel": {
      "enabled": true,
      "langfuse": {
        "host": "https://langfuse.example.com",
        "publicKey": "pk-lf-...",
        "secretKey": "sk-lf-..."
      },
      "captureContent": {
        "enabled": true,
        "inputMessages": true,
        "outputMessages": true,
        "toolInputs": true,
        "toolOutputs": true
      }
    }
  }
}
```

Without raw `captureContent`, traces still carry safe summaries and hashes, but raw prompt,
completion, system prompt, tool argument, and tool schema payloads are omitted. Langfuse OTLP
ingest requires the `/api/public/otel` endpoint and Basic auth; the `langfuse` block derives
both from `host`, `publicKey`, and `secretKey`.

## Configuration reference

### `diagnostics.otel`

| Key | Type | Default | Description |
| - | - | - | - |
| `enabled` | boolean | false | Master switch |
| `endpoint` | string | — | Global OTLP endpoint for all signals. Derived from `langfuse.host` when absent |
| `tracesEndpoint` | string | — | Per-signal trace endpoint override; wins over shared endpoint and Langfuse derivation |
| `metricsEndpoint` | string | — | Per-signal metric endpoint override |
| `logsEndpoint` | string | — | Per-signal log endpoint override |
| `protocol` | `http/protobuf` \| `grpc` | `http/protobuf` | Transport protocol |
| `serviceName` | string | `openclaw` | OTel service name attribute |
| `traces` | boolean | true | Enable trace export |
| `metrics` | boolean | true | Enable metrics export |
| `logs` | boolean | false | Enable log export |
| `sampleRate` | 0.0–1.0 | 1.0 | Trace sample rate |
| `flushIntervalMs` | number | 5000 | Metric export interval |
| `captureContent` | boolean or object | false | Capture prompt/completion/tool content on spans |
| `runtime` | object | — | Runtime metrics collector config. See [Deep Observability](https://docs.wednesdayai.dev/next/admin/gateway/deep-observability.md) |
| `modelPricing` | object | — | Override or extend the built-in USD/M-token price table. Shape: `{provider: {model: {inputPerMToken, outputPerMToken}}}`. Used for `langfuse.trace.cost` attribution. See [Cost attribution](#cost-attribution) |
| `langfuse` | object | — | Langfuse convenience config (see below) |

### `captureContent` (object form)

```json theme={"dark"}
{
  "captureContent": {
    "enabled": true,
    "inputMessages": true, // user/assistant message content on model-call spans
    "outputMessages": true, // model output content
    "toolInputs": true, // tool call arguments
    "toolOutputs": true, // tool call results
    "systemPrompt": false, // system prompt text (often contains sensitive config)
    "toolDefinitions": false // tool schemas (bulky, rarely needed in traces)
  }
}
```

Set `captureContent: true` (bare boolean) to enable input messages, output messages, tool inputs, tool outputs, and tool definitions. It still leaves `systemPrompt` disabled; use object form when you need exact control. Content is character-capped at 128 KB per field; sensitive tokens are redacted via the configured `redactPatterns`.

When capture is disabled, model spans export `openclaw.model.*.summary` and `openclaw.model.*.hash`
attributes instead of raw content.

Captured model content maps to both Langfuse and OTel semantic attributes:

| Field group | Attributes |
| - | - |
| Root trace I/O | `langfuse.trace.input`, `langfuse.trace.output` |
| Generation I/O | `langfuse.observation.input`, `langfuse.observation.output`, `gen_ai.input.messages`, `gen_ai.output.messages` |
| System and tools | `gen_ai.system_instructions`, `gen_ai.tool.definitions` |
| Request metadata | `langfuse.observation.model.name`, `langfuse.observation.model.parameters`, `gen_ai.request.max_tokens`, `gen_ai.request.temperature`, `gen_ai.request.top_p`, `gen_ai.request.frequency_penalty`, `gen_ai.request.presence_penalty` |
| Response metadata | `gen_ai.response.id`, `gen_ai.response.model`, `gen_ai.response.finish_reasons`, `gen_ai.response.time_to_first_chunk`, `langfuse.observation.completion_start_time` |
| Usage and cost | `langfuse.observation.usage_details`, `langfuse.observation.cost_details`, `langfuse.trace.cost` (run-span total), `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.usage.cache_read_input_tokens`, `gen_ai.usage.cache_creation_input_tokens`, `gen_ai.usage.reasoning.output_tokens`, `gen_ai.usage.total_tokens` |

`langfuse.observation.cost_details` is emitted only when the runtime event includes real cost
details from the provider. `langfuse.trace.cost` is the accumulated USD cost for the run, set from
the built-in pricing table or a `diagnostics.otel.modelPricing` override; see [Cost attribution](#cost-attribution).
Response metadata
and time-to-first-chunk attributes are exported when the model-call event carries those provider
fields; the embedded message-stream path currently emits output content, usage, and any known cost
details from the runtime data available at `message_end`.

Retrieval and evaluation telemetry is not emitted yet. Those fields will be added only after
real WednesdayAI retrieval or evaluation events exist.

### `langfuse` convenience block

```json theme={"dark"}
{
  "langfuse": {
    "host": "https://langfuse.example.com",
    "publicKey": "pk-lf-...",
    "secretKey": "sk-lf-...",
    "filterInfraSpans": true
  }
}
```

| Key | Type | Default | Description |
| - | - | - | - |
| `host` | string | — | Langfuse base URL. Derives the OTLP endpoint and Basic-auth header automatically |
| `publicKey` | string | — | Langfuse public key (`pk-lf-…`) |
| `secretKey` | string | — | Langfuse secret key (`sk-lf-…`) |
| `filterInfraSpans` | boolean | `true` | Drop infrastructure/polling spans (Telegram poll, webhook, account-state) from the Langfuse-bound OTLP exporter. Set to `false` to send all spans to Langfuse |
| `sessionSpans` | boolean | `true` | Group runs sharing a `sessionKey` under a parent `openclaw.session` span. Enables the Langfuse session view (per-run costs are visible under the session). See [Session spans](#session-spans) |
| `sessionIdleTimeoutMs` | number | `1800000` | Close the session span after this many ms of inactivity (no new run starts or completions). Timer resets on each run start and completion. Default is 30 minutes |

When `langfuse` is set and no explicit `endpoint` is configured, the plugin derives the global OTLP endpoint as `{host}/api/public/otel` and generates the required Basic-auth header. `host` must be a bare `http` or `https` URL without query string or fragment.

Signal-specific endpoint overrides continue to work independently. If `tracesEndpoint`, `metricsEndpoint`, or `logsEndpoint` is set, that signal routes to the configured target and does not inherit Langfuse auth.

#### `filterInfraSpans`

When `true` (the default), a span-level filter wraps the Langfuse OTLP exporter. Any span carrying `langfuse.trace.name` in the `telegram.*` / `openclaw.telegram.*` namespace, or `ATTR_LANGFUSE_OBSERVATION_TYPE` matching an infrastructure pattern, is dropped before export. All other OTLP backends (Tempo, custom `endpoint`) are unaffected — infrastructure spans still flow there.

Set `filterInfraSpans: false` when you want the full raw span stream in Langfuse, for example during a debugging session where Telegram polling latency matters.

```json theme={"dark"}
{
  "diagnostics": {
    "otel": {
      "langfuse": {
        "host": "https://langfuse.example.com",
        "publicKey": "pk-lf-...",
        "secretKey": "sk-lf-...",
        "filterInfraSpans": false
      }
    }
  }
}
```

## Endpoint precedence

For each signal, the plugin resolves the first available endpoint in this order:

| Signal | Config key | Env override | Fallbacks |
| - | - | - | - |
| Traces | `diagnostics.otel.tracesEndpoint` | `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | `diagnostics.otel.endpoint`, `langfuse.host`, shared `OTEL_EXPORTER_OTLP_ENDPOINT` |
| Metrics | `diagnostics.otel.metricsEndpoint` | `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` | `diagnostics.otel.endpoint`, `langfuse.host`, shared `OTEL_EXPORTER_OTLP_ENDPOINT` |
| Logs | `diagnostics.otel.logsEndpoint` | `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` | `diagnostics.otel.endpoint`, `langfuse.host`, shared `OTEL_EXPORTER_OTLP_ENDPOINT` |

`OTEL_SDK_DISABLED=true` can disable OpenTelemetry SDK behavior. `OPENCLAW_OTEL_PRELOADED=1` tells this extension not to start its own NodeSDK; use it only when another provider has already been configured to export the same signals.

## Trace hierarchy

Spans are structured as a nested tree matching the actual execution path. When `langfuse.sessionSpans` is enabled (the default), turns that share a `sessionKey` are grouped under a parent `openclaw.session` trace root:

```bash theme={"dark"}
openclaw.session                 (one per session key; groups multi-turn conversations)
  openclaw.run                   (one per agent turn)
    openclaw.model.call          (one per LLM invocation)
      execute_tool {name}        (one per tool call, zero or more)
      execute_tool {name}
      openclaw.tool.loop         (loop detector warning/block, if emitted)
    openclaw.model.call          (second invocation if tools triggered a follow-up)
      execute_tool {name}
    openclaw.session_run.running (short lifecycle spans when run ids are available)

# Separate roots (not nested under openclaw.session) — background task / dream lifecycle:
openclaw.task.started | completed | failed | preempted | …
openclaw.task.recovery_exhausted   # error status; boot recovery → dlq
openclaw.dream.cycle_started | cycle_completed | pass_completed | …
```

When `langfuse.sessionSpans: false`, the `openclaw.session` layer is omitted and each `openclaw.run` is its own trace root (the behaviour before Phase 3).

This nesting is built by the plugin using internal span context maps. Tool spans that arrive after `message_end` (common in streaming) are correctly parented to the model-call span, not the run span. If no model call is active, loop spans fall back to the retained run context.

## Task and dream spans

When the task pipeline or Dream Cycle emits diagnostics, `diagnostics-otel` exports a short span per event. Span name is `openclaw.<event.type>` (for example `openclaw.task.completed`, `openclaw.dream.cycle_started`).

**Task attributes** (when present on the event):

| Attribute | Source field |
| - | - |
| `openclaw.task.action` | `action` |
| `openclaw.task.id` | `taskId` |
| `openclaw.task.run_id` | `taskRunId` |
| `openclaw.task.stage` | `stage` |
| `openclaw.task.attempts` | `attempts` |
| `openclaw.task.reason` | `reason` |
| `openclaw.task.error` | `error` |
| `openclaw.task.duration_ms` | `durationMs` |

**Dream attributes** (when present):

| Attribute | Source field |
| - | - |
| `openclaw.dream.action` | `action` |
| `openclaw.dream.cycle_fingerprint` | `cycleFingerprint` |
| `openclaw.dream.stage` | `stage` |
| `openclaw.dream.pass_id` | `passId` |
| `openclaw.dream.reason` | `reason` |
| `openclaw.dream.cap` | `cap` |

**Error span status** is set for: `task.failed`, `task.recovery_exhausted`, `task.boundary_violation`, `dream.contemplation_failed`, `dream.fetch_failed`.

**Task actions exported:** `started`, `completed`, `failed`, `preempted`, `budget_deferred`, `budget_ceiling_reached`, `preemption_ceiling_reached`, `boundary_violation`, `pipeline_disabled`, `idle_monitor_disabled`, `plugin_pass_rejected`, `runlog_retention`, `recovery_exhausted`.

**Dream actions exported:** `cycle_started`, `cycle_completed`, `pass_completed`, `stage_skipped`, `contemplation_unconfigured`, `contemplation_skipped`, `contemplation_failed`, `rate_capped`, `jsonl_parse_skipped`, `fetch_failed`, `crystallisation_candidate`.

## Session spans

When `langfuse.sessionSpans` is `true` (default), the plugin maintains one `openclaw.session` trace-root span per active `sessionKey`. All `openclaw.run` spans for that session are children of it.

* **One session span per session key** — the span is created on the first run for a given session and reused across subsequent turns.
* **`langfuse.trace.name`** is set to `session:<8hex>` on the session span (the Langfuse trace root); `openclaw.run` child spans omit it so they surface as observations under the session, not as separate traces.
* **Idle timeout** — the session span closes automatically after `sessionIdleTimeoutMs` of inactivity (default 30 minutes). The timer resets on each run start and run completion. After the span closes, the next run for that session key opens a new one.
* **Gateway stop** — all open session spans are ended synchronously on gateway shutdown; no graceful drain is performed.
* **Cost roll-up** — `langfuse.trace.cost` is set on each run span to the accumulated cost for that run (summed across model calls). Langfuse aggregates these per-run costs in its session view. There is no cross-run cost attribute on the session span itself. See [Cost attribution](#cost-attribution).

To disable session spans entirely: set `diagnostics.otel.langfuse.sessionSpans: false`.

## Cost attribution

The plugin computes USD cost for each `openclaw.run` and sets `langfuse.trace.cost` on the run span when cost is attributable. Cost is accumulated additively across model calls; the final accumulated value is written once at run completion.

**Built-in pricing table** (USD per million tokens):

| Provider | Model | Input | Output |
| - | - | - | - |
| anthropic | claude-opus-4-8 | \$15 | \$75 |
| anthropic | claude-sonnet-4-6 | \$3 | \$15 |
| anthropic | claude-haiku-4-5 | \$0.80 | \$4 |
| anthropic | claude-fable-5 | \$3 | \$15 |
| anthropic | claude-3-5-sonnet | \$3 | \$15 |
| anthropic | claude-3-5-haiku | \$0.80 | \$4 |
| anthropic | claude-3-opus | \$15 | \$75 |
| anthropic | claude-3-haiku | \$0.25 | \$1.25 |
| openai | gpt-4o | \$2.50 | \$10 |
| openai | gpt-4o-mini | \$0.15 | \$0.60 |
| google | gemini-2.5-pro | \$1.25 | \$10 |
| google | gemini-2.5-flash | \$0.15 | \$0.60 |
| google | gemini-2.0-flash | \$0.10 | \$0.40 |
| google | gemini-1.5-pro | \$1.25 | \$5 |
| google | gemini-1.5-flash | \$0.075 | \$0.30 |

Date-suffix and region-prefix variants (e.g. `claude-sonnet-4-6-20250622`, `us.claude-opus-4-8`) are normalised before lookup.

**Fallback cost** — when the provider event includes token counts but no cost details, the plugin computes fallback cost from the built-in table. When no pricing entry exists for the provider/model, cost is not set and the run span omits `langfuse.trace.cost`.

**Override / extend** — add entries for unlisted models via `diagnostics.otel.modelPricing`:

```json theme={"dark"}
{
  "diagnostics": {
    "otel": {
      "modelPricing": {
        "openai": {
          "o3": { "inputPerMToken": 10, "outputPerMToken": 40 }
        }
      }
    }
  }
}
```

`modelPricing` overrides win over the built-in table for the same provider/model key.

`langfuse.trace.cost` is set only when the accumulated run cost is greater than zero. Zero-cost runs (no model calls, or all model calls with unknown pricing) omit the attribute.

## Langfuse session grouping

Each run span carries:

| Attribute | Value | Purpose |
| - | - | - |
| `gen_ai.thread.id` | `runId` | Groups runs into a Langfuse trace |
| `langfuse.session.id` | session key | Groups traces into a Langfuse session |
| `langfuse.user.id` | user ID derived from session key | User-level filtering |
| `openclaw.sessionKey` | hashed session key | Privacy-safe debug reference |

In Langfuse: each agent run appears as one trace; all runs within a conversation session are grouped under one session view.

## Telegram fleet diagnostics

Telegram fleet supervision emits channel-specific diagnostic events for account
lifecycle, polling, webhook intake, Bot API calls, update handling, and delivery.
The OTel exporter maps these events to spans named `openclaw.telegram.<event>`
and to Telegram-specific metrics.

Event names:

* `telegram.account.state`
* `telegram.api.request`
* `telegram.poll.started`
* `telegram.poll.succeeded`
* `telegram.poll.failed`
* `telegram.poll.backoff`
* `telegram.poll.conflict`
* `telegram.webhook.received`
* `telegram.webhook.rejected`
* `telegram.webhook.reconciled`
* `telegram.update.received`
* `telegram.update.deduped`
* `telegram.update.queued`
* `telegram.update.processed`
* `telegram.delivery.queued`
* `telegram.delivery.sent`
* `telegram.delivery.failed`
* `telegram.delivery.flood_wait`

Representative attributes:

| Attribute | Purpose |
| - | - |
| `openclaw.channel = "telegram"` | Marks Telegram-correlated telemetry |
| `telegram.event.group` / `telegram.event.action` | Groups events by account, API, poll, webhook, update, delivery |
| `telegram.account.hash` / `telegram.bot.hash` | Stable account/bot identity without raw tokens or IDs |
| `telegram.chat.hash` / `telegram.user.hash` | Stable actor/location identity without raw Telegram IDs |
| `telegram.thread.hash` | Stable topic/thread identity when present |
| `telegram.update.id` | Telegram update identifier where safe |
| `telegram.account.state` / `mode` | Fleet account lifecycle state |
| `telegram.api.method` / `telegram.http.status_code` | Bot API method and HTTP result |
| `telegram.retry_after_ms` / `telegram.next_retry_at` | Flood-wait and retry timing |
| `langfuse.session.id` | Stable Telegram session grouping |
| `langfuse.user.id` | Derived from the hashed Telegram actor when available |

Privacy behavior:

* Raw bot tokens, raw chat IDs, raw user IDs, raw thread IDs, raw webhook
  secrets, raw webhook URLs, and Telegram message content are not exported.
* `captureContent` does not override Telegram redaction for Telegram-correlated
  root trace I/O, model input/output, system prompts, tool definitions, tool
  spans, captured tool input, output, blocked reason, and status text. These
  fields are replaced with `telegram-content-redacted`.
* OTLP log records that look Telegram-shaped use a redacted body and drop
  aggregate argument fields that could contain Telegram IDs or message content.

## Splitting traces and metrics to different backends

To send traces to Langfuse and metrics/logs to a local OTel collector:

```json theme={"dark"}
{
  "diagnostics": {
    "otel": {
      "enabled": true,
      "metricsEndpoint": "http://otel-collector:4318",
      "logsEndpoint": "http://otel-collector:4318",
      "langfuse": {
        "host": "https://langfuse.example.com",
        "publicKey": "pk-lf-...",
        "secretKey": "sk-lf-..."
      }
    }
  }
}
```

Traces flow to Langfuse (derived from `langfuse.host`); metrics and logs flow to the local collector via their signal-specific endpoints.

## Troubleshooting

### No traces arrive in Langfuse

1. Confirm the startup summary contains `tracesEnabled=true`, the expected sanitized `traceUrl`, `traceHeaders=Authorization=present` for Langfuse, and `nodeSdk=started`.
2. If it says `nodeSdk=skipped-preloaded`, check why `OPENCLAW_OTEL_PRELOADED=1` is in the gateway environment. The preloaded provider must be configured to export traces to Langfuse.
3. If `otelSdkDisabled=true`, remove or override `OTEL_SDK_DISABLED=true` for the gateway service.
4. Check per-signal environment variables. `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` can route traces away from the configured Langfuse endpoint.
5. If `wednesdayai --version` is current but the extension behavior is stale, check for a managed copy under `~/.openclaw/extensions/diagnostics-otel` and reinstall or refresh it.

### Empty input/output in Langfuse

Trace header input/output is now populated even when `captureContent` is off:

* **Summaries present, raw content missing** — this is expected when `captureContent` is disabled. The trace header shows message-count and role summaries. Enable `captureContent` if you need raw prompt and completion text.
* **Both trace header and generation fields are empty** — the run may have completed before any model call occurred (for example, a routing or command-only turn with no LLM invocation). In that case, the trace header is synthesized from the channel and session identity. Check the `openclaw.run` span attributes for `openclaw.channel` and `openclaw.sessionKeyHash` to confirm the run identity.
* **Trace header truly blank** — confirm the extension version is ≥ 0.4.7 (Phase 3 / Langfuse trace gap fixes). Older builds did not promote summaries to the root span.

### Traces arrive but no session grouping

Check that `gen_ai.thread.id` and `langfuse.session.id` appear on root run spans. These require the `diagnostics-otel` extension to be enabled and on version ≥ 0.4.4 (Phase 2). If attributes are missing, confirm `diagnostics.otel.traces: true` and the extension is active via `wednesdayai gateway status --deep`.

### Tool spans appear as siblings, not children of model-call

This is fixed in ≥ 0.4.4 (Phase 2). In older versions, the `activeModelCallContexts` map was cleared before tool dispatch. Update the installed extension.

### Runs appear as separate traces instead of nested under a session

Confirm `diagnostics.otel.langfuse.sessionSpans` is not set to `false` and the extension version is ≥ 0.4.7 (Phase 3). In older builds, the `openclaw.session` span layer did not exist; each run was its own trace root. Also verify `langfuse.session.id` appears on `openclaw.run` spans — if it is missing, the extension may not be receiving a `sessionKey` from the gateway.

### No `langfuse.trace.cost` on run spans

Cost attribution requires a matching entry in the built-in pricing table or a `diagnostics.otel.modelPricing` override for the provider/model in use. If the model normalises (date-suffix stripped) to a key in the table but cost is still absent, check that `accumulatedCost > 0` — zero-cost runs omit the attribute. The cost panel in the Grafana dashboard requires `openclaw_cost_usd_total` series; confirm the OTLP metrics endpoint is reachable and metrics export is enabled.

### Zero token counts on spans

Fixed in ≥ 0.4.4. Older builds wrote `0` when token fields were absent; current builds omit the attribute instead.

## Metrics

The plugin exports the following OTel metrics:

| Metric | Type | Description |
| - | - | - |
| `openclaw.run.outcome` | counter | Run completions by outcome (`openclaw.outcome`: `ok` or `error`) |
| `gen_ai.client.token.usage` | histogram | Tokens per signal (input/output) |
| `gen_ai.client.operation.duration` | histogram | Model-call duration |
| `openclaw.session.run.active` | gauge | Durable session runs currently marked running |
| `openclaw.session.run.interrupted` | counter | Session runs interrupted by gateway restart |
| `openclaw.session.run.recovery.decisions` | counter | Recovery decisions made for interrupted session runs |
| `openclaw.session.run.recovery.resumed` | counter | Interrupted runs resumed via continue decision |
| `openclaw.session.run.recovery.cancelled` | counter | Interrupted runs cancelled via cancel decision |
| `openclaw.session.system_prompt_flip` | counter | System prompt hash flips between consecutive turns of a session; per-slot flips add `slot` and a normalized `owner` (`core`, `plugin:<id>`, or `custom`) |
| `openclaw.session.system_prompt_stability` | gauge | Cumulative hash comparisons and flips (`openclaw.stat` label) |
| `openclaw.diagnostics_otel.content.truncated` | counter | Content values truncated before export |
| `openclaw.diagnostics_otel.content.redacted` | counter | Content values redacted before export |
| `openclaw.diagnostics_otel.handler.failures` | counter | Diagnostic event handler failures caught by the exporter |
| `session.append.error_count` | counter | Session DB append errors |
| `session.drain.duration_ms` | histogram | Write-queue drain duration |
| `openclaw.telegram.events` | counter | Telegram lifecycle, polling, webhook, update, and delivery events |
| `openclaw.telegram.api.duration_ms` | histogram | Telegram Bot API request duration |
| `openclaw.telegram.retry_after_ms` | histogram | Telegram retry-after and flood-wait delays |
| `openclaw.telegram.webhook.pending_updates` | histogram | Telegram webhook pending-update count |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.