Skip to main content

Grafana and Prometheus setup

This guide wires a WednesdayAI gateway running the diagnostics-otel plugin into Grafana and Prometheus so you get error-rate, latency, token, cost, and session dashboards plus alerting. WednesdayAI pushes metrics over OTLP — there is no scrape-only /metrics endpoint, so Prometheus needs an OTLP receiver (or a collector in front of it). For trace export and the full OTel configuration reference, see OTel and Langfuse diagnostics. For log export over OTLP, see Logging.

Before you begin

  • The diagnostics-otel plugin enabled with metrics export on (diagnostics.otel.metrics: true, the default when OTel is enabled).
  • An OTLP-compatible collector (or Prometheus ≥ 2.47 with its built-in OTLP receiver) reachable from the gateway host.
  • Grafana 10 or newer with a Prometheus data source.

Export metrics

Set the OTLP endpoint in ~/.openclaw/openclaw.json so the plugin exports metrics. OTLP/HTTP uses port 4318:
Restart the gateway for this to take effect (Linux: systemctl --user restart openclaw-gateway; macOS: openclaw gateway restart --deep). Point Prometheus at the collector (or expose the OTLP receiver on Prometheus itself):
Metric names are emitted in OpenTelemetry dotted form and surface in Prometheus with dots converted to underscores: openclaw.run.outcome becomes openclaw_run_outcome_total.

Import the dashboard

A ready-made dashboard ships in the WednesdayAI source tree at extensions/diagnostics-otel/dashboards/grafana-otel.json (uid wednesdayai-otel).
1

Open Grafana import

In Grafana choose Dashboards → New → Import and upload the grafana-otel.json file.
2

Select the data source

Select your Prometheus data source for the DS_PROMETHEUS variable.
The dashboard covers:

Import the alert rules

The alert rules ship at extensions/diagnostics-otel/alerts/prometheus-rules.yaml (group wednesdayai-otel). Add the file to your Prometheus rule_files and reload Prometheus.

Cost panel prerequisites

The cost-per-hour panel and the OtelCostOverrun alert require USD cost attribution (the openclaw_cost_usd_total series). Cost is computed from the diagnostics-otel built-in pricing table or a diagnostics.otel.modelPricing override — see Cost attribution. If your provider/model has no pricing entry, those panels read empty and the cost alert never fires; the rest of the dashboard is unaffected.

Tune alert thresholds

The OtelCostOverrun threshold of 10 USD per hour is a placeholder. Retune it for your workload in your Prometheus rules or Alertmanager configuration before relying on it.

Troubleshooting

  1. Confirm metrics export is on: the gateway startup log line should include metricsEnabled=true.
  2. Confirm the collector receives data: curl -v http://collector-host:4318/v1/metrics from the gateway host.
  3. Confirm Prometheus can query the collector’s output for openclaw_run_outcome_total.
The model in use has no entry in the built-in pricing table. Add one via diagnostics.otel.modelPricing and restart the gateway.
Prometheus converts OTel dots to underscores and adds _total for counters. Query openclaw_tokens_total, not openclaw.tokens.