Skip to main content

Realtime Voice gateway relay

The realtime voice relay streams two-way audio between authenticated clients and a registered provider. The Gateway owns authentication, connection lifecycle, audio framing, tool-call routing, and bounded session state. The currently bundled production provider is OpenAI Realtime; compatible plugins can register other providers. For the web client’s voice workflow, see Using the web client. For provider development, see Realtime Voice Provider API.
This page covers the gateway relay behind the realtime voice providers. The classic console’s Talk tab, which used it, was removed together with the classic console (ADR 1771, dev-docs/adr/1771-classic-console-removal-control-ui-key.md). The /v2/client web client’s Dictate control uses a separate, cascaded design (streaming speech-to-text, then an agent turn, then streaming text-to-speech), and its Voice mode control runs that cascaded loop or - when the operator sets talk.conversation.engine: "realtime" - the same realtime providers through session-bound talk.session.* methods. See Voice conversation sessions.

Requirements

  • A registered realtime voice provider. Enable the bundled OpenAI provider with wednesdayai plugins enable openai-realtime.
  • A nonblank provider key. OpenAI resolves it from talk.providers.openai-realtime.apiKey, plugin defaults, or OPENAI_API_KEY.
  • operator.read for status; operator.write for session control and media methods.
  • operator.admin when consultToolPolicy: "owner" must be honored.
Restart the Gateway after enabling or disabling a plugin or changing an environment variable. On Linux, run systemctl --user restart openclaw-gateway. On macOS, run wednesdayai gateway restart --deep. The provider key stays on the Gateway host. The browser sends audio over its existing authenticated Gateway WebSocket and never receives the provider key.

Enable browser Talk

talk.realtime.enabled defaults to false. Set it to true to expose Talk in the Control UI and report it as enabled in status:
Changes under talk.* apply without a restart to new sessions. Existing sessions retain the configuration resolved when they started. Changes under plugins.* and environment variable changes require a Gateway restart.
talk.realtime.enabled is a UI rollout gate, not an API kill switch. Direct callers with operator.write can still create sessions when it is false. Revoke write scope or disable the provider plugin when you must prevent API access.

OpenAI provider defaults

OpenAI uses gateway relay only and PCM16 signed little-endian mono audio at 24 kHz. The browser has no provider picker; it uses the configured available provider selected by the Gateway. For faster turn detection, test a threshold of 0.58, prefix padding of 300 ms, and a silence duration of 320 ms:

Configure agent consult access

For gateway-relay sessions, the Gateway may expose its reserved openclaw_agent_consult tool to the voice model. The browser never executes this tool. owner is honored only for a connection with operator.admin. Otherwise the Gateway silently downgrades that session to safe-read-only and logs a bounded warning. Message tools and plugin hooks remain disabled under every policy. The reserved tool name cannot be replaced by a client tool, including when the policy is none, and client results for Gateway-owned calls are rejected.
owner consults can use action-taking and owner-only tools, so real changes can happen. The temporary consult session does not preview or undo those actions. Use safe-read-only unless the authenticated operator explicitly needs this access.

Consult privacy and cancellation

A consult can include up to 12 recent final transcript entries and sends them to the configured default agent/model provider, which may differ from the voice provider. The temporary local session file is deleted after the run, but that does not change the model provider’s data-handling terms. The runtime requests cancellation after 30 seconds and cleans up after the embedded runner settles. This is not a hard completion bound for a non-cooperative runner. An accepted interruption cancels an in-flight consult without producing a stray spoken follow-up.

Permissions and methods

owner consult additionally requires explicit operator.admin authorization.

Relay events

Gateway-relay clients receive talk.realtime.relay events: Every pushed mark includes markName. The relay forwards mark acknowledgements to the active provider; the bundled OpenAI provider treats unknown, duplicate, cleared, and old-generation marks as inert. An error event is terminal and is followed by close.

Transport modes

The Gateway-owned consult tool is available only on relay sessions, including auto sessions that fell back to relay.

Session limits

  • Fixed 30-minute TTL per session.
  • Two active sessions per authenticated connection.
  • 64 active sessions globally.
  • Expired sessions close as completed; disconnected-client sessions close as cancelled.

Playback acknowledgements and interruption

Clients acknowledge provider marks only after the associated audio has drained to audible playback. A normal interruption with the bundled OpenAI provider requires at least 250 ms of acknowledged audio. force: true bypasses the duration floor and acknowledgement/echo guards; an assistant output item must have started, but no playback acknowledgement is required. audioPlaybackActive is a provider-defined hint. The bundled OpenAI provider currently ignores it, so it does not replace playback acknowledgements. An accepted interruption emits one clear event with reason barge-in. OpenAI can accept it after response generation has ended when acknowledged audio remains queued for playback.

Tool calls

The relay intercepts openclaw_agent_consult; every other provider tool call is sent to the client. A client can return:
suppressResponse: true submits the result without requesting another voice response. willContinue: true keeps the call ID in the client-owned set so the client can submit another result, but provider support for multiple results is not guaranteed. The bundled OpenAI provider expects one output per call ID.

Session observability

talk.realtime.status reports enabled, available, configured provider IDs, and active sessions with durationMs, audio byte counters, and accepted bargeInCount. These counters exist only in memory for currently active sessions; they are not durable metrics.
startedAt is Unix time in seconds. bargeInCount counts accepted playback clears, not every interruption attempt. Status covers active sessions across authenticated connections. Follow logs with:
The gateway/talk-realtime-relay subsystem emits these structured lifecycle messages: Provider errors are terminal: the relay sends the bounded error message to the client and then closes the session with reason error.

Troubleshooting

Talk is unavailable

If the client reports UNAVAILABLE: No realtime voice provider registered, verify the plugin is enabled. Also verify the API key resolves to a nonblank value and talk.realtime.status includes openai-realtime in configuredProviderIds. Restart after plugin or environment changes.

An owner consult runs read-only

Confirm the connection token has operator.admin. The configured value alone is not authorization; otherwise the Gateway deliberately downgrades it.

Interruptions are ignored

Verify the client acknowledges marks after actual playback. For browser echo problems, use a headset before changing VAD settings. audioPlaybackActive alone is insufficient for the bundled OpenAI provider.

Sessions close immediately

Check relay session error, verify the model and voice are available to the OpenAI account, and confirm the session limits are not exhausted.