Realtime Voice gateway relay
The realtime voice relay streams two-way audio between authenticated clients and a registered provider. The Gateway owns authentication, connection lifecycle, audio framing, tool-call routing, and bounded session state. The currently bundled production provider is OpenAI Realtime; compatible plugins can register other providers. For the web client’s voice workflow, see Using the web client. For provider development, see Realtime Voice Provider API.This page covers the gateway relay behind the realtime voice providers. The classic console’s
Talk tab, which used it, was removed together with the classic console (ADR 1771,
dev-docs/adr/1771-classic-console-removal-control-ui-key.md). The /v2/client web client’s
Dictate control uses a separate, cascaded design (streaming speech-to-text, then an agent
turn, then streaming text-to-speech), and its Voice mode control runs that cascaded loop
or - when the operator sets talk.conversation.engine: "realtime" - the same realtime providers
through session-bound talk.session.* methods. See
Voice conversation sessions.Requirements
- A registered realtime voice provider. Enable the bundled OpenAI provider with
wednesdayai plugins enable openai-realtime. - A nonblank provider key. OpenAI resolves it from
talk.providers.openai-realtime.apiKey, plugin defaults, orOPENAI_API_KEY. operator.readfor status;operator.writefor session control and media methods.operator.adminwhenconsultToolPolicy: "owner"must be honored.
systemctl --user restart openclaw-gateway. On macOS, run
wednesdayai gateway restart --deep.
The provider key stays on the Gateway host. The browser sends audio over its existing
authenticated Gateway WebSocket and never receives the provider key.
Enable browser Talk
talk.realtime.enabled defaults to false. Set it to true to expose Talk in the
Control UI and report it as enabled in status:
talk.* apply without a restart to new sessions. Existing sessions retain
the configuration resolved when they started. Changes under plugins.* and environment
variable changes require a Gateway restart.
OpenAI provider defaults
OpenAI uses gateway relay only and PCM16 signed little-endian mono audio at 24 kHz. The
browser has no provider picker; it uses the configured available provider selected by the
Gateway.
For faster turn detection, test a threshold of
0.58, prefix padding of 300 ms, and a
silence duration of 320 ms:
Configure agent consult access
For gateway-relay sessions, the Gateway may expose its reservedopenclaw_agent_consult tool to the voice model. The browser never executes this tool.
owner is honored only for a connection with operator.admin. Otherwise the Gateway
silently downgrades that session to safe-read-only and logs a bounded warning. Message
tools and plugin hooks remain disabled under every policy. The reserved tool name cannot
be replaced by a client tool, including when the policy is none, and client results for
Gateway-owned calls are rejected.
Consult privacy and cancellation
A consult can include up to 12 recent final transcript entries and sends them to the configured default agent/model provider, which may differ from the voice provider. The temporary local session file is deleted after the run, but that does not change the model provider’s data-handling terms. The runtime requests cancellation after 30 seconds and cleans up after the embedded runner settles. This is not a hard completion bound for a non-cooperative runner. An accepted interruption cancels an in-flight consult without producing a stray spoken follow-up.Permissions and methods
owner consult additionally requires explicit operator.admin authorization.
Relay events
Gateway-relay clients receivetalk.realtime.relay events:
Every pushed
mark includes markName. The relay forwards mark acknowledgements to the
active provider; the bundled OpenAI provider treats unknown, duplicate, cleared, and
old-generation marks as inert. An error event is terminal and is followed by close.
Transport modes
The Gateway-owned consult tool is available only on relay sessions, including
auto
sessions that fell back to relay.
Session limits
- Fixed 30-minute TTL per session.
- Two active sessions per authenticated connection.
- 64 active sessions globally.
- Expired sessions close as
completed; disconnected-client sessions close ascancelled.
Playback acknowledgements and interruption
Clients acknowledge provider marks only after the associated audio has drained to audible playback. A normal interruption with the bundled OpenAI provider requires at least 250 ms of acknowledged audio.force: true bypasses the duration floor and acknowledgement/echo
guards; an assistant output item must have started, but no playback acknowledgement is
required.
audioPlaybackActive is a provider-defined hint. The bundled OpenAI provider currently
ignores it, so it does not replace playback acknowledgements.
An accepted interruption emits one clear event with reason barge-in. OpenAI can accept
it after response generation has ended when acknowledged audio remains queued for playback.
Tool calls
The relay interceptsopenclaw_agent_consult; every other provider tool call is sent to
the client. A client can return:
suppressResponse: true submits the result without requesting another voice response.
willContinue: true keeps the call ID in the client-owned set so the client can submit
another result, but provider support for multiple results is not guaranteed. The bundled
OpenAI provider expects one output per call ID.
Session observability
talk.realtime.status reports enabled, available, configured provider IDs, and active
sessions with durationMs, audio byte counters, and accepted bargeInCount. These counters
exist only in memory for currently active sessions; they are not durable metrics.
startedAt is Unix time in seconds. bargeInCount counts accepted playback clears, not
every interruption attempt. Status covers active sessions across authenticated connections.
Follow logs with:
gateway/talk-realtime-relay subsystem emits these structured lifecycle messages:
Provider errors are terminal: the relay sends the bounded error message to the client and
then closes the session with reason
error.
Troubleshooting
Talk is unavailable
If the client reportsUNAVAILABLE: No realtime voice provider registered, verify the
plugin is enabled. Also verify the API key resolves to a nonblank value and
talk.realtime.status includes openai-realtime in configuredProviderIds. Restart after
plugin or environment changes.
An owner consult runs read-only
Confirm the connection token hasoperator.admin. The configured value alone is not
authorization; otherwise the Gateway deliberately downgrades it.
Interruptions are ignored
Verify the client acknowledges marks after actual playback. For browser echo problems, use a headset before changing VAD settings.audioPlaybackActive alone is insufficient for the
bundled OpenAI provider.
Sessions close immediately
Checkrelay session error, verify the model and voice are available to the OpenAI account,
and confirm the session limits are not exhausted.