Realtime Voice Provider API
Reference for authors implementing aRealtimeVoiceProviderPlugin — the contract
the Gateway uses to bridge client audio to a voice backend.
Building a streaming speech-to-text or streaming text-to-speech provider for the cascaded
stt-tts/Dictate conversation loop instead? Those use two different plugin SDK contracts,
RealtimeTranscriptionProviderPlugin and StreamingSpeechProviderPlugin — see
Register realtime voice, STT and TTS providers
and Voice conversation sessions. All three provider
kinds, plus media-understanding providers, share one id-registrar diagnostic for duplicate and
missing ids (src/plugins/registry.ts).Plugin registration
Register your provider in the plugin entry file:register(). Resolve remote
credentials and open provider connections later in resolveConfig() or connect().
The provider object must implement RealtimeVoiceProviderPlugin:
SDK imports
All types and helpers export fromwednesdayai/plugin-sdk (openclaw/plugin-sdk is the same surface):
RealtimeVoiceBridgeCallbacks
The exact exported contract your bridge invokes (onAudio and onClearAudio are
required; everything else is optional):
RealtimeVoiceAudioClearReason is currently just "barge-in" — pass it when the clear
is caused by an accepted interruption, or omit it for any other buffer clear.
onToolCall may return void or a Promise<void>. onTranscript’s metadata
(itemId, responseId, contentIndex, itemOrder, status) lets the relay stitch
together partial/final transcript ordering across concurrent response items — omit
fields you cannot supply rather than guessing.
onError is treated as terminal by the current gateway relay. Firing it broadcasts
a talk.realtime.relay error event to the client and then always closes the session
with reason "error" (see Close reasons and
Relay events).
There is no path in the shipped relay where firing onError leaves the session open.
The gateway relay (src/gateway/talk-realtime-relay.ts) also does not wire onEvent
through to anything client-visible for gateway-relay sessions, so it is not a working
substitute either — the shipped relay currently has no non-terminal, session-continuing
error channel for providers.
All callbacks must be deferred until after createBridge() returns (see
Callback timing invariant below) — this applies to every
callback in this contract, including onError.
RealtimeVoiceBridgeCreateRequest
The argument to createBridge():
See
RealtimeVoiceBridgeCallbacks above for the exact
signature of every callback field in this table.
cfg is not currently populated on this createBridge() request for gateway-relay sessions
(src/talk/session-runtime.ts omits it) — treat it as undefined in createBridge(). It is
supplied to isConfigured() and resolveConfig() (src/talk/provider-resolver.ts) and to
createBrowserSession() (src/gateway/server-methods/talk.ts), so read the full config there
instead if your provider needs it.
RealtimeVoiceBridge interface
Your createBridge() return value must implement:
acknowledgeMark is called when the client confirms a playback mark you emitted via onMark.
markName echoes the opaque name of the mark being acknowledged. For a gateway-relay realtime
session, playbackTimestamp, when provided, must be a finite, nonnegative, non-regressing integer
on the client’s media clock. That clock is not the cascaded stt-tts mark clock: talk.session.mark
timestamp is milliseconds of audio played for that turn (0 at the first speech sample). Do not
pass performance.now() as an stt-tts speech-mark timestamp. See
Talk sessions.
RealtimeVoiceBargeInOptions is { audioPlaybackActive?: boolean; force?: boolean; speechDetectedAt?: number; playbackPausedAt?: number }.
The last two are client-local performance.now() milliseconds from talk.session.bargeIn and are
only passed for stt-tts sessions. A realtime bridge can ignore them. force bypasses the normal
duration, acknowledgement, and echo guards.
Playback acknowledgement contract
New clients must acknowledge a named mark only after all audio through that mark has drained to audible playback. Never acknowledge when audio is merely received, decoded, or scheduled. Treat mark names as opaque: echo the exact value and never parse or synthesize one. The bundled OpenAI bridge treats named acknowledgements as cumulative boundaries through the matching outstanding mark. An unnamed acknowledgement consumes one FIFO boundary and exists for legacy compatibility only. Unknown, old-generation, cleared, duplicate, fractional, or regressing acknowledgements are inert. The bridge computes truncation from acknowledged PCM16 bytes at 48 bytes per millisecond, not from the client timestamp. It fails closed after 256 outstanding marks.Callback timing invariant
Callbacks must not be fired synchronously duringcreateBridge() execution.
The session runtime assigns the returned bridge to a local variable only after createBridge()
returns. If onReady or onToolCall fires during construction, the guard checks (if (!bridge))
silently discard the event. All callbacks must be deferred to after connect() is called.
Lifecycle and async safety
connect()may resolve before the backend is ready. FireonReadyonly when the bridge can accept audio; clients gate audio on that callback.close()must be idempotent. After it begins, no callback may fire, including callbacks retained by an in-flight promise or socket listener.- A rejected
onToolCallpromise is terminal: the session runtime reports an error and closes the bridge. - Provider callbacks run on the shared Node.js event loop. Avoid synchronous blocking I/O and
serialize state transitions that can race with
close().
Error hygiene
The relay forwardserror.message to the authenticated client before closing. Pass a bounded,
user-safe message to onError; never include credentials, raw audio frames, full prompts, or
private backend diagnostics. Write private details to a redacted subsystem log instead.
Exact-once interruption contract
Manual and server-VAD interruption signals must enter one shared state transition. An interruption is accepted only when the bridge emits exactly oneonClearAudio("barge-in").
Intent alone is not acceptance.
For the bundled OpenAI behavior, the shared transition must:
- Require an assistant output item to have started. Normal interruption also requires an
acknowledged playback snapshot of at least 250 ms;
forcebypasses the duration, acknowledgement, and echo guards. - Install the output-generation fence and clear tracker evidence before outbound cancel, truncate, or client clear side effects.
- Cancel only while a response is active, but truncate queued acknowledged playback even after
response.done. - Emit one reasoned clear and suppress late audio, marks, and stale completion from the old generation.
- Lift the fence only for a genuinely new response after interruption settlement.
- Stop immediately at a re-entrant
close()boundary.
Close reasons
A session-bound realtime session (ADR 0103) whose
talk.conversation.maxSessionMinutes relay TTL
expires closes as "completed", not "session-minutes-exceeded" — only the cascaded loop emits
that reason.
Tool calls
When the voice model triggers a function call, emitonToolCall:
talk.realtime.toolResult. The Gateway calls
bridge.submitToolResult(callId, result, options). Options:
willContinue: boolean— whether the client will send another tool result before expecting a responsesuppressResponse: boolean— submit without asking the provider for a new assistant response
supportsToolResultContinuation is a provider capability hint; the current relay does not
enforce it. willContinue keeps the call ID in the client-owned set so the client can submit
another result. The bundled OpenAI bridge ignores continuation support and expects exactly
one function-call output per call ID. Do not submit interim and final outputs for the same
OpenAI call.
Audio formats
Two format constants are exported fromopenclaw/plugin-sdk:
The Gateway relay always uses
REALTIME_VOICE_AUDIO_FORMAT_PCM16_24KHZ: signed,
little-endian, interleaved mono samples at 24 kHz. Emit onAudio buffers in that exact layout.
If your backend uses a different rate, negotiate it in connect() and resample; the relay
contract is fixed. Declare both formats in capabilities.inputAudioFormats and
capabilities.outputAudioFormats if your provider supports them.
Browser sessions (optional)
For WebRTC / provider-websocket / managed-room transports, implementcreateBrowserSession:
createBrowserSession is absent and a client requests a non-relay transport, the Gateway
returns UNAVAILABLE — it never silently downgrades to relay.
Agent consult tool
The SDK ships a reserved built-in function tool (openclaw_agent_consult, exported as
REALTIME_VOICE_AGENT_CONSULT_TOOL) that, for gateway-relay sessions only, lets the voice
model delegate a request to the configured WednesdayAI agent. Under the default safe-read-only
policy this covers read-only tool use, memory lookups, and current-information retrieval; workspace
actions and any other agent tool are available only when the owner policy is configured and
authorized (see Consult tool policies).
Who installs it (not the provider)
Your provider does not merge, reserve, or opt into this tool. For gateway-relay transport, the gateway handlers that create a relay session —talk.realtime.session and, since ADR 0103, the
session-bound talk.session.create {mode: "realtime"} — resolve the effective policy
(resolveRealtimeVoiceAgentConsultTools / resolveTalkRealtimeConsultPolicy in
src/gateway/server-methods/talk-session-realtime.ts) and merge the built-in descriptor into the
tool list before your createBridge() is ever called. For a session-bound realtime voice
session the consult runs against the owning chat session (its agent and tool session), with
transcript persistence suppressed. Your provider receives the already-merged req.tools like any
other tool and does not see which entries are gateway-reserved. Non-relay (createBrowserSession)
sessions do not receive this merge — the built-in is a gateway-relay-only feature.
resolveConfig() and createBridge() do not receive the same object: resolveConfig({ cfg, rawConfig })
receives rawConfig — the JSON object under talk.providers.<id> as configured, unresolved —
and its return value becomes providerConfig, the object createBridge(req) actually receives on
req.providerConfig. Whether consultToolPolicy survives that trip as-configured depends on
your resolveConfig(). The bundled OpenAI Realtime provider’s resolveConfig()
(extensions/openai-realtime/src/provider.ts) trims whitespace from a valid "safe-read-only" /
"owner" / "none" string (so the exact value survives, just whitespace-normalized) and deletes
consultToolPolicy entirely from providerConfig when the configured value is not one of those
three — an invalid value is never passed through as-is. A resolveConfig() you write yourself may
do something else entirely, including passing an invalid value through unchanged.
Regardless of what resolveConfig()/createBridge() does, never infer authorization from
providerConfig.consultToolPolicy. A connection without operator.admin scope can still see
consultToolPolicy: "owner" in its resolved providerConfig if that is what is configured,
even though the gateway handler silently downgrades the actual tool execution to
safe-read-only for that session (see Consult tool policies). Gateway
auth enforcement (the operator.admin scope check) is entirely separate from — and happens
independently of — whatever value ends up in providerConfig. Treat the configured/requested
value in providerConfig as a hint about what was asked for, never as proof of what was
authorized.
Who handles the call (not the provider)
When the voice backend callsopenclaw_agent_consult, your bridge implementation fires
onToolCall exactly as it would for any other tool — you do not need to detect or special-case
this name. The Gateway’s relay layer (src/gateway/talk-realtime-relay.ts) intercepts calls to
this reserved name before they are broadcast to the browser client and requests cancellation
after 30 seconds for one
transient run on the Analysis lane via runEmbeddedPiAgent (src/talk/agent-consult-runtime.ts)
— a temporary JSONL session removed in finally, with disableMessageTool, disablePluginHooks,
and transcript/event suppression on regardless of policy. It then calls
bridge.submitToolResult(callId, result) with exactly one final result. Cleanup completes after
the runner settles, so 30 seconds is not a hard completion bound for a non-cooperative runner.
No toolCall relay
event reaches the client for this name, and a client-submitted talk.realtime.toolResult for
that call ID is rejected by the relay. Every other tool name keeps the existing behavior: it is
broadcast to the client as a toolCall event, and the client’s talk.realtime.toolResult
response is what your bridge receives through submitToolResult.
An accepted barge-in on a still-live relay aborts an in-flight consult and settles it with a
suppressed generic cancellation result, so submitToolResult is still called exactly once and
the provider is not left waiting. Every other teardown path (relay close, stop, expiry, provider
error, provider close) only aborts the consult — no result is submitted for it, and a late
completion after teardown is silently dropped.
Consult tool policies
Configured viatalk.providers.<id>.consultToolPolicy in openclaw.json, resolved by the
gateway handler — not chosen by the provider:
Public helper contracts
These helpers back the gateway handler’s merge/policy logic. They are exported fromopenclaw/plugin-sdk as documented contracts for other integration surfaces — a provider does
not need to call them to receive gateway-relay consult behavior, which applies automatically:
buildRealtimeVoiceAgentConsultWorkingResponse and buildRealtimeVoiceAgentConsultPolicyInstructions
are also exported but currently unused by the bundled relay — neither controls the shipped
gateway-relay consult behavior described above.
buildRealtimeVoiceAgentConsultWorkingResponsebuilds an interim “working” message, but OpenAI’s Realtime API accepts exactly one function-call output per call ID, so an interim message and a later final result cannot both be submitted for the sameopenclaw_agent_consultcall. Do not build a provider that submits interim and final results as separate outputs for one call ID; the shipped relay submits exactly one final result (see Who handles the call).buildRealtimeVoiceAgentConsultPolicyInstructionsbuilds model-facing instruction text for when the voice model should call the consult tool (it returnsundefinedfor the"auto"consult policy). The bundled relay does not inject these instructions into the session prompt.
Reference implementations
Production provider — the bundled OpenAI Realtime provider is the first production implementation. It covers the full bridge contract: WebSocket lifecycle, server-VAD barge-in signalling, tool calls, audio framing, and post-close race guard. Use it as a starting point when implementing a new production provider. Public test registration — tests can register a provider and create a bridge session using only the public SDK:response.done playback, late-generation audio, re-entrant close, terminal errors,
consult cancellation/ownership, and relay teardown.
RealtimeVoiceTool schema
Tools declared by the client at session start:
createBridge(req.tools) and may register them with the
voice backend. Providers that self-configure their toolset from registered config may ignore
the client-declared tools — declare this in capabilities.supportsToolCalls.
Provider config resolution flow
resolveConfig is optional. If omitted, rawConfig (the JSON object under
talk.providers.<id>) is passed as-is to isConfigured and createBridge.