Skip to main content

Realtime Voice Provider API

Reference for authors implementing a RealtimeVoiceProviderPlugin — the contract the Gateway uses to bridge client audio to a voice backend.
Building a streaming speech-to-text or streaming text-to-speech provider for the cascaded stt-tts/Dictate conversation loop instead? Those use two different plugin SDK contracts, RealtimeTranscriptionProviderPlugin and StreamingSpeechProviderPlugin — see Register realtime voice, STT and TTS providers and Voice conversation sessions. All three provider kinds, plus media-understanding providers, share one id-registrar diagnostic for duplicate and missing ids (src/plugins/registry.ts).

Plugin registration

Register your provider in the plugin entry file:
Registration is synchronous. Do not perform blocking I/O in register(). Resolve remote credentials and open provider connections later in resolveConfig() or connect(). The provider object must implement RealtimeVoiceProviderPlugin:

SDK imports

All types and helpers export from wednesdayai/plugin-sdk (openclaw/plugin-sdk is the same surface):

RealtimeVoiceBridgeCallbacks

The exact exported contract your bridge invokes (onAudio and onClearAudio are required; everything else is optional):
RealtimeVoiceAudioClearReason is currently just "barge-in" — pass it when the clear is caused by an accepted interruption, or omit it for any other buffer clear. onToolCall may return void or a Promise<void>. onTranscript’s metadata (itemId, responseId, contentIndex, itemOrder, status) lets the relay stitch together partial/final transcript ordering across concurrent response items — omit fields you cannot supply rather than guessing. onError is treated as terminal by the current gateway relay. Firing it broadcasts a talk.realtime.relay error event to the client and then always closes the session with reason "error" (see Close reasons and Relay events). There is no path in the shipped relay where firing onError leaves the session open. The gateway relay (src/gateway/talk-realtime-relay.ts) also does not wire onEvent through to anything client-visible for gateway-relay sessions, so it is not a working substitute either — the shipped relay currently has no non-terminal, session-continuing error channel for providers. All callbacks must be deferred until after createBridge() returns (see Callback timing invariant below) — this applies to every callback in this contract, including onError.

RealtimeVoiceBridgeCreateRequest

The argument to createBridge(): See RealtimeVoiceBridgeCallbacks above for the exact signature of every callback field in this table. cfg is not currently populated on this createBridge() request for gateway-relay sessions (src/talk/session-runtime.ts omits it) — treat it as undefined in createBridge(). It is supplied to isConfigured() and resolveConfig() (src/talk/provider-resolver.ts) and to createBrowserSession() (src/gateway/server-methods/talk.ts), so read the full config there instead if your provider needs it.

RealtimeVoiceBridge interface

Your createBridge() return value must implement:
acknowledgeMark is called when the client confirms a playback mark you emitted via onMark. markName echoes the opaque name of the mark being acknowledged. For a gateway-relay realtime session, playbackTimestamp, when provided, must be a finite, nonnegative, non-regressing integer on the client’s media clock. That clock is not the cascaded stt-tts mark clock: talk.session.mark timestamp is milliseconds of audio played for that turn (0 at the first speech sample). Do not pass performance.now() as an stt-tts speech-mark timestamp. See Talk sessions. RealtimeVoiceBargeInOptions is { audioPlaybackActive?: boolean; force?: boolean; speechDetectedAt?: number; playbackPausedAt?: number }. The last two are client-local performance.now() milliseconds from talk.session.bargeIn and are only passed for stt-tts sessions. A realtime bridge can ignore them. force bypasses the normal duration, acknowledgement, and echo guards.

Playback acknowledgement contract

New clients must acknowledge a named mark only after all audio through that mark has drained to audible playback. Never acknowledge when audio is merely received, decoded, or scheduled. Treat mark names as opaque: echo the exact value and never parse or synthesize one. The bundled OpenAI bridge treats named acknowledgements as cumulative boundaries through the matching outstanding mark. An unnamed acknowledgement consumes one FIFO boundary and exists for legacy compatibility only. Unknown, old-generation, cleared, duplicate, fractional, or regressing acknowledgements are inert. The bridge computes truncation from acknowledged PCM16 bytes at 48 bytes per millisecond, not from the client timestamp. It fails closed after 256 outstanding marks.

Callback timing invariant

Callbacks must not be fired synchronously during createBridge() execution. The session runtime assigns the returned bridge to a local variable only after createBridge() returns. If onReady or onToolCall fires during construction, the guard checks (if (!bridge)) silently discard the event. All callbacks must be deferred to after connect() is called.

Lifecycle and async safety

  • connect() may resolve before the backend is ready. Fire onReady only when the bridge can accept audio; clients gate audio on that callback.
  • close() must be idempotent. After it begins, no callback may fire, including callbacks retained by an in-flight promise or socket listener.
  • A rejected onToolCall promise is terminal: the session runtime reports an error and closes the bridge.
  • Provider callbacks run on the shared Node.js event loop. Avoid synchronous blocking I/O and serialize state transitions that can race with close().

Error hygiene

The relay forwards error.message to the authenticated client before closing. Pass a bounded, user-safe message to onError; never include credentials, raw audio frames, full prompts, or private backend diagnostics. Write private details to a redacted subsystem log instead.

Exact-once interruption contract

Manual and server-VAD interruption signals must enter one shared state transition. An interruption is accepted only when the bridge emits exactly one onClearAudio("barge-in"). Intent alone is not acceptance. For the bundled OpenAI behavior, the shared transition must:
  1. Require an assistant output item to have started. Normal interruption also requires an acknowledged playback snapshot of at least 250 ms; force bypasses the duration, acknowledgement, and echo guards.
  2. Install the output-generation fence and clear tracker evidence before outbound cancel, truncate, or client clear side effects.
  3. Cancel only while a response is active, but truncate queued acknowledged playback even after response.done.
  4. Emit one reasoned clear and suppress late audio, marks, and stale completion from the old generation.
  5. Lift the fence only for a genuinely new response after interruption settlement.
  6. Stop immediately at a re-entrant close() boundary.
Do not clear the client before installing the generation fence. Otherwise late predecessor audio can refill the queue after the clear.

Close reasons

A session-bound realtime session (ADR 0103) whose talk.conversation.maxSessionMinutes relay TTL expires closes as "completed", not "session-minutes-exceeded" — only the cascaded loop emits that reason.

Tool calls

When the voice model triggers a function call, emit onToolCall:
The client executes the function and sends back talk.realtime.toolResult. The Gateway calls bridge.submitToolResult(callId, result, options). Options:
  • willContinue: boolean — whether the client will send another tool result before expecting a response
  • suppressResponse: boolean — submit without asking the provider for a new assistant response
supportsToolResultContinuation is a provider capability hint; the current relay does not enforce it. willContinue keeps the call ID in the client-owned set so the client can submit another result. The bundled OpenAI bridge ignores continuation support and expects exactly one function-call output per call ID. Do not submit interim and final outputs for the same OpenAI call.

Audio formats

Two format constants are exported from openclaw/plugin-sdk: The Gateway relay always uses REALTIME_VOICE_AUDIO_FORMAT_PCM16_24KHZ: signed, little-endian, interleaved mono samples at 24 kHz. Emit onAudio buffers in that exact layout. If your backend uses a different rate, negotiate it in connect() and resample; the relay contract is fixed. Declare both formats in capabilities.inputAudioFormats and capabilities.outputAudioFormats if your provider supports them.

Browser sessions (optional)

For WebRTC / provider-websocket / managed-room transports, implement createBrowserSession:
If createBrowserSession is absent and a client requests a non-relay transport, the Gateway returns UNAVAILABLE — it never silently downgrades to relay.

Agent consult tool

The SDK ships a reserved built-in function tool (openclaw_agent_consult, exported as REALTIME_VOICE_AGENT_CONSULT_TOOL) that, for gateway-relay sessions only, lets the voice model delegate a request to the configured WednesdayAI agent. Under the default safe-read-only policy this covers read-only tool use, memory lookups, and current-information retrieval; workspace actions and any other agent tool are available only when the owner policy is configured and authorized (see Consult tool policies).

Who installs it (not the provider)

Your provider does not merge, reserve, or opt into this tool. For gateway-relay transport, the gateway handlers that create a relay session — talk.realtime.session and, since ADR 0103, the session-bound talk.session.create {mode: "realtime"} — resolve the effective policy (resolveRealtimeVoiceAgentConsultTools / resolveTalkRealtimeConsultPolicy in src/gateway/server-methods/talk-session-realtime.ts) and merge the built-in descriptor into the tool list before your createBridge() is ever called. For a session-bound realtime voice session the consult runs against the owning chat session (its agent and tool session), with transcript persistence suppressed. Your provider receives the already-merged req.tools like any other tool and does not see which entries are gateway-reserved. Non-relay (createBrowserSession) sessions do not receive this merge — the built-in is a gateway-relay-only feature. resolveConfig() and createBridge() do not receive the same object: resolveConfig({ cfg, rawConfig }) receives rawConfig — the JSON object under talk.providers.<id> as configured, unresolved — and its return value becomes providerConfig, the object createBridge(req) actually receives on req.providerConfig. Whether consultToolPolicy survives that trip as-configured depends on your resolveConfig(). The bundled OpenAI Realtime provider’s resolveConfig() (extensions/openai-realtime/src/provider.ts) trims whitespace from a valid "safe-read-only" / "owner" / "none" string (so the exact value survives, just whitespace-normalized) and deletes consultToolPolicy entirely from providerConfig when the configured value is not one of those three — an invalid value is never passed through as-is. A resolveConfig() you write yourself may do something else entirely, including passing an invalid value through unchanged. Regardless of what resolveConfig()/createBridge() does, never infer authorization from providerConfig.consultToolPolicy. A connection without operator.admin scope can still see consultToolPolicy: "owner" in its resolved providerConfig if that is what is configured, even though the gateway handler silently downgrades the actual tool execution to safe-read-only for that session (see Consult tool policies). Gateway auth enforcement (the operator.admin scope check) is entirely separate from — and happens independently of — whatever value ends up in providerConfig. Treat the configured/requested value in providerConfig as a hint about what was asked for, never as proof of what was authorized.

Who handles the call (not the provider)

When the voice backend calls openclaw_agent_consult, your bridge implementation fires onToolCall exactly as it would for any other tool — you do not need to detect or special-case this name. The Gateway’s relay layer (src/gateway/talk-realtime-relay.ts) intercepts calls to this reserved name before they are broadcast to the browser client and requests cancellation after 30 seconds for one transient run on the Analysis lane via runEmbeddedPiAgent (src/talk/agent-consult-runtime.ts) — a temporary JSONL session removed in finally, with disableMessageTool, disablePluginHooks, and transcript/event suppression on regardless of policy. It then calls bridge.submitToolResult(callId, result) with exactly one final result. Cleanup completes after the runner settles, so 30 seconds is not a hard completion bound for a non-cooperative runner. No toolCall relay event reaches the client for this name, and a client-submitted talk.realtime.toolResult for that call ID is rejected by the relay. Every other tool name keeps the existing behavior: it is broadcast to the client as a toolCall event, and the client’s talk.realtime.toolResult response is what your bridge receives through submitToolResult. An accepted barge-in on a still-live relay aborts an in-flight consult and settles it with a suppressed generic cancellation result, so submitToolResult is still called exactly once and the provider is not left waiting. Every other teardown path (relay close, stop, expiry, provider error, provider close) only aborts the consult — no result is submitted for it, and a late completion after teardown is silently dropped.

Consult tool policies

Configured via talk.providers.<id>.consultToolPolicy in openclaw.json, resolved by the gateway handler — not chosen by the provider:

Public helper contracts

These helpers back the gateway handler’s merge/policy logic. They are exported from openclaw/plugin-sdk as documented contracts for other integration surfaces — a provider does not need to call them to receive gateway-relay consult behavior, which applies automatically:
buildRealtimeVoiceAgentConsultWorkingResponse and buildRealtimeVoiceAgentConsultPolicyInstructions are also exported but currently unused by the bundled relay — neither controls the shipped gateway-relay consult behavior described above.
  • buildRealtimeVoiceAgentConsultWorkingResponse builds an interim “working” message, but OpenAI’s Realtime API accepts exactly one function-call output per call ID, so an interim message and a later final result cannot both be submitted for the same openclaw_agent_consult call. Do not build a provider that submits interim and final results as separate outputs for one call ID; the shipped relay submits exactly one final result (see Who handles the call).
  • buildRealtimeVoiceAgentConsultPolicyInstructions builds model-facing instruction text for when the voice model should call the consult tool (it returns undefined for the "auto" consult policy). The bundled relay does not inject these instructions into the session prompt.

Reference implementations

Production provider — the bundled OpenAI Realtime provider is the first production implementation. It covers the full bridge contract: WebSocket lifecycle, server-VAD barge-in signalling, tool calls, audio framing, and post-close race guard. Use it as a starting point when implementing a new production provider. Public test registration — tests can register a provider and create a bridge session using only the public SDK:
Cover readiness, real playback acknowledgement, duplicate manual/VAD interruption, post-response.done playback, late-generation audio, re-entrant close, terminal errors, consult cancellation/ownership, and relay teardown.
Import only from openclaw/plugin-sdk. Relative imports from src/talk/* bypass the public contract and can break without a compatibility release.

RealtimeVoiceTool schema

Tools declared by the client at session start:
The provider receives these in createBridge(req.tools) and may register them with the voice backend. Providers that self-configure their toolset from registered config may ignore the client-declared tools — declare this in capabilities.supportsToolCalls.

Provider config resolution flow

resolveConfig is optional. If omitted, rawConfig (the JSON object under talk.providers.<id>) is passed as-is to isConfigured and createBridge.