Voice conversation sessions
This page covers the Talk session modes behind the/v2/client web client’s Dictate and
Voice mode controls: cascaded stt-tts, streaming-transcription, and session-bound
realtime. Most of it describes the cascaded design: a streaming speech-to-text provider, then an
ordinary agent turn in the owning session, then a streaming text-to-speech provider. The
realtime engine runs speech-to-speech instead: the same gateway relay and bundled OpenAI Realtime
provider that the removed classic console’s Talk tab used, but session-bound and driven
through talk.session.*. See
Realtime Voice gateway relay for the relay and the provider itself.
For end-user instructions, see Using the web client.
For the underlying session wire, see
Narrow scope method allowlist.
Three session modes, one admission path
talk.session.create accepts mode: "transcription" (streaming Dictate: STT only, no agent turn,
final-only-on-close like the pre-existing ADR 0065 batch dictation), mode: "stt-tts" (Voice
mode: streaming STT, then a normal agent turn in the owning session under a voice profile, then
streaming TTS), or mode: "realtime" (Voice mode on the speech-to-speech engine, ADR 0103 — see
Realtime engine). All three modes, plus talk.catalog (read-only), are
narrow-admitted: a /v2/client connection holding only
operator.sessions.read/operator.sessions.write can create and drive these sessions without
operator.read/operator.write. See
Narrow scope method allowlist for the exact
method list.
Turn on voice conversation
All voice modes are off until the operator enables the master gate:talk.session.create call for any voice mode (transcription,
stt-tts, realtime) — narrow or operator-scope — fails
UNAVAILABLE: voice conversation disabled, before any provider is resolved. talk.catalog
still reports provider availability in both gate states (via its conversationEnabled field) so a
client can render Dictate/Voice as available-but-off rather than hidden.
A session additionally needs a configured streaming speech-to-text provider (Dictate), both a
configured streaming speech-to-text and streaming text-to-speech provider (cascaded Voice), or
an active realtime provider with its credential (realtime Voice — see
Realtime engine). Without one, talk.session.create fails
UNAVAILABLE: stt-tts requires a configured STT and TTS provider
(or the transcription-mode equivalent). Configure the provider under talk.providers.<id> and
enable its plugin — see ElevenLabs streaming voice.
The voice the cascaded engine speaks with resolves per agent: the session agent’s
agents.list[].tts voice for the speech provider wins, then talk.providers.<id>.voiceId, then
messages.tts.providers.<id>.voice (plugin providers such as Kokoro only), then the provider’s
default voice. Selecting an agent in the web client therefore selects its voice — see
Custom and local providers.
Changes to talk.conversation.* apply without a gateway restart to new sessions; existing
sessions keep the configuration resolved when they started.
Realtime engine
Voice mode can run speech-to-speech instead of the cascaded loop (ADR 0103):engine: "realtime", the web client’s Voice button creates talk.session.create
{ mode: "realtime" }. The gateway relays 24 kHz PCM16 audio to the active talk realtime provider
(talk.catalog realtime.activeProvider, for example openai-realtime), so it needs that
provider’s credential (talk.providers.<id>.apiKey, or the provider’s own default such as
OPENAI_API_KEY) rather than an STT and TTS pair. Voice shows as available
only when the gate is on and a realtime provider is active. mode: "realtime" fails
UNAVAILABLE: realtime voice conversation disabled while engine is "stt-tts".
The session stays bound to the chat:
- The realtime model is seeded with the last 16 messages of the chat, quoted as history, not instructions.
- The realtime model speaks with its provider’s own voice: neither
talk.provider/talk.providers.<id>TTS normessages.tts(or a per-agentagents.list[].tts) applies. - Each final user and assistant utterance is written to the chat, so the thread shows the spoken
conversation. An assistant reply cut off by barge-in is written as
[interrupted]only; the user’s utterance is always written. - Voice obeys the chat’s send policy: a session whose
chat.sendis refused refuses voice too (send blocked by session policy). - Tools run through the agent consult. For
/v2/clientusers the consult is read-only, so a spoken request that needs write tools gets a “can’t do that by voice” answer. The cascaded engine keeps the full agent turn. talk.conversation.maxTurnSecondscaps continuous speech with the sameturn-seconds-exceededclose the cascaded engine uses.maxSessionMinutesbounds the relay session’s lifetime instead of the cascaded loop’s timer, so reaching it closes the session as a normal close rather thansession-minutes-exceeded.- The chat’s recent history and every consult result go to the realtime provider, even when the
chat’s agent uses a different (for example local-only) model. Keep
engine: "stt-tts"where that matters. talk.realtime.*stays operator-only; the session is driven throughtalk.session.*.
talk.catalog
talk.catalog reports what a client can offer, without exposing any secret:
streamingTranscriptionbacks Dictate;speechbacks cascaded Voice mode (Voice additionally needs a configuredstreamingTranscriptionrow).activeProvideris the id named bytalk.conversation.sttProvider(STT) /talk.provider(TTS) when a matching row exists, elsenull— there is no auto-detect order for either seam.conversationEnginemirrorstalk.conversation.engine("stt-tts"default). With"realtime", the web client’s Voice control runs on therealtimesection: it shows Voice as available whenrealtime.activeProvideris set and the gate is on, without any STT/TTS row.realtimereports the gateway-relay realtime providers (for exampleopenai-realtime) with the same row shape, so a client can show a named diagnostic for that bridge too, even thoughtalk.realtime.*itself stays operator-only.conversationEnabledmirrorstalk.conversation.enabled. It is reported unconditionally —streamingTranscription/speech/realtimeare not hidden while the gate is off — so a narrow client can render “available but disabled by your administrator” instead of nothing.- A
streamingTranscriptionrow’sconfiguredis resolved fromtalk.providers.<id>.apiKeythrough the secret resolver (src/secrets/resolve.ts), not from the provider plugin’s ownisConfigured(). Thespeechandrealtimerows instead resolve readiness the waytalk.session.createdoes: secret-ref and environment credentials first, then the provider’s ownisConfigured(), so a keyless provider (a local TTS plugin such as Kokoro) counts as configured as soon as itsisConfigured()says so. An operator-supplied but unusableapiKey(a whitespace string, or a secret ref that resolves empty) keeps the missing-key diagnostic, so a keyed provider never reads as ready without usable key material. Aspeechrow is additionally unconfigured when the provider cannot resolve a voice: notalk.providers.<id>.voiceId, no plugindefaultVoiceId, and nomessages.tts.providers.<id>.voice.diagnostic, present only whenconfiguredisfalseand a cause is known, is one ofmissing-api-key,secret-ref-unresolved,provider-error, ormissing-voice-id— the causes atalk.catalogreader can safely show an operator without a stack trace.
Audio framing
Input (client to gateway) is base64 PCM16, mono, 16 kHz — half the realtime relay’s 24 kHz input, matching ElevenLabs Scribe v2 Realtime’s expected rate. (Arealtime voice session captures and
relays 24 kHz in both directions instead; its create response advertises the rate.) talk.session.appendAudio rejects a frame
over 512 KB of base64 text, non-canonical base64, and (hardening) a non-PCM16-aligned (odd) byte
count or a frame sent after the session closed, without poisoning the connection. Output (gateway
to client) output.audio.* events carry PCM16 mono 24 kHz, tagged kind: "speech" | "filler" | "ack" so a client scheduler can route or exclude non-speech audio from its own metrics.
Session limits
- 2 active Talk relay sessions per connection, 64 globally — shared across every session kind
(
transcription,stt-tts, and the realtime bridge), not a separate stt-tts-only budget. talk.conversation.maxSessionMinutes(default 30): the cascadedstt-ttsloop closes the session withsession.closed { reason: "session-minutes-exceeded" }once reached. The realtime engine uses the value as the relay session’s TTL instead; an expired realtime voice session closes as a normal close, with nosession-minutes-exceededreason.talk.conversation.maxTurnSeconds(default 60): a hard ceiling on one turn’s continuous user speech. The relay closes the session withsession.closed { reason: "turn-seconds-exceeded" }.- A disconnected client’s session closes the same way a realtime-bridge session does.
The voice turn
A committedstt-tts turn runs through the same startChatTurn the typed chat path uses, in the
session’s own history, with its own tools, memory and approvals — not a separate “voice session”.
An additive voice profile on that call:
- forces thinking off;
- forces
CommandAuthorized: false— spoken text can never trigger a slash command or an inline directive (/new,/model, and so on read as plain words, not commands); - adds a short voice note and provenance to the user-role prefix;
- races the memory-recall hook (
context.collect) againsttalk.conversation.recallDeadlineMs(default 250 ms), so one slow recall cannot stall a voice turn the way it can a typed one — a typed chat turn never uses this deadline.
talk.conversation.fillersEnabled: true (default false), a soft-timeout filler phrase plays
while the agent run is in progress if no assistant speech has started within
talk.conversation.fillerAfterMs (default 1200 ms) of end-of-turn. A spoken acknowledgement plays
when the agent starts a tool call. Neither counts toward the stopToFirstAudioMs metric below.
Phrases are per-locale (talk.conversation.{fillerPhrases,ackPhrases,backchannels}, keyed by
talk.conversation.locale, default "en"; an unrecognised locale falls back to "en"). Before
the reply’s first speech audio, a transcript that is a backchannel or an echo of the filler or
acknowledgement just played does not interrupt the turn. Any other words do.
A voice turn that has to wait behind an in-flight run is queued as a follow-up, not merged into a collect batch. The follow-up keeps the same voice profile (thinking off, command-inert text, recall deadline, and the stt-tts bridge kind). A later voice utterance supersedes an older queued voice utterance; a typed message already in that queue is left alone. A voice turn that is still only queued, and is then superseded or cancelled, is not written into the next turn’s model context and never starts a run. Its words are still saved: once the turn that was running finishes, the cancelled utterance’s user text is appended to the session history, so the transcript records what you said even though it produced no reply.
The first turn of a new Talk session stops a leftover voice run left by a closed session on the same chat. A typed run the user started is never aborted for that reason. A run an explicit talk.session.close detached is the exception: it is still finishing and writes its full reply, so the new session’s first turn queues behind it as a follow-up instead of aborting it. If the agent start itself fails, the turn is cancelled with turn.cancelled { reason: "agent-error" } and the session stays open for the next utterance.
Turn-taking
- End of turn. The STT provider ends a turn on silence (
talk.conversation.endOfTurnSilenceMs, default 500 ms for ElevenLabs Scribe’svad_silence_threshold_secs), not on a fused turn-detection signal — ElevenLabs Scribe v2 Realtime does not report one today. - Turn resume. A partial transcript with at least one real word, arriving after end-of-turn
but before the first assistant audio, cancels the in-flight run with
turn.cancelled { reason: "turn-resumed" }and appends no assistant message (nothing was heard yet). The continuation’s own commit starts the next turn. Both fragments normally persist as user messages followed by one reply; when the cancelled run was still queued behind another run, the first fragment may not persist and the reply answers the continuation alone. - Client-local pause, then server confirmation. The web client arms its interruption detector only while reply speech (
kind: "speech") is playing, not during a filler or acknowledgement one-shot, and re-arms it if playback resumes while that reply is still active. Talking through the filler does not confirm an interruption; speech that continues into the reply interrupts as soon as reply audio starts. The browser then pauses playback locally and sendstalk.session.bargeIn. The relay confirms the burst on the real appended audio (−45 dBFS or louder, sustained at least 250 ms), filters backchannels (short per-locale phrases such as “mm-hmm”) and echo (≥60% word-token overlap with the text just pushed to TTS in the last 3 seconds), and requires a qualifying STT word withintalk.conversation.falseInterruptionMs(default 700 ms) of the client’s report. A confirmed interruption emitsoutput.audio.cleared { reason: "barge-in" }; otherwise it emitsoutput.audio.resumedonce the window elapses with no qualifying word, and playback continues. - Heard-text persistence. After a confirmed barge-in, the session’s transcript holds exactly
one assistant message for that turn: only the text whose audio the client acknowledged as
played, plus an
[interrupted]marker.talk.session.marktimestampfor these speech marks is milliseconds of audio played for that turn (0 at the first speech sample), the same domain as the TTS alignment timeline — not the client’sperformance.now()clock. Barge-inspeechDetectedAt/playbackPausedAtstay on that monotonic clock and are not mixed with mark timestamps. An interruption with zero acknowledged characters appends no assistant message. The user’s own utterance is still persisted. This holds on both the JSONL and database session storage backends. On JSONL, the rewrite truncates the transcript file and rewrites the loaded entries, so earlier turns stay; it does not reset the file to a header-only session. - Explicit close detaches.
talk.session.closeon anstt-ttssession stops the outgoing speech and detaches the active turn: noturn.cancelledand no[interrupted]marker. The agent run still finishes, the full reply is written once when it ends, andtranscript.persisted { turnId, payload: { role: "assistant" } }follows the write (it may arrive aftersession.closed, whose reason stayscompleted). A chat delete or reset cancels the turn instead, and a socket drop still cancels with reasoncancelled, persisting the heard prefix plus[interrupted]. transcript.persistedmarks the write. Forstt-ttsturns too, the event is the completion barrier for the turn’s assistant transcript write: it follows a natural finish, a detached finish, or a cancelled turn’s heard-prefix write (a barge-in, for example), including a cancel that heard nothing, which writes no assistant row and still announces. It is never sent when the write throws, and never for a turn cancelled while it was still queued, which only writes the user’s cancelled fragment. Reload the thread on it. The stored rows carry__openclaw.runIdset to the turn id, so a client can pair its voice turn with the durable row.turn.endedis not “written”.turn.endedis broadcast when the TTS stream drains. Session bookkeeping and the assistant transcript write land after it and are not awaited, so a disk write cannot block the next turn. Wait for thechatfinal event ortranscript.persisted, or readchat.history, before treating the transcript as updated.- TTS stall. If a TTS stream goes 30 seconds without audio after it has been asked to finish, or a provider wait never settles, the turn is cancelled with
turn.cancelled { reason: "tts-stall" }. The session stays open. Model latency before any TTS audio is not this watchdog. - STT drop. One unexpected STT close reconnects once. A second fault, or a failure of that reconnect, emits
session.errorand closes the session with reasonerror.
Metrics
talk.metrics (operator-only; narrow clients are refused missing scope: operator.read) reports a
bounded in-process rolling window (500 turns per STT/TTS provider pair) for the cascaded stt-tts
loop’s turns. Realtime engine voice sessions are not recorded in this ring; realtime relay
lifecycle facts stay in the logs listed on
Realtime Voice gateway relay:
falseInterruptionRate is
falseInterruptions / (falseInterruptions + confirmedInterruptions); the key is omitted entirely
(never 0 or NaN) when that denominator is 0, i.e. no interruption has resolved as either
outcome yet for the provider pair. interruptionCandidates counts every time
the burst detector itself qualifies, independent of whether backchannel/echo filtering or word
confirmation later resolved it as false or confirmed — useful for proving the detector engaged even
on a fixture whose mapped text can never distinguish the two outcomes. p50 and p95 may also include bargeInStopMs, the client-reported delay from speech detection to local playback pause; it is omitted when no accepted interruption carried both timestamps. The same per-turn stage
timings are recorded as a voice turn timing log line (createSubsystemLogger("talk/voice-loop"))
and, when extensions/diagnostics-otel is enabled, as OTel histograms named openclaw.talk.turn.*_ms (including openclaw.talk.turn.barge_in_stop_ms) — see
Task and dream spans for the exporter setup pattern.
The pinned stop-to-first-audio ceiling for the fake-provider proxy run that stands in for this
metric during development is 1150 ms; a live run with real provider keys against a deployed
gateway is looser (p50 of at most 1200 ms), since it also carries real network variance the
in-process budget does not model. A live run with real ElevenLabs/OpenAI keys, and the matching
per-provider-pair series this section describes, is operator-owned post-deploy proof — with
OPENCLAW_GATEWAY_URL and ELEVENLABS_API_KEY (or XI_API_KEY) set, run
node scripts/voice-live-smoke.mjs --turns 5 against a deployed gateway to check it.
talk.conversation configuration reference
See Configuration reference for the full
field table and defaults. This block holds the enabled gate and the engine choice
("stt-tts" or "realtime"), both shared by every Voice session, and the session limits. For the
stt-tts engine it also configures the STT seam, turn-taking thresholds, and the
filler/acknowledgement phrases; TTS reuses the existing talk.provider/talk.providers.<id>
settings (including apiKey, which accepts a SecretRef).
Troubleshooting
talk.session.create fails UNAVAILABLE: voice conversation disabled
Set talk.conversation.enabled: true. This gate is checked before any provider is resolved, so a
fully configured provider does not matter until it is on.
talk.session.create fails UNAVAILABLE: realtime voice conversation disabled
talk.conversation.engine is not "realtime". Set it and keep talk.conversation.enabled: true
(both keys live in the same block and apply to new sessions without a restart). A realtime create
also fails UNAVAILABLE with the provider’s message when no talk realtime provider resolves:
check that the plugin is enabled (plugins.entries.openai-realtime.enabled) and that a key
resolves — talk.providers.openai-realtime.apiKey, the plugin’s defaults, or OPENAI_API_KEY.
talk.catalog’s realtime.activeProvider shows what create would select.
The web client’s Voice button is dimmed
Voice is dimmed whentalk.catalog reports it unavailable to that connection: the
talk.conversation.enabled gate is off, or — with engine: "realtime" —
realtime.activeProvider is null because no realtime provider resolved its credential. With the
cascaded engine, Voice also needs a configured STT and TTS pair, and a missing voice id or
unresolved key shows in the speech row’s diagnostic (see below). Dictate is dimmed and enabled
independently of the engine.
talk.session.create fails UNAVAILABLE: stt-tts requires a configured STT and TTS provider
When the message ends with <transcription|speech> provider "<id>" is not configured; check talk.providers.<id>.apiKey, that provider is not ready: its key is missing, does not resolve, or
the provider’s own isConfigured() rejects the resolved config (the same credential chain
talk.catalog’s speech/realtime rows use, so a keyless provider such as Kokoro passes here
while the plugin is enabled). Without the suffix, no registered provider matches
talk.conversation.sttProvider or talk.provider. Either way, talk.catalog’s
streamingTranscription/speech rows show the same cause in their
configured/diagnostic fields: missing-api-key and secret-ref-unresolved both mean the
gateway could not read a usable key for talk.providers.<id>.apiKey. See
ElevenLabs streaming voice.
talk.session.create fails UNAVAILABLE: stt-tts requires talk.providers.<id>.voiceId
talk.catalog’s speech row shows diagnostic: "missing-voice-id" for the same condition: the
provider is ready but no voice resolves. The voice for the turn comes, in order, from the session
agent’s agents.list[].tts voice for this provider, talk.providers.<id>.voiceId,
messages.tts.providers.<id>.voice (plugin providers only), or the speech provider’s default
voice. A missing key is reported first. Set one of those (ElevenLabs ships a default, so this only
affects third-party speech providers without one). The agent voice follows the canonical session
store key, so a session alias such as agent:ops:main speaks with the default agent’s voice.
A turn never speaks, or speaks very late
Checktalk.metrics for the provider pair’s p95.preModelMs and p95.modelFirstTokenMs. A voice
turn races context.collect against talk.conversation.recallDeadlineMs and falls back with no
recalled context rather than hang — the gateway log names the fallback. A consistently slow
modelFirstTokenMs points at the model provider, not this feature.
A real interruption is ignored, or a cough/backchannel stops the assistant
Interruption needs a real sustained burst (−45 dBFS, 250 ms) plus a confirming STT word withintalk.conversation.falseInterruptionMs; a cough or “mm-hmm” should resume, not clear. If
talk.metrics’s interruptionCandidates rises but confirmedInterruptions does not, the burst
detector is working and the word-confirmation or backchannel/echo filter is doing its job — that is
expected on a false interruption, not a bug. If a genuine interruption is not clearing, verify the
client is actually sending talk.session.bargeIn, not just pausing local playback.
A reply stops after about 30 seconds of silence from the speech service
That is the TTS stall watchdog (turn.cancelled { reason: "tts-stall" }), not maxTurnSeconds (that limit is continuous user speech and closes the whole session). Check the speech provider’s socket and talk.metrics ttsFirstByteMs before raising maxTurnSeconds.
Voice mode starts and the reply never speaks, while typed chat in the same chat is fine
A previous Voice session may have left a voice run in flight. The next session’s first turn stops that leftover voice run. A typed run is not aborted. If the agent start failed instead, the turn ends withturn.cancelled { reason: "agent-error" } and the gateway log names the start failure.
Related
- Realtime Voice gateway relay — the relay and the bundled OpenAI
Realtime provider; the
realtimeengine drives the same relay session-bound throughtalk.session.* - ElevenLabs streaming voice — the first streaming STT/TTS provider pair
- Using the web client — end-user instructions
- Configuration reference — full field table
- Gateway protocol — the narrow scope wire