> ## Documentation Index
> Fetch the complete documentation index at: https://docs.wednesdayai.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Realtime Voice gateway relay

> Configure, secure, and operate realtime voice sessions through the WednesdayAI Gateway.

# Realtime Voice gateway relay

The realtime voice relay streams two-way audio between authenticated clients and a
registered provider. The Gateway owns authentication, connection lifecycle, audio framing,
tool-call routing, and bounded session state. The currently bundled production provider is
OpenAI Realtime; compatible plugins can register other providers.

For the web client's voice workflow, see
[Using the web client](https://docs.wednesdayai.dev/next/admin/web/using-the-web-client.md#dictate-and-voice). For provider development,
see [Realtime Voice Provider API](https://docs.wednesdayai.dev/next/users/talk/realtime-voice-api.md).

> [!NOTE]
>
> This page covers the gateway relay behind the realtime voice providers. The classic console's
> **Talk** tab, which used it, was removed together with the classic console (ADR 1771,
> `dev-docs/adr/1771-classic-console-removal-control-ui-key.md`). The `/v2/client` web client's
> **Dictate** control uses a separate, cascaded design (streaming speech-to-text, then an agent
> turn, then streaming text-to-speech), and its **Voice mode** control runs that cascaded loop
> or - when the operator sets `talk.conversation.engine: "realtime"` - the same realtime providers
> through session-bound `talk.session.*` methods. See
> [Voice conversation sessions](https://docs.wednesdayai.dev/next/users/talk/realtime-voice-conversation).

## Requirements

* A registered realtime voice provider. Enable the bundled OpenAI provider with
  `wednesdayai plugins enable openai-realtime`.
* A nonblank provider key. OpenAI resolves it from `talk.providers.openai-realtime.apiKey`,
  plugin defaults, or `OPENAI_API_KEY`.
* `operator.read` for status; `operator.write` for session control and media methods.
* `operator.admin` when `consultToolPolicy: "owner"` must be honored.

Restart the Gateway after enabling or disabling a plugin or changing an environment
variable. On Linux, run `systemctl --user restart openclaw-gateway`. On macOS, run
`wednesdayai gateway restart --deep`.

The provider key stays on the Gateway host. The browser sends audio over its existing
authenticated Gateway WebSocket and never receives the provider key.

## Enable browser Talk

`talk.realtime.enabled` defaults to `false`. Set it to `true` to expose Talk in the
Control UI and report it as enabled in status:

```json5 theme={"dark"}
{
  talk: {
    realtime: {
      enabled: true,
    },
    providers: {
      "openai-realtime": {
        voice: "alloy",
      },
    },
  },
  plugins: {
    entries: {
      "openai-realtime": {
        enabled: true,
      },
    },
  },
}
```

Changes under `talk.*` apply without a restart to new sessions. Existing sessions retain
the configuration resolved when they started. Changes under `plugins.*` and environment
variable changes require a Gateway restart.

> [!WARNING]
>
> `talk.realtime.enabled` is a UI rollout gate, not an API kill switch. Direct callers with
> `operator.write` can still create sessions when it is `false`. Revoke write scope or disable
> the provider plugin when you must prevent API access.

## OpenAI provider defaults

| Setting | Default | Effect |
| - | - | - |
| `model` | `gpt-realtime` | OpenAI Realtime model ID |
| `voice` | OpenAI API default | Spoken voice |
| `consultToolPolicy` | `safe-read-only` | Gateway-owned agent consult access |
| `vadThreshold` | `0.5` | Speech-start sensitivity |
| `prefixPaddingMs` | `300` | Audio retained before detected speech |
| `silenceDurationMs` | `500` | Silence required to end a turn |

OpenAI uses gateway relay only and PCM16 signed little-endian mono audio at 24 kHz. The
browser has no provider picker; it uses the configured available provider selected by the
Gateway.

For faster turn detection, test a threshold of `0.58`, prefix padding of `300` ms, and a
silence duration of `320` ms:

```json5 theme={"dark"}
{
  talk: {
    providers: {
      "openai-realtime": {
        vadThreshold: 0.58,
        prefixPaddingMs: 300,
        silenceDurationMs: 320,
      },
    },
  },
}
```

## Configure agent consult access

For gateway-relay sessions, the Gateway may expose its reserved
`openclaw_agent_consult` tool to the voice model. The browser never executes this tool.

| Value | Effective access |
| - | - |
| Missing, invalid, or `safe-read-only` | Exactly `read`, `web_search`, `web_fetch`, `memory_search`, and `memory_get` |
| `owner` | Removes the consult allowlist and authorizes owner-only tools; remaining agent and sandbox policy still applies |
| `none` | Does not expose the consult tool |

`owner` is honored only for a connection with `operator.admin`. Otherwise the Gateway
silently downgrades that session to `safe-read-only` and logs a bounded warning. Message
tools and plugin hooks remain disabled under every policy. The reserved tool name cannot
be replaced by a client tool, including when the policy is `none`, and client results for
Gateway-owned calls are rejected.

> [!WARNING]
>
> `owner` consults can use action-taking and owner-only tools, so real changes can happen.
> The temporary consult session does not preview or undo those actions. Use `safe-read-only`
> unless the authenticated operator explicitly needs this access.

```json5 theme={"dark"}
{
  talk: {
    providers: {
      "openai-realtime": {
        consultToolPolicy: "safe-read-only",
      },
    },
  },
}
```

### Consult privacy and cancellation

A consult can include up to 12 recent final transcript entries and sends them to the
configured default agent/model provider, which may differ from the voice provider. The
temporary local session file is deleted after the run, but that does not change the model
provider's data-handling terms.

The runtime requests cancellation after 30 seconds and cleans up after the embedded runner
settles. This is not a hard completion bound for a non-cooperative runner. An accepted
interruption cancels an in-flight consult without producing a stray spoken follow-up.

## Permissions and methods

| Method | Required scope | Purpose |
| - | - | - |
| `talk.realtime.status` | `operator.read` | Report availability and active sessions |
| `talk.realtime.session` | `operator.write` | Start a relay session |
| `talk.realtime.audio` | `operator.write` | Send a PCM16 audio frame |
| `talk.realtime.mark` | `operator.write` | Acknowledge a playback mark |
| `talk.realtime.toolResult` | `operator.write` | Submit a client-owned tool result |
| `talk.realtime.bargeIn` | `operator.write` | Request interruption |
| `talk.realtime.stop` | `operator.write` | Close a session |

`owner` consult additionally requires explicit `operator.admin` authorization.

## Relay events

Gateway-relay clients receive `talk.realtime.relay` events:

| Event type | Meaning |
| - | - |
| `ready` | The provider can accept audio. |
| `audio` | Base64-encoded PCM16 audio from the provider. |
| `clear` | Clear queued playback; reason is `barge-in` for an accepted interruption. |
| `mark` | Opaque provider playback boundary. |
| `transcript` | Partial or final user or assistant text. |
| `toolCall` | A client-owned tool call is ready to execute. |
| `vadBargeIn` | Provider server VAD detected speech start. |
| `error` | The provider reported a terminal error. |
| `close` | The session ended as `completed`, `error`, or `cancelled`. |

Every pushed `mark` includes `markName`. The relay forwards mark acknowledgements to the
active provider; the bundled OpenAI provider treats unknown, duplicate, cleared, and
old-generation marks as inert. An `error` event is terminal and is followed by `close`.

## Transport modes

| Transport | Behavior |
| - | - |
| `gateway-relay` | Default. Audio and events pass through the Gateway. |
| `auto` | Uses `createBrowserSession()` when available, otherwise falls back to relay. |
| `webrtc`, `provider-websocket`, `managed-room` | Requires provider support; explicit requests never fall back. |

The Gateway-owned consult tool is available only on relay sessions, including `auto`
sessions that fell back to relay.

## Session limits

* Fixed 30-minute TTL per session.
* Two active sessions per authenticated connection.
* 64 active sessions globally.
* Expired sessions close as `completed`; disconnected-client sessions close as `cancelled`.

## Playback acknowledgements and interruption

Clients acknowledge provider marks only after the associated audio has drained to audible
playback. A normal interruption with the bundled OpenAI provider requires at least 250 ms of
acknowledged audio. `force: true` bypasses the duration floor and acknowledgement/echo
guards; an assistant output item must have started, but no playback acknowledgement is
required.

`audioPlaybackActive` is a provider-defined hint. The bundled OpenAI provider currently
ignores it, so it does not replace playback acknowledgements.

An accepted interruption emits one `clear` event with reason `barge-in`. OpenAI can accept
it after response generation has ended when acknowledged audio remains queued for playback.

## Tool calls

The relay intercepts `openclaw_agent_consult`; every other provider tool call is sent to
the client. A client can return:

```json theme={"dark"}
{
  "relaySessionId": "<id>",
  "callId": "<callId>",
  "result": { "answer": "..." },
  "willContinue": false,
  "suppressResponse": false
}
```

`suppressResponse: true` submits the result without requesting another voice response.
`willContinue: true` keeps the call ID in the client-owned set so the client can submit
another result, but provider support for multiple results is not guaranteed. The bundled
OpenAI provider expects one output per call ID.

## Session observability

`talk.realtime.status` reports `enabled`, `available`, configured provider IDs, and active
sessions with `durationMs`, audio byte counters, and accepted `bargeInCount`. These counters
exist only in memory for currently active sessions; they are not durable metrics.

```json theme={"dark"}
{
  "enabled": true,
  "available": true,
  "configuredProviderIds": ["openai-realtime"],
  "sessions": [
    {
      "id": "<relaySessionId>",
      "provider": "openai-realtime",
      "connId": "<connectionId>",
      "startedAt": 1710000000,
      "durationMs": 12000,
      "audioBytesIn": 12345,
      "audioBytesOut": 67890,
      "bargeInCount": 1
    }
  ]
}
```

`startedAt` is Unix time in seconds. `bargeInCount` counts accepted playback clears, not
every interruption attempt. Status covers active sessions across authenticated connections.

Follow logs with:

```bash theme={"dark"}
wednesdayai logs --follow
```

The `gateway/talk-realtime-relay` subsystem emits these structured lifecycle messages:

| Message | Useful fields |
| - | - |
| `relay session started` | relay session, provider, connection |
| `relay barge-in` | relay session and interruption context |
| `relay session error` | relay session, provider, bounded error |
| `relay session ended` | relay session, reason, duration, audio counters, barge-in count |

Provider errors are terminal: the relay sends the bounded error message to the client and
then closes the session with reason `error`.

## Troubleshooting

### Talk is unavailable

If the client reports `UNAVAILABLE: No realtime voice provider registered`, verify the
plugin is enabled. Also verify the API key resolves to a nonblank value and
`talk.realtime.status` includes `openai-realtime` in `configuredProviderIds`. Restart after
plugin or environment changes.

### An owner consult runs read-only

Confirm the connection token has `operator.admin`. The configured value alone is not
authorization; otherwise the Gateway deliberately downgrades it.

### Interruptions are ignored

Verify the client acknowledges marks after actual playback. For browser echo problems, use
a headset before changing VAD settings. `audioPlaybackActive` alone is insufficient for the
bundled OpenAI provider.

### Sessions close immediately

Check `relay session error`, verify the model and voice are available to the OpenAI account,
and confirm the session limits are not exhausted.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.