Audio / Voice Notes — 2026-01-17
Audio transcription providers (OpenAI, Groq, Deepgram, Google, Mistral, and others) ship as plugin extensions and are disabled by default. Enable the vendor you want withplugins.entries.<vendor>.enabled: true before configuring tools.media.audio. A config-provided
OpenAI-compatible proxy (models.providers.<id> with api: "openai-completions" or "openai-responses")
needs no plugin extension, but voice notes are never auto-detected on it — name a speech-to-text model in
tools.media.audio.models (see Config-provided proxy bindings).
See Media Understanding for the full list and
auto-detect order.
What works
- Media understanding (audio): If audio understanding is enabled (or auto‑detected), WednesdayAI:
- Locates the first audio attachment (local path or URL) and downloads it if needed.
- Enforces
maxBytesbefore sending to each model entry. - Runs the first eligible model entry in order (provider or CLI).
- If it fails or skips (size/timeout), it tries the next entry.
- On success, it replaces
Bodywith an[Audio]block and sets{{Transcript}}.
- Command parsing: When transcription succeeds,
CommandBody/RawBodyare set to the transcript so slash commands still work. - Verbose logging: In
--verbose, we log when transcription runs and when it replaces the body.
Auto-detection (default)
If you don’t configure models andtools.media.audio.enabled is not set to false,
WednesdayAI auto-detects in this order and stops at the first working option:
- Local CLIs (if installed)
sherpa-onnx-offline(requiresSHERPA_ONNX_MODEL_DIRwith encoder/decoder/joiner/tokens)whisper-cli(fromwhisper-cpp; usesWHISPER_CPP_MODELor the bundled tiny model)whisper(Python CLI; downloads models automatically)
- Gemini CLI (
gemini) usingread_many_files - Provider keys, tried in
autoDetectOrderamong enabled provider extensions only (shipped audio order: OpenAI → Groq → Deepgram → Google → Mistral → ElevenLabs → OpenRouter → DeepInfra → Bifrost; see Media Understanding)
tools.media.audio.enabled: false.
To customize, set tools.media.audio.models.
Note: Binary detection is best-effort across macOS/Linux/Windows; ensure the CLI is on PATH (we expand ~), or set an explicit CLI model with a full command path.
Config examples
Provider + CLI fallback (OpenAI + Whisper CLI)
Provider-only with scope gating
Provider-only (Deepgram)
Requires thedeepgram extension enabled (plugins.entries.deepgram.enabled: true).
Provider-only (Mistral Voxtral)
Requires themistral extension enabled (plugins.entries.mistral.enabled: true).
Echo transcript to chat (opt-in)
Notes & limits
- Provider auth follows the standard model auth order: auth profiles, then environment variables
(the one core maps for the provider, then any the provider extension declares for itself), then
models.providers.*.apiKey. A provider whose key resolves nowhere is skipped, not attempted. - Each provider reads its own key (
OPENAI_API_KEY,DEEPGRAM_API_KEY,ELEVENLABS_API_KEY, and so on). Full table: Environment variables. - Deepgram setup details: Deepgram (audio transcription).
- Mistral setup details: Mistral.
- Audio providers can override
baseUrl,headers, andproviderOptionsviatools.media.audio. - Default size cap is 20MB (
tools.media.audio.maxBytes). Oversize audio is skipped for that model and the next entry is tried. - Tiny/empty audio files below 1024 bytes are skipped before provider/CLI transcription.
- Default
maxCharsfor audio is unset (full transcript). Settools.media.audio.maxCharsor per-entrymaxCharsto trim output. - OpenAI auto default is
gpt-transcribe. Avoidgpt-4o-transcribe: on a byte-identical 15.8s clip it returned a transcript truncated at the first sentence boundary in 4 of 11 runs, with no error, whilegpt-transcribereturned the full text in 9 of 9.gpt-4o-mini-transcribeis the fastest of the three and was also reliable in testing. - Use
tools.media.audio.attachmentsto process multiple voice notes (mode: "all"+maxAttachments). - Transcript is available to templates as
{{Transcript}}. tools.media.audio.echoTranscriptis off by default; enable it to send transcript confirmation back to the originating chat before agent processing.tools.media.audio.echoFormatcustomizes the echo text (placeholder:{transcript}).- CLI stdout is capped (5MB); keep CLI output concise.
Proxy environment support
Provider-based audio transcription honors standard outbound proxy env vars:HTTPS_PROXYHTTP_PROXYhttps_proxyhttp_proxy
Mention Detection in Groups
WhenrequireMention: true is set for a group chat, WednesdayAI now transcribes audio before checking for mentions. This allows voice notes to be processed even when they contain mentions.
When a voice note is transcribed, your assistant is told the text came from speech, so it
can allow for mis-heard names and similar-sounding words. This note is not shown in the
chat.
How it works:
- If a voice message has no text body and the group requires mentions, WednesdayAI performs a “preflight” transcription.
- The transcript is checked for mention patterns (e.g.,
@BotName, emoji triggers). - If a mention is found, the message proceeds through the full reply pipeline.
- The transcript is used for mention detection so voice notes can pass the mention gate.
- If transcription fails during preflight (timeout, API error, etc.), the message is processed based on text-only mention detection.
- This ensures that mixed messages (text + audio) are never incorrectly dropped.
- Set
channels.telegram.groups.<chatId>.disableAudioPreflight: trueto skip preflight transcript mention checks for that group. - Set
channels.telegram.groups.<chatId>.topics.<threadId>.disableAudioPreflightto override per-topic (trueto skip,falseto force-enable). - Default is
false(preflight enabled when mention-gated conditions match).
requireMention: true. The voice note is transcribed, the mention is detected, and the agent replies.
Troubleshooting
- Every voice note comes back empty, on every provider. Check the gateway log at
warn. Each failed provider now logs its status and message, and the exhausted chain logs one line listing every attempt. If the log shows nothing at all, upgrade: releases up to and including 0.4.10 could send transcription uploads with no form fields at all, which providers reported as a missingmodelparameter, a bad multipart boundary, or a type error depending on the vendor. - One provider fails but the log is quiet. Set
logging.leveltodebugfor per-step lines: which chain was resolved, each attempt, and the length of the returned transcript. See Logging. - A provider is skipped without being tried. Its API key did not resolve. Check the auth order and the per-provider variable names in the notes above.
Gotchas
- Scope rules use first-match wins.
chatTypeis normalized todirect,group, orroom. - Ensure your CLI exits 0 and prints plain text; JSON needs to be massaged via
jq -r .text. - For
parakeet-mlx, if you pass--output-dir, WednesdayAI reads<output-dir>/<media-basename>.txtwhen--output-formatistxt(or omitted); non-txtoutput formats fall back to stdout parsing. - Keep timeouts reasonable (
timeoutSeconds, default 60s) to avoid blocking the reply queue. - Preflight transcription only processes the first audio attachment for mention detection. Additional audio is processed during the main media understanding phase.