Media Understanding (Inbound) — 2026-01-17
WednesdayAI can summarize inbound media (image/audio/video) before the reply pipeline runs. It auto‑detects when local tools or provider keys are available, and can be disabled or customized. If understanding is off, models still receive the original files/URLs as usual. Providers as extensions: media understanding providers are shipped as plugin extensions (e.g.@wednesdayai/openai, @wednesdayai/google), not built into core. By default, all provider extensions are disabled — enable them in config by setting plugins.entries.<vendor>.enabled: true. See Plugin Extensions below.
Goals
- Optional: pre‑digest inbound media into short text for faster routing + better command parsing.
- Preserve original media delivery to the model (always).
- Support provider APIs and CLI fallbacks.
- Allow multiple models with ordered fallback (error/size/timeout).
High‑level behavior
- Collect inbound attachments (
MediaPaths,MediaUrls,MediaTypes). - For each enabled capability (image/audio/video), select attachments per policy (default: first).
- Choose the first eligible model entry (size + capability + auth).
- If a model fails or the media is too large, fall back to the next entry.
- On success:
Bodybecomes[Image],[Audio], or[Video]block.- Audio sets
{{Transcript}}; command parsing uses caption text when present, otherwise the transcript. - Captions are preserved as
User text:inside the block.
Config overview
tools.media supports shared models plus per‑capability overrides:
tools.media.models: shared model list (usecapabilitiesto gate).tools.media.image/tools.media.audio/tools.media.video:- defaults (
prompt,maxChars,maxBytes,timeoutSeconds,language) - provider overrides (
baseUrl,headers,providerOptions) - Deepgram audio options via
tools.media.audio.providerOptions.deepgram - audio transcript echo controls (
echoTranscript, defaultfalse;echoFormat) - optional per‑capability
modelslist (preferred before shared models) attachmentspolicy (mode,maxAttachments,prefer)scope(optional gating by channel/chatType/session key)
- defaults (
tools.media.concurrency: max concurrent capability runs (default 2).
Model entries
Eachmodels[] entry can be provider or CLI:
Credential borrowing (apiKeyProvider)
UseapiKeyProvider to apply credentials from a different provider. This is useful when
a proxy provider (e.g., Bifrost) handles authentication for underlying models:
Config-provided proxy bindings (OpenAI-compatible)
An OpenAI-compatible API inmodels.providers can be used directly as a media
understanding provider without a plugin extension:
openai-completions or openai-responses.
Audio is never auto-detected on a config-provided proxy: core cannot know which transcription model the proxy routes, so you must name one in tools.media.audio.models. A shared tools.media.models entry with capabilities: ["audio"] works for both the main pass and the pre-mention transcription of requireMention group voice notes, with or without a tools.media.audio key. tools.media.audio.models covers both paths on its own.
Without that entry the gateway logs media auto-selection exhausted: voice notes are never auto-detected on an OpenAI-compatible models.providers entry (my-proxy) … and wednesdayai doctor names the provider and the same fix. When nothing usable resolves for audio, the pre-mention transcription pass for requireMention group voice notes warns with the same remedy on the media-understanding/audio-preflight logger.
Credentials resolve under the proxy’s raw models.providers key, in the standard order: stored auth profile, core env map, provider-declared env var, then models.providers.<id>.apiKey (first hit wins). The key and the endpoint always come from the same entry, so a proxy key whose id aliases a canonical media id (for example gemini vs google) is never borrowed from the canonical provider’s credentials.
Core sends the configured model id on the request as written in config. Proxy-side model aliasing and routing are operator configuration on the proxy — WednesdayAI does not rewrite or map model ids for the hop.
{{MediaDir}}(directory containing the media file){{OutputDir}}(scratch dir created for this run){{OutputBase}}(scratch file base path, no extension)
Defaults and limits
Recommended defaults:maxChars: 500 for image/video (short, command‑friendly)maxChars: unset for audio (full transcript unless you set a limit)maxBytes:- image: 10MB
- audio: 20MB
- video: 50MB
- If media exceeds
maxBytes, that model is skipped and the next model is tried. - Audio files smaller than 1024 bytes are treated as empty/corrupt and skipped before provider/CLI transcription.
- If the model returns more than
maxChars, output is trimmed. promptdefaults to simple “Describe the .” plus themaxCharsguidance (image/video only).- If
<capability>.enabled: truebut no models are configured, WednesdayAI determines the image/media model using the chat’s session or channel model rather than just the agent default. When the turn’s active model is text-only, selection follows the text-only fallback order.
Describing media when the primary model is text-only
Media understanding evaluates whether the model handling the turn can perceive images or media natively. This check inspects the turn’s session or channel model override (if set) rather than only the agent default model. If the active model is text-only, media understanding automatically selects a capable vision model in this order:tools.media.image.models(or sharedtools.media.models) — explicit configuration replaces the automatic chain below entirelyagents.defaults.imageModel- The active provider’s first image-capable model in
models.providers.<id>.models[] - The provider’s declared default model (
defaultModels.image) - The Gemini CLI, when the
geminibinary is on PATH - Auto-detect across enabled media providers (
autoDetectOrder)
agents.defaults.imageModel does not displace it: the active model is used first, and
the pin keeps its fallback slot — after the active-model probe and local audio, before the
Gemini CLI and auto-detect. This keeps a provider that authenticates through its own declared
environment variables reachable on the active-model probe.
Auto-detect media understanding (default)
Iftools.media.<capability>.enabled is not set to false and you haven’t
configured models, WednesdayAI auto-detects enabled providers and chooses based on
per-provider autoDetectOrder metadata. Shipped order (lower numbers tried first):
Audio:
- OpenAI (10)
- Groq (20)
- Deepgram (30)
- Google (40)
- Mistral (50)
- ElevenLabs (60)
- OpenRouter (70)
- DeepInfra (80)
- Bifrost (90)
- OpenAI (10)
- Anthropic (20)
- Google (30)
- MiniMax (40)
- Z.ai (50)
- Google (10)
- Moonshot (20)
sherpa-onnx-offline(requiresSHERPA_ONNX_MODEL_DIRwith encoder/decoder/joiner/tokens)whisper-cli(whisper-cpp; usesWHISPER_CPP_MODELor the bundled tiny model)whisper(Python CLI; downloads models automatically)
gemini) using read_many_files is available as a fallback CLI entry if configured.
Migration from pre-0.5
If you upgraded WednesdayAI and media understanding stopped working, the provider extension you need is now disabled by default. Check the gateway warning logs for a message like:OPENAI_API_KEY or the equivalent environment variable for your chosen provider if not
already set. Credentials resolve in this order, first hit wins:
- A stored auth profile for the provider.
- An environment variable core maps for that provider id.
- Any variable the provider extension declares for itself, such as
ELEVENLABS_API_KEYorDEEPINFRA_API_KEY. models.providers.<id>.apiKeyinopenclaw.json.
media auto-selection exhausted: voice notes are never auto-detected on an OpenAI-compatible models.providers entry, your estate is a config-provided proxy rather than a provider extension — see Config-provided proxy bindings and set tools.media.audio.models.
Step 3 did not work in releases up to and including 0.4.10: a provider whose id core did not
recognise could never authenticate, no matter which variable was exported. ElevenLabs and DeepInfra
were affected, as was Google when only
GOOGLE_API_KEY was set rather than GEMINI_API_KEY.PATH (we expand ~), or set an explicit CLI model with a full command path.
Proxy environment support (provider models)
When provider-based audio and video media understanding is enabled, WednesdayAI honors standard outbound proxy environment variables for provider HTTP calls:HTTPS_PROXYHTTP_PROXYhttps_proxyhttp_proxy
Capabilities (optional)
If you setcapabilities, the entry only runs for those media types. For shared
lists, WednesdayAI can infer defaults:
openai,anthropic,minimax: imagegoogle(Gemini API): image + audio + videogroq: audiodeepgram: audio
capabilities explicitly to avoid surprising matches.
If you omit capabilities, the entry is eligible for the list it appears in.
Provider support matrix (WednesdayAI integrations)
Model selection guidance
- Prefer the strongest latest-generation model available for each media capability when quality and safety matter.
- For tool-enabled agents handling untrusted inputs, avoid older/weaker media models.
- Keep at least one fallback per capability for availability (quality model + faster/cheaper model).
- CLI fallbacks (
whisper-cli,whisper,gemini) are useful when provider APIs are unavailable. parakeet-mlxnote: with--output-dir, WednesdayAI reads<output-dir>/<media-basename>.txtwhen output format istxt(or unspecified); non-txtformats fall back to stdout.
Provider extensions
Media understanding providers are plugin extensions that must be explicitly enabled. Enable a provider extension by adding it to your config underplugins.entries:
- Provider ID: used in
provider:field when explicitly configuring a model entry. - Auto-detect metadata:
autoDetectOrder(priority for auto-selection) anddefaultModels(fallback model when none is configured). - Wire functions: capabilities-specific implementations (audio transcription, image description, video understanding).
- Declared env vars:
envVars, the environment variables the provider can authenticate from. These are honoured as a credential source, so an extension can ship support for its own key without a core change.
Attachment policy
Per‑capabilityattachments controls which attachments are processed:
mode:first(default) orallmaxAttachments: cap the number processed (default 1)prefer:first,last,path,url
mode: "all", outputs are labeled [Image 1/2], [Audio 2/2], etc.
To describe every photo in an album, configure attachments under tools.media.image:
Config examples
1) Shared models list + overrides
2) Audio + Video only (image off)
3) Optional image understanding
4) Multi‑modal single entry (explicit capabilities)
Status output
When media understanding runs,/status includes a short summary line:
Notes
- Understanding is best‑effort. Errors do not block replies.
- Attachments are still passed to models even when understanding is disabled.
- Use
scopeto limit where understanding runs (e.g. only DMs). - A failure is quiet in the reply but not in the log. Each provider that is reached and fails logs
one
warnline with its status and message, and an exhausted chain logs a singlewarnsummarising every attempt. Setlogging.leveltodebugfor per-step detail: the resolved chain, each attempt, and the transcript length on success.