Plugin media understanding providers
WednesdayAI plugins can register media understanding providers that handle audio transcription, image description, and video understanding. Each provider implements wire-protocol logic for one or more capabilities and is automatically discoverable via auto-detection metadata. Media understanding providers are disabled by default. Users enable them in config:Basic provider
A minimal audio-only provider:openclaw/plugin-sdk specifier still loads (ADR 2057).
Provider contract
AMediaUnderstandingProviderPlugin defines:
- id: unique provider identifier (used in
provider:config and auto-detection). - label: human-readable provider name for logging and UI.
- capabilities: array of
"audio","image","video"— one or more. - autoDetectOrder (optional): metadata for auto-detection ordering per capability
(e.g.
{ audio: 50, image: 40 }). Lower numbers are tried first. - defaultModels (optional): default model name per capability when not explicitly
configured (e.g.
{ audio: "whisper-large-v3" }). - envVars (optional): array of environment variable names holding credentials only (e.g.
["MY_API_KEY", "MY_LEGACY_API_KEY"]). The runner reads these when resolving credentials, so listing a variable here is enough to make it work without a core change. See Credential resolution. They are also named in error messages to guide operators. The first one that resolves to a non-empty value wins, so list the preferred key first and list nothing that is not a key: aMY_BASE_URLentry would be handed to the provider asrequest.apiKeywhenever the real key is unset. Base URLs and other non-secret settings belong in the plugin’s ownconfigSchema, not here. - transcribeAudio (optional):
(request: AudioTranscriptionRequest) => Promise<AudioTranscriptionResult>. - describeImage (optional):
(request: ImageDescriptionRequest) => Promise<ImageDescriptionResult>. - describeVideo (optional):
(request: VideoDescriptionRequest) => Promise<VideoDescriptionResult>.
api.registerMediaUnderstandingProvider(...) — there is no functions wrapper. Implement one
handler per entry in capabilities.
Shared wire helpers
The SDK exports wire-protocol helpers and test utilities for common patterns:Multipart uploads
PreferpostTranscriptionRequest (or transcribeOpenAiCompatibleAudio, which wraps it) over a
hand-rolled fetch for any multipart/form-data upload. All outbound requests pass through the
SSRF egress guard, which dispatches on a bundled HTTP client that only recognises its own
FormData class. A body built from globalThis.FormData is not recognised, and the request goes
out with no boundary and no fields — the endpoint answers with a confusing complaint about a
missing parameter rather than a transport error. The shared helpers rebuild the body for the
dispatching client, so they are always safe.
If you must build a multipart body yourself, do not set content-type or content-length
manually. Let the client derive the boundary.
OpenAI-compatible provider (audio)
For audio transcription via OpenAI-compatible APIs (Groq, OpenRouter, DeepInfra, etc.):OpenAI-compatible provider (image and video)
For image description and video understanding:Auto-detection ordering
TheautoDetectOrder field controls which provider is tried first when multiple
providers are enabled and no explicit model entry is configured.
groq.autoDetectOrder.audio < openai.autoDetectOrder.audio.
Test utilities
The SDK exports test helpers for mocking provider responses:Required fields per capability
Audio transcription (AudioTranscriptionRequest):
buffer:Buffercontaining the raw audio bytes.fileName: original file name.apiKey: resolved credential for the provider.timeoutMs: request timeout in milliseconds.mime(optional): MIME type of the audio.model(optional): transcription model identifier.language(optional): BCP 47 language tag.prompt(optional): transcription hint/prompt.baseUrl(optional): base URL override for the provider endpoint.headers(optional): custom headers to merge into the request.query(optional): extra query-string parameters.fetchFn(optional): fetch override, used in tests.
AudioTranscriptionResult: { text: string; model?: string }.
Image description (ImageDescriptionRequest):
buffer:Buffercontaining the raw image bytes.fileName: original file name.model: vision model identifier.provider: id of the provider the request is being resolved for.timeoutMs: request timeout in milliseconds.agentDir: agent working directory.cfg: the resolvedOpenClawConfig.mime(optional): MIME type of the image.prompt(optional): description instruction.maxTokens(optional): response token cap.profile/preferredProfile(optional): model profile hints.apiKey(optional): credential for the provider — populated by the runner when resolved.baseUrl(optional): base URL override for the provider endpoint.headers(optional): custom headers to merge into the request.fetchFn(optional): fetch override, used in tests.
ImageDescriptionResult: { text: string; model?: string }.
Video understanding (VideoDescriptionRequest):
buffer:Buffercontaining the raw video bytes.fileName: original file name.apiKey: resolved credential for the provider.timeoutMs: request timeout in milliseconds.mime(optional): MIME type of the video.model(optional): video understanding model identifier.prompt(optional): description instruction.baseUrl(optional): base URL override for the provider endpoint.headers(optional): custom headers to merge into the request.fetchFn(optional): fetch override, used in tests.
VideoDescriptionResult: { text: string; model?: string }.
Credential resolution
Your handler receives the resolved credential inrequest.apiKey. The runner fills it from the
first source that yields a value:
- A stored auth profile for the provider (
auth-profiles.json). - An environment variable that core maps for well-known provider ids.
- Any variable named in your registration’s
envVars, in the order you list them. models.providers.<id>.apiKeyinopenclaw.json.
GEMINI_API_KEY for google, for example, but an operator who exported only GOOGLE_API_KEY
depends on it appearing in envVars.
Releases up to and including 0.4.10 treated
envVars as documentation only, so a provider absent
from core’s internal map could never authenticate no matter what the operator exported. If you
maintain a provider plugin, no change is needed on your side beyond declaring envVars.Borrowing a credential from another provider
When a model entry specifiesapiKeyProvider: "other-provider", the media understanding
runner resolves credentials from that other provider instead of the implementation
provider, and your own envVars are not consulted.