Web client components
apps/web is the end-user web client (client id openclaw-user-client). Its view layer is a small
native React component layer under apps/web/src/ui/, styled over the shared
Brand tokens. This page is the contract for changing it. For hosting and
configuration see Web client hosting.
Layout
ui/ are Shell, Sidebar, NavigationMenus, TopBar, ModelMenu, ThinkMenu,
DisplayMenu, ToolDisplay, AssistantPicker, AccountMenu, AttentionMenu (the top-bar unread
bell), Thread, TypingIndicator,
AssistantMessage, UserMessage, EmptyState, Composer, RightPanel, DisabledControl,
Icon and markdown/MarkdownRenderer. Shell owns layout mechanics (mobile
breakpoint at 860px, sidebar open and collapsed state) so AppShell supplies data, not chrome. The
assistant and chat controls client-chat-v3 P2 made live (model, Think, assistant picker, queue,
reply, rename) are covered under Chat controls and hooks.
MarkdownRenderer memoises the parsed markdown subtree on the content string (MarkdownBody), so
each streamed delta parses the markdown once and re-renders of unchanged rows skip the parse.
Streaming content renders urgently, not through useDeferredValue. A deferred prose pass can
still be rendering when the next frame’s tool-step chip commits, which would show the chip before
the sentence it interrupts (#2014). Keep prose and chip commits in frame order when touching the
renderer.
Styling conventions
- Every class in
apps/web/src/ui/client.cssis prefixedcl-(cl-sidebar,cl-composer,cl-msg). There is no shadow DOM and no CSS-in-JS. Boot views (loading, pairing, configuration error) live inapp/app.csswith aboot-prefix. - Colours, spacing, radii, motion and fonts come from
brand-tokenscustom properties. Never hard-code a colour or font family. Text on an accent fill uses--accent-foreground. - A token only this app needs (
--personalise-subtle,--composer,--shadow-md) is declared at the top ofclient.css, not in the shared package. - Icons are inline SVG paths in
ui/Icon.tsx(Lucide-style, ISC notice inapps/web/NOTICE). Add a glyph toIconNamerather than an icon-library dependency. - Markdown replies render under a
.prosewrapper. Style prose elements under.prose, not by element selector on the page. - Wide content never widens the chat column.
.cl-conv-col,.cl-thread,.cl-msg-bodyand.cl-bubblecarrymin-width: 0, and.prosewraps (overflow-wrap: anywhere); code blocks and tables scroll inside their own boxes (.prose pre,.prose [data-table-wrap]getoverflow-x: auto) and images cap atmax-width: 100%. Any new message-level content must contain its own width, or the phone layout overflows horizontally. - Honour
prefers-reduced-motion: entrance animation is disabled under it. - Tests and e2e suites select by
data-testidanddata-role, never bycl-class. Keep an existing hook stable when you restyle; rename it in the same change as the suites that use it.
DisabledControl and the phase contract
A control whose backing feature ships later renders throughDisabledControl
(apps/web/src/ui/DisabledControl.tsx), visible but inert:
- It renders a
<button type="button" aria-disabled="true">with atitletooltip. It usesaria-disabled, not thedisabledattribute, so it stays focusable and the tooltip is reachable from the keyboard. - The click handler only calls
preventDefault(). A disabled control must never reach a gateway RPC, whatever capability flag is set.features.voiceis deprecated and ignored: it never enabled Dictate or Voice mode, and now has no reader at all — both controls’ live/disabled state istalk.catalog-driven (ADR 0088 item 12, amended): Dictate needs a configured streaming transcription provider and the gate on, Voice additionally needs a configured streaming speech provider. - The props are a union: pass exactly one of
phase,findingorreason. Usefindingfor controls that wait on a tracked finding rather than a phase: Calendar and Memory cite"F1, GitHub #634"(their fallback while no chat UI plugin provides those panels), Add extension cites the extension marketplace item"F1, GitHub #1331", and the P2 retags (message feedback, message regenerate, account-menu settings, account-menu keyboard shortcuts, composer search) each cite their own finding ("F-2026-09-29-1x, GitHub #69x", ADR 0085 M9); thefindingtooltip wording is fixed, so it reads “coming with the extension panel SDK” whichever finding it cites. Memory and Calendar render as live sidebar buttons instead when a chat UI plugin registers those slots; the DisabledControl is the fallback while no plugin is allowlisted (see Chat UI plugins). Usereasonfor capability-driven unavailability that is not roadmap work: thereasontext is rendered verbatim after the label, so it names why this connection cannot use the control — Dictate and Voice mode pass"unavailable on this gateway"whentalk.catalogdoes not offer them, Read aloud passes the same reason strings when the gateway gate is off or the client is offline (Read aloud), and the account menu’s notifications toggle passes"Not supported in this browser"where there is noNotificationconstructor. - Give it a
testId. The brand e2e asserts that each disabled control hasaria-disabled="true", a tooltip naming its phase, finding or reason, and sends no gateway frame when activated.
DisabledControl with a real element, update
the allowlist ADR if it needs a new method, and update the disabled-control e2e in the same change.
Assistant UI surfaces and the renderer registry
The assistant can compose inline UI (show_ui, the ui-surfaces plugin, ADR 0106) using the 14
declarative components from the shared contract catalog. The renderer that maps those component
names to React lives under apps/web/src/ui/surface/ (registry in SurfaceView.tsx,
components in components/).
Import rule. apps/web imports only ui-contract/types — never the zod-bearing modules and
never another validator library. The contract package owns the shape and the budgets; the browser
only consumes the parsed snapshot.
How to add a component.
- Add the component to the catalog in
packages/ui-contract/src/types.ts(COMPONENT_NAMESand the node interface) and its prop checks topackages/ui-contract/src/catalog.tsandsurface.ts. - Map it to a React component in
apps/web/src/ui/surface/components/and register it in the registry inapps/web/src/ui/surface/SurfaceView.tsxunder the samecomponentname. - Add tests: the contract package test proves the catalog + validation, per-component tests
apps/web/src/ui/surface/components/*.test.tsxprove keyboard operability and axe accessibility (SC11), and the web e2e proves the render (test/user-client-web-surfaces.e2e.test.tsviatest/helpers/user-client-web-harness.ts).
__openclaw.ui = {surfaceId, revision} is projected from the show_ui tool result’s
details.ui at chat.history read time, and the snapshot itself is fetched via
chat.surfaces.get. __openclaw.uiAction = {surfaceId, revision, actionId, label, title} rides
the user turn (stored as the openclaw:ui-action custom entry) and renders as the
“You submitted” chip; the submitted data is not in it — it reaches the model only in the composed
[ui-action] text. Both must survive a reload, which is why they persist as history rows rather
than in a scratch store.
Rendering follows the same identity: Thread builds a surfaceByCall map keyed by
toolCallIdentity (run key plus call id), stored refs first, and AssistantMessage renders a
surface inside its turn right after the tool block that published it, in every tool display mode.
A live ref (LiveSurfaceRef in state/surfaces.ts) carries the runId and toolCallId from
chat.activity; a live ref whose tool step is not drawn in the thread (another device’s run, or
the run a uiAction form submission starts, which the runtime does not track as its own) does not
render until the turn lands from history.
Session vocabulary
The UI says chat; the code says session. The gateway’ssessions.* rows are what the user
sees as chats.
Do not reintroduce “Space” or “Spaces” in UI copy or identifiers; that word is reserved for a
possible external Spaces extension, and the brand e2e asserts the rendered page contains neither. Session keys
look like
agent:<id>:<rest>; session-key-shape.ts parses them client-side without importing core
code, and ui/recency.ts groups rows into Today, Yesterday and Earlier by local midnight.
State and provider seams
Data flows in one direction.providers/web-client-runtime.ts(WebClientRuntime) owns the connection and implements both seams: theGatewayContextValue(providers/gateway.tsx, request-style calls such assendMessage,switchSession,listSessions,uploadFile,abortMessage) and theEventsContextValue(providers/events.tsx, subscriptions such asonChatChunk,onSessionChanged,onSessionsList,onConnectionStatus). It also has WednesdayAI-native members the vendored seams have no room for, which theapp/hooks call directly:listAgents,listModels,patchSession,prepareSessionTitle,resolveDirectSession,sendChatMessage(the onechat.sendseam),steerMessage,queueMessage,listActiveRunIds,adoptRestoredRun, thesessionListruntime, and theonSidebarSessionsChange,onActiveSessionRowChange,onQueuedRunStarted,onRunResumed,onToolStep,onOwnTurnStarted,onOwnTurnQueued,onSteerTurnSettled,onAssistantReplyandonActiveLeafConflictsubscriptions.onToolStepturns eachchat.activityframe of a run this client owns into a tool row in the reply (useChatStore.appendToolStep); a start frame’s optionalverb/detail(redacted, length-capped display strings resolved from the shared tool-display spec; the wire contract is thechat.activitybullet in Protocol) are stored with the row and never cleared by a later frame of the same call, andToolDisplayrenders them as the live chip’s label — for a call with no stored arguments yet, full mode shows the same title/verb/detail line instead of an invented{}dump.onRunResumedrestores the typing indicator on the first live frame of a run adopted after a reload;ui/assistant-turns.tsgroups a run’s rows into one reply block forui/Thread.tsx.Threadannounces each newly drawn tool step in a visually hiddenaria-live="polite"region (data-testid="tool-step-announcer"): the tool name only, no status cues, and nothing while the display hides tools.app/AppShell.tsxsubscribes to the events seam once per runtime, writes into the store (state/chat.ts,useChatStore) and composes the hooks described under Chat controls and hooks. Components read the store or receive props; they do not subscribe to the connection themselves.- The seam interfaces are kept in the shape of their upstream Saturday origin, including members
the web client no-ops (desktop context, accessibility). Provenance is in
apps/web/NOTICE. The Saturday-era seams with no web client consumer are gone, not no-oped: the events seam’sonContextUpdate,onSSEStatus,onCanvasUpdateandonCanvasSync, their payload types instate/types.ts(ContextUpdatePayload,SSEStatus, the Canvas shapes), and the store fields and actions they fed (sseStatus/setSSEStatus,contextEnabled/setContextEnabled,currentContext/setCurrentContext) are deleted. The wire and preference shapes (StatusResponse.context_enabled,Preferences.context_enabled) stay — they describe gateway data, not UI state. - Runs live in
runs/behindRunsController. Itsstart()andstop()are one boolean, so components take a reference-counted lease withuseRunsLease(controller)(ui/use-runs-lease.ts); the busy-state projection inapp/use-busy-sessions.tsholds one so the sidebar’s busy dots stay live.
Threaddoes not scroll itself. The scrolling element is the.cl-scrollwrapperAppShellplaces around it, so scroll code must drive that element. Auto-follow applies only while the reader is within 80px of the bottom.
Reconnect and send integrity
A socket drop or gateway restart during a turn must not show a reply twice, lose a sent message, or return the user to onboarding. The contracts live ingateway/connection.ts (transport) and
providers/web-client-runtime.ts (recovery), and the store guarantees live in
state/chat.ts and gateway/message-text.ts:
- Socket identity. Every listener in
openSocketis bound to its own socket, andhandleClose(ws)returns immediately when the closing socket is no longer the current one. A lateclosefrom an old socket therefore cannot reject the new socket’s pending requests, andforceReconnect(which closes the socket itself and runs the close path) schedules exactly one reconnect even though the browser also fires its owncloseevent. - Liveness watchdog.
hello-ok’spolicy.tickIntervalMs(default 30 000) arms a chainedsetTimercheck: no frame for two tick intervals closes the socket with code4000(tick timeout) and reconnects, mirroringsrc/gateway/client.ts.online,pageshowandvisibilitychange(to visible) retry immediately whilereconnecting, and force a reconnect when the resumed page’s socket is stale. A resumed page (also the Page Lifecycleresumeevent) whose connected socket has been silent for more than 5 s is probed withverifyLive(): onehealthrequest, and no answer within 5 s closes the socket with4000(liveness probe timeout) and reconnects. There is no per-RPC response timeout; the watchdog and the probe bound every hang on a dead socket. - Transient connect classification.
isTransientConnectFailuretreats anything that is not aGatewayRequestError, anerror.retryableflag,UNAVAILABLE,RECOVERY_NOT_READY, or a detail code inTRANSIENT_CONNECT_DETAIL_CODES(AUTH_RATE_LIMITED,DEVICE_AUTH_SIGNATURE_EXPIRED,DEVICE_AUTH_NONCE_REQUIRED,DEVICE_AUTH_NONCE_MISMATCH) as transient. After onboarding, a transient failure keeps the device credential and only reschedules the reconnect, honouringretryAfterMs(AUTH_RATE_LIMITEDbacks off toMAX_BACKOFF_MS). Any otherconnectrefusal — a credential the gateway no longer accepts (AUTH_DEVICE_TOKEN_MISMATCH) is the common example — clears the credential and re-runs onboarding. - Idempotent sends.
sendChatMessagegenerates oneidempotencyKeyper logical send andrequestWithSendRetry(providers/send-retry.ts) re-sends the identical params until the ack arrives, the turn stops being this client’s pending one (session switch, stop, abandon),connection.isStopped(), orSEND_RETRY_DEADLINE_MS(60 s) has elapsed.sessions.steer(also onCONFLICT/duplicate_in_flight, and voided by Stop) and the empty-statesessions.directuse the same loop. A send is retryable when the socket died with the request in flight (not aGatewayRequestError),error.retryableis set, or the code isRECOVERY_NOT_READY; the delay between attempts isMath.max(250, retryAfterMs). The gateway’s dedupe answers a repeat whose first attempt landed within_flightor the cached result instead of a second run, and a cached result for a turn whose final was missed settles the run locally throughsettleRunMissedDuringGap. - Dispatch guard.
request(type, params, { canDispatch })(packages/gateway-browser-client/src/connection.ts) re-checks a caller’s precondition after the readiness wait and immediately before dispatch:falserejects withgateway connection closedinstead of sending. A precondition that died while the request waited for reconnect readiness must not still fire - the steer path passes() => !preSteer.cancelled, so a Stop that lands during the wait voids the steer (web-client-runtime.ts). - Run reconcile.
reconcileOwnRuns(trigger)lists the open chat’s runs (sessions.runs.list) and settles every locally-owned run:runningorqueuedkeeps tracking and streaming,terminalor absent settles against the transcript, andaction-needed(interrupted by a restart) settles as aborted and emits the restart notice as achatError. It runs after every reconnect ("reconnect"), when a tracked run’s frames go quiet for 15 s ("stall",RunStallMonitorinproviders/run-stall-monitor.ts- a single re-armable timer the runtime touches on own-run registration and on every live frame of a tracked run, fired after averifyLive()probe), and when the tab returns with an own run in flight ("resume"). Afinalsettlement first reads the transcript and replays the run’s durable assistant text through the live stream path beforedone, so the bubble (and auto read aloud) get the whole reply at once. A failed status read changes nothing: tracking and streaming continue and Stop remains the user’s escape hatch. Queued follow-ups and an unacked pending turn are left to the existingloadHistoryand send-retry paths. - Store and stream dedupe.
prependMessagesdrops incoming rows whoseentryIdoridalready exists and sorts the merge withorderThread.addSyncedMessagepositions the run before it dedupes (applyPositionRun: the run’s queued turn un-queues andturnSeqanchors the rows that have no stored position), and a deduped sync still returnsorderThread(base), so the positioning lands even when the row does not. The sync is dropped only when the answer itself is already on screen: the run’s live text row carrying identical content (a final that extends the streamed prose updates the bubble in place, one with new text after a tool step appends after the step), or a provenance match — same-run assistant prose, or an earlier sync of the same event; the run’s question, its tool rows and a still-queued turn never suppress it.streamUpdate(gateway/message-text.ts) opens no second bubble when the final text differs from the streamed text only in leading or trailing whitespace. - Draft safety. Every outgoing
chat.sendandsessions.steertext is recorded in alocalStoragesend outbox (apps/web/src/providers/send-outbox.ts, keyopenclaw-user-client:outbox:v1, 24 h expiry) before the RPC leaves, and removed when the RPC answers. Each entry carries the text, its session key, the run ids its send attempts used, and a lease the sending tab renews every 5 s while the RPC is pending. A failed send restores its text to the composer from the outbox (or, with no entry, the in-hand text) in the chat it was sent from, next to the failure’s own error row, and consumes only that send’s own settled entry (restoreAfterFailureplusconsumeOutboxEntryinapps/web/src/app/use-outbox-restore.ts). Entries left behind by a reload, crash or gateway restart are restored after the chat’s next history load (restoreUnsent, “Restored a message that was not sent.”); entries whose run id appears in the loaded history are dropped silently, and entries still pending (this tab’s in-flight set, or a live lease another tab renewed) are left untouched. Queued follow-ups that have not started yet are deliberately not restored after a reload: the gateway’s own followup queue holds them across connection resets, and an unconfirmed record is not evidence of loss, so restoring would invite a duplicate send (dropUnconfirmedPendingQueuedTurns). The composer persists its text per session key underopenclaw-user-client:draft:<sessionKey>inlocalStorage(the empty state uses@new; 300 ms debounce plus a flush onpagehideand on hide, removed when emptied, dropped after 7 days), reads it back on mount, and merges a restore behind text typed meanwhile instead of clobbering it. Logout clears every draft and the outbox (clearAllComposerDrafts). Attachments are not persisted.
Chat controls and hooks
client-chat-v3 P2 made the assistant and chat controls live and splitAppShell.tsx into one hook or
provider module per concern, each testable without a gateway connection.
Completed history boundary and transcript controls
These contracts keep completed runs, session-list responses, and browser-local presentation state from overwriting newer or connection-specific state.ChatMessage is the client-owned message type in state/types.ts; the map relates current row
indexes to loaded row indexes. This is an internal client contract, not a plugin SDK export.
resolveCompletedHistoryBoundary accepts a terminal outcome of "final" or "aborted". It
finds the last loaded row matched to the completed run. When there is no completed-run match, it
falls back to the loaded row matched to the user message that preceded that run’s assistant turn.
If the completed assistant was not adopted, the outcomes diverge:
- For
"final", the resolver advancesloadedBoundarythrough subsequent loaded rows until the next user turn. This admits the server’s completed assistant output into the loaded boundary. - For
"aborted", it stops at the existingloadedBoundaryand does not scan forward. The merge therefore preserves the stopped text observed in local state instead of adopting a durable server suffix that was ahead of the stream when the abort landed.
pageCoversCompletedRun reports whether the loaded page actually reaches the completed run’s
durable assistant row: true when the run’s assistant output was adopted from the page, or when the
run’s turn is anchored in the page (its preceding user row matched) and the rows after that anchor
reach an assistant row. A final page that ends at the run’s user row — which is what the gateway
delivered before ADR 0089 made terminal events post-durable — does not cover it. When the outcome
is final and the page does not cover the run, reconcileCompletedHistory treats the streamed
assistant reply as still live instead of dropping it as already-represented; the next re-read whose
page contains the durable row replaces it through the ordinary entry and content matching. An
aborted run keeps the observed stopped text regardless, as above.
SessionListRuntime tags each request with a monotonically increasing seq. A response is applied
only while its captured mySeq equals the current sequence; otherwise it is silently dropped as
stale after a newer filter, search, sort, or page request. dispose() clears the pending 250 ms
search debounce and increments seq, preventing an in-flight Gateway response from landing on a
torn-down component. On sessions.changed, refetchLoadedWindow() starts at offset 0 and uses
limit = Math.max(sessions.length, pageSize), preserving every page the user already loaded.
Navigation and transcript preferences are per connection:
- Navigation uses
openclaw.web.navigation:<encodedGatewayUrl>:<clientId>, where the storage scope contains bothgatewayUrlandclientId. Defaults areassistantSort: "newest",activityWindowHours: 4,includeInternal: false,chatSortBy: "updatedAt",chatSortOrder: "desc",selectedAgentId: null, andlastChatByAgent: {}. - Transcript display uses the
openclaw.web.transcript-display.v1object, keyed within that object by<gatewayUrl>|<clientId>. Defaults areshowReasoning: falseandtools: "minimum". A stored legacytools: "minimal"preference loads as"collapsed".
showReasoning and tools: "hide" | "minimum" | "collapsed" | "full" are
strictly browser-local presentation state in DisplayMenu; they do not alter Gateway execution or the
session’s reasoningLevel. Tool detail strings render through the shared shortenHomePaths helper
(src/agents/tool-display-common.ts), which shortens absolute user-home paths to ~ (POSIX and
Windows) without Node-only APIs; both this app and the TUI ui package import it. Tool failures
remain visible even in "hide" mode:
null, so a tool-only assistant turn must not leave an empty
assistant bubble.
Web-client-specific footguns and their safe patterns:
- Do not apply a late session-list response after a filter or sort changes, or after unmount. Keep
the
mySeq === seqguard and invalidate the sequence indispose(). - Do not refetch only
pageSizerows onsessions.changed; doing so drops an already loaded second or later page. Refetch the full loaded window from offset 0. - Do not treat aborted and final reconciliation as equivalent. Advancing an aborted boundary can replace the exact stopped text the user saw with a durable suffix they never observed.
- Do not persist navigation or display preferences under an unscoped key. The same browser can connect to multiple gateways or client identities.
- Do not mutate the active row optimistically for model, thinking, reasoning, or label changes. Wait for the patch and refreshed session row; otherwise a refusal can leak one chat’s pending state into another.
- Do not hide tool errors with ordinary tool details. Error output is an alert even when tool display is hidden.
- Settings are not optimistic.
ModelMenu,ThinkMenuand the rename field patch the open chat throughWebClientRuntime.patchSessionand only ever show that chat’s own row (sessions.describe, refreshed on everysessions.changedfor it). A refused patch therefore needs no rollback: nothing on screen changed, and the control shows the gateway’s message (chat-controls-error). “Assistant default” in the model menu sendsmodel: null; Think sendsoff,low,mediumorhigh; Send reasoning sendsreasoningLevelonoroff.ThinkMenuis inert (aria-disabled, a tooltip naming the model, no frame sent) whenmodels.listreportsreasoning: falsefor the chat’s model. - Suggest never saves.
rename-suggestsendssessions.title.preparewith the chat’s assistant id and the first user message currently loaded (its first 1000 characters), and only fills the field (60 characters at most). Enter or blur saves throughsessions.patch {label}. A refusal, the rate limit or anulltitle showsrename-error. - Assistant picker.
AssistantPickeris opened bysidebar-browse-assistants. A pick sends exactly onesessions.direct {agentId}(a ref guard, not just React state, stops a double click); the gateway returns the caller’s dmScope-correct direct session key (ADR 0109). There is no model select and no create step. - History and the chat leaf.
useChatHistoryloads the newest page (HISTORY_PAGE_SIZE, 50) and pages older with the returnedcursor. When that page starts inside a reply (its oldest row carries a run id and the page holds no user row of that run), the load keeps paging older — capped atMAX_BISECTED_HEAD_EXTRA_PAGES(20 extra pages, 1,000 rows); beyond the cap it publishes what it read with the cursor of the last page read, so “Load older” continues from there — until it reaches the reply’s user row, a row of another run, or the start of the history, and publishes the combined rows at once, so a reload never shows a reply without its question.WebClientRuntimerecords the chat’s leaf from its newest-pagechat.historyreads (the thinking reconcile opts out withnoteLeaf: false); abeforepage never updates it.LeafTrackertells “never learned” (undefined: the send omitsexpectedLeafEntryId) from “known empty” (null: the send asserts an empty chat). After a final, aborted or error event for a run this client owns,readLeafUntilStablere-reads withlimit: 1until two reads agree (every 25 ms, at most 40 attempts) and a send waits for it. A final for a run this client did not start never settles the leaf directly; it schedules a transcript re-read (settleThenRefreshRows), and thatforeign-turn/foreign-turn-preserve-localpage does note the leaf — only the own-runcompleted-turnreconcile opts out (AppShellsetsskipLeafNoteRefforcompleted-turnalone) — so a client that has shown the other device’s turn stops tripping the guard, while an unseen turn is still caught. Returning to a visible tab with a tracked leaf, an active chat and no own turn in flight reads the leaf once and emitsforeign-turnwhen it moved. On a firstactive-leaf-changedrefusal the send is still accepted (VAL-LEAF-002 reversal): the runtime re-learns the leaf (refreshAfterUnseenChanges), merges the unseen rows while keeping the optimistic user row (foreign-turn-preserve-local, no notice), and resends once automatically with a freshidempotencyKey(sendChatMessageOnce(params, "leaf-retry")); switching chats while that refresh is in flight cancels the resend (selectionSupersededError) and the caller restores the draft. Only a second consecutive refusal takes the pre-reversal path —recoverFromActiveLeafConflict, the thread reload,activeLeafConflict(thechat-changed-noticebanner) andActiveLeafChangedError, whose draft restoreuseChatSendowns — so two back-to-back foreign appends cannot loop. - Steer and Queue. While a reply streams, or a queued turn is still waiting, Enter or Send
steers into the current turn via
sessions.steerwithmode: "inject"andfallback: "queue"(unless attachments are staged, which inject refuses, in which case it sendsmode: "queue"). The send menu item “Send after reply” (composer-send-menu-queue) and desktop Ctrl/Cmd+Enter sendsessions.steerwithmode: "queue"and show the turn at once withmessage-queued-badge. Both sendfallback: "queue". The badge is presentation only; the turn’s place in the thread isorderThread’s derived tail until the run’s first frame clears it (see the order bullet below). Neither can carry a reply: the composer refuses both while a reply chip is set, becausesessions.steerhas noreplyToId. On touch-primary devices (coarse pointer), plain Enter inserts a native newline, and Ctrl/Cmd+Enter runs the primary action (steer while streaming, queue with a staged file, send when idle) instead of queueing. A queued turn is saved insessionStorage(openclaw.web.pendingQueuedTurns.v1, dropped after 24 hours) until its run drains, so it survives a reload or reconnect;useChatHistoryshows it again only whensessions.runs.liststill reports its run. - Replies. Only a message with a real entry id (
__openclaw.id) offers Reply. It sets thereplyChipin the store (composer-reply, excerpt up to 140 characters), and the nextchat.sendcarries its entry id asreplyToIdand clears the chip, whether or not the send succeeds. The quote renders from__openclaw.replyToasmessage-reply-quote, live and after a reload. - Chat list.
SessionListRuntimeowns the assistant filter, the search (debounced 250 ms),loadMore(the nextoffsetis the number of rows already loaded) and thesessions.changedrefetch, which re-reads offset 0 up to the rows already loaded so later pages are not dropped. Every request carries a sequence number and a superseded response is dropped. The open chat’s own row is fetched separately, so the top bar stays live when a filter or search hides the chat from the list. Rows show the exact session key instead of a message preview; the client no longer requests unused last-message previews.sessions.listrows carryinternal: truefor internal sessions (cron, hook, heartbeat, subagent and acp keys), and the Sidebar groups those rows under an Internal label. They are hidden by default (includeInternal: false); the toggle includes them again. Gateway sorting precedes pagination; title sort uses labels before derived titles. Global title enrichment caps transcript reads at 16 at a time and yields between batches; normalized sort keys are computed once and are not returned.includeDerivedTitlesremains opt-in: callers displaying the effective sorted titles should request it explicitly.agentActivityaggregates the newest eligible session timestamp before assistant, search, activity-window and pagination narrowing. The assistant list defaults to a 4-hour window, pins the selected assistant, and refreshes its cutoff once per minute. An agent-filtered page merges itsagentActivityinto the existing map (only the requested agent’s slot is refreshed) instead of replacing it, so filtering never blanks the other assistants’ activity.selectAgentalso records the id inpickedAgentIds, a session-only list that is never written to preferences; theSidebaralways lists picked assistants past the activity cutoff, whatever the current filter, and only a reload clears the record. The default Recent sort (assistantSort: "newest") lists assistants whose activity falls withinRECENT_ASSISTANT_WINDOW_MS(24 h) newest first, then everyone else alphabetically (case-insensitive, locale-aware, id as the last tie-break); equal timestamps sort by name.useAssistantOrderfreezes the rendered order:sessions.changedrefetches update the map without moving rows. It re-sorts on the first loaded page, a windowfocus, avisibilitychangeto visible, the resume refetch landing (refetchForResumebumpsassistantOrderEpoch), and a sort or activity-window menu change. Between re-sorts, newcomers append at the bottom and removed rows drop without moving the rest. - Assistant navigation. Clicking an assistant scopes Sessions and opens that assistant’s
dmScope direct session (
sessions.direct { agentId }). The resolved key becomes the remembered session that startup restore reopens. Re-clicking while already on that key re-affirms the filter without a history reload. A failed resolve clears the selection and showsCould not open that assistant: …. Picking an assistant in the picker and first-sendsessions.directresolution keep the assistant selected too. Every actual activation invalidates older restoration callbacks, so delayed startup lookups cannot replace a newly selected session. The selection is visible during lookup; the draft remains editable, but submission waits for lookup to finish or be superseded. Changing visibility only refilters a settled chat; an already pending assistant lookup restarts with the new eligibility rules. Creation revisions reject late creation replies. Separate send-navigation revisions invalidate leaf-gated sends on genuine navigation, but not onAll assistants, the already-selected assistant, or the same open chat.All assistantsexplicitly clears the scope. Navigation preferences and remembered selections are scoped by gateway URL and client ID in browser storage; malformed remembered entries are discarded. - Display is not delivery.
DisplayMenustoresshowReasoning: falseandtools: "minimum"by default, separately fromreasoningLevel.extractTranscriptBlockspreserves ordered text, thinking, tool calls, results and media;Threadpairs calls and results by their run-scoped identity (run key plus call id) across loaded pages. Legacytoolroles and snake_case result metadata remain tools, not assistant prose. Hidden tool details retain errors, and wholly hidden execution turns leave no empty assistant row. Local final/aborted turns trigger acompleted-turnreconciliation after the leaf settles, even while a newer local or queued run exists. The completed run ID and terminal outcome bound that snapshot: a final replaces its stale partial, while an aborted run keeps the observed stopped text rather than an ahead-of-stream durable suffix.completed-history-boundary.tsexcludes newer assistant/execution data but admits matched newer durable users. Those rows keep their client IDs and run ownership; the queued marker clears on the run’s first frame (positionRun), not on storage adoption. The merge retains newer partials, optimistic users, queued turns, older pages and errors without advancing the leaf guard. Reconcile decides only which current rows survive the merge:placeUnmatchedRowsreturnskeptin arrival order and places nothing. The assembled page isorderThread([...reconciledLoaded, ...kept]), so a kept client notice (run error, failed send) lands right after the run it reports on through itsanchorRunId/anchorSeq, never below a later reply of that run. Legacy identity-less matches are one-to-one FIFO within durable-entry anchor intervals, so identical distinct turns remain distinct. Scoped loaded-history and older-page IDs distinguish replaceable heads from optimistic rows and preserve imported legacy entries. Superseded older-page responses cannot splice into a replacement load. Events received beforechat.sendacknowledges its run are buffered for exact attribution. A rejected send recovers the foreign question/reply through the non-destructive merge, including first-send-created chats, without deleting its own optimistic question or refusal. A durable foreign terminal event clears the fresh-empty-chat skip guard just like a local completion. Send failures belong to their original optimistic turn: an obsolete rejection cannot rewrite a newer chat or foreign reply. Distinct assistant message IDs remain distinct even with identical text. Media links and derived attachments sharenormalizeMediaUrl: HTTP(S) and inline raster images/WAV are allowed; unsupported schemes and data types remain non-interactive metadata. - Order is server sequence. The thread order is a pure function of gateway positions, never of
insertion order.
gateway-mappers.tsstamps each history row’s transcript position ontoChatMessage.seq(__openclaw.seq, falling back to the integer form of__openclaw.rowId); livechatandchat.activityframes carryturnSeq, the transcript position of the run’s first user entry, whichweb-client-runtime.tscopies into every store write. One comparator owns placement,orderThread(messages)instate/thread-order.ts: a row with its ownseqsorts there; a still-queued turn sorts into the derived tail (after every positioned row, in send order among themselves); a live user row whose run’s reply is loaded but whose stored row sits on an older page sorts just above the run’s first positioned row; any other unpositioned row follows its run’s last positioned row withanchorSeqas the fallback; a client notice follows the run named byanchorRunId; a row with no positioning input keeps its arrival slot. Every store action that changesmessagesreturnsorderThread(next), and an already-ordered input returns the same array reference (render isolation).seqWatermarkanchors new optimistic rows after everything positioned. A sent message therefore always renders above its reply and above later turns, through streaming, queue and steer, reconnects and reloads. “Queued” is a badge only: it clears on any frame of that run (positionRun), when a history load carries the stored message, or when a later own turn takes its transcript position (the gateway drains queued turns first in, first out, so a positioned turn proves the turns queued before it started). - Colours.
ui/assistant-color.tsmaintains a browser-safe palette identical, in order, toASSISTANT_IDENTITY_COLORSinsrc/shared/assistant-identity-values.ts. A configuredidentity.colorwins; otherwise FNV-1a 32-bit of the agent id modulo 8 picks one, so an unconfigured assistant keeps its colour across reloads and devices. The--wai-assistant-<name>and-fgtokens are inpackages/brand-tokens/tokens.css; ADR 0084 section 6 records the contrast guarantee. - Names and avatars follow the open chat.
AppShellderives the active chat’s assistant once (activeAssistantName/activeAssistantAvatar): the top bar, the thread’s bubbles and typing indicator, the empty state, both composers and the reply-chip sender all take it, and an agent with no configured name shows its id. While the chat’s assistant cannot be resolved (agents.liststill loading, or a session key with no agent id) those surfaces fall back to the brand name, and the turn avatar falls back to the brand logo image. The avatar glyph comes fromui/assistant-avatar.ts(assistantAvatarText: the configuredemoji, else the name’s first two graphemes uppercased) — the one source theSidebarandAssistantPickerrender too. It reaches a turn through the optionalassistantAvatarprop onThreadandAssistantMessage: a coloured, aria-hiddencl-msg-avatar cl-agent-avspan (data-testid="assistant-avatar") when set, the logoimgwhen undefined. Do not inlineslice(0, 2)initials or read the brand constant in a chat-scoped component; take the props. - Menus.
useDismissibleMenucloses a menu on an outside mousedown and on Escape, returning focus to its trigger.ModelMenu,ThinkMenuand the Composer send menu all use it; do not copy its body into a new menu.
Dictate and Voice
Dictate and Voice mode are Composer controls (test idscomposer-dictate and composer-voice),
not phase placeholders. When talk.catalog does not offer a control it renders through
DisabledControl with reason="unavailable on this gateway"; when it is offered it is a real
toggle button: aria-pressed plus the cl-composer-btn--active style while it is on
(recording for Dictate; listening, thinking or speaking for Voice mode), and a
data-dictate-status / data-voice-status attribute mirroring the hook state — the error value
gets the danger outline from client.css, and the next click retries. features.voice is
ignored.
Availability comes from talk.catalog, and updates on reconnect:
- Dictate is live when
conversationEnabledis true andstreamingTranscriptionhas aconfigured: truerow. - Voice follows
conversationEngine. With"stt-tts"(the default, also used when the field is absent) Voice is live only when Dictate’s conditions hold andspeechalso has a configured row. With"realtime"Voice is live whenconversationEnabledis true andrealtime.activeProvideris not null;talk.session.createthen sendsmode: "realtime"and capture runs at the result’sinputSampleRateHz(24 kHz).
useDictate returns status unavailable | idle | recording | transcribing | error, a live cumulative
transcript while recording, commit: { text: string; nonce: number; autoSend: boolean } | null,
consumeCommit(): void,
and errorMessage (mic denied or no input device, a microphone that stops sending audio, or a
transcription relay that ended the take). The live transcript previews in the textarea as a
read-only draft with an accent border and muted text (.cl-composer--dictating), without breaking the
textarea height cap. When dictation stops or encounters a capture error, commit receives the final or
last-captured transcript and commits into the draft at the caret position via insertDictation; the composer
calls onDictateCommitConsumed to clear the commit.
VoiceSeam (providers/voice.ts) starts the mic inside the toggle click and requests the session
concurrently: capture.start runs before any await, so AudioContext creation and resume()
happen while the tap’s user activation is fresh (packages/talk-client/src/audio-capture.ts
constructs and resumes the context before the getUserMedia await — a context created after it
can stay suspended on a slow phone link and produce zero frames). Frames captured before the
session id lands and session.ready arrives buffer in order and are sent once ready, so the relay
never drops a take’s first words; stopping a not-yet-ready session waits up to 10 s
(SESSION_READY_TIMEOUT_MS) before it closes, so the buffered frames are not lost. A dead take
ends visibly: a 3 s frame-silence watchdog (CAPTURE_SILENCE_TIMEOUT_MS, re-armed per frame)
retires the session with talk.session.cancelTurn and reports Your microphone stopped sending
audio. Tap Dictate to try again., and a session.closed that arrives mid-take with no stop in
flight reports Transcription stopped unexpectedly. Tap Dictate to try again. without closing
the session. A rejected talk.session.create stops the capture it already owns without waiting
for its acquisition, so a hung getUserMedia cannot leave the take recording; the next tap issues
a fresh create.
One-tap dictation (useDictate(runtime, { autoSend, autoSendDelayMs })). Both options are
take-start snapshots: autoSend is the auto-read preference (AppShell passes
readAloud.autoRead) and autoSendDelayMs is the persisted auto-send pause
(providers/auto-send-delay-preference.ts). DictateCommit.autoSend records that value, so
toggling either option mid-take cannot change what happens to text the take already produced.
While autoSend is true at take start:
packages/talk-client’screateSilenceEndpointer(exported from the package index) ends the take without a second tap.providers/voice.tscreates it per recording with the take’ssilenceMsand feeds every PCM16 capture frame; it firesonEndpointonce, after at least 250 ms of above-threshold speech (pcm16RmsDbfsat the same −45 dBFS threshold the barge-in detector uses) is followed by the configured silence window. A burst shorter than the minimum never endpoints, andreset()re-arms for a fresh utterance. The endpointer is advisory capture-side timing only - the transcription stream and the gateway session are untouched.- The silence window is the composer’s auto-send pause choice
(
ui/AutoSendDelayMenu.tsx, chipdata-testid="composer-auto-send-delay", menucomposer-auto-send-delay-menu, optionscomposer-auto-send-delay-option-1500|2500|4000), rendered right after the Read replies aloud toggle and only while that toggle is on. The choice is 1.5 s, 2.5 s (default) or 4 s, stored per gateway URL and client id underopenclaw.web.auto-send-delay:<encodedGatewayUrl>:<clientId>; anything else in storage loads the default. It is browser-local only; there is no operator default. ui/use-dictate-autosend.tssubmits the committed text exactly once per commit, and only when three things hold: the commit’s text is non-empty and not whitespace (a denied mic or an empty take never sends, and never sends the user’s pre-existing draft), the dictated text is observable in the draft (the insertion is staged asynchronously, so the armed send waits for the re-render that contains it), and the composer can deliver it - while a reply is streaming the commit routes throughsubmitWhileStreaming, the same path Enter takes (steer into the current turn, or queue when attachments are staged) instead of stalling in the draft, whilesubmissionDisabledoruploadingleave the text in the draft.- The composer marks the draft dictated before the auto-send runs, so the send carries
{ dictated: true }provenance like a manual send after dictation.
AppShell’s dictate toggle runs readAloud.unlock() inside the
click (the AudioContext gesture unlock, which auto-read also needs) and then, when a take is
starting and a read is active, readAloud.pause(). Stopping the take never touches playback, so
the read stays paused; only the message control’s Resume reading, or starting another read,
continues it.
Screen wake lock. While any voice path is live — a dictate take that is recording or
transcribing, active Voice mode, or read-aloud playback that is not paused —
app/use-screen-wake-lock.ts holds a best-effort navigator.wakeLock request. A missing
wakeLock (unsupported browsers, iOS home-screen apps before 18.4, insecure contexts) is a plain
no-op, request rejections (for example Low Power Mode) are swallowed, and the lock re-acquires on
visibilitychange when the page becomes visible again.
useVoice returns status unavailable | idle | listening | thinking | speaking | error,
captions: { user, assistant }, and engine: "stt-tts" | "realtime" (the voice engine the
gateway resolved). Toggle starts or stops the session; thinking counts as on. A start rejection
is swallowed here; the seam’s error status is what the button shows.
Cascaded stt-tts turns render as chat turns. The gateway opens an agent run for each spoken
utterance and the client registers that run as its own, so when the turn starts the spoken words
become a normal user row in the thread and the reply streams as a normal assistant bubble - no
browser chat.send is issued for a voice turn. Until then the live user transcript previews
inside the composer textarea with Dictate’s read-only preview styling: a display-only composition
over the untouched draft, never written to the draft or its storage, cleared when the turn starts
and the user row commits. stt-tts renders no caption box, and reply text never echoes into the
composer. Turning Voice off mid-reply stops the microphone and the speech, but the agent run
still finishes: the full reply persists to the chat exactly once and appears in the thread. A
reload or disconnect mid-reply keeps the heard prefix plus an [interrupted] marker instead.
With engine: "realtime", while Voice is listening, thinking or speaking, the Composer
renders VoiceCaptions
(ui/VoiceCaptions.tsx, data-testid="voice-captions", aria-live="polite") above the input
row: an orb cl-voice-orb with a --listening, --thinking or --speaking modifier (its pulse
stops under prefers-reduced-motion), the status label, and two caption lines,
cl-voice-caption--user and cl-voice-caption--assistant (rendered only while its text is
non-empty). The caption region
(cl-voice-captions__scroll) is height-capped against the shell canvas
(--shell-canvas-height), not the layout viewport, so a shrunken keyboard canvas still leaves
the thread a scrollable floor; it scrolls instead of growing the composer, and its
column-reverse layout pins the scroll origin to the newest line, so an active conversation
stays in view without yanking a reader who scrolled up. The user caption is replaced by each
transcript.delta/transcript.done; the assistant caption is cleared on turn.started and
appended from output.text.delta (realtime deltas are raw increments concatenated verbatim).
tool.call sets thinking. On transcript.persisted the
runtime re-reads the active chat, so the persisted turns appear in the thread without a reload.
While either feature is live, switching chats is locked. The runtime derives the lock
synchronously from the live talk seams (navigationLockReason: a dictate take from its first
toggle click, including the lazy create of a first chat, until its stop settles, whether the
stop succeeds, fails, or the gateway drops; a Voice conversation while listening or
speaking (or thinking); dictation wins when both are live), so a guard holds in the same tick the feature
starts — no React effect latency. A user-initiated switch — a sidebar-session row, the unread
bell, an assistant chip or the assistant picker — is then refused: no session change and no
sessions.direct/history traffic, plus a role="alert" notice (chat-switch-blocked) saying
exactly what to stop: “Stop dictation to switch chats.” or “Turn off voice to switch chats.”.
The runtime emits onNavigationBlocked for every refusal and onNavigationLockChanged when the
lock changes; the notice’s reason follows the lock (a dictation→voice handoff re-labels it), and
it clears when the runtime lock clears. A sessions.direct
response that lands after the lock leaves the resolved row unselected in the sidebar
instead of switching. The lock only gates user intent: startup restore (forceSwitchSession),
send-create of a first message, and a chat deleted elsewhere still switch — a live Voice
conversation is stopped cleanly when its chat is deleted elsewhere, while a Dictation take
(session-independent) keeps running and commits into the draft. An assistant lookup that settles
while locked just releases the pending navigation without touching the open thread.
Playback and barge-in come from the talk-client package (createTalkAudioPlayback,
createBargeInDetector), not from wednesdayai/plugin-sdk. Call playback with preRollMs: 40
and echoSafeSink: true for Voice. Both options default off (preRollMs 0, echoSafeSink false),
which is what the operator console keeps. preRollMs delays only the first chunk of a fresh burst.
echoSafeSink routes through a MediaStreamAudioDestinationNode
into an <audio> element so echo cancellation has a real media element.
The barge-in detector stays disarmed except while kind: "speech" audio is playing. Do not arm it
for filler or ack. Re-arm it on output.audio.resumed while speech is still active, and do
not disarm it in the gap between speech chunks. Voice uses a 150 ms minimum burst (the package
default is 250 ms) at the package’s −45 dBFS threshold. On detection, pause playback locally
before talk.session.bargeIn. Those two timestamps are performance.now(). Speech
talk.session.mark timestamps are played-audio milliseconds on the turn timeline (0 at the first
speech sample), attached when the chunk is enqueued, not at turn.ended. Realtime sessions
differ: attach the mark once, at output.audio.done, and acknowledge with a session-monotonic
timestamp that never decreases — per-turn offsets would drop every ack after the first
(shorter) reply.
If the gateway socket drops, stop the microphone immediately and return Voice to idle, including
when the drop happens while start is still in flight. Do not wait on talk.session.close: that
RPC blocks until the socket is ready again. Dictate stops capture the same way, without waiting
for a transcript that can no longer arrive.
Frame shapes are in Talk sessions.
Composer paste, rich copy and attachment prep
Four composer-side contracts shipped with the 2026-10-25 chat-flow slice. None adds a gateway method; they shape what the browser sends on the existingchat.send attachments and composer
draft paths.
Paste: use-composer-paste.ts and html-to-markdown.ts
ui/use-composer-paste.ts returns the composer’s onPaste handler with a fixed priority order:
HTML → markdown at the caret, files → the same uploadFiles staging path the picker and drop
use, anything else → the browser default (the event is not prevented, so paste without
text/html — Shift+Cmd+V — and plain-text paste keep working).
- The HTML conversion resolves asynchronously, so the paste captures the active session key and
the draft at paste time (
getSessionKey) and drops the completion if either changed: a chat switch during the wait would otherwise insert into the destination chat’s draft at the source chat’s caret. - The undo-preserving insert path is
document.execCommand("insertText")— a real editing transaction that keeps Cmd/Ctrl+Z history and firesinputfor the controlled draft. It needs a focused element and is absent in jsdom, so the fallback writes the controlled value withsetText: same text and caret, but no undo history. The caret is restored after the inserted text once React has committed (next frame) in both paths. ui/markdown/html-to-markdown.tsconverts clipboard HTML to GFM markdown: it parses withDOMParser, prunes comment nodes plusscript, style, meta, link, title, template, iframe, object(the walker must come from the parsedDocumentitself — aDocument.ownerDocumentis null, so walking a bare node yields no walker and nothing would be pruned), drops images whosesrcis nothttp(s), then converts throughhast-util-from-dom/hast-util-to-mdast/mdast-util-to-markdown/mdast-util-gfmwith-bullets,_emphasis and backtick fences, collapsing 3+ newlines.
hast-util-from-dom and hast-util-to-mdast load through one Promise.all
of dynamic import()s inside html-to-markdown.ts; only import type references to them may
appear anywhere else in the app graph. A static import would pull them into the entry chunk and
break the lazy-chunk budget (V5): they must stay out of index.html’s transitive static imports
and be reachable only via dynamicImports. mdast-util-to-markdown and mdast-util-gfm are
imported statically, because remark-gfm already ships them in the entry chunk and a dynamic
import of them cannot split (Rolldown reports INEFFECTIVE_DYNAMIC_IMPORT).
Rich copy: clipboard-html.ts
cleanClipboardHtml(source: DocumentFragment | Element[])clones the copied prose and strips everything non-semantic: onlyhref,src,alt,colspanandrowspansurvive as attributes,href/srconly forhttp(s)andmailto:URLs (SAFE_URL_PATTERN), andbutton, svg, script, style, [aria-hidden='true'], .cl-msg-actionschrome is dropped. The result pastes into Outlook or Docs with the structure (headings, lists, tables, bold) and the receiving document’s own styling — no colours or fonts from the chat.writeRichClipboard({ html, text })writestext/htmlplustext/plainthroughClipboardItemwhen the browser supports it and falls back towriteTextwhen it does not — or when the rich write itself is rejected. Best effort in both directions; a failed copy is never surfaced as an error. The plain slot carries the reply’s markdown source, not the rendered text.
Attachment prep: image-resize.ts
prepareImageAttachment(file, codec?) runs at the top of Composer’s uploadFiles, before the
byte reservation, and returns a possibly re-encoded File:
- Only
image/(jpe?g|png|webp|heic|heif|avif)MIME types are touched; everything else — GIFs included — attaches untouched. - A failed decode returns the original file unchanged (the ordinary over-cap checks then apply).
- Downscale to fit 2048 px on the long edge; never upscale — a small image over the byte cap (a huge 1×1 PNG) re-encodes at its own dimensions.
- PNG and WebP encode as PNG first; a PNG result still over
CHAT_ATTACHMENT_MAX_BYTESre-encodes as JPEG at the same size. JPEG quality is 0.85 (IMAGE_JPEG_QUALITY). - The output keeps the original name with the matching extension (
photo.pngcan come back asphoto.jpg). An over-cap image that no codec run can bring under the cap is returned unchanged and is then rejected by the ordinary checks (finding #1330 tracks a real upload path for those). - The codec is injectable (
ImageCodec) so unit tests pin the policy;browserImageCodecdecodes with EXIF orientation and usesOffscreenCanvaswith an HTML-canvas fallback.
Read aloud
The Read aloud action on assistant messages speaks that reply through the gateway’s per-agent TTS config (messages.tts merged with agents.list[].tts; ADR 0102, wire shapes in
TTS read aloud). app/use-read-aloud.ts
owns the state: one tts.readAloud.status read per runtime, agent and connection, one shared
ReadAloudPlayer, and a ReadAloudControls {unavailableReason, activeKey, paused, phase, error, toggle, pause, resume, stop}
value. AssistantMessage takes it as readAloud plus readAloudKeys from Thread: the reply
block’s turn key first, then each of its rows’ ids. The buttons target the first key unless
playback is active under another member (auto-read streams under a row id), and an error or the
playing state shows on the reply when any of its keys matches.
unavailableReason is first match wins: unavailable while offline when disconnected,
unavailable during Voice mode, unavailable on this gateway when the status reports
enabled: false. A failed status read gates the control off instead of guessing the feature is
live, and only a status for the current agent counts. When the player reports a
read_aloud_disabled refusal, the hook bumps its status revision so the gate re-reads instead of
staying stale. Playback stops when the session changes or Voice mode engages; a connection drop
does not stop it — the player keeps playing across a reconnect, and the manual control still
reports unavailable while offline until the socket is back.
providers/read-aloud-text.ts turns the reply into speech chunks:
messageMarkdownForSpeechprefers the message transcript’stextblocks over the streamed content.extractSpeakableUnitsparses the markdown with remark (GFM) and keeps one unit per paragraph, heading and list item, recursing into blockquotes, while dropping code blocks, inline and block html, tables, rules and metadata nodes. Unpunctuated units get a trailing period so the voice pauses at the unit end.packSpeechChunksjoins units with blank lines under the gateway’smaxTextLength, splitting an oversized unit on sentence boundaries (Intl.Segmenter), then on whitespace. Every chunk is 1..max characters and no non-whitespace text is lost.
providers/read-aloud-player.ts is the framework-free playback engine. start runs synchronously
inside the click handler — it creates and resumes the shared AudioContext there, which unlocks
Safari/iOS playback from the user gesture — then fetches chunk 0 and prefetches the next chunk
while the current one plays, with at most one tts.speak request in flight. pause() suspends
the shared AudioContext, freezing the sounding buffer mid-playback so the play loop parks
instead of ending; resume() resumes the context and runs inside a click, satisfying Safari’s
gesture rule. paused is a flag separate from the play loop’s phase, so the loop’s own state
writes can never clear it. unlock() is a no-op while paused, so the enabling gestures (turning
auto-read on, the dictate gesture’s unlock) cannot silently un-suspend a paused read. A
generation counter makes stop(), or starting another message, retire in-flight work silently: a
late synth promise never surfaces audio or errors. stop() clears paused without resuming the
context — the next read’s unlock() does that. A transient synth failure — a socket close,
UNAVAILABLE, or an error the gateway marked retryable — is retried up to two extra times with
500 ms, then 1 s, backoff (honouring retryAfterMs), so a blip mid-reply does not end the read;
stop() cancels a waiting retry through the generation counter, and a read_aloud_disabled
refusal is never retried. describeReadAloudError maps a
read_aloud_disabled refusal to
Read aloud is turned off for this assistant., any other gateway error to
Couldn't read aloud: <message>, and a decodeAudioData failure to
Couldn't play the audio in this browser. The manual start is now built on the same primitives as
the streaming queue: startStream(key, agentId) opens an empty queue, pushChunks(key, chunks)
feeds it (prefetching the next chunk as soon as one exists), and endStream(key) closes it once the
loop has drained. unlock() must run synchronously inside the enabling gesture, the same
AudioContext-resume contract as start.
Markup contract:
- The main button carries
data-testid="message-read-aloud",aria-labelandtitleRead aloudwhen idle,Pause readingwhile this message reads, andResume readingwhile the read is paused;aria-pressedwhile active,aria-busywhile loading,data-read-aloud-phaseset toloadingorplayingwhile this message reads (idleotherwise), anddata-read-aloud-paused="true"while the read is paused. - While this message reads or is paused, a second button
data-testid="message-read-aloud-stop"labelled and titledStop readingstops the read outright: it clearspausedwithout resuming the context, and the next read’sunlock()resumes it. - The actions row gets
data-reading="true"while this message reads or shows an error. - An error renders a
role="status"span withdata-testid="message-read-aloud-error".
Auto-read (Read replies aloud)
The composer’s Read replies aloud toggle (data-testid="composer-auto-read", speaker icon,
aria-pressed plus cl-composer-btn--active while on) turns streamed read-aloud on. Its props are
optional (autoRead, onToggleAutoRead, autoReadUnavailableReason), so an unwired composer
renders it off. When the reason is set it renders through DisabledControl with the same reason
strings the message button uses; when live it plays every new reply of the open chat while it
streams.
- Preference.
providers/auto-read-preference.tsstores the toggle per gateway URL and client id underopenclaw.web.auto-read:<encodedGatewayUrl>:<clientId>("1"/"0", default off) - the same scoping rule as the navigation preferences. It is browser-local only; there is no operator default (finding #1347). - Chunking.
providers/read-aloud-stream.tsis the incremental counterpart ofpackSpeechChunks:update(markdown)returns chunks for every newly completed sentence (one perextractSpeakableUnitsunit, split by the sharedsplitSentences), holding back the last sentence of a still-growing unit;finish(markdown)flushes the remainder. It re-parses only when the appended tail can complete a sentence, survives a markdown reflow that reparses to fewer sentences than already emitted, and packs through the samepackSpeechChunks— so streamed and manual reads of the same text produce the same speech. - Driver.
app/auto-read-driver.tsturns the chat store intopushChunkscalls. One stream per run: the stream key is the run’s first tracked assistant id, and the stream stays open across every assistant message the run appends (a message boundary flushes the previous message’s pending sentences into the same stream and replaces only the chunker — a secondstartStreamwould stop playback and drop the queued tail). At the run boundary it flushes and closes.reset(seed)marks every assistant id on screen as known, so a reply that was already streaming when auto-read turned on is never spoken from its beginning; the driver abandons a run the moment the player stops serving its stream (toggle off, chat switch, Voice mode, a manual read). - Gates.
use-read-aloud.tssubscribes the driver only whileautoReadis on, the status reportsenabledand Voice mode is off — not on the connection: a blip mid-reply must not tear the subscription down, or the reconnect’s re-seed would mark the in-flight reply known and silence the rest of it; the cleanup unsubscribes, resets the driver and callsplayer.stop(), so queued speech never outlives the gates. The streaming chunk limits (first chunk 200, later chunks 600 characters) are clamped to the status’smaxTextLengthand refreshed whenever the status or agent changes, so a low gateway limit never produces atts.speakrejection. - Gesture. Enabling the toggle, and starting Dictate with auto-read on, unlock the audio
context synchronously inside the click (
readAloud.unlock()), keeping Safari/iOS playback audible.
player.stop(), and so does the message’s Stop reading button: the driver abandons the
run the moment the player stops serving its stream, so its queued and remaining sentences are never
spoken. A disconnect does not stop playback; the stream continues across the reconnect. Pressing
the message’s main button during a streaming read only pauses it; Resume
reading continues from the same point. A normal run ending (no stop) flushes the reply’s
remaining sentences into the stream and the queue drains to the end of the reply. An in-flight
tts.speak request is not cancelled on stop; the generation counter retires its result instead of
surfacing the audio.
Busy state, unread badges and reply notifications
Three layers, kept separate: the runs projection says which chats have a run in flight, the store carries the unread chat keys the sidebar renders, anduse-reply-notifications owns the
per-device unread state and turns hidden-tab replies into optional desktop notifications.
Busy state
selectBusySessionKeys(rows)(runs/runs-model.ts) maps thesessions.runs.listprojection the sidebar already reads to the session keys with a run in flight: rows whosegroupForSessionRunStatus(status)isqueuedorrunning, excludingkind === "heartbeat"(heartbeat output is suppressed in chat; counting it would pulse the busy state on every tick).useBusySessionKeys(controller)(app/use-busy-sessions.ts) reads the projection withuseSyncExternalStoreand holds auseRunsLease, so it stays live while the shell is mounted even when the Sidebar is not.AppShellORs the open chat’s key into the composer’s streaming state, so Stop sendschat.abort { sessionKey }and Enter steers viasessions.steerfor a run this client never started — one running in the same chat from another device, or one that predates a reload.ui/TypingIndicator.tsxis the three-dot indicator:role="status"witharia-label="<assistant> is working", the dotsaria-hidden.Threadshows it while itsisBusyprop is true (not only before the first streamed text) and setsaria-busyon the thread element;EmptyStateshows it while a still-empty chat’s first turn runs elsewhere. The sidebar marks busy chats withsidebar-session-busy(a pulsingcl-sb-busydot,role="img",aria-label="Working"); both pulses are disabled underprefers-reduced-motion.
Unread badges and the title count
- The runtime broadcasts
assistantReply { sessionKey, runId, text, at }on its events bus for achatfinalthat carries non-empty assistant text, once perrunId(bounded FIFO of 200 run ids), fromhandleEventbeforehandleChatEvent— so a pre-ack buffered final replayed after the send acknowledgement still fires exactly once. Heartbeat finals (message: undefined), silent finals and user-role finals never fire.atis the gateway-stampedmessage.timestampof the final, ornullfrom a gateway that does not stamp timestamps; unread detection is then live-only. AppShell’suseReplyNotificationsowns unread state per browser or installed app: a chat is unread when an assistant reply arrived after this device last had the chat open. Detection is durable — it compares the session row’slastReplyAt(see Gateway protocol) against a per-device read cursor, and the state survives a reload. It publishes the unread keys to the store (setUnreadSessionsonunreadBySession, rendered as the staticsidebar-session-unreaddot,aria-label="New reply") and clears a chat when it is activated, when the tab becomes visible with it open, or through the top-bar bell’s Mark all as read. The document title gets a(N)prefix counting unread chats against the title captured at mount. Badges and the count run whether or not the notification opt-in is on.providers/unread-state.tsis the pure state machine behind that durability (UNREAD_STATE_STORAGE_KEYisopenclaw.web.unread.v1, scoped like every other preference): per chat a seen floor and an unread timestamp, a baseline seeded from the first attention window (replies older than it never light), an optional read-all timestamp, a monotonic cross-tab merge in which a read always beats an unread of the same or older reply, and pruning that sheds the oldest seen entries past 2000 keys by raising the baseline. Timestamps only — never message text or titles. A chat deleted elsewhere is dropped by tombstoning it seen and forgetting the key, both from the livesessionDeletedevent and from a completed read (completeAt) whose window no longer lists it.providers/attention-runtime.tsis the attention window the unread owner reconciles against. In the default Sessions view it adopts the sidebar’s own completed page (SESSIONS_LIST_LIMIT, 100 rows); anywhere elserefresh()reads the newestATTENTION_LIST_LIMIT(50) user-facing chats withincludeDerivedTitles, coalescing to one fetch in flight plus one trailing and never emitting on failure. The runtime publishesattentionRows { sessions, completeAt };completeAtis the responsetsonly when the read returned the whole window (hasMore: false), so only a complete window can clear chats that vanished from it. A hidden-to-visible resume refetches the sidebar and attention windows at most once perRESUME_REFRESH_MIN_MS(10 s).ui/AttentionMenu.tsxrenders the bell: an icon trigger whose badge caps at99+(aria-label="Notifications, N unread") opening a dismissible menu that lists unread chats (assistant colour marker, chat name, an assistant and recency line) and offers Mark all as read only when the list is non-empty; an empty menu reads “You’re all caught up”.- Operator decision: nothing from the reply itself may reach the OS.
AssistantReply.textexists only so empty finals can be skipped — never render it in a notification, a tooltip or a title.
Desktop notifications (opt-in)
providers/reply-notifications.tsowns the browser seam:notificationSupport()(returns"unsupported"when there is noNotificationconstructor, else the live permission), the scoped opt-in flag underopenclaw.web.notifications.v1keyed${gatewayUrl}|${clientId}(the same per-connection scoping rule as the theme and display preferences),formatUnreadTitle(base, total), andshowReplyNotification.- A notification is shown only while the tab is hidden, the opt-in is on and permission is
granted. Its title is<assistant name> replied(the chat’s agent name when it resolves, else the configured assistant name), its body is the chat’s title, andtagis the session key, so a newer reply replaces the older notification for that chat.showReplyNotificationreturnsfalsewhen the constructor throws (Android Chrome and iOS throw without a service worker); the caller then recordsunsupportedso the account menu stops offering the toggle. AccountMenurenders the Notifications radiogroup (account-menu-notifications-off|on).Notification.requestPermission()runs inside the click handler — it requires a user gesture. Adeniedresult keeps the toggle Off and showsBlocked in browser settings; anunsupportedbrowser renders aDisabledControlwithreason="Not supported in this browser".
Stay inside the ADR 0077 allowlist
The client connects with onlyoperator.sessions.read and operator.sessions.write, and the
gateway admits a fixed method and event set for them (listed under
narrow scope method allowlist and
event allowlist; ADR 0077 records the decision; the
sources of truth are NARROW_SESSION_READ_METHODS and NARROW_SESSION_WRITE_METHODS in
src/gateway/method-scopes.ts and NARROW_SESSION_EVENT_ALLOWLIST in
src/gateway/server-broadcast.ts).
- Methods: reads such as
agents.list,sessions.list,chat.history,sessions.runs.*,tasks.runs.*,talk.catalog, andtts.readAloud.status; writeschat.send,chat.abort,sessions.create,sessions.steer,sessions.abort,sessions.runs.cancel,sessions.runs.continue,sessions.patch,sessions.title.prepare(ADR 0083),talk.session.create,talk.session.appendAudio,talk.session.close,talk.session.cancelTurn,talk.session.mark,talk.session.bargeIn(ADR 0086), andtts.speak(ADR 0102).sessions.patchis field-guarded even for this narrow client: onlylabel,model,thinkingLevelandreasoningLevelare admitted (NARROW_SESSION_PATCH_FIELDS); every other field still needsoperator.adminand is refused withmissing scope: operator.admin, the same shape as a fully unlisted method. The patch response is redacted for this client (nopath, none of the admin-only entry fields), so never read those from a patch result. - Events:
chat,sessions.changed,runs.invalidated,talk.event(this connection’s own Talk sessions only),tick,heartbeat. - Not admitted:
talk.realtime.*,talk.config,talk.mode,talk.metrics,config.*,logs.*,cron.*,node.*,send, and anything not listed. A method outside the list is refused by the gateway, so a feature that needs one is an ADR amendment, not a client-only change.
Test at the real entry point
Unit tests underapps/web/src/** (pnpm test:web) prove components in jsdom. They cannot see
scrolling, fonts, color-scheme or gateway frames. For a behaviour change, also drive the built
bundle in real Chromium against a real gateway using the harness in
test/helpers/user-client-web-harness.ts; see
Testing for the suites and their caveats.
Related
- Brand tokens - the shared tokens and fonts under
client.css - Web client hosting - deployment, origins,
config.json - Using the web client - what users see
- Gateway protocol - client identity and scopes