1# Voice input
2
3Transcription edits a composer draft. It does not submit an agent turn. Audio is
4temporary client input, and only normal message submission sends the resulting
5text to the agent. Mobile defaults to on-device transcription. Sending audio off
6the device requires an explicit grant naming one environment and, for cloud
7transcription, one provider.
8
9The [shared controller](../../packages/client-runtime/src/voice-input/controller.ts)
10owns the operation while the client supplies capture and transcription. Preparation
11binds the transcriber and resolved locale for the whole recording. Draft ownership,
12text, and revision are captured before recording and checked before insertion, so
13a late transcript cannot overwrite a draft that was edited or replaced. A
14transcriber may expose a streaming session, in which case audio leaves during
15capture and `finish` replaces the file-based `transcribe`.
16
17## Where audio may go
18
19On mobile, on-device transcription needs no audio grant. Environment and cloud
20transcription share a recorder and upload only finished recordings. The client
21keeps the grant in device-bound secure storage, outside preference backups, and
22rechecks it after native decoding and before upload.
23
24The [web grant](../../apps/web/src/speech/voiceConsent.ts) lives outside the preferences snapshot so another
25window saving unrelated settings cannot resurrect a revoked grant. Each new choice
26gets a distinct grant, even if the destination was previously selected. The
27[web hook](../../apps/web/src/speech/useEnvironmentSpeechInput.ts) binds the
28controller to that grant and the selected environment's connection. It requires
29`voiceTranscription` to use the environment's model or `cloudVoiceTranscription`
30to forward audio to the selected provider. Changing the grant or losing the
31connection, permission, or required capability discards both the active recording
32and any recording retained for retry. It never chooses a fallback destination.
33The settings page's transcription test uses the same hook. The microphone loopback
34test captures separately for local playback and sends nothing. The transport
35refuses redirects so credentials and audio cannot follow one to another host.
36
37Transcript cleanup requires its own grant, off by default.
38It sends text, never audio, to the text generation model of the draft's
39own environment, because that is where its credentials live.
40
41A cloud destination binds both the forwarding environment and the named provider.
42It uses a separate, batch-only endpoint: changing a server key cannot upgrade
43an environment-local grant, and a cloud recording never uses the streaming
44socket or falls back to another provider or the environment's native model.
45
46OpenRouter routes to many upstreams, so its grant also names the exact model and
47the model travels with every upload as a query parameter. The server accepts only
48ids present in OpenRouter's public ZDR-eligible transcription catalog,
49which it caches for ten minutes after a successful read;
50router aliases such as `openrouter/auto` or `:nitro` fail schema validation before
51that. Requests set `provider.zdr` and carry no `models` fallback list. OpenRouter
52applies the account's safety-retention policy when choosing an endpoint. The catalog
53holds no credential or consent state, which is why it may be cached while key
54presence is read fresh on every request.
55
56## Server side
57
58The [speech service](../../apps/server/src/speech/SpeechService.ts) runs
59transcribe-cpp in a child process so a hung native call can be killed without
60taking the server down; see [native.ts](../../apps/server/src/speech/native.ts)
61for the packaging constraints (Node SEA, Electron ASAR). One native request runs
62at a time. Batch uploads carry their options in the body framing defined in
63[shared/speech.ts](../../packages/shared/src/speech.ts); streaming sends them in
64the socket's first command. Model files and the selected model, hardware
65placement, and unload timeout are environment state in server settings.
66A model file from another app is installed as a symlink only after its SHA-256
67matches a catalog entry, so the store never loads a file the catalog does not pin
68and removing a model never deletes a file T3 Code did not download.
69
70The Silero silence detector is optional. If its download fails, recordings use
71raw audio during the retry cooldown so a detector outage cannot block dictation.
72Explicitly installing a speech model also retries the detector without waiting.
73
74## Cancellation and resource ownership
75
76Cancellation invalidates a result immediately, but resources stay owned until the
77underlying work settles. Apple's native transcription call cannot be interrupted
78once started. Releasing the session or deleting its recording when the abort signal
79fires would race that work. The [transcription contract](../../packages/client-runtime/src/voice-input/transcription.ts)
80therefore requires implementations to settle only after their work has stopped;
81the [Apple binding](../../apps/mobile/src/native/voiceTranscription.ios.ts) checks
82cancellation between native calls and discards late results. On the server,
83interrupting a batch request kills the worker process instead; the next request
84reloads the model.