feat(ai): live speech-to-text sessions with Deepgram, ElevenLabs, OpenAI, xAI, Gemini, Mistral, and Cloudflare adapters #1964
- docs/media/live-transcription-advanced.md line 28Outdated@@ -0,0 +8,21 @@8+live transcription9+mulaw10+alaw11+pcm12+audio conversion13+custom adapter14+---15+16+Set `audio` to the format of the chunks you send. Each adapter lists the formats it takes in `adapter.audioInputs`, and the session converts anything else to PCM.17+18+This page builds on [Live Transcription](./live-transcription).19+20+## Audio formats21+22+`audio.encoding` takes three values:23+24+`mulaw`: G.711 μ-law. Twilio, Telnyx, Plivo, and Vonage send this in North America.25+`alaw`: G.711 A-law. Most carriers outside North America send this.26+`linear16`: 16-bit signed little-endian PCM, at any sample rate.27+28+Set `sampleRate` (8000 for a phone call), and `channels` when the audio is not mono:docs/media/live-transcription-advanced.md line 77Outdated@@ -0,0 +57,21 @@57+ { encoding: 'mulaw', sampleRate: 8000 },58+ { encoding: 'linear16', sampleRate: 16000 },59+)60+61+const pcm = toPcm(new Uint8Array([0xff, 0x7f]))62+```63+64+## Provider options65+66+Pass provider settings in `modelOptions`. The type follows the adapter.67+68+Deepgram Nova-3, with a turn-end after one second of no new words:69+70+```typescript71+import { transcriptionSession } from '@tanstack/ai'72+import { deepgramLiveTranscription } from '@tanstack/ai-deepgram'73+74+const session = await transcriptionSession({75+ adapter: deepgramLiveTranscription('nova-3'),76+ audio: { encoding: 'mulaw', sampleRate: 8000 },77+ modelOptions: { utteranceEndMs: 1000, smartFormat: true, keyterms: ['TanStack'] },docs/media/live-transcription-advanced.md line 98Outdated@@ -0,0 +78,21 @@78+})79+```80+81+Deepgram Flux detects turns itself. Tune it with `eotThreshold` and `eotTimeoutMs`:82+83+```typescript84+import { transcriptionSession } from '@tanstack/ai'85+import { deepgramLiveTranscription } from '@tanstack/ai-deepgram'86+87+const session = await transcriptionSession({88+ adapter: deepgramLiveTranscription('flux-general-en'),89+ audio: { encoding: 'linear16', sampleRate: 16000 },90+ modelOptions: { eotThreshold: 0.8 },91+})92+```93+94+`turn-end` support differs by provider:95+96+Deepgram, ElevenLabs, xAI, and Gemini send `turn-end` when the speaker pauses.97+OpenAI sends it with `gpt-transcribe` and the `gpt-4o-*` models, which detect voice activity.98+OpenAI `gpt-live-transcribe` streams text with no `turn-end`. It sends one `final` when the session closes.docs/media/live-transcription.md line 77Outdated@@ -0,0 +57,21 @@57+ 'media' in message &&58+ typeof message.media === 'object' &&59+ message.media !== null &&60+ 'payload' in message.media &&61+ typeof message.media.payload === 'string'62+ ) {63+ return message.media.payload64+ }65+ return undefined66+}67+```68+69+Open a session when the call connects. Send each payload to it, and close it when the call ends:70+71+```typescript72+import { transcriptionSession } from '@tanstack/ai'73+import { deepgramLiveTranscription } from '@tanstack/ai-deepgram'74+import { mediaPayload } from './twilio'75+76+export async function handleCall(twilio: WebSocket) {77+ const session = await transcriptionSession({docs/media/live-transcription.md lines 114 to 117Outdated@@ -0,0 +97,21 @@97+Call the number and talk. Each finished phrase prints as you say it.98+99+## Reply to the caller100+101+Collect the `final` text of a turn. When the provider sends `turn-end`, give the turn to `chat()`:102+103+```typescript104+import { chat, transcriptionSession } from '@tanstack/ai'105+import type { ModelMessage } from '@tanstack/ai'106+import { deepgramLiveTranscription } from '@tanstack/ai-deepgram'107+import { openaiText } from '@tanstack/ai-openai'108+109+export async function runAgent(callId: string, audio: AsyncIterable<string>) {110+ const session = await transcriptionSession({111+ adapter: deepgramLiveTranscription('nova-3'),112+ audio: { encoding: 'mulaw', sampleRate: 8000 },113+ })114+ void (async () => {115+ for await (const chunk of audio) session.send(chunk)116+ await session.close()117+ })()docs/media/live-transcription.md line 147@@ -0,0 +113,21 @@113+ })114+ void (async () => {115+ for await (const chunk of audio) session.send(chunk)116+ await session.close()117+ })()118+119+ const messages: Array<ModelMessage> = []120+ let turn = ''121+ for await (const event of session) {122+ if (event.type === 'final') turn = `${turn} ${event.text}`.trim()123+ if (event.type === 'turn-end' && turn) {124+ messages.push({ role: 'user', content: turn })125+ turn = ''126+ const { text } = await chat({127+ adapter: openaiText('gpt-6.1-sol'),128+ messages,129+ threadId: callId,130+ stream: false,131+ })132+ messages.push({ role: 'assistant', content: text })133+ console.log('Agent:', text)packages/ai-gemini/src/adapters/live-transcription.ts lines 81 to 82Outdated@@ -0,0 +62,21 @@62+ * })63+ * ```64+ */65+export function geminiLiveTranscription<66+ TModel extends GeminiLiveTranscriptionModel,67+>(68+ model: TModel,69+ config: GeminiLiveTranscriptionConfig = {},70+): LiveTranscriptionAdapter<TModel, GeminiLiveTranscriptionProviderOptions> {71+ return {72+ kind: 'liveTranscription',73+ name: 'gemini',74+ model,75+ audioInputs: [{ encoding: 'linear16', sampleRates: [16000] }],76+ '~types': {} as never,77+ connect: async (options) => {78+ const apiKey = config.apiKey ?? getGeminiApiKeyFromEnv()79+ const opts = options.modelOptions ?? {}80+ const mimeType = `audio/pcm;rate=${options.audio.sampleRate}`81+ // Interim text with no final yet. When there is none, close does not wait.82+ let pending = falsepackages/ai-openai/src/adapters/live-transcription.ts lines 108 to 109Outdated@@ -0,0 +89,21 @@89+ model,90+ audioInputs: [91+ { encoding: 'linear16', sampleRates: [24000] },92+ { encoding: 'mulaw', sampleRates: [8000] },93+ { encoding: 'alaw', sampleRates: [8000] },94+ ],95+ '~types': {} as never,96+ connect: async (options) => {97+ const apiKey = config.apiKey ?? getOpenAIApiKeyFromEnv()98+ const opts = options.modelOptions ?? {}99+ const noVad = NO_VAD_MODELS.includes(options.model)100+ const format =101+ options.audio.encoding === 'mulaw'102+ ? { type: 'audio/pcmu' }103+ : options.audio.encoding === 'alaw'104+ ? { type: 'audio/pcma' }105+ : { type: 'audio/pcm', rate: 24000 }106+ // Deltas per item, joined into one interim text.107+ const interim = new Map<string, string>()108+ // Speech or text with no final yet. When there is none, close does not wait.109+ let pending = falsepackages/ai/src/activities/transcriptionSession/socket.ts lines 131 to 132@@ -0,0 +83,21 @@83+ if (ws.readyState === 1) ws.send(bytes as Uint8Array<ArrayBuffer>)84+ },85+ get closing() {86+ return closing87+ },88+ finish: async (lastMessage, timeoutMs = 5000) => {89+ if (closing) return closed90+ closing = true91+ if (lastMessage !== undefined) live.sendJson(lastMessage)92+ const timer = setTimeout(() => ws.close(1000), timeoutMs)93+ await closed94+ clearTimeout(timer)95+ },96+ close: () => {97+ closing = true98+ if (ws.readyState < 2) ws.close(1000)99+ },100+ }101+102+ const onAbort = () => live.close()103+ abortSignal?.addEventListener('abort', onAbort, { once: true })packages/ai/src/activities/transcriptionSession/socket.ts lines 134 to 173@@ -0,0 +111,21 @@111+ options.onOpen?.(live)112+ if (!options.readyWhen) markReady()113+ })114+ ws.addEventListener('message', (event: MessageEvent) => {115+ // Some providers (Gemini Live) send JSON in binary frames.116+ const text =117+ typeof event.data === 'string'118+ ? event.data119+ : event.data instanceof ArrayBuffer120+ ? decoder.decode(event.data)121+ : undefined122+ if (text === undefined) return123+ let message: unknown124+ try {125+ message = JSON.parse(text)126+ } catch {127+ return128+ }129+ if (!ready && options.readyWhen?.(message)) markReady()130+ options.onMessage(message, live)131+ })
github-actions[bot](GitHub profile, opens in a new tab) assignedAlemTuzlak
github-actions[bot](GitHub profile, opens in a new tab) added labelswaiting-on: maintainer
tombeckenham changed the title
- docs/adapters/cloudflare.md lines 555 to 558Outdated@@ -533,0 +538,21 @@538+```typescript539+import { transcriptionSession } from "@tanstack/ai";540+import { cloudflareLiveTranscription } from "@tanstack/ai-cloudflare";541+import type { Ai } from "@cloudflare/workers-types";542+543+interface Env {544+ AI: Ai;545+}546+547+export async function transcribeCall(env: Env, audio: AsyncIterable<Uint8Array>) {548+ const session = await transcriptionSession({549+ adapter: cloudflareLiveTranscription("@cf/deepgram/nova-3", {550+ binding: env.AI,551+ }),552+ audio: { encoding: "mulaw", sampleRate: 8000 },553+ });554+555+ void (async () => {556+ for await (const chunk of audio) session.send(chunk);557+ await session.close();558+ })();docs/adapters/mistral.md lines 245 to 247Outdated@@ -229,3 +229,19 @@229229230230 See the [Embeddings guide](../embeddings.md).231231232+## Live transcription233+234+`mistralLiveTranscription` streams speech-to-text with Voxtral for [`transcriptionSession()`](../media/live-transcription). Send audio as it arrives, for example from a phone call:235+236+```typescript237+import { transcriptionSession } from "@tanstack/ai";238+import { mistralLiveTranscription } from "@tanstack/ai-mistral";239+240+const session = await transcriptionSession({241+ adapter: mistralLiveTranscription("voxtral-mini-transcribe-realtime-2602"),242+ audio: { encoding: "mulaw", sampleRate: 8000 },243+});244+245+for await (const event of session) {246+ if (event.type === "final") console.log(event.text);247+}docs/media/live-transcription.md line 177@@ -0,0 +140,21 @@140+141+| Event | Meaning |142+| ---------- | ----------------------------------------------------------------------------- |143+| `interim` | Text that can still change. Use it for live captions, and to stop playback when the caller talks over the agent. |144+| `final` | Stable text for one segment. It does not change again. |145+| `turn-end` | The provider thinks the speaker finished their turn. |146+| `error` | A provider or connection error. The session can end after it. |147+148+## Pick a provider149+150+Swap the adapter to change provider. The rest of the code stays the same.151+152+| Provider | Adapter | Phone audio (8 kHz) |153+| ---------- | ------------------------------------------------------- | -------------------- |154+| Deepgram | `deepgramLiveTranscription('nova-3')` from `@tanstack/ai-deepgram` | μ-law, A-law |155+| ElevenLabs | `elevenlabsLiveTranscription('scribe_v2_realtime')` from `@tanstack/ai-elevenlabs` | μ-law |156+| OpenAI | `openaiLiveTranscription('gpt-transcribe')` from `@tanstack/ai-openai` | μ-law, A-law |157+| xAI | `grokLiveTranscription('grok-voice-transcribe-2.0')` from `@tanstack/ai-grok` | μ-law, A-law |158+| Gemini | `geminiLiveTranscription('gemini-3.5-transcribe-live')` from `@tanstack/ai-gemini` | Converted to PCM |159+| Mistral | `mistralLiveTranscription('voxtral-mini-transcribe-realtime-2602')` from `@tanstack/ai-mistral` | μ-law, A-law |160+| Cloudflare | `cloudflareLiveTranscription('@cf/deepgram/nova-3', { binding: env.AI })` from `@tanstack/ai-cloudflare` | μ-law |
github-actions[bot](GitHub profile, opens in a new tab) added labelsmerge-conflicts
github-actions[bot](GitHub profile, opens in a new tab) added labelswaiting-on: author
github-actions[bot](GitHub profile, opens in a new tab) removed labelswaiting-on: maintainer
- docs/adapters/cloudflare.md line 579@@ -548,0 +559,21 @@559+ AI: Ai;560+}561+562+export async function transcribeCall(env: Env, audio: AsyncIterable<Uint8Array>) {563+ const session = await transcriptionSession({564+ adapter: cloudflareLiveTranscription("@cf/deepgram/nova-3", {565+ binding: env.AI,566+ }),567+ audio: { encoding: "mulaw", sampleRate: 8000 },568+ });569+570+ const feeding = (async () => {571+ try {572+ for await (const chunk of audio) session.send(chunk);573+ } finally {574+ await session.close();575+ }576+ })();577+578+ for await (const event of session) {579+ if (event.type === "final") console.log(event.text);docs/adapters/cloudflare.md line 587@@ -548,0 +567,21 @@567+ audio: { encoding: "mulaw", sampleRate: 8000 },568+ });569+570+ const feeding = (async () => {571+ try {572+ for await (const chunk of audio) session.send(chunk);573+ } finally {574+ await session.close();575+ }576+ })();577+578+ for await (const event of session) {579+ if (event.type === "final") console.log(event.text);580+ }581+ await feeding;582+}583+```584+585+`@cf/deepgram/nova-3` takes PCM and μ-law. Other formats are converted to PCM.586+`@cf/deepgram/flux` takes PCM and detects turns itself.587+Both take the Deepgram `modelOptions`, such as `keyterms` and `endpointing`.docs/adapters/openai.md line 195@@ -178,3 +178,18 @@178178179179 For speaker labels, use `gpt-4o-transcribe-diarize`. See [Speaker diarization](./openai-advanced#speaker-diarization).180180181+182+## Live transcription183+184+`openaiLiveTranscription` streams speech-to-text for [`transcriptionSession()`](../media/live-transcription). Send audio as it arrives, for example from a phone call:185+186+```typescript187+import { transcriptionSession } from '@tanstack/ai'188+import { openaiLiveTranscription } from '@tanstack/ai-openai'189+190+const session = await transcriptionSession({191+ adapter: openaiLiveTranscription('gpt-transcribe'),192+ audio: { encoding: 'mulaw', sampleRate: 8000 },193+})194+195+for await (const event of session) {packages/ai/skills/ai-core/media-generation/SKILL.md line 701@@ -673,0 +681,21 @@681+682+Use `transcriptionSession()` when audio arrives live (Twilio/Telnyx/Vonage683+Media Streams, an SFU track, a browser WebSocket). It runs on the server684+(Node, Bun, Workers, Durable Objects). `generateTranscription()` only takes a685+finished file; `useRealtimeChat` reads the browser microphone itself.686+687+```typescript688+import { transcriptionSession } from '@tanstack/ai'689+import type { TranscriptionSession } from '@tanstack/ai'690+import { deepgramLiveTranscription } from '@tanstack/ai-deepgram'691+import { mediaPayload } from './twilio' // parses Twilio `media` frames692+693+export async function handleCall(twilio: WebSocket) {694+ // Listen before the session opens: Twilio sends audio right away.695+ const early: Array<string> = []696+ let session: TranscriptionSession | undefined697+ twilio.addEventListener('message', (event) => {698+ const payload = mediaPayload(event.data) // base64 string or bytes699+ if (payload) session ? session.send(payload) : early.push(payload)700+ })701+ twilio.addEventListener('close', () => void session?.close()) // flushespackages/ai/src/activities/transcriptionSession/socket.ts lines 83 to 89@@ -0,0 +69,21 @@69+ * Open a provider WebSocket for a live transcription adapter. It reports70+ * unexpected closes as `error` events, calls `end()` once on close, and closes71+ * on abort. Resolves when the socket is open.72+ */73+export function openLiveSocket(74+ options: LiveSocketOptions,75+): Promise<LiveSocket> {76+ const { emit, end, abortSignal } = options.connect77+ const Ctor = options.WebSocket ?? globalThis.WebSocket78+ if (!options.socket && (typeof Ctor !== 'function' || !options.url)) {79+ return Promise.reject(80+ new Error('No WebSocket available. Pass a WebSocket constructor.'),81+ )82+ }83+ if (abortSignal?.aborted) {84+ return Promise.reject(85+ abortSignal.reason instanceof Error86+ ? abortSignal.reason87+ : new Error('The session was aborted before it connected.'),88+ )89+ }
github-actions[bot](GitHub profile, opens in a new tab) added labelswaiting-on: maintainer
github-actions[bot](GitHub profile, opens in a new tab) removed labelswaiting-on: author
github-actions[bot](GitHub profile, opens in a new tab) removed labelsmerge-conflicts
github-actions[bot](GitHub profile, opens in a new tab) added labelsmerge-conflicts
github-actions[bot](GitHub profile, opens in a new tab) added labelswaiting-on: author
github-actions[bot](GitHub profile, opens in a new tab) removed labelswaiting-on: maintainer
github-actions[bot](GitHub profile, opens in a new tab) added labelswaiting-on: maintainer
github-actions[bot](GitHub profile, opens in a new tab) removed labelswaiting-on: author
github-actions[bot](GitHub profile, opens in a new tab) removed labelsmerge-conflicts
github-actions[bot](GitHub profile, opens in a new tab) added labelsmerge-conflicts
github-actions[bot](GitHub profile, opens in a new tab) added labelswaiting-on: author
github-actions[bot](GitHub profile, opens in a new tab) removed labelswaiting-on: maintainer
github-actions[bot](GitHub profile, opens in a new tab) added labelswaiting-on: maintainer
github-actions[bot](GitHub profile, opens in a new tab) removed labelswaiting-on: author
github-actions[bot](GitHub profile, opens in a new tab) removed labelsmerge-conflicts
github-actions[bot](GitHub profile, opens in a new tab) added labelsmerge-conflicts
github-actions[bot](GitHub profile, opens in a new tab) added labelswaiting-on: author
github-actions[bot](GitHub profile, opens in a new tab) removed labelswaiting-on: maintainer
tombeckenham linked this to fix #1944
github-actions[bot](GitHub profile, opens in a new tab) added labelswaiting-on: maintainer
github-actions[bot](GitHub profile, opens in a new tab) removed labelswaiting-on: author
github-actions[bot](GitHub profile, opens in a new tab) removed labelsmerge-conflicts
github-actions[bot](GitHub profile, opens in a new tab) added labelsmerge-conflicts
github-actions[bot](GitHub profile, opens in a new tab) added labelswaiting-on: author
github-actions[bot](GitHub profile, opens in a new tab) removed labelswaiting-on: maintainer
github-actions[bot](GitHub profile, opens in a new tab) added labelswaiting-on: maintainer
github-actions[bot](GitHub profile, opens in a new tab) removed labelswaiting-on: author
github-actions[bot](GitHub profile, opens in a new tab) removed labelsmerge-conflicts
github-actions[bot](GitHub profile, opens in a new tab) added labelsmerge-conflicts
github-actions[bot](GitHub profile, opens in a new tab) added labelswaiting-on: author
github-actions[bot](GitHub profile, opens in a new tab) removed labelswaiting-on: maintainer
Sign in to join this conversation.
🦋 Changeset detected#
Latest commit: 4725e44c97d5b562bf69019411245ff1ebf47f1a
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 8 packages
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR