Open History to see the last commit
1# Unified Chat API
3## Overview
5The `chat()` method has been unified to support three different return types through the `as` configuration option:
7- **`"promise"`** (default): Returns `Promise<ChatCompletionResult>` - standard non-streaming chat
8- **`"stream"`**: Returns `AsyncIterable<StreamChunk>` - streaming chunks for manual handling
9- **`"response"`**: Returns `Response` - HTTP response with proper streaming headers (SSE format)
11This eliminates the need for separate `chat()` and `streamChat()` methods.
13## Migration Guide
15### Before (Separate Methods)
17```typescript
18// For non-streaming
19const result = await ai.chat({
20 adapter: "openai",
21 model: "gpt-4",
22 messages: [{ role: "user", content: "Hello" }]
23});
25// For streaming
26const stream = ai.streamChat({
27 adapter: "openai",
28 model: "gpt-4",
29 messages: [{ role: "user", content: "Hello" }]
30});
31for await (const chunk of stream) {
32 console.log(chunk);
33}
35// For HTTP response
36import { toStreamResponse } from "@ts-poc/ai";
37const stream = ai.streamChat({
38 adapter: "openai",
39 model: "gpt-4",
40 messages: [{ role: "user", content: "Hello" }]
41});
42return toStreamResponse(stream);
43```
45### After (Unified Method)
47```typescript
48// For non-streaming (default, "as" is optional)
49const result = await ai.chat({
50 adapter: "openai",
51 model: "gpt-4",
52 messages: [{ role: "user", content: "Hello" }],
53 as: "promise" // or omit this line - "promise" is the default
54});
56// For streaming
57const stream = ai.chat({
58 adapter: "openai",
59 model: "gpt-4",
60 messages: [{ role: "user", content: "Hello" }],
61 as: "stream"
62});
63for await (const chunk of stream) {
64 console.log(chunk);
65}
67// For HTTP response (no need to import toStreamResponse!)
68return ai.chat({
69 adapter: "openai",
70 model: "gpt-4",
71 messages: [{ role: "user", content: "Hello" }],
72 as: "response"
73});
74```
76## Usage Examples
78### 1. Promise Mode (Default)
80Standard non-streaming chat completion:
82```typescript
83const result = await ai.chat({
84 adapter: "openai",
85 model: "gpt-4",
86 messages: [
87 { role: "system", content: "You are a helpful assistant." },
88 { role: "user", content: "What is TypeScript?" }
89 ],
90 temperature: 0.7,
91 // as: "promise" is implicit
92});
94console.log(result.content);
95console.log(`Tokens used: ${result.usage.totalTokens}`);
96```
98### 2. Stream Mode
100Manual streaming for custom handling:
102```typescript
103const stream = ai.chat({
104 adapter: "openai",
105 model: "gpt-4",
106 messages: [{ role: "user", content: "Write a story" }],
107 as: "stream"
108});
110for await (const chunk of stream) {
111 if (chunk.type === "content") {
112 process.stdout.write(chunk.delta);
113 } else if (chunk.type === "tool_call") {
114 console.log(`Tool: ${chunk.toolCall.function.name}`);
115 } else if (chunk.type === "done") {
116 console.log(`\nFinished: ${chunk.finishReason}`);
117 console.log(`Tokens: ${chunk.usage?.totalTokens}`);
118 }
119}
120```
122### 3. Response Mode (HTTP Streaming)
124Perfect for API endpoints:
126```typescript
127// TanStack Start API Route
128export const POST = async ({ request }: { request: Request }) => {
129 const { messages } = await request.json();
131 // Returns a Response object with proper SSE headers
132 return ai.chat({
133 adapter: "openai",
134 model: "gpt-4o",
135 messages,
136 temperature: 0.7,
137 as: "response", // ← Returns Response directly!
138 fallbacks: [
139 { adapter: "ollama", model: "llama2" }
140 ]
141 });
142};
143```
145## With Fallbacks
147All three modes support fallbacks:
149```typescript
150// Promise mode with fallbacks
151const result = await ai.chat({
152 adapter: "openai",
153 model: "gpt-4",
154 messages: [{ role: "user", content: "Hello" }],
155 as: "promise",
156 fallbacks: [
157 { adapter: "anthropic", model: "claude-3-sonnet-20240229" },
158 { adapter: "ollama", model: "llama2" }
159 ]
160});
162// Stream mode with fallbacks
163const stream = ai.chat({
164 adapter: "openai",
165 model: "gpt-4",
166 messages: [{ role: "user", content: "Hello" }],
167 as: "stream",
168 fallbacks: [
169 { adapter: "anthropic", model: "claude-3-sonnet-20240229" }
170 ]
171});
173// Response mode with fallbacks (seamless failover in HTTP streaming!)
174return ai.chat({
175 adapter: "openai",
176 model: "gpt-4",
177 messages: [{ role: "user", content: "Hello" }],
178 as: "response",
179 fallbacks: [
180 { adapter: "ollama", model: "llama2" }
181 ]
182});
183```
185## Tool Execution
187Tool execution works in all three modes:
189```typescript
190const tools = [
191 {
192 type: "function" as const,
193 function: {
194 name: "get_weather",
195 description: "Get weather for a location",
196 parameters: {
197 type: "object",
198 properties: {
199 location: { type: "string" }
200 }
201 }
202 },
203 execute: async (args: { location: string }) => {
204 return JSON.stringify({ temp: 72, condition: "sunny" });
205 }
206 }
207];
209// Promise mode - waits for all tool executions to complete
210const result = await ai.chat({
211 adapter: "openai",
212 model: "gpt-4",
213 messages: [{ role: "user", content: "What's the weather in SF?" }],
214 tools,
215 toolChoice: "auto",
216 maxIterations: 5,
217 as: "promise"
218});
220// Stream mode - see tool execution happen in real-time
221const stream = ai.chat({
222 adapter: "openai",
223 model: "gpt-4",
224 messages: [{ role: "user", content: "What's the weather in SF?" }],
225 tools,
226 toolChoice: "auto",
227 maxIterations: 5,
228 as: "stream"
229});
231for await (const chunk of stream) {
232 if (chunk.type === "tool_call") {
233 console.log(`Calling tool: ${chunk.toolCall.function.name}`);
234 }
235}
237// Response mode - stream tool execution to client
238return ai.chat({
239 adapter: "openai",
240 model: "gpt-4",
241 messages: [{ role: "user", content: "What's the weather in SF?" }],
242 tools,
243 toolChoice: "auto",
244 maxIterations: 5,
245 as: "response"
246});
247```
249## Type Safety
251TypeScript automatically infers the correct return type based on the `as` option:
253```typescript
254// Type: Promise<ChatCompletionResult>
255const promise = ai.chat({ adapter: "openai", model: "gpt-4", messages: [] });
257// Type: AsyncIterable<StreamChunk>
258const stream = ai.chat({ adapter: "openai", model: "gpt-4", messages: [], as: "stream" });
260// Type: Response
261const response = ai.chat({ adapter: "openai", model: "gpt-4", messages: [], as: "response" });
262```
264## Benefits
2661. **Simpler API**: One method instead of two
2672. **Consistent Interface**: Same options across all modes
2683. **HTTP Streaming Made Easy**: No need to manually call `toStreamResponse()`
2694. **Fallbacks Everywhere**: All modes support the same fallback mechanism
2705. **Type Safety**: TypeScript infers the correct return type
2716. **Backward Compatible**: Default behavior (promise) matches old `chat()` method
273## Real-World Example: TanStack Start API
275```typescript
276import { createAPIFileRoute } from "@tanstack/start/api";
277import { ai } from "~/lib/ai-client";
279export const Route = createAPIFileRoute("/api/chat")({
280 POST: async ({ request }) => {
281 const { messages, tools } = await request.json();
283 // That's it! Just return the result - it's already a Response
284 return ai.chat({
285 adapter: "openAi",
286 model: "gpt-4o",
287 messages,
288 tools,
289 toolChoice: "auto",
290 maxIterations: 5,
291 temperature: 0.7,
292 as: "response", // ← Returns Response with SSE headers
293 fallbacks: [
294 { adapter: "ollama", model: "llama2" }
295 ]
296 });
297 }
298});
299```
301Client-side consumption:
303```typescript
304const response = await fetch("/api/chat", {
305 method: "POST",
306 body: JSON.stringify({ messages, tools })
307});
309const reader = response.body!.getReader();
310const decoder = new TextDecoder();
312while (true) {
313 const { done, value } = await reader.read();
314 if (done) break;
316 const text = decoder.decode(value);
317 const lines = text.split("\n\n");
319 for (const line of lines) {
320 if (line.startsWith("data: ")) {
321 const data = line.slice(6);
322 if (data === "[DONE]") continue;
324 const chunk = JSON.parse(data);
325 if (chunk.type === "content") {
326 console.log(chunk.delta); // Stream content to UI
327 }
328 }
329 }
330}
331```
333## Summary
335The unified `chat()` API provides:
336- **One method** for all chat use cases
337- **Three modes**: promise, stream, response
338- **Same options** across all modes
339- **Built-in HTTP streaming** (no manual conversion needed)
340- **Full fallback support** in all modes
341- **Type-safe** return types
342- **Simpler code** for common patterns