Skip to content
Source
Sign inSign up
TanStack/ai
Open History to see the last commit

README.md

839 lines19.2 KB

1# @tanstack/ai
2​
3A powerful, open-source AI SDK with a unified interface across multiple providers. No vendor lock-in, no proprietary formats, just clean TypeScript and honest open source.
4​
5## Features
6​
7- **Multi-Provider Support** - OpenAI, Anthropic, Ollama, Google Gemini
8- **Unified API** - Same interface across all providers
9- **Structured Streaming** - JSON chunks with token deltas, tool calls, and usage stats
10- **Tool/Function Calling** - First-class support for AI function calling (OpenAI & Anthropic)
11- **React Hooks** - Simple `useChat` hook with v5 API (you control input state)
12- **TypeScript First** - Full type safety throughout
13- **Zero Lock-in** - Switch providers at runtime without code changes
14​
15## Installation
16​
17```bash
18# Core library
19npm install @tanstack/ai
20​
21# Provider adapters (install what you need)
22npm install @tanstack/ai-openai
23npm install @tanstack/ai-anthropic
24npm install @tanstack/ai-ollama
25npm install @tanstack/ai-gemini
26​
27# React hooks (for frontend chat UIs)
28npm install @tanstack/ai-react
29```
30​
31## Architecture
32​
33### Core Concepts
34​
35**1. Provider-Agnostic Design**
36​
37All providers implement the same `AIAdapter` interface:
38​
39```typescript
40interface AIAdapter {
41 // Chat
42 chatCompletion(options: ChatCompletionOptions): Promise<ChatCompletionResult>;
43 chatStream(options: ChatCompletionOptions): AsyncIterable<StreamChunk>;
44​
45 // Text generation
46 generateText(options: TextGenerationOptions): Promise<TextGenerationResult>;
47​
48 // Summarization
49 summarize(options: SummarizationOptions): Promise<SummarizationResult>;
50​
51 // Embeddings
52 createEmbeddings(options: EmbeddingOptions): Promise<EmbeddingResult>;
54```
55​
56**2. Structured Streaming**
57​
58Streams return typed JSON chunks instead of raw strings:
59​
60```typescript
61type StreamChunk =
62 | ContentStreamChunk // Text tokens with delta + accumulated content
63 | ToolCallStreamChunk // Function call information
64 | DoneStreamChunk // Completion signal with token usage
65 | ErrorStreamChunk; // Error information
66```
67​
68**3. Adapter Pattern**
69​
70The `AI` class wraps any adapter and provides a consistent API:
71​
72```typescript
73const ai = new AI(adapter); // Initialize with adapter
74await ai.chat(options); // Standard chat
75ai.streamChat(options); // Structured streaming
76ai.setAdapter(newAdapter); // Switch providers
77```
78​
79## API Reference
80​
81### Core Library
82​
83#### AI Class
84​
85**Constructor**
86​
87```typescript
88constructor(adapter: AIAdapter)
89```
90​
91Create an AI instance with a specific provider adapter.
92​
93#### Methods
94​
95##### `chat(options): Promise<ChatCompletionResult>`
96​
97Non-streaming chat completion.
98​
99```typescript
100const result = await ai.chat({
101 model: "gpt-3.5-turbo",
102 messages: [
103 { role: "system", content: "You are helpful" },
104 { role: "user", content: "Hello!" },
105 ],
106 temperature: 0.7,
107 maxTokens: 1000,
108});
109​
110console.log(result.content);
111console.log(result.usage.totalTokens);
112```
113​
114##### `streamChat(options): AsyncIterable<StreamChunk>`
115​
116Structured streaming with JSON chunks. **Automatically executes tools** if they have `execute` functions.
117​
118```typescript
119for await (const chunk of ai.streamChat({
120 model: "gpt-3.5-turbo",
121 messages: [{ role: "user", content: "Hello" }],
122 tools: [...], // Optional: Tools with execute functions
123 toolChoice: "auto", // Optional: "auto" | "none" | { type, function }
124 maxIterations: 5, // Optional: Max tool calling loops (default: 5)
125})) {
126 switch (chunk.type) {
127 case "content":
128 console.log("Delta:", chunk.delta);
129 console.log("Full:", chunk.content);
130 break;
131 case "tool_call":
132 console.log("Tool:", chunk.toolCall.function.name);
133 console.log("Args:", chunk.toolCall.function.arguments);
134 // Tool is executed automatically if execute function is defined!
135 break;
136 case "done":
137 console.log("Finish:", chunk.finishReason);
138 console.log("Usage:", chunk.usage);
139 break;
140 case "error":
141 console.error("Error:", chunk.error.message);
142 break;
143 }
145```
146​
147**Automatic Tool Execution:** If tools have `execute` functions, `streamChat` will:
148​
1491. Detect tool calls from the AI
1502. Execute the tools automatically
1513. Add results to the conversation
1524. Continue streaming the final response
1535. Repeat up to `maxIterations` times
154​
155##### `generateText(options): Promise<TextGenerationResult>`
156​
157Generate text from a prompt.
158​
159```typescript
160const result = await ai.generateText({
161 model: "gpt-3.5-turbo-instruct",
162 prompt: "Once upon a time",
163 maxTokens: 100,
164});
165​
166console.log(result.text);
167```
168​
169##### `summarize(options): Promise<SummarizationResult>`
170​
171Summarize text.
172​
173```typescript
174const result = await ai.summarize({
175 model: "gpt-3.5-turbo",
176 text: "Long text to summarize...",
177 style: "bullet-points", // "bullet-points" | "paragraph" | "concise"
178 maxLength: 200,
179});
180​
181console.log(result.summary);
182```
183​
184##### `embed(options): Promise<EmbeddingResult>`
185​
186Generate embeddings.
187​
188```typescript
189const result = await ai.embed({
190 model: "text-embedding-ada-002",
191 input: ["Text 1", "Text 2"],
192});
193​
194console.log(result.embeddings); // number[][]
195```
196​
197##### `setAdapter(adapter): void`
198​
199Switch providers at runtime.
200​
201```typescript
202ai.setAdapter(new AnthropicAdapter({ apiKey: "..." }));
203```
204​
205#### Helper Functions
206​
207##### `toStreamResponse(stream): Response`
208​
209Convert a `StreamChunk` async iterable to an HTTP Response with proper SSE headers.
210​
211```typescript
212import { toStreamResponse } from "@tanstack/ai";
213​
214const stream = ai.streamChat({ model, messages });
215return toStreamResponse(stream);
216```
217​
218Returns a `Response` with:
219​
220- `Content-Type: text/event-stream`
221- `Cache-Control: no-cache`
222- `Connection: keep-alive`
223- Automatic JSON encoding of chunks
224- Error handling
225- `[DONE]` completion marker
226​
227##### `toReadableStream(stream): ReadableStream`
228​
229Convert a `StreamChunk` async iterable to a `ReadableStream` for more control.
230​
231```typescript
232import { toReadableStream } from "@tanstack/ai";
233​
234const stream = ai.streamChat({ model, messages });
235const readableStream = toReadableStream(stream);
236​
237// Use in custom Response
238return new Response(readableStream, {
239 headers: { "Custom-Header": "value" },
240});
241```
242​
243### Types
244​
245#### Message
246​
247```typescript
248interface Message {
249 role: "system" | "user" | "assistant" | "tool";
250 content: string | null;
251 name?: string;
252 toolCalls?: ToolCall[]; // For assistant messages with tool calls
253 toolCallId?: string; // For tool response messages
255```
256​
257#### Tool
258​
259```typescript
260interface Tool {
261 type: "function";
262 function: {
263 name: string;
264 description: string;
265 parameters: Record<string, any>; // JSON Schema
266 };
267 execute?: (args: any) => Promise<string> | string; // Optional: auto-execute
269```
270​
271**Note:** If `execute` is provided, `streamChat` will automatically:
272​
2731. Detect when the AI calls this tool
2742. Execute the function with the parsed arguments
2753. Add the result back to the conversation
2764. Continue streaming the final response
277​
278#### StreamChunk Variants
279​
280**ContentStreamChunk**
281​
282```typescript
284 type: "content"
285 id: string
286 model: string
287 timestamp: number
288 delta: string // New token(s)
289 content: string // Full accumulated content
290 role?: "assistant"
292```
293​
294**ToolCallStreamChunk**
295​
296```typescript
298 type: "tool_call"
299 id: string
300 model: string
301 timestamp: number
302 toolCall: {
303 id: string
304 type: "function"
305 function: {
306 name: string
307 arguments: string // Incremental JSON arguments
308 }
309 }
310 index: number
312```
313​
314**DoneStreamChunk**
315​
316```typescript
318 type: "done"
319 id: string
320 model: string
321 timestamp: number
322 finishReason: "stop" | "length" | "content_filter" | "tool_calls" | null
323 usage?: {
324 promptTokens: number
325 completionTokens: number
326 totalTokens: number
327 }
329```
330​
331**ErrorStreamChunk**
332​
333```typescript
335 type: "error"
336 id: string
337 model: string
338 timestamp: number
339 error: {
340 message: string
341 code?: string
342 }
344```
345​
346### React Hooks
347​
348#### useChat
349​
350The `useChat` hook provides complete chat functionality for React applications.
351​
352```typescript
353import { useChat } from "@tanstack/ai-react";
354​
355const {
356 messages, // Current message list
357 sendMessage, // Send a message (you manage input state)
358 isLoading, // Is generating response
359 error, // Current error
360 append, // Add message programmatically
361 reload, // Reload last response
362 stop, // Stop generation
363 clear, // Clear all messages
364} = useChat({
365 api: "/api/chat",
366 onChunk: (chunk) => console.log(chunk),
367});
368​
369// Simple usage
370await sendMessage("Hello!");
371```
372​
373**Key Features:**
374​
375- Maintains message state automatically
376- Sends messages to your API endpoint
377- Parses streaming `StreamChunk` responses
378- Updates messages in real-time
379- You control input state (v5 API style)
380​
381See `packages/ai-react/README.md` for full documentation and backend examples.
382​
383## Advanced Usage
384​
385### Automatic Tool Calling
386​
387Define tools with `execute` functions and `streamChat` handles everything automatically:
388​
389```typescript
390import type { Tool } from "@tanstack/ai";
391​
392const tools: Tool[] = [
393 {
394 type: "function",
395 function: {
396 name: "get_weather",
397 description: "Get current weather",
398 parameters: {
399 type: "object",
400 properties: {
401 location: { type: "string", description: "City name" },
402 },
403 required: ["location"],
404 },
405 },
406 execute: async ({ location }: { location: string }) => {
407 const weather = await fetchWeather(location);
408 return JSON.stringify(weather);
409 },
410 },
411 {
412 type: "function",
413 function: {
414 name: "calculate",
415 description: "Perform calculations",
416 parameters: {
417 type: "object",
418 properties: {
419 expression: { type: "string" },
420 },
421 required: ["expression"],
422 },
423 },
424 execute: async ({ expression }: { expression: string }) => {
425 const result = eval(expression);
426 return JSON.stringify({ result });
427 },
428 },
429];
430​
431// Tools are executed automatically!
432for await (const chunk of ai.streamChat({
433 model: "gpt-3.5-turbo",
434 messages: [
435 { role: "user", content: "What's the weather in Paris and what's 2+2?" },
436 ],
437 tools,
438 toolChoice: "auto",
439 maxIterations: 5, // Limit tool calling loops (default: 5)
440})) {
441 if (chunk.type === "content") {
442 process.stdout.write(chunk.delta);
443 }
444 // Tools are called and executed automatically - no manual handling needed!
446```
447​
448**Key Features:**
449​
450- ✅ Define `execute` function with each tool
451- ✅ `streamChat` automatically detects tool calls
452- ✅ Executes tools automatically
453- ✅ Adds results back to conversation
454- ✅ Continues until final response (up to `maxIterations`)
455- ✅ You just handle the stream - no manual tool logic!
456​
457### Backend Helper - toStreamResponse
458​
459Convert streaming responses to HTTP responses with one line:
460​
461```typescript
462import { AI, toStreamResponse } from "@tanstack/ai";
463import { OpenAIAdapter } from "@tanstack/ai-openai";
464​
465const ai = new AI(new OpenAIAdapter({ apiKey: "..." }));
466​
467// Express/Node.js
468app.post("/api/chat", async (req, res) => {
469 const { messages } = req.body;
470​
471 const stream = ai.streamChat({
472 model: "gpt-3.5-turbo",
473 messages,
474 tools, // Optional
475 });
476​
477 // One line to convert to HTTP response!
478 return toStreamResponse(stream);
479});
480​
481// Next.js App Router
482export async function POST(req: Request) {
483 const { messages } = await req.json();
484​
485 const stream = ai.streamChat({
486 model: "gpt-3.5-turbo",
487 messages,
488 });
489​
490 return toStreamResponse(stream);
492​
493// TanStack Start
494export const Route = createFileRoute("/api/chat")({
495 server: {
496 handlers: {
497 POST: async ({ request }) => {
498 const { messages } = await request.json();
499​
500 return toStreamResponse(
501 ai.streamChat({ model: "gpt-3.5-turbo", messages })
502 );
503 },
504 },
505 },
506});
507```
508​
509The helper handles:
510​
511- ✅ Server-Sent Events formatting
512- ✅ Proper headers (Content-Type, Cache-Control, Connection)
513- ✅ JSON encoding of chunks
514- ✅ Error handling and error chunks
515- ✅ Completion marker `[DONE]`
516​
517### Provider-Specific Configuration
518​
519Each adapter accepts its own configuration:
520​
521```typescript
522// OpenAI
523new OpenAIAdapter({
524 apiKey: string,
525 organization: string,
526 baseURL: string,
527 timeout: number,
528 maxRetries: number,
529});
530​
531// Anthropic
532new AnthropicAdapter({
533 apiKey: string,
534 baseUrl: string,
535 timeout: number,
536 maxRetries: number,
537});
538​
539// Ollama (local)
540new OllamaAdapter({
541 host: string, // Default: "http://localhost:11434"
542});
543​
544// Gemini
545new GeminiAdapter({
546 apiKey: string,
547});
548```
549​
550## Examples
551​
552### Basic Chat
553​
554```typescript
555import { AI } from "@tanstack/ai";
556import { OpenAIAdapter } from "@tanstack/ai-openai";
557​
558const ai = new AI(new OpenAIAdapter({ apiKey: process.env.OPENAI_API_KEY }));
559​
560const response = await ai.chat({
561 model: "gpt-3.5-turbo",
562 messages: [{ role: "user", content: "Explain quantum computing" }],
563});
564​
565console.log(response.content);
566```
567​
568### Streaming
569​
570```typescript
571for await (const chunk of ai.streamChat({
572 model: "gpt-3.5-turbo",
573 messages: [{ role: "user", content: "Tell me a story" }],
574})) {
575 if (chunk.type === "content") {
576 process.stdout.write(chunk.delta);
577 }
578 if (chunk.type === "done") {
579 console.log(`\nTokens: ${chunk.usage?.totalTokens}`);
580 }
582```
583​
584### Switching Providers
585​
586```typescript
587// Start with OpenAI
588const ai = new AI(new OpenAIAdapter({ apiKey: "..." }));
589​
590// Switch to Anthropic - same code works!
591ai.setAdapter(new AnthropicAdapter({ apiKey: "..." }));
592const response = await ai.chat({ model: "claude-3-sonnet-20240229", messages });
593​
594// Switch to local Ollama
595ai.setAdapter(new OllamaAdapter());
596const response2 = await ai.chat({ model: "llama2", messages });
597```
598​
599### Text Generation
600​
601```typescript
602const result = await ai.generateText({
603 model: "gpt-3.5-turbo-instruct",
604 prompt: "Write a haiku about TypeScript",
605 maxTokens: 50,
606});
607​
608console.log(result.text);
609```
610​
611### Summarization
612​
613```typescript
614const longText = `...`; // Your long text
615​
616const summary = await ai.summarize({
617 model: "gpt-3.5-turbo",
618 text: longText,
619 style: "bullet-points",
620 maxLength: 300,
621});
622​
623console.log(summary.summary);
624```
625​
626### Embeddings
627​
628```typescript
629const result = await ai.embed({
630 model: "text-embedding-ada-002",
631 input: "Semantic search query",
632});
633​
634const vector = result.embeddings[0]; // number[]
635console.log(`Dimensions: ${vector.length}`);
636```
637​
638### React Hook - useChat
639​
640Build chat interfaces with the `useChat` hook:
641​
642```typescript
643import { useChat } from "@tanstack/ai-react";
644​
645function ChatComponent() {
646 const { messages, sendMessage, isLoading, error } = useChat({
647 api: "/api/chat",
648 });
649​
650 const [input, setInput] = useState("");
651​
652 return (
653 <div>
654 {/* Message list */}
655 {messages.map((message) => (
656 <div key={message.id}>
657 <strong>{message.role}:</strong> {message.content}
658 </div>
659 ))}
660​
661 {/* Input - you control the state */}
662 <input
663 value={input}
664 onChange={(e) => setInput(e.target.value)}
665 onKeyDown={(e) => {
666 if (e.key === "Enter") {
667 sendMessage(input);
668 setInput("");
669 }
670 }}
671 placeholder="Type a message..."
672 disabled={isLoading}
673 />
674 <button
675 onClick={() => {
676 sendMessage(input);
677 setInput("");
678 }}
679 disabled={isLoading || !input.trim()}
680 >
681 {isLoading ? "Sending..." : "Send"}
682 </button>
683​
684 {error && <div>Error: {error.message}</div>}
685 </div>
686 );
688```
689​
690The hook handles:
691​
692- Message state management
693- Sending messages to your API endpoint
694- Streaming response parsing
695- Loading and error states
696- Automatic message updates as chunks arrive
697​
698**You control the input** - the hook only manages message state and API communication.
699​
700## Examples
701​
702### Standalone Scripts
703​
704```bash
705# Install and build
706pnpm install && pnpm build
707​
708# Run examples (set API keys first)
709cd examples
710​
711# See basic usage of all providers
712pnpm quick-start
713​
714# See streaming in action
715pnpm streaming-demo
716​
717# See tool calling with both OpenAI & Anthropic (same code!)
718pnpm tool-calling
719```
720​
721**Note:** Set `OPENAI_API_KEY` and/or `ANTHROPIC_API_KEY` environment variables to see the examples work with real API calls.
722​
723### Full TanStack Start Application
724​
725A complete chat application with tool calling, built with TanStack Start and our React hooks:
726​
727```bash
728cd examples/ts-chat
729​
730# Set your API key
731echo "ANTHROPIC_API_KEY=sk-ant-..." > .env
732​
733# Install and run
734pnpm install
735pnpm dev
736```
737​
738Open http://localhost:3000/demo/tanchat to see:
739​
740- Real-time streaming chat with Claude
741- Tool calling for guitar recommendations
742- Beautiful UI with markdown formatting
743- All powered by `@tanstack/ai-react`
744​
745### Interactive CLI
746​
747We provide a full-featured CLI for testing and demos:
748​
749```bash
750# Interactive chat
751pnpm cli chat --provider openai
752​
753# Tool calling demo (OpenAI & Anthropic)
754pnpm cli tools --provider openai
755pnpm cli tools --provider anthropic
756​
757# See JSON stream chunks
758pnpm cli chat --provider openai --debug
759​
760# Other commands
761pnpm cli generate --provider anthropic --prompt "..."
762pnpm cli summarize --provider gemini --text "..." --style concise
763pnpm cli embed --provider openai --text "..."
764```
765​
766See `examples/cli/README.md` for full CLI documentation.
767​
768## Package Structure
769​
770```
771@tanstack/ai/
772├── packages/
773│ ├── ai/ # Core library
774│ │ ├── types.ts # Type definitions
775│ │ ├── ai.ts # Main AI class
776│ │ ├── base-adapter.ts # Adapter base class
777│ │ └── stream-utils.ts # Streaming utilities
778│ ├── ai-openai/ # OpenAI adapter
779│ ├── ai-anthropic/ # Anthropic adapter
780│ ├── ai-ollama/ # Ollama adapter
781│ ├── ai-gemini/ # Google Gemini adapter
782│ └── ai-react/ # React hooks
783│ ├── use-chat.ts # Chat hook
784│ └── types.ts # React-specific types
785└── examples/
786 ├── cli/ # Interactive CLI demo
787 ├── ts-chat/ # TanStack Start full app
788 ├── quick-start.js # Basic examples
789 ├── streaming-demo.js # Streaming demo
790 └── tool-calling-example.js # Tool calling demo
791```
792​
793## Provider Support Matrix
794​
795| Feature | OpenAI | Anthropic | Ollama | Gemini |
796| --------------- | ------ | --------- | ------ | ------ |
797| Chat | ✅ | ✅ | ✅ | ✅ |
798| Streaming | ✅ | ✅ | ✅ | ✅ |
799| Tool Calling | ✅ | ✅ | ⏳ | ⏳ |
800| Text Generation | ✅ | ✅ | ✅ | ✅ |
801| Summarization | ✅ | ✅ | ✅ | ✅ |
802| Embeddings | ✅ | ❌ | ✅ | ✅ |
803​
804✅ = Fully supported | ⏳ = Planned | ❌ = Not supported by provider
805​
806## Development
807​
808```bash
809# Install dependencies
810pnpm install
811​
812# Build all packages
813pnpm build
814​
815# Run in dev mode
816pnpm dev
817​
818# Type checking
819pnpm typecheck
820​
821# Clean build artifacts
822pnpm clean
823```
824​
825## Contributing
826​
827We welcome contributions! This is a community-driven project providing a truly open alternative to proprietary AI SDKs.
828​
829## License
830​
831MIT - Use freely, modify, share. No strings attached.
832​
833## Philosophy
834​
835Unlike certain companies that use open source as marketing only to lock you into paid services, @tanstack/ai is committed to remaining truly open source. No enshittification, no bait-and-switch, just honest software that respects developers.
836​
837---
838​
839Built with ❤️ by the open-source community.