Skip to content
Source
Sign inSign up
TanStack/ai
Open History to see the last commit

http-stream-protocol.md

437 lines11.3 KB

1---
2title: HTTP Stream Protocol
3id: http-stream-protocol
4description: "TanStack AI's HTTP streaming protocol spec using newline-delimited JSON (NDJSON) — an alternative to SSE for simpler line-based transport."
5keywords:
6 - tanstack ai
7 - http stream
8 - ndjson
9 - newline-delimited json
10 - streaming protocol
11 - protocol spec
12---
13​
14HTTP streaming with newline-delimited JSON (NDJSON) is a simpler protocol than SSE that sends one JSON object per line. It's useful when:
15​
16- SSE event prefixes add unwanted overhead
17- You need more control over the streaming format
18- Working in environments that don't support SSE well
19- Building custom protocols on top of the stream
20​
21This protocol is **less common** than SSE for TanStack AI applications, but supported for flexibility.
22​
23This document describes how TanStack AI transmits StreamChunks over raw HTTP streaming (newline-delimited JSON), an alternative to Server-Sent Events.
24​
25---
26​
27## Protocol Specification
28​
29### HTTP Request
30​
31**Method:** `POST`
32​
33**Headers:**
34```http
35Content-Type: application/json
36```
37​
38**Body:**
39```json
41 "messages": [
42 {
43 "role": "user",
44 "content": "Hello, how are you?"
45 }
46 ],
47 "data": {
48 // Optional additional data
49 }
51```
52​
53### HTTP Response
54​
55**Status:** `200 OK`
56​
57**Headers:**
58```http
59Content-Type: application/x-ndjson
60Transfer-Encoding: chunked
61```
62​
63Or alternatively:
64```http
65Content-Type: application/json
66Transfer-Encoding: chunked
67```
68​
69**Body:** Stream of newline-delimited JSON chunks
70​
71---
72​
73## Stream Format
74​
75Each StreamChunk is transmitted as a single line of JSON followed by a newline (`\n`):
76​
77```
78{JSON_ENCODED_CHUNK}\n
79```
80​
81### Key Points
82​
831. **One JSON object per line**
842. **Each line ends with `\n`**
853. **No prefixes** (unlike SSE's `data:` prefix)
864. **No blank lines between chunks** (unlike SSE's `\n\n`)
875. **Stream ends when connection closes** (no `[DONE]` marker)
88​
89### Examples
90​
91#### Content Chunks
92​
93```json
94{"type":"content","id":"chatcmpl-abc123","model":"gpt-5.2","timestamp":1701234567890,"delta":"Hello","content":"Hello","role":"assistant"}
95{"type":"content","id":"chatcmpl-abc123","model":"gpt-5.2","timestamp":1701234567891,"delta":" world","content":"Hello world","role":"assistant"}
96{"type":"content","id":"chatcmpl-abc123","model":"gpt-5.2","timestamp":1701234567892,"delta":"!","content":"Hello world!","role":"assistant"}
97```
98​
99#### Tool Call
100​
101```json
102{"type":"tool_call","id":"chatcmpl-abc123","model":"gpt-5.2","timestamp":1701234567893,"toolCall":{"id":"call_xyz","type":"function","function":{"name":"get_weather","arguments":"{\"location\":\"SF\"}"}},"index":0}
103{"type":"tool_result","id":"chatcmpl-abc123","model":"gpt-5.2","timestamp":1701234567894,"toolCallId":"call_xyz","content":"{\"temperature\":72,\"conditions\":\"sunny\"}"}
104```
105​
106#### Stream Completion
107​
108```json
109{"type":"done","id":"chatcmpl-abc123","model":"gpt-5.2","timestamp":1701234567895,"finishReason":"stop","usage":{"promptTokens":10,"completionTokens":15,"totalTokens":25}}
110```
111​
112---
113​
114## Stream Lifecycle
115​
116### 1. Client Initiates Connection
117​
118```typescript
119const response = await fetch('/api/chat', {
120 method: 'POST',
121 headers: {
122 'Content-Type': 'application/json',
123 },
124 body: JSON.stringify({ messages }),
125});
126```
127​
128### 2. Server Sends Response Header
129​
130```http
131HTTP/1.1 200 OK
132Content-Type: application/x-ndjson
133Transfer-Encoding: chunked
134```
135​
136### 3. Server Streams Chunks
137​
138The server sends newline-delimited JSON:
139​
140```json
141{"type":"content","id":"msg_1","model":"gpt-5.2","timestamp":1701234567890,"delta":"The","content":"The"}
142{"type":"content","id":"msg_1","model":"gpt-5.2","timestamp":1701234567891,"delta":" weather","content":"The weather"}
143{"type":"content","id":"msg_1","model":"gpt-5.2","timestamp":1701234567892,"delta":" is","content":"The weather is"}
144{"type":"content","id":"msg_1","model":"gpt-5.2","timestamp":1701234567893,"delta":" sunny","content":"The weather is sunny"}
145{"type":"done","id":"msg_1","model":"gpt-5.2","timestamp":1701234567894,"finishReason":"stop"}
146```
147​
148### 4. Stream Completion
149​
150Server closes the connection. No special marker needed (unlike SSE's `[DONE]`).
151​
152---
153​
154## Error Handling
155​
156### Server-Side Errors
157​
158If an error occurs during generation, send an error chunk:
159​
160```json
161{"type":"error","id":"msg_1","model":"gpt-5.2","timestamp":1701234567895,"error":{"message":"Rate limit exceeded","code":"rate_limit_exceeded"}}
162```
163​
164Then close the connection.
165​
166### Connection Errors
167​
168Unlike SSE, HTTP streaming does not provide automatic reconnection:
169- Client must detect connection drops
170- Client must implement retry logic
171- Use exponential backoff for retries
172​
173---
174​
175## Implementation
176​
177### Server-Side (Node.js/TypeScript)
178​
179#### Using TanStack AI (Custom Stream)
180​
181TanStack AI doesn't provide a built-in NDJSON formatter, but you can create one easily:
182​
183```typescript
184import { chat } from '@tanstack/ai';
185import { openaiText } from '@tanstack/ai-openai';
186​
187export async function POST(request: Request) {
188 const { messages } = await request.json();
189 const encoder = new TextEncoder();
190​
191 const stream = chat({
192 adapter: openaiText('gpt-5.2'),
193 messages,
194 });
195​
196 const readableStream = new ReadableStream({
197 async start(controller) {
198 try {
199 for await (const chunk of stream) {
200 // Send as newline-delimited JSON
201 const line = JSON.stringify(chunk) + '\n';
202 controller.enqueue(encoder.encode(line));
203 }
204 controller.close();
205 } catch (error: any) {
206 const errorChunk = {
207 type: 'error',
208 error: {
209 message: error.message || 'Unknown error',
210 code: error.code,
211 },
212 };
213 controller.enqueue(encoder.encode(JSON.stringify(errorChunk) + '\n'));
214 controller.close();
215 }
216 },
217 });
218​
219 return new Response(readableStream, {
220 headers: {
221 'Content-Type': 'application/x-ndjson',
222 'Cache-Control': 'no-cache',
223 },
224 });
226```
227​
228#### Using Express.js
229​
230```typescript
231import express from 'express';
232import { chat } from '@tanstack/ai';
233import { openaiText } from '@tanstack/ai-openai';
234​
235const app = express();
236app.use(express.json());
237​
238app.post('/api/chat', async (req, res) => {
239 const { messages } = req.body;
240​
241 res.setHeader('Content-Type', 'application/x-ndjson');
242 res.setHeader('Cache-Control', 'no-cache');
243 res.setHeader('Transfer-Encoding', 'chunked');
244​
245 try {
246 const stream = chat({
247 adapter: openaiText('gpt-5.2'),
248 messages,
249 });
250​
251 for await (const chunk of stream) {
252 res.write(JSON.stringify(chunk) + '\n');
253 }
254 } catch (error: any) {
255 const errorChunk = {
256 type: 'error',
257 error: { message: error.message },
258 };
259 res.write(JSON.stringify(errorChunk) + '\n');
260 } finally {
261 res.end();
262 }
263});
264```
265​
266### Client-Side (Browser/Node.js)
267​
268TanStack AI provides `fetchHttpStream()` connection adapter:
269​
270```typescript
271import { useChat, fetchHttpStream } from '@tanstack/ai-react';
272​
273const { messages, sendMessage } = useChat({
274 connection: fetchHttpStream('/api/chat'),
275});
276```
277​
278**What `fetchHttpStream()` does:**
2791. Makes POST request with messages
2802. Reads response body as stream
2813. Splits by newlines
2824. Parses each line as JSON
2835. Yields StreamChunk objects
284​
285### Manual Implementation (Advanced)
286​
287#### Client
288```typescript
289const response = await fetch('/api/chat', {
290 method: 'POST',
291 headers: { 'Content-Type': 'application/json' },
292 body: JSON.stringify({ messages }),
293});
294​
295const reader = response.body!.getReader();
296const decoder = new TextDecoder();
297let buffer = '';
298​
299while (true) {
300 const { done, value } = await reader.read();
301 if (done) break;
303 buffer += decoder.decode(value, { stream: true });
304 const lines = buffer.split('\n');
306 // Keep incomplete line in buffer
307 buffer = lines.pop() || '';
309 for (const line of lines) {
310 if (line.trim()) {
311 try {
312 const chunk = JSON.parse(line);
313 // Handle chunk...
314 console.log(chunk);
315 } catch (error) {
316 console.warn('Failed to parse chunk:', line);
317 }
318 }
319 }
321​
322// Process any remaining data in buffer
323if (buffer.trim()) {
324 try {
325 const chunk = JSON.parse(buffer);
326 console.log(chunk);
327 } catch (error) {
328 console.warn('Failed to parse final chunk:', buffer);
329 }
331```
332​
333---
334​
335## Comparison: HTTP Stream vs SSE
336​
337| Feature | HTTP Stream (NDJSON) | Server-Sent Events (SSE) |
338|---------|---------------------|--------------------------|
339| Format | `{json}\n` | `data: {json}\n\n` |
340| Overhead | Lower (no prefixes) | Higher (`data:` prefix) |
341| Auto-reconnect | ❌ No | ✅ Yes |
342| Browser API | ❌ No (manual) | ✅ Yes (EventSource) |
343| Completion marker | ❌ No (close connection) | ✅ Yes (`[DONE]`) |
344| Debugging | Easy (plain JSON lines) | Easy (plain text) |
345| Use case | Custom protocols, lower overhead | Standard streaming, reconnection needed |
346​
347**Recommendation:** Use SSE (`fetchServerSentEvents`) for most applications. Use HTTP streaming when you need lower overhead or have specific protocol requirements.
348​
349---
350​
351## Debugging
352​
353### Inspecting HTTP Stream Traffic
354​
355**Browser DevTools:**
3561. Open Network tab
3572. Look for POST request to `/api/chat`
3583. View response as it streams in
359​
360**cURL:**
361```bash
362curl -N -X POST http://localhost:3000/api/chat \
363 -H "Content-Type: application/json" \
364 -d '{"messages":[{"role":"user","content":"Hello"}]}'
365```
366​
367The `-N` flag disables buffering to see real-time output.
368​
369**Example Output:**
370```json
371{"type":"content","id":"msg_1","model":"gpt-5.2","timestamp":1701234567890,"delta":"Hello","content":"Hello"}
372{"type":"content","id":"msg_1","model":"gpt-5.2","timestamp":1701234567891,"delta":" there","content":"Hello there"}
373{"type":"done","id":"msg_1","model":"gpt-5.2","timestamp":1701234567892,"finishReason":"stop"}
374```
375​
376### Validating NDJSON
377​
378Each line must be valid JSON. Test with:
379​
380```bash
381# Validate each line
382curl -N http://localhost:3000/api/chat | while read line; do
383 echo "$line" | jq . > /dev/null || echo "Invalid JSON: $line"
384done
385```
386​
387---
388​
389## Advantages of HTTP Streaming
390​
3911. **Lower Overhead** - No `data:` prefixes or double newlines
3922. **Simpler Protocol** - Just JSON + newline
3933. **Flexible** - Easy to extend or modify
3944. **Standard Format** - NDJSON is widely used
395​
396---
397​
398## Disadvantages vs SSE
399​
4001. **No Auto-Reconnect** - Must implement manually
4012. **No Browser API** - Can't use EventSource
4023. **No Completion Marker** - Must rely on connection close
4034. **Less Common** - SSE is more standard for streaming
404​
405---
406​
407## Best Practices
408​
4091. **Use `\n` consistently** - Don't mix `\r\n` and `\n`
4102. **Set proper Content-Type** - Use `application/x-ndjson` or `application/json`
4113. **Handle partial lines** - Buffer incomplete data
4124. **Validate JSON** - Catch parsing errors gracefully
4135. **Flush regularly** - Don't buffer chunks server-side
4146. **Implement retry logic** - Client should handle connection drops
415​
416---
417​
418## Alternative: JSON Lines (.jsonl)
419​
420HTTP streaming in TanStack AI follows the [JSON Lines](http://jsonlines.org/) specification (also called NDJSON):
421​
422- One JSON value per line
423- Each line is terminated with `\n`
424- UTF-8 encoding
425- File extension: `.jsonl` or `.ndjson`
426​
427This makes streams compatible with standard NDJSON tools and libraries.
428​
429---
430​
431## See Also
432​
433- [Chunk Definitions](./chunk-definitions) - StreamChunk type reference
434- [SSE Protocol](./sse-protocol) - Recommended protocol (with auto-reconnect)
435- [Connection Adapters Guide](../chat/connection-adapters) - Client implementation
436- [JSON Lines Specification](http://jsonlines.org/)
437- [NDJSON Specification](http://ndjson.org/)