drainAfterTerminal misses openai-base's own client stream (follow-up to #1965) #2090
Send to my agentFollow-up to #1965 (closed by #1966).
Thanks for #1966: the drain after a terminal event is exactly what we needed. One gap remains with the adapters' own client in @tanstack/ai-openai 1.1.x (@tanstack/openai-base 1.0.2, @tanstack/ai-utils 1.1.0), and the 499 race from this issue comes back there.
What happens. drainResponsesStream passes drainAfterTerminal an abort only when the stream is the SDK's Stream (stream instanceof Stream), and drainAfterTerminal can also stop a ReadableStream itself. The stream the Responses adapter gets from openai-base's own client (buildClient → call → readEvents/mapEvents) is neither: it is a plain async generator. So there is no stop, the finally takes the last branch, and the stream is closed (return(), which cancels the body) as soon as the consumer stops at response.completed. A body that would have ended a few ms later is cut, and a gateway logs the finished, billed call as a client cancel (499).
Repro (Node 22+, @tanstack/ai 1.2.0, @tanstack/ai-openai 1.1.1, vitest):
import { chat } from "@tanstack/ai";
import { createOpenaiChat } from "@tanstack/ai-openai";
import { expect, it } from "vitest";
const sse = (type: string, rest: object) => `event: ${type}\ndata: ${JSON.stringify({ type, ...rest })}\n\n`;
const response = { id: "resp_1", model: "m", status: "completed" };
const events = [
sse("response.created", { sequence_number: 0, response: { ...response, status: "in_progress", output: [] } }),
sse("response.output_text.delta", { sequence_number: 1, item_id: "msg_1", output_index: 0, content_index: 0, delta: "Hi." }),
sse("response.completed", {
sequence_number: 2,
response: {
...response,
output: [{ type: "message", id: "msg_1", role: "assistant", content: [{ type: "output_text", text: "Hi.", annotations: [] }] }],
usage: { input_tokens: 1, output_tokens: 1, total_tokens: 2 },
},
}),
];
it("reads a body that ends 20 ms after response.completed to its end", async () => {
const seen = { cancelled: false, aborted: false, readToEnd: false };
const fetch = async (_url: unknown, init?: RequestInit) => {
init?.signal?.addEventListener("abort", () => (seen.aborted = true));
let sent = 0;
const body = new ReadableStream<Uint8Array>({
cancel: () => void (seen.cancelled = true),
async pull(controller) {
if (sent < events.length) return controller.enqueue(new TextEncoder().encode(events[sent++]));
await new Promise((r) => setTimeout(r, 20)); // the body ends a moment after the last event
seen.readToEnd = true;
controller.close();
},
});
return new Response(body, { headers: { "content-type": "text/event-stream" } });
};
const adapter = createOpenaiChat("m" as never, "key", { baseURL: "https://provider.test/v1", fetch: fetch as never });
for await (const _ of chat({ adapter, threadId: "t", runId: "r", messages: [{ role: "user", content: "Hi" }] as never }));
await new Promise((r) => setTimeout(r, 100));
expect(seen).toEqual({ cancelled: false, aborted: false, readToEnd: true }); // fails: { cancelled: true, aborted: false, readToEnd: true } (the body is cancelled before its end; this fake body's own timer still runs out)
});
A possible fix, which we run as a local patch: give the client's stream a way to stop its request. In buildClient's call, a stream's request gets its own AbortController linked to the caller's signal, and the stream returned carries abort(). drainResponsesStream then passes () => stream.abort() when there is one, so drainAfterTerminal drains it for DRAIN_AFTER_TERMINAL_MS and aborts it past that, as it does for the SDK's Stream. (Passing the abort from the client also ends a read still waiting on a body that stays open, which return() on a generator would queue behind.)
Happy to open a PR with this and a test like the one above, if that direction suits you.
tanstack-source added labelsbug
github-actions[bot](GitHub profile, opens in a new tab) assignedjherr
github-actions[bot](GitHub profile, opens in a new tab) added labelswaiting-on: maintainer
Sign in to join this conversation.