OpenAI-compatible Chat Completions adapter drops an assistant message's thinking when replaying it (reasoning_content is read from the stream but never sent back) #2202
Send to my agentTanStack AI version#
@tanstack/ai 1.7.0, @tanstack/ai-openai 1.2.0, @tanstack/openai-base 1.1.0 (same code on main at e13e919)
Framework/Library version#
Node 24.20.0 (no UI framework involved)
Describe the bug and the steps to reproduce it#
OpenAIBaseChatCompletionsTextAdapter.convertMessage builds an assistant request message from its text and tool calls only. message.thinking is left out, so the thinking an OpenAI-compatible provider streamed on one step of the agent loop is missing from the request for the next step.
The inbound half works:
OpenAICompatibleChatAdapter.extractReasoningreadsdelta.reasoning_content/delta.reasoning/delta.reasoning_textand emits theREASONING_*events: https://github.com/TanStack/ai/blob/e13e919892fbab97f809c85c9fc60e8fc121a5ab/packages/ai-openai/src/compatible/adapter.ts#L99-L112chat()puts that text on the assistantModelMessageasthinking: [{ content }]for the next step: https://github.com/TanStack/ai/blob/e13e919892fbab97f809c85c9fc60e8fc121a5ab/packages/ai/src/activities/chat/index.ts#L3453-L3465
The outbound half drops it:
return {
role: 'assistant',
content: hasToolCalls && !textContent ? null : textContent,
...(hasToolCalls ? { tool_calls: toolCalls } : {}),
}
The adapters on this base that I read (@tanstack/ai-byteplus, @tanstack/ai-cloudflare, @tanstack/ai-llmgateway) don't send plain-text thinking back either. ai-byteplus overrides convertMessage, but only for encrypted_content.
Steps to reproduce#
The script below needs no network or API key: a stub fetch plays the provider. Step 1 streams reasoning_content and a tool call, step 2 an answer. It prints the messages of the second request.
import { chat, toolDefinition } from '@tanstack/ai'
import { openaiCompatible } from '@tanstack/ai-openai/compatible'
import { z } from 'zod'
const requests = []
const sse = (chunks) =>
new Response(
chunks
.map((delta, i) =>
`data: ${JSON.stringify({
id: 'c1',
object: 'chat.completion.chunk',
created: 0,
model: 'local',
choices: [
{
index: 0,
delta,
finish_reason:
i < chunks.length - 1 ? null : delta.tool_calls ? 'tool_calls' : 'stop',
},
],
})}\n\n`,
)
.join('') + 'data: [DONE]\n\n',
{ headers: { 'content-type': 'text/event-stream' } },
)
const fetchStub = async (_url, init) => {
requests.push(JSON.parse(init.body))
return requests.length === 1
? sse([
{ role: 'assistant', reasoning_content: 'The user wants the weather. ' },
{ reasoning_content: 'I will call get_weather for Paris.' },
{
tool_calls: [
{
index: 0,
id: 'call_1',
type: 'function',
function: { name: 'get_weather', arguments: '{"city":"Paris"}' },
},
],
},
])
: sse([{ role: 'assistant', content: 'It is sunny in Paris.' }])
}
const local = openaiCompatible({
baseURL: 'http://localhost:8080/v1',
apiKey: 'none',
models: ['local'],
fetch: fetchStub,
})
const getWeather = toolDefinition({
name: 'get_weather',
description: 'Get the weather for a city',
inputSchema: z.object({ city: z.string() }),
}).server(async () => ({ weather: 'sunny' }))
for await (const _ of chat({
adapter: local('local'),
messages: [{ role: 'user', content: 'What is the weather in Paris?' }],
tools: [getWeather],
threadId: 'repro',
})) {
// drain
}
console.log(JSON.stringify(requests[1].messages, null, 2))
Actual: the assistant message of the second request has no thinking.
[
{ "role": "user", "content": "What is the weather in Paris?" },
{
"role": "assistant",
"content": null,
"tool_calls": [
{
"id": "call_1",
"type": "function",
"function": { "name": "get_weather", "arguments": "{\"city\":\"Paris\"}" }
}
]
},
{ "role": "tool", "tool_call_id": "call_1", "content": "{\"weather\":\"sunny\"}" }
]
Expected: a way to have that assistant message carry "reasoning_content": "The user wants the weather. I will call get_weather for Paris.".
Why it matters#
Models that think and call tools expect their thinking back for the turn in progress (the steps after the last user message).
Local OpenAI-compatible servers hand an input message's
reasoning_contentto the model's chat template. llama.cpp reads it here: https://github.com/ggml-org/llama.cpp/blob/b94959a64d011915d77a5381c7d4846dcda99530/common/chat.cpp#L436-L438The templates keep it for the turn in progress and drop it for earlier turns themselves. LFM3's template:
{%- set thinking = message.thinking if message.thinking is defined else message.reasoning if message.reasoning is defined else message.reasoning_content -%} {%- set keep_thinking = preserve_thinking or loop.index0 > ns.last_user_index -%} {%- if thinking is defined and keep_thinking -%} {{- "<think>" + (thinking or "") + "</think>" -}} {%- elif keep_thinking and "</think>" not in (message.get("content") if message.get("content") is string else "") -%} {{- "<think></think>" -}} {%- endif -%}With the field missing, the step that brings a tool's result renders
<think></think>for the assistant step that made the call. The model can't read what it thought before it called the tool.DeepSeek's thinking mode guide says that for requests with
tools,reasoning_content"must be fully passed back to the API in all subsequent requests", and that the API returns a 400 otherwise. Withouttoolsit says the field is ignored. I have not run this against DeepSeek's API.
Proposed change#
Let an OpenAI-compatible Chat Completions adapter send an assistant message's thinking text back, as an opt-in:
- A protected hook on the base next to
extractReasoning, for examplereplayReasoning(message), whose resultconvertMessageadds to the assistant message. The base default returns nothing, asextractReasoningdoes. - An option on
openaiCompatible/openaiCompatibleTextthat turns it on forOpenAICompatibleChatAdapter, sendingreasoning_content. The field name could be a setting, since the stream side already reads three names.
Opt-in rather than always on, because reasoning_content is not part of OpenAI's Chat Completions message schema and I have not checked which hosted endpoints reject unknown fields on a message. Providers also differ on when they want it (DeepSeek above: only with tools).
Today the only way is to subclass and override the protected convertMessage:
protected override convertMessage(message: ModelMessage) {
const converted = super.convertMessage(message)
const thinking =
message.role === 'assistant'
? (message.thinking ?? []).map((part) => part.content).join('')
: ''
return thinking ? { ...converted, reasoning_content: thinking } : converted
}
With that override the script above prints the field. But it means leaving the openaiCompatible() factory, which constructs OpenAICompatibleChatAdapter itself.
Related: #2191 reports the same gap in @tanstack/ai-openrouter, which has its own convertMessage. #1836 (closed) did this for MiniMax on the gateway adapters.
Your Minimal, Reproducible Example - (Sandbox Highly Recommended)#
The script in the description. It runs with npm install @tanstack/ai @tanstack/ai-openai zod and node repro.mjs; a stub fetch serves the stream, so it needs no provider.
Screenshots or Videos (Optional)#
No response
Do you intend to try to help solve this bug with your own PR?#
Yes, I think I know how to fix it and will discuss it in the comments of this issue
Terms & Code of Conduct#
- I agree to follow this project's Code of Conduct
- I understand that if my bug cannot be reliable reproduced in a debuggable environment, it will probably not be fixed and this issue may even be closed.
tanstack-source added labelsbug
github-actions[bot](GitHub profile, opens in a new tab) assignedtombeckenham
github-actions[bot](GitHub profile, opens in a new tab) added labelswaiting-on: maintainer
Sign in to join this conversation.