Skip to content
Source
Sign inSign up
TanStack/ai

OpenAI-compatible Chat Completions adapter drops an assistant message's thinking when replaying it (reasoning_content is read from the stream but never sent back) #2202

Send to my agent
Opendaveycodez(GitHub profile, opens in a new tab) opened this issue 0 comments

TanStack AI version

@tanstack/ai 1.7.0, @tanstack/ai-openai 1.2.0, @tanstack/openai-base 1.1.0 (same code on main at e13e919)

Framework/Library version

Node 24.20.0 (no UI framework involved)

Describe the bug and the steps to reproduce it

OpenAIBaseChatCompletionsTextAdapter.convertMessage builds an assistant request message from its text and tool calls only. message.thinking is left out, so the thinking an OpenAI-compatible provider streamed on one step of the agent loop is missing from the request for the next step.

The inbound half works:

The outbound half drops it:

https://github.com/TanStack/ai/blob/e13e919892fbab97f809c85c9fc60e8fc121a5ab/packages/openai-base/src/adapters/chat-completions-text.ts#L1684-L1709

return {
  role: 'assistant',
  content: hasToolCalls && !textContent ? null : textContent,
  ...(hasToolCalls ? { tool_calls: toolCalls } : {}),
}

The adapters on this base that I read (@tanstack/ai-byteplus, @tanstack/ai-cloudflare, @tanstack/ai-llmgateway) don't send plain-text thinking back either. ai-byteplus overrides convertMessage, but only for encrypted_content.

Steps to reproduce

The script below needs no network or API key: a stub fetch plays the provider. Step 1 streams reasoning_content and a tool call, step 2 an answer. It prints the messages of the second request.

import { chat, toolDefinition } from '@tanstack/ai'
import { openaiCompatible } from '@tanstack/ai-openai/compatible'
import { z } from 'zod'

const requests = []
const sse = (chunks) =>
  new Response(
    chunks
      .map((delta, i) =>
        `data: ${JSON.stringify({
          id: 'c1',
          object: 'chat.completion.chunk',
          created: 0,
          model: 'local',
          choices: [
            {
              index: 0,
              delta,
              finish_reason:
                i < chunks.length - 1 ? null : delta.tool_calls ? 'tool_calls' : 'stop',
            },
          ],
        })}\n\n`,
      )
      .join('') + 'data: [DONE]\n\n',
    { headers: { 'content-type': 'text/event-stream' } },
  )

const fetchStub = async (_url, init) => {
  requests.push(JSON.parse(init.body))
  return requests.length === 1
    ? sse([
        { role: 'assistant', reasoning_content: 'The user wants the weather. ' },
        { reasoning_content: 'I will call get_weather for Paris.' },
        {
          tool_calls: [
            {
              index: 0,
              id: 'call_1',
              type: 'function',
              function: { name: 'get_weather', arguments: '{"city":"Paris"}' },
            },
          ],
        },
      ])
    : sse([{ role: 'assistant', content: 'It is sunny in Paris.' }])
}

const local = openaiCompatible({
  baseURL: 'http://localhost:8080/v1',
  apiKey: 'none',
  models: ['local'],
  fetch: fetchStub,
})

const getWeather = toolDefinition({
  name: 'get_weather',
  description: 'Get the weather for a city',
  inputSchema: z.object({ city: z.string() }),
}).server(async () => ({ weather: 'sunny' }))

for await (const _ of chat({
  adapter: local('local'),
  messages: [{ role: 'user', content: 'What is the weather in Paris?' }],
  tools: [getWeather],
  threadId: 'repro',
})) {
  // drain
}

console.log(JSON.stringify(requests[1].messages, null, 2))

Actual: the assistant message of the second request has no thinking.

[
  { "role": "user", "content": "What is the weather in Paris?" },
  {
    "role": "assistant",
    "content": null,
    "tool_calls": [
      {
        "id": "call_1",
        "type": "function",
        "function": { "name": "get_weather", "arguments": "{\"city\":\"Paris\"}" }
      }
    ]
  },
  { "role": "tool", "tool_call_id": "call_1", "content": "{\"weather\":\"sunny\"}" }
]

Expected: a way to have that assistant message carry "reasoning_content": "The user wants the weather. I will call get_weather for Paris.".

Why it matters

Models that think and call tools expect their thinking back for the turn in progress (the steps after the last user message).

  • Local OpenAI-compatible servers hand an input message's reasoning_content to the model's chat template. llama.cpp reads it here: https://github.com/ggml-org/llama.cpp/blob/b94959a64d011915d77a5381c7d4846dcda99530/common/chat.cpp#L436-L438

  • The templates keep it for the turn in progress and drop it for earlier turns themselves. LFM3's template:

    {%- set thinking = message.thinking if message.thinking is defined else message.reasoning if message.reasoning is defined else message.reasoning_content -%}
    {%- set keep_thinking = preserve_thinking or loop.index0 > ns.last_user_index -%}
    {%- if thinking is defined and keep_thinking -%}
        {{- "<think>" + (thinking or "") + "</think>" -}}
    {%- elif keep_thinking and "</think>" not in (message.get("content") if message.get("content") is string else "") -%}
        {{- "<think></think>" -}}
    {%- endif -%}

    With the field missing, the step that brings a tool's result renders <think></think> for the assistant step that made the call. The model can't read what it thought before it called the tool.

  • DeepSeek's thinking mode guide says that for requests with tools, reasoning_content "must be fully passed back to the API in all subsequent requests", and that the API returns a 400 otherwise. Without tools it says the field is ignored. I have not run this against DeepSeek's API.

Proposed change

Let an OpenAI-compatible Chat Completions adapter send an assistant message's thinking text back, as an opt-in:

  • A protected hook on the base next to extractReasoning, for example replayReasoning(message), whose result convertMessage adds to the assistant message. The base default returns nothing, as extractReasoning does.
  • An option on openaiCompatible / openaiCompatibleText that turns it on for OpenAICompatibleChatAdapter, sending reasoning_content. The field name could be a setting, since the stream side already reads three names.

Opt-in rather than always on, because reasoning_content is not part of OpenAI's Chat Completions message schema and I have not checked which hosted endpoints reject unknown fields on a message. Providers also differ on when they want it (DeepSeek above: only with tools).

Today the only way is to subclass and override the protected convertMessage:

protected override convertMessage(message: ModelMessage) {
  const converted = super.convertMessage(message)
  const thinking =
    message.role === 'assistant'
      ? (message.thinking ?? []).map((part) => part.content).join('')
      : ''
  return thinking ? { ...converted, reasoning_content: thinking } : converted
}

With that override the script above prints the field. But it means leaving the openaiCompatible() factory, which constructs OpenAICompatibleChatAdapter itself.

Related: #2191 reports the same gap in @tanstack/ai-openrouter, which has its own convertMessage. #1836 (closed) did this for MiniMax on the gateway adapters.

The script in the description. It runs with npm install @tanstack/ai @tanstack/ai-openai zod and node repro.mjs; a stub fetch serves the stream, so it needs no provider.

Screenshots or Videos (Optional)

No response

Do you intend to try to help solve this bug with your own PR?

Yes, I think I know how to fix it and will discuss it in the comments of this issue

Terms & Code of Conduct

  • I agree to follow this project's Code of Conduct
  • I understand that if my bug cannot be reliable reproduced in a debuggable environment, it will probably not be fixed and this issue may even be closed.
  1. tanstack-source added labelsbug

  2. github-actions[bot](GitHub profile, opens in a new tab) added labelswaiting-on: maintainer

Sign in to join this conversation.