Skip to content
Source
Sign inSign up
TanStack/ai

openai-base: a Responses max_output_tokens cutoff ends as RUN_ERROR without usage instead of finishReason 'length' #1916

Send to my agent
Opengrayashh(GitHub profile, opens in a new tab) opened this issue 1 comment

What happens

When a Responses stream hits max_output_tokens, it ends with response.incomplete (status: "incomplete", incomplete_details.reason: "max_output_tokens"), and that event carries the response's usage. The Responses adapter in @tanstack/openai-base turns it into RUN_ERROR (message: "max_output_tokens", code: "incomplete") and drops the usage:

https://github.com/TanStack/ai/blob/ae976552710726b13b524c307c1b5bbe78478eb0/packages/openai-base/src/adapters/responses-text.ts#L1748-L1806

So with the Responses adapter:

  • the run ends in onError, not onFinish, right after the partial text the user has already seen;
  • onUsage never fires, although the provider billed the tokens;
  • finishReason: 'length' never appears, so untilFinishReason(['length']) and the core stop-on-length path (#1874) never see a capped answer.

The response.completed branch already maps incomplete_details.reason (max_output_tokens → length, content_filter → content_filter), "mirroring the chat-completions adapter". A capped response does not arrive as response.completed, so that mapping is not reached for it:

https://github.com/TanStack/ai/blob/ae976552710726b13b524c307c1b5bbe78478eb0/packages/openai-base/src/adapters/responses-text.ts#L2442-L2473

The Chat Completions adapter reports the same cutoff as RUN_FINISHED with usage and finishReason: 'length'.

Repro

@tanstack/ai 1.0.0, @tanstack/openai-base 1.0.0, @tanstack/ai-vercel-gateway 1.0.0, server-side chat() on Node 24

for await (const chunk of chat({
  adapter: createVercelGatewayText('openai/gpt-4.1-nano', key), // Responses API by default
  messages: [{ role: 'user', content: 'Count from 1 to 300, separated by commas.' }],
  threadId: 'repro',
  modelOptions: { max_output_tokens: 16 },
  middleware: [{ name: 'probe', onUsage, onFinish, onError }],
})) {
  if (chunk.type === 'RUN_FINISHED' || chunk.type === 'RUN_ERROR') console.log(chunk)
}
AdapterTerminal chunkonUsageTerminal hook
Responses (default)RUN_ERROR { message: 'max_output_tokens', code: 'incomplete' }not calledonError
Chat Completions (api: 'chat', max_tokens: 16)RUN_FINISHED, usage 19 / 16called onceonFinish, finishReason: 'length'

On the Responses adapter the chunks arrive as RUN_STARTED, TEXT_MESSAGE_START, TEXT_MESSAGE_CONTENT (62 characters), TEXT_MESSAGE_END, RUN_ERROR. anthropic/claude-sonnet-5.5 gives the same RUN_ERROR. A direct POST https://ai-gateway.vercel.sh/v1/responses with stream: true and max_output_tokens: 16 ends like this for both models (gpt-4.1-nano shown):

{
  "type": "response.incomplete",
  "response": {
    "status": "incomplete",
    "incomplete_details": { "reason": "max_output_tokens" },
    "usage": { "input_tokens": 19, "output_tokens": 16, "…": "…" }
  }
}

The openai SDK types describe response.incomplete as "An event that is emitted when a response finishes as incomplete", with incomplete_details.reason one of max_output_tokens, max_messages, content_filter, steered.

Expected

response.incomplete with incomplete_details.reason max_output_tokens or content_filter ends the run the way the Chat Completions adapter does: RUN_FINISHED with the response's usage and finishReason: 'length' or 'content_filter'. response.failed, and an incomplete response without a known reason, stay RUN_ERROR.

Why

We stop our agent loop with untilFinishReason(['length']) and record usage per model call. On the Responses adapter, a capped answer shows the user an error after a partial answer, and its tokens are not recorded.

  1. Confirmed on ae97655, the SHA your permalinks point at. The cause is that response.incomplete is only ever handled by the response.failed || response.incomplete terminal branch, which unconditionally yields RUN_ERROR and never looks at chunk.response.usage. The incomplete_details.reason to finishReason mapping exists only in the response.completed branch, and a capped response never arrives as response.completed, so nothing maps it.

    I reproduced it offline with no API key, reusing the package's own stub-client pattern: one canned response.incomplete carrying incomplete_details.reason: 'max_output_tokens' plus usage, on the Responses adapter, against the equivalent finish_reason: 'length' plus usage-tail chunk on the Chat Completions adapter. Both are driven through chat() with an onUsage / onFinish / onError middleware, and I also read the adapter's own chatStream() output directly.

    $ corepack pnpm vitest run packages/openai-base/tests/repro-1916.test.ts --silent=false --disable-console-intercept
    
    --- Responses adapter (response.incomplete) ---
    chunk types     : RUN_STARTED -> TEXT_MESSAGE_START -> TEXT_MESSAGE_CONTENT -> TEXT_MESSAGE_END -> RUN_ERROR
    terminal chunk  : {"type":"RUN_ERROR","message":"max_output_tokens","code":"incomplete"}
    onUsage calls   : 0
    terminal hooks  : onError
    adapter terminal: {"type":"RUN_ERROR","message":"max_output_tokens","code":"incomplete"}
    
    --- Chat Completions adapter (finish_reason: length) ---
    chunk types     : RUN_STARTED -> TEXT_MESSAGE_START -> TEXT_MESSAGE_CONTENT -> TEXT_MESSAGE_END -> RUN_FINISHED
    terminal chunk  : {"type":"RUN_FINISHED","usage":{"promptTokens":19,"completionTokens":16,"totalTokens":35}}
    onUsage calls   : 1 {"promptTokens":19,"completionTokens":16,"totalTokens":35}
    terminal hooks  : onFinish(finishReason="length")
    adapter terminal: {"type":"RUN_FINISHED","finishReason":"length","usage":{"promptTokens":19,"completionTokens":16,"totalTokens":35}}
    
     Test Files  1 passed (1)
          Tests  1 passed (1)

    That matches your report line for line. One detail worth knowing for the fix: finishReason is carried on the adapter's own RUN_FINISHED chunk and in the FinishInfo passed to onFinish, not on the RUN_FINISHED chunk chat() re-emits, which is what untilFinishReason reads through the agent-loop strategy.

    Fix

    In that branch, after the lifecycle close (closeReasoning() and the TEXT_MESSAGE_END pair) and before the RUN_ERROR payload is built, treat a known incomplete_details.reason as a completed-but-capped response:

              const cappedFinishReason =
                chunk.type === 'response.incomplete'
                  ? chunk.response.incomplete_details?.reason ===
                    'max_output_tokens'
                    ? ('length' as const)
                    : chunk.response.incomplete_details?.reason ===
                        'content_filter'
                      ? ('content_filter' as const)
                      : undefined
                  : undefined
              if (cappedFinishReason) {
                yield {
                  type: EventType.RUN_FINISHED,
                  ...(responseId ? { responseId } : {}),
                  runId: aguiState.runId,
                  threadId: aguiState.threadId,
                  model: chunk.response.model || options.model,
                  timestamp: Date.now(),
                  ...(chunk.response.usage && {
                    usage: this.responsesTokenUsage(
                      chunk.response.usage,
                      chunk.response,
                    ),
                  }),
                  finishReason: cappedFinishReason,
                }
                runFinishedEmitted = true
                return
              }

    With only that applied, the two adapters produce identical output on the same cutoff:

    $ corepack pnpm vitest run packages/openai-base/tests/repro-1916.test.ts --silent=false --disable-console-intercept
    
    --- Responses adapter (response.incomplete) ---
    chunk types     : RUN_STARTED -> TEXT_MESSAGE_START -> TEXT_MESSAGE_CONTENT -> TEXT_MESSAGE_END -> RUN_FINISHED
    terminal chunk  : {"type":"RUN_FINISHED","usage":{"promptTokens":19,"completionTokens":16,"totalTokens":35}}
    onUsage calls   : 1 {"promptTokens":19,"completionTokens":16,"totalTokens":35}
    terminal hooks  : onFinish(finishReason="length")
    adapter terminal: {"type":"RUN_FINISHED","finishReason":"length","usage":{"promptTokens":19,"completionTokens":16,"totalTokens":35}}
    
    --- Chat Completions adapter (finish_reason: length) ---
    chunk types     : RUN_STARTED -> TEXT_MESSAGE_START -> TEXT_MESSAGE_CONTENT -> TEXT_MESSAGE_END -> RUN_FINISHED
    terminal chunk  : {"type":"RUN_FINISHED","usage":{"promptTokens":19,"completionTokens":16,"totalTokens":35}}
    onUsage calls   : 1 {"promptTokens":19,"completionTokens":16,"totalTokens":35}
    terminal hooks  : onFinish(finishReason="length")
    adapter terminal: {"type":"RUN_FINISHED","finishReason":"length","usage":{"promptTokens":19,"completionTokens":16,"totalTokens":35}}

    And the whole package suite:

    $ cd packages/openai-base && corepack pnpm vitest run     # with the patch
    
     ❯ |@tanstack/openai-base| tests/responses-text.test.ts (101 tests | 2 failed) 117ms
           × flushes completed web searches before RUN_ERROR on incomplete response
           × emits RUN_ERROR on response.incomplete event
    
     Test Files  1 failed | 33 passed (34)
          Tests  2 failed | 402 passed (404)

    402 pass. The 2 failures are the tests that currently pin the old behaviour, so a fix is a deliberate behaviour change with a known footprint:

    • error handling > emits RUN_ERROR on response.incomplete event (its fixture is exactly max_output_tokens)
    • error handling > flushes completed web searches before RUN_ERROR on incomplete response — its TOOL_CALL_START / TOOL_CALL_END flush still happens, because emitProviderWebSearchCalls runs earlier in the same branch; only the terminal chunk it asserts on changes.

    Two things the minimal patch above does not cover:

    1. packages/ai-openrouter/src/adapters/responses-text.ts overrides processStreamChunks and repeats this branch twice (at lines 630 and 1099), keyed on camelCase incompleteDetails. OpenRouter keeps the current RUN_ERROR behaviour unless it is mirrored there.
    2. The response.completed branch also drains function_call / user-tool / shell_call items and can resolve finishReason to 'tool_calls'; the incomplete branch does not. A cutoff that interrupts a tool call probably wants to route into that shared path rather than duplicate the RUN_FINISHED emission.

    response.failed, and response.incomplete with a missing or unrecognised incomplete_details.reason, stay RUN_ERROR under this patch, which is what you asked for.

    The repro file (packages/openai-base/tests/repro-1916.test.ts, no API key needed)
    // Scratch repro for TanStack/ai#1916 — no API key, canned streams only.
    import { describe, it } from 'vitest'
    import OpenAI from 'openai'
    import { chat } from '@tanstack/ai'
    import type { AnyChatMiddleware, UsageInfo } from '@tanstack/ai'
    import { resolveDebugOption } from '@tanstack/ai/adapter-internals'
    import { OpenAIBaseResponsesTextAdapter } from '../src/adapters/responses-text'
    import { OpenAIBaseChatCompletionsTextAdapter } from '../src/adapters/chat-completions-text'
    
    function asyncIterable<T>(chunks: Array<T>): AsyncIterable<T> {
      return {
        async *[Symbol.asyncIterator]() {
          for (const c of chunks) yield c
        },
      }
    }
    
    // ---- Responses adapter wired to a canned `response.incomplete` stream ----
    const RESPONSES_STREAM = [
      {
        type: 'response.created',
        response: { id: 'resp_repro', model: 'gpt-4.1-nano', status: 'in_progress' },
      },
      { type: 'response.output_text.delta', delta: '1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16' },
      {
        type: 'response.incomplete',
        response: {
          id: 'resp_repro',
          model: 'gpt-4.1-nano',
          status: 'incomplete',
          incomplete_details: { reason: 'max_output_tokens' },
          output: [],
          usage: {
            input_tokens: 19,
            input_tokens_details: { cached_tokens: 0 },
            output_tokens: 16,
            output_tokens_details: { reasoning_tokens: 0 },
            total_tokens: 35,
          },
        },
      },
    ]
    
    class ReproResponsesAdapter extends OpenAIBaseResponsesTextAdapter<string> {
      constructor() {
        super('gpt-4.1-nano', 'repro-responses', {
          responses: { create: () => Promise.resolve(asyncIterable(RESPONSES_STREAM)) },
        } as unknown as OpenAI)
      }
    }
    
    // ---- Chat Completions adapter wired to the equivalent `length` cutoff ----
    const CHAT_STREAM = [
      {
        id: 'chatcmpl_repro',
        model: 'gpt-4.1-nano',
        choices: [{ index: 0, delta: { content: '1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16' }, finish_reason: null }],
      },
      {
        id: 'chatcmpl_repro',
        model: 'gpt-4.1-nano',
        choices: [{ index: 0, delta: {}, finish_reason: 'length' }],
        usage: { prompt_tokens: 19, completion_tokens: 16, total_tokens: 35 },
      },
    ]
    
    class ReproChatAdapter extends OpenAIBaseChatCompletionsTextAdapter<string> {
      constructor() {
        const client = new OpenAI({ apiKey: 'test' })
        client.chat.completions.create = (() =>
          Promise.resolve(asyncIterable(CHAT_STREAM))) as typeof client.chat.completions.create
        super('gpt-4.1-nano', 'repro-chat', client)
      }
    }
    
    async function probe(adapter: ReproResponsesAdapter | ReproChatAdapter, label: string) {
      const usage: Array<UsageInfo> = []
      const hooks: Array<string> = []
      const middleware: AnyChatMiddleware = {
        name: 'probe',
        onUsage: (_ctx, u) => { usage.push(u) },
        onFinish: (_ctx, info) => { hooks.push(`onFinish(finishReason=${JSON.stringify(info.finishReason)})`) },
        onError: () => { hooks.push('onError') },
      }
      const chunks: Array<any> = []
      try {
        for await (const chunk of chat({
          adapter: adapter as any,
          threadId: 'repro',
          messages: [{ role: 'user', content: 'Count from 1 to 300, separated by commas.' }],
          middleware: [middleware],
        })) {
          chunks.push(chunk)
        }
      } catch (err) {
        hooks.push(`threw:${(err as Error).message}`)
      }
      const adapterChunks: Array<any> = []
      for await (const c of (adapter as any).chatStream({
        threadId: 'repro', model: 'gpt-4.1-nano', logger: resolveDebugOption(false),
        messages: [{ role: 'user', content: 'Count from 1 to 300, separated by commas.' }],
      })) adapterChunks.push(c)
      const adapterTerminal = adapterChunks.at(-1)
      const terminal = chunks.at(-1)
      console.log(`\n--- ${label} ---`)
      console.log('chunk types     :', chunks.map((c) => c.type).join(' -> '))
      console.log('terminal chunk  :', JSON.stringify(
        terminal && { type: terminal.type, message: terminal.message, code: terminal.code,
                      finishReason: terminal.finishReason, usage: terminal.usage }))
      console.log('onUsage calls   :', usage.length, usage.length ? JSON.stringify(usage[0]) : '')
      console.log('terminal hooks  :', hooks.join(',') || '(none)')
      console.log('adapter terminal:', JSON.stringify(
        adapterTerminal && { type: adapterTerminal.type, message: adapterTerminal.message,
                             code: adapterTerminal.code, finishReason: adapterTerminal.finishReason,
                             usage: adapterTerminal.usage }))
      return { chunks, terminal, usage, hooks, adapterTerminal }
    }
    
    describe('#1916 Responses max_output_tokens cutoff', () => {
      it('Responses adapter vs Chat Completions adapter on the same cutoff', async () => {
        await probe(new ReproResponsesAdapter(), 'Responses adapter (response.incomplete)')
        await probe(new ReproChatAdapter(), 'Chat Completions adapter (finish_reason: length)')
      })
    })

    The usage half of this is a failure mode we have written up separately: a terminal error path that drops the usage record, so per-model token totals quietly under-count work the provider already billed. Our note on it is item 5 of our catalogue (ours): https://github.com/aron-intframe/agent-guards

  2. github-actions[bot](GitHub profile, opens in a new tab) added labelswaiting-on: maintainer

Sign in to join this conversation.