Skip to content
Source
Sign inSign up
TanStack/ai

fix(ai-anthropic, ai-gemini, ai-mistral, ai-ollama, ai-bedrock): report max-token truncation in structuredOutput #1548

Mergedkartsan03(GitHub profile, opens in a new tab)merged 4 commits intomain fromrefs/pull/1548/headmerged by AlemTuzlak View on GitHub

When a structured-output response stops at the output token limit, Anthropic, Bedrock, Gemini, Mistral, and Ollama show a JSON parse error, or return the partial result as valid data. This PR makes each adapter check its stop signal first and report a truncation error, as openai-base and ai-openrouter already do (#1426 / #1427).

Author: @kartsan03. A maintainer merged the latest main into the branch and made the changeset text shorter.

🎯 Changes

AdapterStop signalBefore this PR
ai-anthropicstop_reason: 'max_tokens'The forced structured_output tool call comes back with its partial input as the result.
ai-bedrockstopReason: 'max_tokens'structuredOutput(): same as Anthropic. structuredOutputStream(): parse-error.
ai-geminifinishReason: MAX_TOKENSJSON parse error.
ai-mistralfinishReason: 'length'JSON parse error.
ai-ollamadone_reason: 'length'JSON parse error.
  • Each adapter checks its stop signal before the parse step. The error text matches openai-base and names the option to raise (max_tokens, max_completion_tokens, maxOutputTokens, options.num_predict).
  • Bedrock structuredOutputStream() yields RUN_ERROR { code: 'max_tokens' }, like the openai-base stream.
  • Gemini structuredOutputStream() already reports this through the MAX_TOKENS RUN_ERROR in chatStream(), so only its non-stream path changes.
  • Docs: no change. Only the error for a response that already failed changes. #1427 also had no docs change.
  • Changeset: patch for the five packages.

✅ Checklist

  • I have followed the steps in the Contributing guide.
  • I have tested code changes locally with pnpm run test:pr, or these tests do not apply to this pull request. (Package-scoped checks only. See Testing.)
  • I fully understand the code in this pull request, including any code generated with AI assistance.
  • Docs: I updated docs/ for this change, or this change is not user-facing.
  • Changeset: I added a changeset (pnpm changeset), or this PR does not change a published package.

🚀 Release Impact

  • This change affects published code, and I have generated a changeset.
  • This change is docs/CI/dev-only (no release).

Root cause

Issue. A caller uses structured output with Anthropic, Bedrock, Gemini, Mistral, or Ollama, and the response stops at the output token limit. The caller gets a JSON parse error, or partial data that looks valid. Nothing says that the token limit was the cause.

Cause. Each structuredOutput() reads the content and does not look at the stop signal (stop_reason, stopReason, finishReason, done_reason) first. Bedrock structuredOutputStream() sets finishReason = 'length', but no code reads it before the parse step.

Fix. Each adapter checks its stop signal before the parse step. It throws, or for the Bedrock stream yields RUN_ERROR, with the text the response was cut off because the maximum token limit was reached.

Possible alternatives

  • One shared helper in @tanstack/ai. One function for all adapters. Each adapter has a different stop signal and option name, so a local check per adapter is simpler.
  • Check in the chat() engine. chat() does not get the stop signal from structuredOutput(), so this needs a change to the adapter contract. Too large for a bug fix.

Testing

Commands run

Agent-written repro tests (not in the PR), one per path. Each feeds a response that stops at the token limit with partial JSON. Each expects cut off because the maximum token limit was reached. Run with vitest run --maxWorkers=2, first with the 5 source files from main (630ec86e9), then with the PR source:

PathOn mainOn this PR
anthropic structuredOutputpromise resolved "{ data: { name: 'Fen' }, …(2) }" instead of rejectingpass
bedrock structuredOutputpromise resolved "{ data: { name: 'Fen' }, …(1) }" instead of rejectingpass
bedrock structuredOutputStreamRUN_ERROR Failed to parse structured output as JSON. Content: {"name":"Fenpass
gemini structuredOutputFailed to parse structured output as JSON. Content: {"name":"Fenpass
mistral structuredOutputFailed to parse structured output as JSON. Content: {"name":"Fenpass
ollama structuredOutputStructured output generation failed: Failed to parse structured output as JSON. Content: {"name":"Fenpass

The 6 new unit tests in this PR also fail on the main source and pass on the PR source.

Per package (ai-anthropic, ai-gemini, ai-mistral, ai-ollama, ai-bedrock): vitest run, test:types, test:oxlint, and build pass (184, 370, 66, 58, and 118 tests). testing/e2e test:types passes. oxfmt --check passes on all changed files. The E2E suite did not run locally. CI runs it.

Manual test

  1. On main, call adapter.structuredOutput() (for example createMistralText) against a mock response with finishReason: 'length' and content {"name":"Fen. You get Failed to parse structured output as JSON.
  2. On this branch, run the same call. You get mistral.structuredOutput: the response was cut off because the maximum token limit was reached (finish_reason=length); raise modelOptions.max_tokens.

How this PR makes testing easy

  • One unit test per adapter, plus one for the Bedrock stream, in each package tests/ folder.
  • E2E: testing/e2e/tests/structured-output-length-wire.spec.ts now also covers Anthropic, Mistral, and Ollama through chat({ outputSchema }). Bedrock Converse cannot be mocked in the E2E app yet. Gemini chat() uses structuredOutputStream(), which already reported truncation. Both stay covered by unit tests.

Risk / rollback

Low risk. Only responses that stop at the token limit change, and those already failed or returned partial data. To undo, revert this PR.

    1. Review in Change Stack →

      No actionable comments were generated in the recent review. 🎉

      ℹ️ Recent review info
      ⚙️ Run configuration
      • Configuration used: Repository: TanStack/ai/.coderabbit.yaml
      • Review profile: CHILL
      • Plan: Advanced
      • Run ID: 28e1d9f7-cb3f-4f84-bff3-81063564f24f
      📥 Commits

      Reviewing files that changed from the base of the PR and between adbd60cd61fede3f24de65df023fb90d6539646b and fc2d94e5ba0435259e51bbd1a2d9434303d97f98.

      📒 Files selected for processing (1)
      • packages/ai-anthropic/src/adapters/text.ts

      Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 1 remain after this review.


      📝 Walkthrough

      Walkthrough

      Five adapters now detect provider responses stopped at the output-token limit during structured-output generation. They report truncation errors instead of parsing incomplete output. Bedrock streaming emits a RUN_ERROR with code max_tokens.

      Changes

      Structured-output truncation

      Layer / File(s)Summary
      Adapter truncation checks and coverage
      packages/ai-anthropic/src/adapters/text.ts, packages/ai-anthropic/tests/anthropic-adapter.test.ts, packages/ai-bedrock/src/adapters/converse-text.ts, packages/ai-bedrock/tests/converse/adapter.test.ts, packages/ai-gemini/src/adapters/text.ts, packages/ai-gemini/tests/gemini-adapter.test.ts, packages/ai-mistral/src/adapters/text.ts, packages/ai-mistral/tests/mistral-adapter.test.ts, packages/ai-ollama/src/adapters/text.ts, packages/ai-ollama/tests/text-adapter.test.ts, .changeset/structured-output-truncation-other-adapters.md
      The adapters check provider stop reasons before parsing structured output. Bedrock streaming emits a RUN_ERROR with code max_tokens. Unit tests cover the responses, and a changeset adds patch release entries for the five adapters.
      End-to-end truncation coverage
      testing/e2e/src/routes/api.structured-output-length-wire.ts, testing/e2e/tests/structured-output-length-wire.spec.ts
      The route adds truncated-response cases for Anthropic, Mistral, and Ollama. Local HTTP fixtures serve Mistral and Ollama responses, and the test expects the three new error keys.

      Priority: ➖ Normal

      Estimated code review effort: 3 (Moderate) | ~20 minutes

      Change: Bug fix

      Suggested reviewers: alemtuzlak

      Merge Risk: ⚪ Minimal · up to fc2d9

      Structured-output truncation is reported explicitly across the changed adapters, including Bedrock streaming. The inspected changes reveal no specific user-impacting issue that should block merging.

      Security Architecture Review

      Security architecture risk: 🔵 Low · up to adbd6

      The change makes token-limited structured output fail explicitly instead of appearing malformed or, for some forced-tool responses, returning partial data. No new security exposure was established. The test application’s deployment exposure remains unverified.

      Retained concerns No architecture-level concerns identified.

      Security review details

      Security Blast Radius

      • inferred — The demonstrated effects are confined to callers receiving structured-output failures and to fixed test-route invocations. The added route cases do not show request-controlled provider calls or use of real provider credentials.

      Trust Boundaries and Controls

      • observed — Provider completion reasons determine whether non-stream output is returned or rejected; Bedrock’s stream carries the corresponding failure as a terminal error event after any raw fragments.

      Resilience and Maintainability Implications

      • inferred — Normal and rejected test callbacks have per-invocation server cleanup, but a listen failure or callback that never settles is not shown to reach it. The security significance depends on whether the test route is exposed outside its intended environment, which was not established.

      Hardening Proposals

      • proposed — Consumers that require validated structured data should withhold trust from streaming text fragments until successful completion and discard them when RUN_ERROR arrives.
      🚥 Pre-merge checks | ✅ 4 | ❌ 1

      ❌ Failed checks (1 warning)

      Check nameStatusExplanationResolution
      Docstring Coverage⚠️ WarningDocstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 12 files.Write docstrings for the functions missing them to satisfy the coverage threshold.
      ✅ Passed checks (4 passed)
      Check nameStatusExplanation
      Linked Issues check✅ PassedCheck skipped because no linked issues were found for this pull request.
      Out of Scope Changes check✅ PassedCheck skipped because no linked issues were found for this pull request.
      Title check✅ PassedThe title clearly identifies the affected adapters and the primary change: reporting maximum-token truncation in structured output. It is specific and relevant.
      Description check✅ PassedThe description follows the required template and explains the changes, root cause, testing, release impact, changeset, and rollback risk. It also states that the full test:pr suite was not run while …
      • Fix all pre-merge checks with AI
      ✨ Finishing Touches
      🧪 Generate unit tests (beta)
      • Create a new PR

      Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

      ❤️ Share
      Comment `@coderabbitai help` to get the list of available commands.
    2. Thanks for the PR, @kartsan03! 🙌 @jherr will take a look.

      Automated pre-review checks

      • ✅ CI passing
      • ✅ No merge conflicts
      • ✅ Changeset present
      • ⚠️ No E2E test changes detected — behavior changes need coverage under testing/e2e/ (see CONTRIBUTING)
      Automated triage — a human review follows.
    3. github-actions[bot](GitHub profile, opens in a new tab) added labelswaiting-on: maintainer

    4. Added E2E coverage in adbd60cd: structured-output-length-wire now also runs chat({ outputSchema }) through Anthropic (claude-opus-4-1, which takes the forced-tool structuredOutput() path), Mistral and Ollama against a response cut off at the output cap. Against main's adapters the three new cases fail: Anthropic returns the partial tool input as the result, and Mistral and Ollama throw a JSON parse error. Mistral and Ollama take no custom fetch, so they point at a local server in the route.

      Bedrock's change is on the Converse path, which the E2E README lists as not mockable in aimock yet, so it stays covered by the unit tests. Gemini's chat() path goes through structuredOutputStream(), which already reported this; the change is in its non-stream structuredOutput(), covered by the unit test.

    5. 🦋 Changeset detected

      Latest commit: eddc70c60dac158a61e22ac80ce6ae0e51e16739

      The changes in this PR will be included in the next version bump.

      This PR includes changesets to release 5 packages
      NameType
      @tanstack/ai-anthropicPatch
      @tanstack/ai-geminiPatch
      @tanstack/ai-mistralPatch
      @tanstack/ai-ollamaPatch
      @tanstack/ai-bedrockPatch

      Not sure what this means? Click here to learn what changesets are.

      Click here if you're a maintainer who wants to add another changeset to this PR

    6. Maintainer review: findings

      Thank you for this fix, @kartsan03. I reviewed the PR and ran my own checks. The fix is correct. I will push one small change and then the PR is ready for a maintainer decision.

      Security

      Clean. The PR changes no package.json, lockfile, workflow, or script. The only network code is a test-only local node:http server on 127.0.0.1 in the E2E route.

      Main sync

      I merged the latest main (630ec86e9) into the branch. There were no conflicts. I pushed the merge (fc2d94e5b).

      Root cause

      • Issue. When a structured-output response stops at the output token limit, Anthropic, Bedrock, Gemini, Mistral, and Ollama do not report truncation. Gemini, Mistral, Ollama, and the Bedrock stream show a JSON parse error. Anthropic and Bedrock structuredOutput() return the partial tool input as a valid result.
      • Cause. Each structuredOutput() reads the content and does not look at the stop signal first (stop_reason, stopReason, finishReason, done_reason). Bedrock structuredOutputStream() sets finishReason = 'length', but no code reads it before the parse step.
      • Fix. Each adapter now checks its stop signal before the parse step and throws (or, for the Bedrock stream, yields RUN_ERROR with code: 'max_tokens'). The message text is the same as in openai-base and ai-openrouter (#1427): <name>.structuredOutput: the response was cut off because the maximum token limit was reached (...). Each message names the real option to raise. I compared each name with the adapter code.

      Repro (my own tests, not the PR tests)

      I wrote one small test per path. Each test gives a response that stops at the token limit with partial JSON. Each test expects the text cut off because the maximum token limit was reached. I ran the tests with the 5 source files from main, then with the PR source.

      AdapterOn mainOn this PR
      anthropic structuredOutputfail: promise resolved "{ data: { name: 'Fen' }, …(2) }" instead of rejectingpass
      bedrock structuredOutputfail: promise resolved "{ data: { name: 'Fen' }, …(1) }" instead of rejectingpass
      bedrock structuredOutputStreamfail: RUN_ERROR Failed to parse structured output as JSON. Content: {"name":"Fenpass
      gemini structuredOutputfail: Failed to parse structured output as JSON. Content: {"name":"Fenpass
      mistral structuredOutputfail: Failed to parse structured output as JSON. Content: {"name":"Fenpass
      ollama structuredOutputfail: Structured output generation failed: Failed to parse structured output as JSON. Content: {"name":"Fenpass

      The 6 new unit tests in the PR also fail on the main source and pass on the PR source.

      Keep check

      Each source change is the smallest check that stops the bug. I found no extra hunks. The E2E route uses a local server for Mistral and Ollama because those adapters take no custom fetch. This is necessary.

      Possible alternative: one shared helper for the check in all adapters. Each adapter has a different stop signal and option name, so one local check per adapter is simpler.

      CodeRabbit and reviewers

      • CodeRabbit: no actionable comments. The docstring coverage warning is a style item (keep-fail, not applied). The note about server cleanup in the test route is test-only and has no repro (keep-fail, not applied).
      • No human review comments.
      • The triage bot asked for E2E coverage. Done in adbd60cd.

      Checks I ran (per package, one at a time)

      vitest run, test:types, test:oxlint, and build pass for ai-anthropic (184 tests), ai-gemini (370), ai-mistral (66), ai-ollama (58), and ai-bedrock (118). testing/e2e test:types passes. oxfmt --check passes on all changed files. No existing test broke. I did not run the E2E suite locally. CI runs it.

      What I will change

      • Rewrite the changeset text into shorter sentences. No other change is necessary.

      Possible follow-up (not in this PR): the experimental Gemini text-interactions adapter has its own structuredOutput() that does not check for truncation.

    7. Maintainer update: pushed

      • fc2d94e5b: merged the latest main (630ec86e9). No conflicts.
      • eddc70c60: rewrote the changeset text into shorter sentences. No code change.

      All package checks pass (vitest, test:types, test:oxlint, build for the 5 adapters, test:types for testing/e2e, oxfmt --check). I also updated the PR description with Root cause, Possible alternatives, and the repro results. The PR now waits for a maintainer decision.

    8. AlemTuzlak approved these changes

    9. View your CI Pipeline Execution ↗ for commit eddc70c60dac158a61e22ac80ce6ae0e51e16739

      CommandStatusDurationResult
      nx run-many --targets=build --exclude=examples/...✅ Succeeded16sView ↗

      ☁️ Nx Cloud last updated this comment at 2026-10-07 16:51:02 UTC

    10. AlemTuzlak merged this68aeada

    11. AlemTuzlak closed this

Sign in to join this conversation.