fix(ai-anthropic, ai-gemini, ai-mistral, ai-ollama, ai-bedrock): report max-token truncation in structuredOutput #1548
main fromrefs/pull/1548/head·merged by AlemTuzlak View on GitHubWhen a structured-output response stops at the output token limit, Anthropic, Bedrock, Gemini, Mistral, and Ollama show a JSON parse error, or return the partial result as valid data. This PR makes each adapter check its stop signal first and report a truncation error, as openai-base and ai-openrouter already do (#1426 / #1427).
Author: @kartsan03. A maintainer merged the latest main into the branch and made the changeset text shorter.
🎯 Changes#
| Adapter | Stop signal | Before this PR |
|---|---|---|
ai-anthropic | stop_reason: 'max_tokens' | The forced structured_output tool call comes back with its partial input as the result. |
ai-bedrock | stopReason: 'max_tokens' | structuredOutput(): same as Anthropic. structuredOutputStream(): parse-error. |
ai-gemini | finishReason: MAX_TOKENS | JSON parse error. |
ai-mistral | finishReason: 'length' | JSON parse error. |
ai-ollama | done_reason: 'length' | JSON parse error. |
- Each adapter checks its stop signal before the parse step. The error text matches
openai-baseand names the option to raise (max_tokens,max_completion_tokens,maxOutputTokens,options.num_predict). - Bedrock
structuredOutputStream()yieldsRUN_ERROR { code: 'max_tokens' }, like theopenai-basestream. - Gemini
structuredOutputStream()already reports this through theMAX_TOKENSRUN_ERRORinchatStream(), so only its non-stream path changes. - Docs: no change. Only the error for a response that already failed changes. #1427 also had no docs change.
- Changeset: patch for the five packages.
✅ Checklist#
- I have followed the steps in the Contributing guide.
- I have tested code changes locally with
pnpm run test:pr, or these tests do not apply to this pull request. (Package-scoped checks only. See Testing.) - I fully understand the code in this pull request, including any code generated with AI assistance.
- Docs: I updated
docs/for this change, or this change is not user-facing. - Changeset: I added a changeset (
pnpm changeset), or this PR does not change a published package.
🚀 Release Impact#
- This change affects published code, and I have generated a changeset.
- This change is docs/CI/dev-only (no release).
Root cause#
Issue. A caller uses structured output with Anthropic, Bedrock, Gemini, Mistral, or Ollama, and the response stops at the output token limit. The caller gets a JSON parse error, or partial data that looks valid. Nothing says that the token limit was the cause.
Cause. Each structuredOutput() reads the content and does not look at the stop signal (stop_reason, stopReason, finishReason, done_reason) first. Bedrock structuredOutputStream() sets finishReason = 'length', but no code reads it before the parse step.
Fix. Each adapter checks its stop signal before the parse step. It throws, or for the Bedrock stream yields RUN_ERROR, with the text the response was cut off because the maximum token limit was reached.
Possible alternatives#
- One shared helper in
@tanstack/ai. One function for all adapters. Each adapter has a different stop signal and option name, so a local check per adapter is simpler. - Check in the
chat()engine.chat()does not get the stop signal fromstructuredOutput(), so this needs a change to the adapter contract. Too large for a bug fix.
Testing#
Commands run
Agent-written repro tests (not in the PR), one per path. Each feeds a response that stops at the token limit with partial JSON. Each expects cut off because the maximum token limit was reached. Run with vitest run --maxWorkers=2, first with the 5 source files from main (630ec86e9), then with the PR source:
| Path | On main | On this PR |
|---|---|---|
anthropic structuredOutput | promise resolved "{ data: { name: 'Fen' }, …(2) }" instead of rejecting | pass |
bedrock structuredOutput | promise resolved "{ data: { name: 'Fen' }, …(1) }" instead of rejecting | pass |
bedrock structuredOutputStream | RUN_ERROR Failed to parse structured output as JSON. Content: {"name":"Fen | pass |
gemini structuredOutput | Failed to parse structured output as JSON. Content: {"name":"Fen | pass |
mistral structuredOutput | Failed to parse structured output as JSON. Content: {"name":"Fen | pass |
ollama structuredOutput | Structured output generation failed: Failed to parse structured output as JSON. Content: {"name":"Fen | pass |
The 6 new unit tests in this PR also fail on the main source and pass on the PR source.
Per package (ai-anthropic, ai-gemini, ai-mistral, ai-ollama, ai-bedrock): vitest run, test:types, test:oxlint, and build pass (184, 370, 66, 58, and 118 tests). testing/e2e test:types passes. oxfmt --check passes on all changed files. The E2E suite did not run locally. CI runs it.
Manual test
- On
main, calladapter.structuredOutput()(for examplecreateMistralText) against a mock response withfinishReason: 'length'and content{"name":"Fen. You getFailed to parse structured output as JSON. - On this branch, run the same call. You get
mistral.structuredOutput: the response was cut off because the maximum token limit was reached (finish_reason=length); raise modelOptions.max_tokens.
How this PR makes testing easy
- One unit test per adapter, plus one for the Bedrock stream, in each package
tests/folder. - E2E:
testing/e2e/tests/structured-output-length-wire.spec.tsnow also covers Anthropic, Mistral, and Ollama throughchat({ outputSchema }). Bedrock Converse cannot be mocked in the E2E app yet. Geminichat()usesstructuredOutputStream(), which already reported truncation. Both stay covered by unit tests.
Risk / rollback#
Low risk. Only responses that stop at the token limit change, and those already failed or returned partial data. To undo, revert this PR.
- 1
github-actions[bot](GitHub profile, opens in a new tab) assignedjherr
Thanks for the PR, @kartsan03! 🙌 @jherr will take a look.
Automated pre-review checks
- ✅ CI passing
- ✅ No merge conflicts
- ✅ Changeset present
- ⚠️ No E2E test changes detected — behavior changes need coverage under
testing/e2e/(see CONTRIBUTING)
github-actions[bot](GitHub profile, opens in a new tab) added labelswaiting-on: maintainer
Added E2E coverage in adbd60cd:
structured-output-length-wirenow also runschat({ outputSchema })through Anthropic (claude-opus-4-1, which takes the forced-toolstructuredOutput()path), Mistral and Ollama against a response cut off at the output cap. Againstmain's adapters the three new cases fail: Anthropic returns the partial tool input as the result, and Mistral and Ollama throw a JSON parse error. Mistral and Ollama take no customfetch, so they point at a local server in the route.Bedrock's change is on the Converse path, which the E2E README lists as not mockable in aimock yet, so it stays covered by the unit tests. Gemini's
chat()path goes throughstructuredOutputStream(), which already reported this; the change is in its non-streamstructuredOutput(), covered by the unit test.🦋 Changeset detected#
Latest commit: eddc70c60dac158a61e22ac80ce6ae0e51e16739
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 5 packages
Name Type @tanstack/ai-anthropic Patch @tanstack/ai-gemini Patch @tanstack/ai-mistral Patch @tanstack/ai-ollama Patch @tanstack/ai-bedrock Patch Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
Maintainer review: findings#
Thank you for this fix, @kartsan03. I reviewed the PR and ran my own checks. The fix is correct. I will push one small change and then the PR is ready for a maintainer decision.
Security#
Clean. The PR changes no
package.json, lockfile, workflow, or script. The only network code is a test-only localnode:httpserver on127.0.0.1in the E2E route.Main sync#
I merged the latest
main(630ec86e9) into the branch. There were no conflicts. I pushed the merge (fc2d94e5b).Root cause#
- Issue. When a structured-output response stops at the output token limit, Anthropic, Bedrock, Gemini, Mistral, and Ollama do not report truncation. Gemini, Mistral, Ollama, and the Bedrock stream show a JSON parse error. Anthropic and Bedrock
structuredOutput()return the partial tool input as a valid result. - Cause. Each
structuredOutput()reads the content and does not look at the stop signal first (stop_reason,stopReason,finishReason,done_reason). BedrockstructuredOutputStream()setsfinishReason = 'length', but no code reads it before the parse step. - Fix. Each adapter now checks its stop signal before the parse step and throws (or, for the Bedrock stream, yields
RUN_ERRORwithcode: 'max_tokens'). The message text is the same as inopenai-baseandai-openrouter(#1427):<name>.structuredOutput: the response was cut off because the maximum token limit was reached (...). Each message names the real option to raise. I compared each name with the adapter code.
Repro (my own tests, not the PR tests)#
I wrote one small test per path. Each test gives a response that stops at the token limit with partial JSON. Each test expects the text
cut off because the maximum token limit was reached. I ran the tests with the 5 source files frommain, then with the PR source.Adapter On mainOn this PR anthropic structuredOutputfail: promise resolved "{ data: { name: 'Fen' }, …(2) }" instead of rejectingpass bedrock structuredOutputfail: promise resolved "{ data: { name: 'Fen' }, …(1) }" instead of rejectingpass bedrock structuredOutputStreamfail: RUN_ERROR Failed to parse structured output as JSON. Content: {"name":"Fenpass gemini structuredOutputfail: Failed to parse structured output as JSON. Content: {"name":"Fenpass mistral structuredOutputfail: Failed to parse structured output as JSON. Content: {"name":"Fenpass ollama structuredOutputfail: Structured output generation failed: Failed to parse structured output as JSON. Content: {"name":"Fenpass The 6 new unit tests in the PR also fail on the
mainsource and pass on the PR source.Keep check#
Each source change is the smallest check that stops the bug. I found no extra hunks. The E2E route uses a local server for Mistral and Ollama because those adapters take no custom
fetch. This is necessary.Possible alternative: one shared helper for the check in all adapters. Each adapter has a different stop signal and option name, so one local check per adapter is simpler.
CodeRabbit and reviewers#
- CodeRabbit: no actionable comments. The docstring coverage warning is a style item (keep-fail, not applied). The note about server cleanup in the test route is test-only and has no repro (keep-fail, not applied).
- No human review comments.
- The triage bot asked for E2E coverage. Done in
adbd60cd.
Checks I ran (per package, one at a time)#
vitest run,test:types,test:oxlint, andbuildpass forai-anthropic(184 tests),ai-gemini(370),ai-mistral(66),ai-ollama(58), andai-bedrock(118).testing/e2etest:typespasses.oxfmt --checkpasses on all changed files. No existing test broke. I did not run the E2E suite locally. CI runs it.What I will change#
- Rewrite the changeset text into shorter sentences. No other change is necessary.
Possible follow-up (not in this PR): the experimental Gemini text-interactions adapter has its own
structuredOutput()that does not check for truncation.- Issue. When a structured-output response stops at the output token limit, Anthropic, Bedrock, Gemini, Mistral, and Ollama do not report truncation. Gemini, Mistral, Ollama, and the Bedrock stream show a JSON parse error. Anthropic and Bedrock
Maintainer update: pushed#
fc2d94e5b: merged the latestmain(630ec86e9). No conflicts.eddc70c60: rewrote the changeset text into shorter sentences. No code change.
All package checks pass (
vitest,test:types,test:oxlint,buildfor the 5 adapters,test:typesfortesting/e2e,oxfmt --check). I also updated the PR description with Root cause, Possible alternatives, and the repro results. The PR now waits for a maintainer decision.AlemTuzlak approved these changes
View your CI Pipeline Execution ↗ for commit eddc70c60dac158a61e22ac80ce6ae0e51e16739
Command Status Duration Result nx run-many --targets=build --exclude=examples/...✅ Succeeded 16s View ↗
☁️ Nx Cloud last updated this comment at
2026-10-07 16:51:02UTCAlemTuzlak merged this68aeada
AlemTuzlak closed this
Sign in to join this conversation.
No actionable comments were generated in the recent review. 🎉
ℹ️ Recent review info
⚙️ Run configuration
28e1d9f7-cb3f-4f84-bff3-81063564f24f📥 Commits
Reviewing files that changed from the base of the PR and between adbd60cd61fede3f24de65df023fb90d6539646b and fc2d94e5ba0435259e51bbd1a2d9434303d97f98.
📒 Files selected for processing (1)
packages/ai-anthropic/src/adapters/text.tsIncluded review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 1 remain after this review.
📝 Walkthrough
Walkthrough#
Five adapters now detect provider responses stopped at the output-token limit during structured-output generation. They report truncation errors instead of parsing incomplete output. Bedrock streaming emits a
RUN_ERRORwith codemax_tokens.Changes#
Structured-output truncation
packages/ai-anthropic/src/adapters/text.ts,packages/ai-anthropic/tests/anthropic-adapter.test.ts,packages/ai-bedrock/src/adapters/converse-text.ts,packages/ai-bedrock/tests/converse/adapter.test.ts,packages/ai-gemini/src/adapters/text.ts,packages/ai-gemini/tests/gemini-adapter.test.ts,packages/ai-mistral/src/adapters/text.ts,packages/ai-mistral/tests/mistral-adapter.test.ts,packages/ai-ollama/src/adapters/text.ts,packages/ai-ollama/tests/text-adapter.test.ts,.changeset/structured-output-truncation-other-adapters.mdRUN_ERRORwith codemax_tokens. Unit tests cover the responses, and a changeset adds patch release entries for the five adapters.testing/e2e/src/routes/api.structured-output-length-wire.ts,testing/e2e/tests/structured-output-length-wire.spec.tsPriority: ➖ Normal
Estimated code review effort: 3 (Moderate) | ~20 minutes
Change: Bug fix
Suggested reviewers:
alemtuzlakMerge Risk: ⚪ Minimal · up to
fc2d9Structured-output truncation is reported explicitly across the changed adapters, including Bedrock streaming. The inspected changes reveal no specific user-impacting issue that should block merging.
Security Architecture Review#
Security architecture risk: 🔵 Low · up to
adbd6The change makes token-limited structured output fail explicitly instead of appearing malformed or, for some forced-tool responses, returning partial data. No new security exposure was established. The test application’s deployment exposure remains unverified.
Retained concerns No architecture-level concerns identified.
Security review details
Security Blast Radius
Trust Boundaries and Controls
Resilience and Maintainability Implications
Hardening Proposals
🚥 Pre-merge checks | ✅ 4 | ❌ 1
❌ Failed checks (1 warning)#
✅ Passed checks (4 passed)
✨ Finishing Touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
❤️ Share