Skip to content
Source
Sign inSign up
TanStack/ai

chat({ outputSchema }) runs final structured output request outside middleware #390

Send to my agent
Closedimsherrill(GitHub profile, opens in a new tab) opened this issue 1 comment

TanStack AI version

0.8.0

Framework/Library version

TanStack Start 1.149.3, @tanstack/react-router 1.149.3, @tanstack/ai-gemini ^0.8.2

Describe the bug and the steps to reproduce it

When chat() is used with outputSchema, the final provider structured-output request appears to execute outside the normal middleware path.

In runAgenticStructuredOutput(), middleware is applied to the TextEngine.run() phase, but then the actual provider request is made separately via adapter.structuredOutput(...).

That means middleware-based behaviors can miss the real structured-output provider call, even though they work for normal chat flows.

Relevant core path:

  const engine = new TextEngine({ adapter, params, middleware, context });
  for await (const _chunk of engine.run()) {}

  const result = await adapter.structuredOutput({
    chatOptions: { ...structuredTextOptions, model, messages: finalMessages },
    outputSchema: jsonSchema,
  });
}

This seems to affect multiple adapters, not just one provider:

Gemini structuredOutput() -> provider call inside adapter OpenAI structuredOutput() -> provider call inside adapter Anthropic structuredOutput() -> provider call inside adapter Expected behavior:

middleware should wrap the actual final structured-output provider request too, or there should be a dedicated structured-output middleware phase/hook, or docs should explicitly call out that outputSchema final provider calls are outside middleware coverage Use case on our side was observability/tracing middleware, but I think the gap would also matter for any middleware that expects to see the real provider request.

https://gist.github.com/imsherrill/af39137a58fbfc12c26f8b894e207506

Screenshots or Videos (Optional)

No response

Do you intend to try to help solve this bug with your own PR?

Yes, I think I know how to fix it and will discuss it in the comments of this issue

Terms & Code of Conduct

  • I agree to follow this project's Code of Conduct
  • I understand that if my bug cannot be reliable reproduced in a debuggable environment, it will probably not be fixed and this issue may even be closed.
  1. it feels to me that nothing, especially not llm calls, should happen outside of the middleware.

  2. AlemTuzlak closed this

Sign in to join this conversation.