Skip to content
Source
Sign inSign up
TanStack/schemas

Fill OpenRouter-shaped catalog facts from the bound schema, with per-field provenance #53

Send to my agent
Closedtombeckenham opened this issue 1 comment
tombeckenhamcommented

Follow-up to #50 (activity + schemaEndpointId). Every listed model on GET /v1/models should carry the same facts OpenRouter publishes on its Models API standard, filled from this service’s sources — the bound request schema first, then the provider’s own listing/docs — with per-field provenance so a consumer can see why a value is there.

#58 lands the listing/docs slice for chat/text hosts. It skipped the thing we uniquely have: the generation schema the model is already bound to.

OpenRouter is compare-only. Never copy an OpenRouter value into another provider’s catalog, labeled or not, and not even to fill a null. Use it only as a discrepancy signal. If no first-party source states a fact, the field stays null.

Target shape

Mirror OpenRouter’s model object, in our camelCase catalog fields:

OpenRouterUsPrimary source
context_lengthcontextWindowlisting / docs (schema almost never has this)
top_provider.max_completion_tokensmaxOutputlisting / docs (max_tokens.maximum is a weak ceiling)
architecture.input_modalities / output_modalitiesmodalitiesbound schema content-parts or listing
supported_parameterscapabilities (string[] for chat/text)bound request schema properties, then listing flags
default_parametersnot stored yetJSON Schema default on those properties
pricing (+ overrides)pricinglisting / pricing page / extras. Never the schema
tokenizer / instruct_typenot stored yetdocs, if we care later
benchmarksout of scopethird party

Chat/text capabilities use OpenRouter’s parameter names: tools, tool_choice, max_tokens, temperature, top_p, reasoning, include_reasoning, structured_outputs, response_format, stop, frequency_penalty, presence_penalty, seed (and top_k / reasoning_effort when the native schema has them).

Speech / image / video / world-model hosts keep host-native capability objects where those flags do not apply. Schema walk still fills modalities and the request properties that are on that model’s route.

Fill order (per field, never overwrite a stronger rung)

Strongest first. A weaker source only writes when the field (or flag) is still null.

RungWhat it isSelf-heals?
listingA field on the provider’s own /models row or extras JSONyes, every poll
upstream-spec / generatedA property on the bound generation request/response schema (schemaEndpointId → current input schema). Inherits that schema version’s derivationyes, every spec sync
docs-derivedDeterministic parse of a provider docs page (markdown table / bullet). Same fail-closed rule as #58yes, every poll (memoised)
docs-extractedExtraction model over leftover prose, each value carrying the verbatim sentenceno

generated specs that are OpenAI’s document with the host swapped (DeepSeek, Cerebras, DashScope, …) do not stamp OpenAI’s tools onto that host’s models. Only a spec the provider publishes (Groq Stainless, Anthropic, Gemini, xAI, BytePlus Ark, FAL per-model, …) counts as upstream-spec / honest generated.

Grain:

  • Model-grained (FAL, Reactor, …): the schema is the model. Walking it is the whole supported_parameters + modalities answer.
  • Provider-grained, one family per route (Anthropic v1/messages, Gemini :predict vs :generateContent): stamp the route’s properties onto every model bound to that route. This is how Anthropic tools / tool_choice / temperature get filled — they are on the messages schema, not the Models API row.
  • Provider-grained, mixed family on one route (OpenAI chat/completions): schema is an upper bound. Activity + schemaEndpointId already keep whisper off chat flags. Per-model listing/docs overlay where a family on the same route does not honor a field (o-series vs gpt).

We store presence, not absence. If the schema has temperature and the model docs never mention it, the flag is present from upstream-spec. A later docs-extracted “this model ignores temperature” is a separate, explicit deny — do not invent denies.

Fine-grained provenance

Not one source on the row. Each stored fact names its winning rung, and capabilities is per flag.

Reuse the schema provenance idea (sourceUrl, sourceHash, fetchedAt, derivation) with extra locators:

interface FactSource {
  derivation:
    | 'listing'
    | 'upstream-spec'
    | 'generated'
    | 'docs-derived'
    | 'docs-extracted'
  sourceUrl?: string
  sourceHash?: string
  fetchedAt?: number
  /** Bound generation route, when the winner is the schema. */
  endpointId?: string
  /** JSON pointer or field name in the source document. */
  path?: string
  /** Verbatim sentence; required for `docs-extracted`. */
  quote?: string
}

interface ModelFactSources {
  contextWindow?: FactSource
  maxOutput?: FactSource
  modalities?: FactSource
  pricing?: FactSource
  /** Per OpenRouter parameter name (or per host-native key). */
  capabilities?: Record<string, FactSource>
}

Store as JSON on models (new column, e.g. fact_sources). The poller / schema-sync writes it next to the values. A flag that moved from docs-derived to upstream-spec is a real model.updated.

Expose

  • GET /v1/models — values only (list stays small). ?provenance=1 includes factSources.
  • GET /v1/models/{provider}/{id} — always includes factSources.
  • Same provenance object shape as GET /v1/schemas/... so agents already know how to read a trust ladder.
  • _links.schema already points at the bound input schema; factSources.capabilities.tools.endpointId should match schemaEndpointId when the winner is the schema.

OpenRouter: compare only

Join native provider/rawId (and undated aliases) to OpenRouter’s id. The join produces one thing:

  1. Discrepancy. After first-party rungs, if OpenRouter has a different value (or has a flag we lack, or we have a flag they lack), record it. Do not change our value.
    • On the detail payload: discrepancies: [{ field, ours, openrouter, oursDerivation, openrouterId }].
    • On the changes feed: model.fact_mismatch (or a dedicated /v1/models/discrepancies) so it is watchable.
    • Typical: we have Anthropic tools from v1/messages and they agree; we lack Grok tools from extras and they have them — that mismatch is the cue to walk xAI’s chat schema.

No fill-nulls rung. A catalog that copied OpenRouter, even labeled, would cap accuracy at theirs and blur what the provider itself states. A catalog that shows the diff uses them as a checker. A null with an OpenRouter discrepancy is the honest answer when the provider publishes nothing.

What #58 already did

Listing/docs fill for openai, anthropic, gemini, grok, groq, jina, fireworks, mistral, moonshot, sambanova, hyperbolic, novita, cohere. Together already has context_length + pricing. OpenRouter is the reference catalog. Gemini schemaEndpointId is the bind the schema walk needs.

Still empty on purpose until this path: DeepSeek / Cerebras / DashScope / Perplexity (bare rows and generated OpenAI specs — schema walk is invalid until they have their own document). Grok capabilities: null is a #58 listing gap; the chat schema should fill tools / temperature / … for language models.

Speech/media (elevenlabs, fal, replicate, reactor, bfl, kling, …): schema walk is the right next fill for request properties and modalities; pricing stays listing/docs.

Implementation slices (each a PR)

  1. fact_sources column + detail API (?provenance=1 on the list). No new facts. Backfill listing-derived #58 values as derivation: listing (or docs-derived for OpenAI pages / Grok context table).
  2. Bound-schema walker — given schemaEndpointId + current input (and output) schema, emit supported_parameters + modalities. Skip generated OpenAI-borrowed specs. Merge under the ladder (listing wins on conflict of presence from a per-model field; schema fills nulls).
  3. OpenRouter compare — join, write discrepancies, changes event. No writes to fact columns.
  4. Pricing — listing/docs/overrides; same factSources.pricing.
  5. docs-extracted — leftover prose only (reasoning.mandatory, Gemini modalities, o-series temperature deny, bare-row host docs), quote required.

Fail closed stays: a docs parse of zero rows throws; a schema walk of a missing bind writes nothing (does not null listing facts).

Consumer

TanStack/ai#1308. Ids and activity already come from modelschemas. Facts still come from our OpenRouter catalog until first-party rungs cover the provider being synced. Where modelschemas has a null, the consumer decides whether to fall back. modelschemas never makes that choice for it.

Out of scope

  • New adapters from docs/providers-to-add.md. Day-one they still walk the bound schema and record factSources.
  • Any OpenRouter copy into another provider’s facts, labeled or unlabeled (decided on #75, 2026-09-26).
  • Benchmarks / tokenizer unless a later issue asks.
  1. tombeckenham added labelsenhancement

  2. tombeckenham changed the title

  3. tombeckenham changed the title

  4. tombeckenham closed this

  5. Slice 5 (pricing as an OpenRouter-shaped blob) is superseded. A blob you cannot evaluate is the wrong target — see Openstory #1605 / #1606 rate cards.

    New issues (no parent; attach to a milestone when one exists):

Sign in to join this conversation.