Fill OpenRouter-shaped catalog facts from the bound schema, with per-field provenance #53
Send to my agentFollow-up to #50 (activity + schemaEndpointId). Every listed model on GET /v1/models should carry the same facts OpenRouter publishes on its Models API standard, filled from this service’s sources — the bound request schema first, then the provider’s own listing/docs — with per-field provenance so a consumer can see why a value is there.
#58 lands the listing/docs slice for chat/text hosts. It skipped the thing we uniquely have: the generation schema the model is already bound to.
OpenRouter is compare-only. Never copy an OpenRouter value into another provider’s catalog, labeled or not, and not even to fill a null. Use it only as a discrepancy signal. If no first-party source states a fact, the field stays null.
Target shape#
Mirror OpenRouter’s model object, in our camelCase catalog fields:
| OpenRouter | Us | Primary source |
|---|---|---|
context_length | contextWindow | listing / docs (schema almost never has this) |
top_provider.max_completion_tokens | maxOutput | listing / docs (max_tokens.maximum is a weak ceiling) |
architecture.input_modalities / output_modalities | modalities | bound schema content-parts or listing |
supported_parameters | capabilities (string[] for chat/text) | bound request schema properties, then listing flags |
default_parameters | not stored yet | JSON Schema default on those properties |
pricing (+ overrides) | pricing | listing / pricing page / extras. Never the schema |
tokenizer / instruct_type | not stored yet | docs, if we care later |
benchmarks | out of scope | third party |
Chat/text capabilities use OpenRouter’s parameter names: tools, tool_choice, max_tokens, temperature, top_p, reasoning, include_reasoning, structured_outputs, response_format, stop, frequency_penalty, presence_penalty, seed (and top_k / reasoning_effort when the native schema has them).
Speech / image / video / world-model hosts keep host-native capability objects where those flags do not apply. Schema walk still fills modalities and the request properties that are on that model’s route.
Fill order (per field, never overwrite a stronger rung)#
Strongest first. A weaker source only writes when the field (or flag) is still null.
| Rung | What it is | Self-heals? |
|---|---|---|
listing | A field on the provider’s own /models row or extras JSON | yes, every poll |
upstream-spec / generated | A property on the bound generation request/response schema (schemaEndpointId → current input schema). Inherits that schema version’s derivation | yes, every spec sync |
docs-derived | Deterministic parse of a provider docs page (markdown table / bullet). Same fail-closed rule as #58 | yes, every poll (memoised) |
docs-extracted | Extraction model over leftover prose, each value carrying the verbatim sentence | no |
generated specs that are OpenAI’s document with the host swapped (DeepSeek, Cerebras, DashScope, …) do not stamp OpenAI’s tools onto that host’s models. Only a spec the provider publishes (Groq Stainless, Anthropic, Gemini, xAI, BytePlus Ark, FAL per-model, …) counts as upstream-spec / honest generated.
Grain:
- Model-grained (FAL, Reactor, …): the schema is the model. Walking it is the whole
supported_parameters+ modalities answer. - Provider-grained, one family per route (Anthropic
v1/messages, Gemini:predictvs:generateContent): stamp the route’s properties onto every model bound to that route. This is how Anthropictools/tool_choice/temperatureget filled — they are on the messages schema, not the Models API row. - Provider-grained, mixed family on one route (OpenAI
chat/completions): schema is an upper bound. Activity +schemaEndpointIdalready keep whisper off chat flags. Per-model listing/docs overlay where a family on the same route does not honor a field (o-series vs gpt).
We store presence, not absence. If the schema has temperature and the model docs never mention it, the flag is present from upstream-spec. A later docs-extracted “this model ignores temperature” is a separate, explicit deny — do not invent denies.
Fine-grained provenance#
Not one source on the row. Each stored fact names its winning rung, and capabilities is per flag.
Reuse the schema provenance idea (sourceUrl, sourceHash, fetchedAt, derivation) with extra locators:
interface FactSource {
derivation:
| 'listing'
| 'upstream-spec'
| 'generated'
| 'docs-derived'
| 'docs-extracted'
sourceUrl?: string
sourceHash?: string
fetchedAt?: number
/** Bound generation route, when the winner is the schema. */
endpointId?: string
/** JSON pointer or field name in the source document. */
path?: string
/** Verbatim sentence; required for `docs-extracted`. */
quote?: string
}
interface ModelFactSources {
contextWindow?: FactSource
maxOutput?: FactSource
modalities?: FactSource
pricing?: FactSource
/** Per OpenRouter parameter name (or per host-native key). */
capabilities?: Record<string, FactSource>
}
Store as JSON on models (new column, e.g. fact_sources). The poller / schema-sync writes it next to the values. A flag that moved from docs-derived to upstream-spec is a real model.updated.
Expose#
GET /v1/models— values only (list stays small).?provenance=1includesfactSources.GET /v1/models/{provider}/{id}— always includesfactSources.- Same
provenanceobject shape asGET /v1/schemas/...so agents already know how to read a trust ladder. _links.schemaalready points at the bound input schema;factSources.capabilities.tools.endpointIdshould matchschemaEndpointIdwhen the winner is the schema.
OpenRouter: compare only#
Join native provider/rawId (and undated aliases) to OpenRouter’s id. The join produces one thing:
- Discrepancy. After first-party rungs, if OpenRouter has a different value (or has a flag we lack, or we have a flag they lack), record it. Do not change our value.
- On the detail payload:
discrepancies: [{ field, ours, openrouter, oursDerivation, openrouterId }]. - On the changes feed:
model.fact_mismatch(or a dedicated/v1/models/discrepancies) so it is watchable. - Typical: we have Anthropic
toolsfromv1/messagesand they agree; we lack Groktoolsfrom extras and they have them — that mismatch is the cue to walk xAI’s chat schema.
- On the detail payload:
No fill-nulls rung. A catalog that copied OpenRouter, even labeled, would cap accuracy at theirs and blur what the provider itself states. A catalog that shows the diff uses them as a checker. A null with an OpenRouter discrepancy is the honest answer when the provider publishes nothing.
What #58 already did#
Listing/docs fill for openai, anthropic, gemini, grok, groq, jina, fireworks, mistral, moonshot, sambanova, hyperbolic, novita, cohere. Together already has context_length + pricing. OpenRouter is the reference catalog. Gemini schemaEndpointId is the bind the schema walk needs.
Still empty on purpose until this path: DeepSeek / Cerebras / DashScope / Perplexity (bare rows and generated OpenAI specs — schema walk is invalid until they have their own document). Grok capabilities: null is a #58 listing gap; the chat schema should fill tools / temperature / … for language models.
Speech/media (elevenlabs, fal, replicate, reactor, bfl, kling, …): schema walk is the right next fill for request properties and modalities; pricing stays listing/docs.
Implementation slices (each a PR)#
fact_sourcescolumn + detail API (?provenance=1on the list). No new facts. Backfill listing-derived #58 values asderivation: listing(ordocs-derivedfor OpenAI pages / Grok context table).- Bound-schema walker — given
schemaEndpointId+ current input (and output) schema, emitsupported_parameters+ modalities. SkipgeneratedOpenAI-borrowed specs. Merge under the ladder (listing wins on conflict of presence from a per-model field; schema fills nulls). - OpenRouter compare — join, write
discrepancies, changes event. No writes to fact columns. - Pricing — listing/docs/
overrides; samefactSources.pricing. docs-extracted— leftover prose only (reasoning.mandatory, Gemini modalities, o-series temperature deny, bare-row host docs), quote required.
Fail closed stays: a docs parse of zero rows throws; a schema walk of a missing bind writes nothing (does not null listing facts).
Consumer#
TanStack/ai#1308. Ids and activity already come from modelschemas. Facts still come from our OpenRouter catalog until first-party rungs cover the provider being synced. Where modelschemas has a null, the consumer decides whether to fall back. modelschemas never makes that choice for it.
Out of scope#
- New adapters from
docs/providers-to-add.md. Day-one they still walk the bound schema and recordfactSources. - Any OpenRouter copy into another provider’s facts, labeled or unlabeled (decided on #75, 2026-09-26).
- Benchmarks / tokenizer unless a later issue asks.
tombeckenham added labelsenhancement
tombeckenham changed the title
tombeckenham changed the title
tombeckenham closed this
Sign in to join this conversation.
Slice 5 (pricing as an OpenRouter-shaped blob) is superseded. A blob you cannot evaluate is the wrong target — see Openstory #1605 / #1606 rate cards.
New issues (no parent; attach to a milestone when one exists):
@modelschemas/rate-cardpackagePOST /v1/estimate