Skip to content
Source
Sign inSign up
TanStack/schemas

docs/source-silent

README.md

Source-silent ledger

Facts a provider does not publish in any native source (listing, docs, or spec). pnpm run gap:report leaves these out of that provider's score and lists them under silent. We never fill them from another catalog.

One file per provider, <provider>.md, so two branches never edit the same file. One entry per list line: - <provider>: <fact> — <why, with the issue>. The fact is a gap-report key (contextWindow, maxOutput, modalities, outputModalities, priced, cacheRead, capabilities, reasoning, efforts, requestMap, endpoint, replayReasoningContent, originLink), or serverTools, which is evidence only: it is served as factSources.serverTools on a null row and never scored. A poll that read the model's own tool statement and found it says no already stores that evidence (#707), so a serverTools entry is only for a model with no such page. activity is scored but cannot be ledgered. Remove an entry when the provider starts publishing the fact.

Each activity has its own fact set (ACTIVITY_FACTS in src/lib/completeness.ts). Chat rows are scored on the chat keys above. Image, video and audio rows are scored on priced (any stored rate card that is not a models.dev price), endpoint, modalities (input) and outputModalities. A row with a null activity is scored on activity, which it always lacks: classify the row, do not ledger it. Rows of an activity with no fact set (embeddings, moderation, decision, world, search) are listed with facts: null and not scored. originLink is scored on every row: a row needs an upstream link when its raw id or an alias, also with dots read as hyphens, matches a row at another provider and no row links to it; a row with sameAs set has one. A provider-wide entry applies to every activity that scores that key.

For a fact missing only on particular models, use the exact native raw model id: - provider/maker/model-id: fact — why, https://provider.example/model, checked YYYY-MM-DD. The first slash separates the provider; the rest is the raw id, including any slashes, @, or colons. Backticks may surround the entire scope. No aliases, slug normalization, family prefixes, or wildcard matching are applied. A scoped entry never exempts sibling models. Do not use a provider-wide entry for a partially sourced fact.

replayReasoningContent is a separate gap-report key for the requestMap.replayReasoningContent leaf. It applies to reasoning rows whose endpoint id is a chat/completions route (reasoning_content is that wire). Anthropic messages, Gemini generateContent, OpenAI Responses, and Bedrock Converse are not scored for it. requestMap applies only when the endpoint id is chat/completions, completions, or responses. A provider with nothing scored (every row of an activity with no fact set, no row needing a link) scores n/a.

A sourced true or false fills replayReasoningContent; null remains unknown. A ledger entry adds read-time evidence at factSources.requestMapFields.replayReasoningContent, never a boolean or a request map. Populated facts and their provenance are preserved.

A "-" or blank cached-input cell is a stated "not offered". Store it as factSources.cacheRead with path: 'not-offered'. Never store a made-up price. The scorer counts that path as filled. A table with no cached-input column stays unknown.

A statement a provider makes about its whole API applies to every model on that route unless it names an exception. Behaviour that varies by model still needs a per-model source. Evidence records the API-wide URL.

A field missing from the provider's own assistant message schema, when that schema sets additionalProperties: false, is a sourced false for replayReasoningContent. A schema that does not close additional properties states nothing by omission.

A model's own exhaustive effort list, with no off value, and no other reasoning control, means mandatory: true. A list shared by many models states nothing about one model. A gateway can drop the off value, so a gateway list stays strict.

The report retains raw have/need totals and provider-wide silent keys. modelSilent counts applicable missing rows with exact-model exceptions; these counts are subtracted from the scoring denominator and displayed explicitly in the table. Entries for facts that have become populated do not remove those facts from the model-scoped score. Remove obsolete entries after sourcing them. Recorded URLs and checked dates are copied literally into API evidence; omitted metadata is not invented. Entries require verified native silence, not merely a failed fetch, a missing credential, or an upstream model maker's claim.