Skip to content
Source
Sign inSign up
TanStack/schemas

Add a gap report script for the facts ai-models needs #201

Send to my agent
Closedtombeckenham opened this issue 0 comments
tombeckenhamcommented

Problem

TanStack AI's @tanstack/ai-models wants to read modelschemas and stop reading models.dev. We have no way to see what is still missing. The check is done by hand today with curl and jq.

An agent loop will fill the gaps one provider at a time. It needs one command that says what is missing and when the work is done.

Ask

Add scripts/gap-report.ts and a gap:report script in package.json.

It reads GET /v1/models?pricing=1&limit=20000 (base URL from --base, default https://modelschemas.com) and prints JSON to stdout. --table prints a readable table.

For each provider, count the chat rows (activity === "chat") and, of those, how many have each fact:

KeyA row has it when
contextWindowcontextWindow is not null
maxOutputmaxOutput is not null
modalitiesmodalities.input is a non-empty array
pricedpricing.tables.rate.base has an input and an output rate
cacheReadthe base rate has a cache read rate
capabilitiescapabilities is an array
reasoningreasoning is not null, counted only over rows whose capabilities has reasoning or whose reasoning is set
effortsreasoning.efforts is non-empty, counted only over rows with reasoning.mode of effort or adaptive
requestMaprequestMap is not null
endpointschemaEndpointId is not null

Also report per provider:

  • rows, chat, and noActivity (rows with activity: null). A provider with rows and no chat rows is a gap, not a pass.
  • fromModelsDev: rows whose pricing.source.url is on models.dev. These do not count as filled.
  • score: filled facts over needed facts, 0 to 1.

Rate keys differ by provider (input_tokens and prompt, cache_read_tokens and input_cache_read). Handle both.

Source-silent ledger

Read docs/source-silent.md if it exists. Each line names a provider and a fact that the provider does not publish (for example Grok maxOutput, see #75). A fact in the ledger is left out of that provider's score and is listed under silent in the output. Create the file with a header and the entries that #75 already settled.

Exit code

--check exits 1 when a provider in --providers a,b,c is below --target (default 1). With no --providers, it checks every provider that has chat rows.

Acceptance

  • bun run gap:report --table prints one row per provider, worst score first.
  • bun run gap:report prints JSON that an agent can read: { generatedAt, providers: [{ provider, rows, chat, noActivity, fromModelsDev, facts: { <key>: { have, need } }, silent: [], score }] }.
  • A unit test runs the scoring on a small fixture: a complete row, a row with a models.dev price, a row with a ledger entry, and a provider with rows but no chat rows.
  • The script makes one HTTP request. It does not call the detail endpoint per model.

Out of scope

  • Comparing values with models.dev or any other catalog.
  • Filling any data.
  • A page or an API route for the report. #106 covers the homepage score.
  1. tombeckenham added labelsenhancement

  2. tombeckenham closed this

Sign in to join this conversation.