Add a gap report script for the facts ai-models needs #201
Send to my agentProblem#
TanStack AI's @tanstack/ai-models wants to read modelschemas and stop reading models.dev. We have no way to see what is still missing. The check is done by hand today with curl and jq.
An agent loop will fill the gaps one provider at a time. It needs one command that says what is missing and when the work is done.
Ask#
Add scripts/gap-report.ts and a gap:report script in package.json.
It reads GET /v1/models?pricing=1&limit=20000 (base URL from --base, default https://modelschemas.com) and prints JSON to stdout. --table prints a readable table.
For each provider, count the chat rows (activity === "chat") and, of those, how many have each fact:
| Key | A row has it when |
|---|---|
contextWindow | contextWindow is not null |
maxOutput | maxOutput is not null |
modalities | modalities.input is a non-empty array |
priced | pricing.tables.rate.base has an input and an output rate |
cacheRead | the base rate has a cache read rate |
capabilities | capabilities is an array |
reasoning | reasoning is not null, counted only over rows whose capabilities has reasoning or whose reasoning is set |
efforts | reasoning.efforts is non-empty, counted only over rows with reasoning.mode of effort or adaptive |
requestMap | requestMap is not null |
endpoint | schemaEndpointId is not null |
Also report per provider:
rows,chat, andnoActivity(rows withactivity: null). A provider with rows and no chat rows is a gap, not a pass.fromModelsDev: rows whosepricing.source.urlis onmodels.dev. These do not count as filled.score: filled facts over needed facts, 0 to 1.
Rate keys differ by provider (input_tokens and prompt, cache_read_tokens and input_cache_read). Handle both.
Source-silent ledger#
Read docs/source-silent.md if it exists. Each line names a provider and a fact that the provider does not publish (for example Grok maxOutput, see #75). A fact in the ledger is left out of that provider's score and is listed under silent in the output. Create the file with a header and the entries that #75 already settled.
Exit code#
--check exits 1 when a provider in --providers a,b,c is below --target (default 1). With no --providers, it checks every provider that has chat rows.
Acceptance#
bun run gap:report --tableprints one row per provider, worst score first.bun run gap:reportprints JSON that an agent can read:{ generatedAt, providers: [{ provider, rows, chat, noActivity, fromModelsDev, facts: { <key>: { have, need } }, silent: [], score }] }.- A unit test runs the scoring on a small fixture: a complete row, a row with a models.dev price, a row with a ledger entry, and a provider with rows but no chat rows.
- The script makes one HTTP request. It does not call the detail endpoint per model.
Out of scope#
- Comparing values with models.dev or any other catalog.
- Filling any data.
- A page or an API route for the report. #106 covers the homepage score.
tombeckenham added labelsenhancement
tombeckenham closed this
Sign in to join this conversation.