Score image, video, audio and unclassified rows and origin links in completeness #729
main fromfeat/714-completeness-per-activity·merged by tombeckenham View on GitHubCompleteness now scores every live row, not only chat. src/lib/completeness.ts is still the one scorer behind pnpm run gap:report, recordCompleteness and GET /v1/status.
Fact sets (ACTIVITY_FACTS)#
- chat: the eleven chat facts, unchanged.
- image, video, audio:
priced,endpoint,modalities(input) and the newoutputModalities. For these,pricedis any stored rate card that is not a models.dev price, since they do not bill tokens. unclassified(null activity): one fact,activity, which the row always lacks. This counts on every provider, not only chat ones. The ledger refuses anactivityentry: classify the row instead.- embeddings, moderation, decision, world, search have no fact set. They appear in
activitieswithfacts: nullandscore: null, are counted underunscoredin the table, and do not move the score. originLinkis scored on every row. A row needs a link when its raw id or an alias (also with dots read as hyphens, thereconcileSameAstiers) matches a row at another provider, and no row links to it. A link target is the origin, so it does not need a link itself. A row withsameAsset needs one and has it.
Provider-wide ledger entries apply to every activity that scores the key. The provider score is filled over needed across all scored activities plus originLink. It is null only when nothing is scored.
Exports for the UI (#715)#
ACTIVITY_FACTS, ScoredActivity, UNCLASSIFIED, CHAT_FACT_KEYS, the ActivityReport and FactTally types, ProviderReport.activities / originLink, scoreRow(row, ledger, origins?) returning RowFact[], rowCompleteness(facts) returning { filled, needed, missing }, originIndex(rows) and originIdKeys(row).
/v1/status shape (breaking)#
completeness drops chat and gains rows, activities and originLink:
{ score, rows, filled, needed, silent,
activities: { [activity | 'unclassified']: { rows, score, filled, needed } },
originLink: { have, need } }
The cron now scores the rows listModelsCatalog(db, { pricing: true }) serves, which is the same read /v1/models?pricing=1 does, because origin links match across the whole catalog. A stored score is served only while the provider's live row count matches the count it scored. The cache_meta key moves to status:completeness:v3, and isCompleteness also rejects the old chat-only shape. After the deploy every score is null until the first models poll. docs/api-changes.md notes the change. The OpenAPI document, generated client and SKILL.md are regenerated, and a changeset (client minor) is added. MODEL_ROW_SHAPE is unchanged because no model row field changed.
The home page column header is now "complete" and its note reads rows instead of chat. The /models column still shows chat facts only. #715 does the rest of the UI.
Sample rows (live service, 2026-11-21)#
Run as pnpm run gap:report --table --base https://modelschemas.openstory.workers.dev. Under pnpm 11, pnpm run gap:report -- --table passes the -- through and parseArgs rejects it. That was already broken before this change.
provider score rows chat image video audio noAct unscored originLink media.priced media.endpoint media.modalities media.outputModalities
anthropic 1.00 15 1.00 (15) - - - 0 0 0/0 0/0 0/0 0/0 0/0
elevenlabs 0.25 17 - - - 0.25 (17) 0 0 0/0 0/17 17/17 0/17 0/17
cartesia 0.75 943 - - - 0.75 (943) 0 0 0/0 0/943 943/943 943/943 943/943
openrouter 0.91 477 0.93 (475) - 1.00 (2) - 0 0 108/187 2/2 2/2 2/2 2/2
llmgateway 0.76 1058 0.76 (878) 0.75 (29) 0.75 (50) 0.75 (43) 27 31 448/647 0/122 122/122 122/122 122/122
replicate 0.23 500 0.78 (39) 0.15 (68) 0.15 (51) 0.06 (28) 314 0 0/89 77/147 0/147 0/147 0/147
Known limits of the stated rule (for #719)#
Matching uses only exact id, alias and dots as hyphens, so a prefixed reseller id (anthropic/claude-opus-4.5) needs a link only when sameAs is already set. Two region providers listing the same ids (dashscope and dashscope-cn, 0/177) both "need" links. An origin row that another provider lists without linking to it counts as needing one until some row links to it (openai 0/1). Read-time source-silent evidence on model rows is still written for chat rows only.
Checks#
pnpm run typecheck, pnpm run lint, pnpm run check, pnpm run check:client, and pnpm run test (257 files, 2560 tests) all pass.
Fixes #714