Skip to content
Source
Sign inSign up
TanStack/schemas

Score image, video, audio and unclassified rows and origin links in completeness #729

Mergedtombeckenhammerged 2 commits intomain fromfeat/714-completeness-per-activitymerged by tombeckenham View on GitHub

tombeckenhamcommented

Completeness now scores every live row, not only chat. src/lib/completeness.ts is still the one scorer behind pnpm run gap:report, recordCompleteness and GET /v1/status.

Fact sets (ACTIVITY_FACTS)

  • chat: the eleven chat facts, unchanged.
  • image, video, audio: priced, endpoint, modalities (input) and the new outputModalities. For these, priced is any stored rate card that is not a models.dev price, since they do not bill tokens.
  • unclassified (null activity): one fact, activity, which the row always lacks. This counts on every provider, not only chat ones. The ledger refuses an activity entry: classify the row instead.
  • embeddings, moderation, decision, world, search have no fact set. They appear in activities with facts: null and score: null, are counted under unscored in the table, and do not move the score.
  • originLink is scored on every row. A row needs a link when its raw id or an alias (also with dots read as hyphens, the reconcileSameAs tiers) matches a row at another provider, and no row links to it. A link target is the origin, so it does not need a link itself. A row with sameAs set needs one and has it.

Provider-wide ledger entries apply to every activity that scores the key. The provider score is filled over needed across all scored activities plus originLink. It is null only when nothing is scored.

Exports for the UI (#715)

ACTIVITY_FACTS, ScoredActivity, UNCLASSIFIED, CHAT_FACT_KEYS, the ActivityReport and FactTally types, ProviderReport.activities / originLink, scoreRow(row, ledger, origins?) returning RowFact[], rowCompleteness(facts) returning { filled, needed, missing }, originIndex(rows) and originIdKeys(row).

/v1/status shape (breaking)

completeness drops chat and gains rows, activities and originLink:

{ score, rows, filled, needed, silent,
  activities: { [activity | 'unclassified']: { rows, score, filled, needed } },
  originLink: { have, need } }

The cron now scores the rows listModelsCatalog(db, { pricing: true }) serves, which is the same read /v1/models?pricing=1 does, because origin links match across the whole catalog. A stored score is served only while the provider's live row count matches the count it scored. The cache_meta key moves to status:completeness:v3, and isCompleteness also rejects the old chat-only shape. After the deploy every score is null until the first models poll. docs/api-changes.md notes the change. The OpenAPI document, generated client and SKILL.md are regenerated, and a changeset (client minor) is added. MODEL_ROW_SHAPE is unchanged because no model row field changed.

The home page column header is now "complete" and its note reads rows instead of chat. The /models column still shows chat facts only. #715 does the rest of the UI.

Sample rows (live service, 2026-11-21)

Run as pnpm run gap:report --table --base https://modelschemas.openstory.workers.dev. Under pnpm 11, pnpm run gap:report -- --table passes the -- through and parseArgs rejects it. That was already broken before this change.

provider    score  rows        chat       image       video       audio  noAct  unscored  originLink  media.priced  media.endpoint  media.modalities  media.outputModalities
anthropic    1.00    15   1.00 (15)           -           -           -      0         0         0/0           0/0             0/0               0/0                     0/0
elevenlabs   0.25    17           -           -           -   0.25 (17)      0         0         0/0          0/17           17/17              0/17                    0/17
cartesia     0.75   943           -           -           -  0.75 (943)      0         0         0/0         0/943         943/943           943/943                 943/943
openrouter   0.91   477  0.93 (475)           -    1.00 (2)           -      0         0     108/187           2/2             2/2               2/2                     2/2
llmgateway   0.76  1058  0.76 (878)   0.75 (29)   0.75 (50)   0.75 (43)     27        31     448/647         0/122         122/122           122/122                 122/122
replicate    0.23   500   0.78 (39)   0.15 (68)   0.15 (51)   0.06 (28)    314         0        0/89        77/147           0/147             0/147                   0/147

Known limits of the stated rule (for #719)

Matching uses only exact id, alias and dots as hyphens, so a prefixed reseller id (anthropic/claude-opus-4.5) needs a link only when sameAs is already set. Two region providers listing the same ids (dashscope and dashscope-cn, 0/177) both "need" links. An origin row that another provider lists without linking to it counts as needing one until some row links to it (openai 0/1). Read-time source-silent evidence on model rows is still written for chat rows only.

Checks

pnpm run typecheck, pnpm run lint, pnpm run check, pnpm run check:client, and pnpm run test (257 files, 2560 tests) all pass.

Fixes #714

    1. tombeckenham pushed03d4b12

    2. tombeckenham merged this254279b

    3. tombeckenham closed this

Sign in to join this conversation.