Skip to content
Source
Sign inSign up
TanStack/schemas

Changes feed: write model.updated only when a fact changed (96% of recent events are noise) #309

Send to my agent
Closedtombeckenham opened this issue 0 comments
tombeckenhamcommented

Problem

Most model.updated events in /v1/changes are not changes to a model. Of the 4,000 most recent feed entries (2026-10-12 17:20 to 23:04 UTC), about 96% carry a before and after that describe the same model.

This is the second time. #92 was the same class (Replicate runCount ticking on every poll) and was fixed inside one adapter. Fixing each case where it shows up does not stop the next one, so this issue asks for a guard at the one place every event is written.

Evidence

What differs between before and afterEventsA real change?
Only pricing.source.hash and pricing.source.extractedAt3,468No
The operands of the price + in a different order, same terms and same rate tables341No
Other fields (prices, context window, capabilities, deprecated, display name)123Mostly yes
model.added56Yes
model.removed12Yes
  • Stamp-only updates by provider: vercel 2,425 (about 273 rows on each of 9 polls), replicate 448, cloudflare-workers-ai 234 (30 rows roughly hourly), deepgram 206, amazon-bedrock 68, mistral 56, google-vertex-anthropic 26, meta 3, kling 2.
  • Reorders: one burst at 17:30 to 17:48 UTC across openai 128, gemini 61, google-vertex 57, byteplus 36, mistral 40, anthropic 19. Example: input_tokens, cache_read_tokens, output_tokens became cache_read_tokens, input_tokens, output_tokens. Many rows flipped there and back within the same minute.
  • BytePlus Seedance rows also reordered an input's values list (ratio, resolution) with the same members.

Cause

pollProviderModels decides a row is dirty by comparing stableStringify(before) with stableStringify(after) (src/server/ingest/poll-models.ts, const dirty). That comparison treats two kinds of non-fact as fact:

  1. Provenance stored inside the value. pricing.source holds the hash of the text the card was read from and the read time. storeListedPricing (src/server/rate-card.ts) keeps the stored card only while that hash is unchanged. For Vercel the hash covers the whole listing (adapters/vercel.ts, sha256Text(text)), so one byte changing anywhere restamps every card and every row emits.
  2. Representation. stableStringify sorts object keys but not arrays. The shared token card takes its term order from the key order of the rates object it is given (packages/rate-card/src/token-card.ts, levers), so the same rates in a different key order produce a different card. Why the key order alternated on 10-12 is not confirmed; a cached copy with sorted keys alternating with a fresh one is the likely reason.

Neither is specific to one adapter. Any adapter that stamps a clock, hashes a whole page, or builds a list from an unordered source reproduces it.

Proposal

One rule: a model.updated is written only when a fact changed, enforced where the event is written, not in each adapter.

  1. Compare facts, not bytes. Add one function that reduces a row snapshot to its facts, and use it for dirty:

    • drop pricing.source.hash and pricing.source.extractedAt, and the same two on each input's estimate.source;
    • sort the operands of every + in pricing.price, and every input's values list.

    A row whose facts match is clean: no row write, no event. The stored source then records when the price last changed, not when the page was last read.

  2. Each event says what changed. Add payload.changed: the field paths that differ between the two fact forms (for example ["contextWindow", "pricing.tables.rate.base.input_tokens"]). The insert refuses an empty list, so an event with nothing to report cannot be written. This also gives feed and webhook consumers the diff without comparing two snapshots themselves.

  3. A test that fails for the whole class. In poll-models.worker.test.ts: poll a provider, then poll again with the clock advanced, a new source hash, object keys shuffled, and + operands and values reversed. Expect zero model.updated. Run the same second-poll check over each adapter that has a listing fixture.

  4. A mark for what the test cannot foresee. When one poll updates more than half of a provider's live rows (and at least 20), log an ingest signal and hold an update-storm:<providerId> row in cache_meta, served on GET /v1/status beside docsFailing and priceClearsRefused. It only marks. A real repricing of a whole catalog is still written.

Parts 1 and 2 are the fix. Part 3 stops a regression. Part 4 is how we find out within one poll if a new kind of noise appears.

Acceptance

  • Re-polling an unchanged provider with a new clock and new source hashes writes no model.updated and no row update beyond lastSeenAt.
  • Two cards with the same terms and rates in a different operand order compare equal.
  • A changed rate, a new or removed price term, and a changed expiresAt each still write model.updated.
  • Every new model.updated has a non-empty payload.changed.
  • After deploy, a full day of the feed has no model.updated whose only difference is pricing.source, and Vercel no longer emits its whole catalog on a poll.
  • payload.changed is in openapi.json, the generated client, and the README notes the addition.

Not in this issue

  • Sources that really do move. Hugging Face host changes, OpenRouter price moves, and sambanova-gemma-4-31b-it flipping its context window between 262144 and 131072 eight times in the window. These are what the source said on each poll. The SambaNova flip deserves its own look.
  • The roughly 3,800 noise rows already in changes. They have gone out to webhook subscribers. Deleting them is a separate decision.
  • Moving provenance out of the rate card. Keeping source out of pricing would remove cause 1 at the root, but it changes the published @modelschemas/rate-card schema. The fact comparison gets the same result without that.
  1. tombeckenham added labelsbug

  2. tombeckenham closed this

Sign in to join this conversation.