Changes feed: write model.updated only when a fact changed (96% of recent events are noise) #309
Send to my agentProblem#
Most model.updated events in /v1/changes are not changes to a model. Of the 4,000 most recent feed entries (2026-10-12 17:20 to 23:04 UTC), about 96% carry a before and after that describe the same model.
This is the second time. #92 was the same class (Replicate runCount ticking on every poll) and was fixed inside one adapter. Fixing each case where it shows up does not stop the next one, so this issue asks for a guard at the one place every event is written.
Evidence#
What differs between before and after | Events | A real change? |
|---|---|---|
Only pricing.source.hash and pricing.source.extractedAt | 3,468 | No |
The operands of the price + in a different order, same terms and same rate tables | 341 | No |
| Other fields (prices, context window, capabilities, deprecated, display name) | 123 | Mostly yes |
model.added | 56 | Yes |
model.removed | 12 | Yes |
- Stamp-only updates by provider: vercel 2,425 (about 273 rows on each of 9 polls), replicate 448, cloudflare-workers-ai 234 (30 rows roughly hourly), deepgram 206, amazon-bedrock 68, mistral 56, google-vertex-anthropic 26, meta 3, kling 2.
- Reorders: one burst at 17:30 to 17:48 UTC across openai 128, gemini 61, google-vertex 57, byteplus 36, mistral 40, anthropic 19. Example:
input_tokens, cache_read_tokens, output_tokensbecamecache_read_tokens, input_tokens, output_tokens. Many rows flipped there and back within the same minute. - BytePlus Seedance rows also reordered an input's
valueslist (ratio,resolution) with the same members.
Cause#
pollProviderModels decides a row is dirty by comparing stableStringify(before) with stableStringify(after) (src/server/ingest/poll-models.ts, const dirty). That comparison treats two kinds of non-fact as fact:
- Provenance stored inside the value.
pricing.sourceholds the hash of the text the card was read from and the read time.storeListedPricing(src/server/rate-card.ts) keeps the stored card only while that hash is unchanged. For Vercel the hash covers the whole listing (adapters/vercel.ts,sha256Text(text)), so one byte changing anywhere restamps every card and every row emits. - Representation.
stableStringifysorts object keys but not arrays. The shared token card takes its term order from the key order of the rates object it is given (packages/rate-card/src/token-card.ts,levers), so the same rates in a different key order produce a different card. Why the key order alternated on 10-12 is not confirmed; a cached copy with sorted keys alternating with a fresh one is the likely reason.
Neither is specific to one adapter. Any adapter that stamps a clock, hashes a whole page, or builds a list from an unordered source reproduces it.
Proposal#
One rule: a model.updated is written only when a fact changed, enforced where the event is written, not in each adapter.
Compare facts, not bytes. Add one function that reduces a row snapshot to its facts, and use it for
dirty:- drop
pricing.source.hashandpricing.source.extractedAt, and the same two on each input'sestimate.source; - sort the operands of every
+inpricing.price, and every input'svalueslist.
A row whose facts match is clean: no row write, no event. The stored
sourcethen records when the price last changed, not when the page was last read.- drop
Each event says what changed. Add
payload.changed: the field paths that differ between the two fact forms (for example["contextWindow", "pricing.tables.rate.base.input_tokens"]). The insert refuses an empty list, so an event with nothing to report cannot be written. This also gives feed and webhook consumers the diff without comparing two snapshots themselves.A test that fails for the whole class. In
poll-models.worker.test.ts: poll a provider, then poll again with the clock advanced, a new source hash, object keys shuffled, and+operands andvaluesreversed. Expect zeromodel.updated. Run the same second-poll check over each adapter that has a listing fixture.A mark for what the test cannot foresee. When one poll updates more than half of a provider's live rows (and at least 20), log an ingest signal and hold an
update-storm:<providerId>row incache_meta, served onGET /v1/statusbesidedocsFailingandpriceClearsRefused. It only marks. A real repricing of a whole catalog is still written.
Parts 1 and 2 are the fix. Part 3 stops a regression. Part 4 is how we find out within one poll if a new kind of noise appears.
Acceptance#
- Re-polling an unchanged provider with a new clock and new source hashes writes no
model.updatedand no row update beyondlastSeenAt. - Two cards with the same terms and rates in a different operand order compare equal.
- A changed rate, a new or removed price term, and a changed
expiresAteach still writemodel.updated. - Every new
model.updatedhas a non-emptypayload.changed. - After deploy, a full day of the feed has no
model.updatedwhose only difference ispricing.source, and Vercel no longer emits its whole catalog on a poll. payload.changedis inopenapi.json, the generated client, and the README notes the addition.
Not in this issue#
- Sources that really do move. Hugging Face host changes, OpenRouter price moves, and
sambanova-gemma-4-31b-itflipping its context window between 262144 and 131072 eight times in the window. These are what the source said on each poll. The SambaNova flip deserves its own look. - The roughly 3,800 noise rows already in
changes. They have gone out to webhook subscribers. Deleting them is a separate decision. - Moving provenance out of the rate card. Keeping
sourceout ofpricingwould remove cause 1 at the root, but it changes the published@modelschemas/rate-cardschema. The fact comparison gets the same result without that.
tombeckenham added labelsbug
tombeckenham closed this
Sign in to join this conversation.