Skip to content
Source
Sign inSign up
TanStack/schemas

replicate: one upstream 502 on any of up to 20 catalog pages fails the whole models poll #662

Send to my agent
Closedtombeckenham opened this issue 0 comments
tombeckenhamcommented

What happens

The Replicate models poll fails outright whenever any page of the catalog walk returns a transient 502. On 2026-11-15 that happened on 5 of about 14 polls since 10:00 UTC. Before that, the rate was about one a day.

PostHog ingest_failed events (job models-poll, provider replicate):

  • 10:04 GET /v1/models → 502 Bad Gateway
  • 10:34 GET /v1/models?cursor=… → 502 Bad Gateway (a later page)
  • 10:49 GET /v1/models?cursor=… → 502 Bad Gateway (a later page)
  • 13:03 GET /v1/models → 502 Bad Gateway
  • 13:18 GET /v1/models → 502 Bad Gateway

Over the previous 7 days there was one a day or less: 11-11, 11-12, 11-13, 11-14. The endpoint answers normally between failures.

Why

listCatalog (src/server/providers/adapters/replicate.ts:525) walks up to MAX_MODEL_PAGES = 20 pages. Each page is one fetchJson call with no retry. Both fetchSpec (line 571) and listModels (line 794) use it. So one 502 on any page sinks the poll. The more pages there are, the more often that happens.

FAL already solved the same problem. fetchPageWithRetry (src/server/providers/fal.ts:237) retries statuses 408, 429, 500, 502, 503 and 504 with exponential backoff, honours Retry-After, and rethrows anything else as-is.

Fix

  • Retry each Replicate catalog page on the same transient statuses, with bounded backoff. Lift the FAL helper into a shared spot if that keeps it one implementation. Don't copy it.
  • After the last attempt, throw with the URL and status, the same as now (fail loud). Never return a partial catalog: a short walk would read as routes being removed.
  • Keep the per-page FETCH_TIMEOUT_MS, and keep the total wait within the cron invocation's budget.
  • Add tests:
    • a page that returns 502 once and then 200 completes the walk;
    • a page that returns 502 on every attempt throws with that page's URL.
  1. tanstack-source added labelsbug

  2. tombeckenham linked this to #663

  3. tombeckenham closed this

Sign in to join this conversation.