Skip to content
Source
Sign inSign up
TanStack/db

fix(db): filter out-of-window SSE inserts in collection-subscriber #1555

Closedv-anton(GitHub profile, opens in a new tab)wants to merge 2 commits intomain fromrefs/pull/1555/headopened

🎯 Changes

sendChangesInRange in CollectionSubscriber passed all SSE changes to D2 unfiltered. Inserts that sort outside the current orderBy + limit window were tracked by D2's TopKState. When an in-window item was later deleted, D2 promoted the tracked out-of-window item instead of triggering loadNextItems to fetch the correct next item from the BTree.

The fix adds a pre-filter that drops new inserts whose sort value falls outside the current window boundary. Deletes, updates, already-sent keys, and inserts within the window pass through unchanged.

✅ Checklist

  • I have tested this code locally with pnpm test:pr.
  • I have followed the steps in the Contributing guide.

🚀 Release Impact

  • This change affects published code, and I have generated a changeset.
  • This change is docs/CI/dev-only (no release).

Summary by CodeRabbit

  • Bug Fixes

    • SSE filter now prevents out-of-window insert events from incorrectly replacing items in an active ordered query window.
  • Tests

    • Added tests covering out-of-window insert handling, behavior when window isn’t full, and unbounded (no-limit) query behavior.
  • Chore

    • Added a changeset entry marking a patch release.

Review Change Stack

    1. No actionable comments were generated in the recent review. 🎉

      ℹ️ Recent review info
      ⚙️ Run configuration

      Configuration used: defaults

      Review profile: CHILL

      Plan: Pro

      Run ID: 40278cd8-72df-468d-9972-a992f9a3a918

      📥 Commits

      Reviewing files that changed from the base of the PR and between ae146bce3377e4292fe042e5ced4bb4059a04a47 and 7269fd4ab4cdb7d3e8264e280c8eb44393a2310f.

      📒 Files selected for processing (1)
      • packages/db/tests/collection-subscriber-out-of-window-sse.test.ts

      📝 Walkthrough

      Walkthrough

      sendChangesInRange now computes the ordered window (limit+offset) and a window boundary, filters incoming SSE changes to exclude out-of-window inserts (with exceptions), and uses the filtered set for tracking and downstream updates. Tests and a changeset entry validate and document the fix.

      Changes

      Out-of-window SSE Insert Filtering

      Layer / File(s)Summary
      Window filtering for ordered changes
      packages/db/src/query/live/collection-subscriber.ts, .changeset/fix-out-of-window-sse-filter.md
      sendChangesInRange computes window size and boundary from the current largest value, filters incoming changes so non-inserts are always kept and inserts are kept only when already-sent, within window capacity, or at/after the boundary. Filtered changes are used for value tracking and downstream dispatch; release notes updated.
      Out-of-window SSE filter tests
      packages/db/tests/collection-subscriber-out-of-window-sse.test.ts
      Adds Vitest scenarios: verify an out-of-window SSE insert is not promoted when an in-window item is deleted, verify inserts pass through while the window is not yet full, and verify all inserts appear when there is no limit.

      Estimated code review effort

      🎯 3 (Moderate) | ⏱️ ~20 minutes

      Poem

      🐰 I nibble at windows, counting each hop, Filtering inserts that try to pop. When bounds are set, I guard the view, Only in-window friends get through. Hop on — the order stays true. 🥕

      🚥 Pre-merge checks | ✅ 5
      ✅ Passed checks (5 passed)
      Check nameStatusExplanation
      Title check✅ PassedThe title accurately describes the main change: filtering out-of-window SSE inserts in the collection-subscriber component.
      Description check✅ PassedThe description covers all required template sections: changes are clearly explained with motivation, all checklist items are addressed, and release impact is documented with a changeset generated.
      Docstring Coverage✅ PassedNo functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
      Linked Issues check✅ PassedCheck skipped because no linked issues were found for this pull request.
      Out of Scope Changes check✅ PassedCheck skipped because no linked issues were found for this pull request.
      ✏️ Tip: You can configure your own custom pre-merge checks in the settings.
      ✨ Finishing Touches
      🧪 Generate unit tests (beta)
      • Create PR with unit tests

      Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

      ❤️ Share
      Comment `@coderabbitai help` to get the list of available commands and usage tips.
    2. Actionable comments posted: 2

      🤖 Prompt for all review comments with AI agents
      Verify each finding against current code. Fix only still-valid issues, skip the
      rest with a brief reason, keep changes minimal, and validate.
      
      Inline comments:
      In `@packages/db/tests/collection-subscriber-out-of-window-sse.test.ts`:
      - Around line 13-86: Add explicit tests that cover limit/offset edge cases and
      async event-ordering races: create new test cases using the same helpers
      (createCollection, createLiveQueryCollection,
      sourceCollection.utils.write/commit, liveQueryCollection.preload) to assert
      behavior when limit is 0 and when offset is non-zero (verify window is empty or
      starts at correct item), and add race-condition tests that fire concurrent SSE
      inserts and deletes with controlled timing (use setTimeout/promises to vary
      ordering) to ensure replacements come from the storage BTree (e.g., expect item
      with value 70) not an out-of-window SSE insert; include clear assertions on
      liveQueryCollection.values() after each sequence and small waits (e.g.,
      setTimeout) to let loadNextItems run.
      - Around line 72-74: The test currently uses a fixed 50ms setTimeout to wait for
      loadNextItems to fetch a replacement, which is flaky; replace this with a
      condition-based wait that polls until the expected window state is observed (or
      a timeout elapses). Specifically, poll the collection subscriber's
      observable/window (e.g., check collectionSubscriber.window,
      collectionSubscriber.getWindowItems(), or the items array that loadNextItems
      should update) until the replacement item is present, using a short interval
      (e.g., 10–50ms) and a reasonable overall timeout (e.g., 1–2s), and fail the test
      if the condition is not met; you can extract a small helper
      waitForCondition(pollFn, timeout) to reuse across tests.
      🪄 Autofix (Beta)

      Fix all unresolved CodeRabbit comments on this PR:

      • Push a commit to this branch (recommended)
      • Create a new PR with the fixes

      ℹ️ Review info
      ⚙️ Run configuration

      Configuration used: defaults

      Review profile: CHILL

      Plan: Pro

      Run ID: 50122b4b-71f1-4239-8840-dfcd3ee2bad3

      📥 Commits

      Reviewing files that changed from the base of the PR and between be656be5c5fa65cc1c2bf76bc0b60be3d9d2a577 and ae146bce3377e4292fe042e5ced4bb4059a04a47.

      📒 Files selected for processing (3)
      • .changeset/fix-out-of-window-sse-filter.md
      • packages/db/src/query/live/collection-subscriber.ts
      • packages/db/tests/collection-subscriber-out-of-window-sse.test.ts
      packages/db/tests/collection-subscriber-out-of-window-sse.test.ts lines 13 to 89
      @@ -0,0 +66,21 @@
      66+ sourceCollection.utils.write({
      67+ type: `delete`,
      68+ value: { id: `2`, value: 90 },
      69+ })
      70+ sourceCollection.utils.commit()
      71+
      72+ // Wait for loadNextItems to fetch the replacement
      73+ await new Promise((resolve) => setTimeout(resolve, 50))
      74+
      75+ results = Array.from(liveQueryCollection.values())
      76+ expect(results).toHaveLength(3)
      77+
      78+ // The replacement should be item 4 (value 70) — the next item from the
      79+ // BTree — NOT the out-of-window SSE insert (value 10).
      80+ expect(
      81+ results.map((r) => r.id),
      82+ `Expected item 4 (value 70) to replace deleted item 2, ` +
      83+ `not the out-of-window SSE insert (value 10). ` +
      84+ `Got: ${JSON.stringify(results.map((r) => ({ id: r.id, value: r.value })))}`,
      85+ ).toEqual([`1`, `3`, `4`])
      86+ })

      🛠️ Refactor suggestion | 🟠 Major | ⚡ Quick win

      Add explicit edge-case tests for limit/offset and async ordering.

      This test reproduces the core bug path well, but it still misses dedicated corner-case coverage for limit/offset boundaries (for example limit: 0 and non-zero offset) and async event ordering/race scenarios, which are high-value for this fix area.

      As per coding guidelines, “Test corner cases including: … async race conditions, and limit/offset edge cases”.

      🤖 Prompt for AI Agents
      Verify each finding against current code. Fix only still-valid issues, skip the
      rest with a brief reason, keep changes minimal, and validate.
      
      In `@packages/db/tests/collection-subscriber-out-of-window-sse.test.ts` around
      lines 13 - 86, Add explicit tests that cover limit/offset edge cases and async
      event-ordering races: create new test cases using the same helpers
      (createCollection, createLiveQueryCollection,
      sourceCollection.utils.write/commit, liveQueryCollection.preload) to assert
      behavior when limit is 0 and when offset is non-zero (verify window is empty or
      starts at correct item), and add race-condition tests that fire concurrent SSE
      inserts and deletes with controlled timing (use setTimeout/promises to vary
      ordering) to ensure replacements come from the storage BTree (e.g., expect item
      with value 70) not an out-of-window SSE insert; include clear assertions on
      liveQueryCollection.values() after each sequence and small waits (e.g.,
      setTimeout) to let loadNextItems run.
      Resolvedby coderabbitai[bot](GitHub profile, opens in a new tab)
    3. Closing as superseded by RFC #1657's planned TotalOrder/WindowState work. The reported out-of-window insert bug is confirmed and will remain an acceptance case; the current point patch is too narrow and regresses valid refill/window movement paths.

Sign in to join this conversation.