Skip to content
Source
Sign inSign up
TanStack/db

fix(db): preserve exact D2 rows during sync reconciliation #1794

Closedobeattie(GitHub profile, opens in a new tab)wants to merge 4 commits intomain fromrefs/pull/1794/headopened

🎯 Changes

Fixes #1783.

D2 applies an update by removing the old row and adding the new one. The row being removed must exactly match the row previously added. The state diverged like this:

  1. An optimistic insert added one snapshot of the row to D2.
  2. Sync confirmation emitted another insert with the same key. This was correctly skipped so D2 would not count the row twice.
  3. D2 therefore kept the optimistic snapshot, while the collection later described the row using its sync-confirmed snapshot.
  4. A later update tried to remove the collection’s snapshot, but D2 held the optimistic snapshot.
  5. The snapshots represented the same key but were not identical, so D2 threw Query contributors with the same row key are not congruent.

Any differing property can cause this; it is not specific to $synced or $origin. A focused test demonstrates the same mismatch with a normal status property. Changing the hashing would only hide the mismatch by making different rows appear equal. The correct fix is to remove the exact row that was previously added.

I bisected the regression to #1740. That PR added the congruence check which exposed the incorrect removal. This check is correct; the source bookkeeping was not.

The fix remembers the exact row sent to D2 for each key and uses it for later updates and deletes. The same bookkeeping also prevents duplicate inserts.

The regression test covers the original optimistic insert and synchronous sync confirmation, then verifies that the collection and live query each contain exactly one updated row.

✅ Checklist

  • I have tested this code locally with pnpm test.

🚀 Release Impact

  • This change affects published code, and I have generated a changeset.
  • This change is docs/CI/dev-only (no release).

Summary by CodeRabbit

  • Bug Fixes

    • Fixed live queries throwing when optimistically inserted rows are updated after synchronous sync confirmation.
    • Improved handling of row updates, deletions, and duplicate inserts to keep query and collection results consistent.
    • Ensured updates correctly replace previously received row values without stale or duplicate data.
  • Tests

    • Added coverage for optimistic insert updates and accurate row reconciliation.
    1. Review Change Stack

      No actionable comments were generated in the recent review. 🎉

      ℹ️ Recent review info
      ⚙️ Run configuration

      Configuration used: defaults

      Review profile: CHILL

      Plan: Team

      Run ID: 0769a2a9-3cfa-48e8-9002-cb8e12f189d0

      📥 Commits

      Reviewing files that changed from the base of the PR and between 68366ecaeef6c12a13402b558bd4a68d7519442f and 8abd32e080818e4f143dc192ca72429b1e9e702b.

      📒 Files selected for processing (5)
      • .changeset/sixty-nights-refuse.md
      • packages/db/src/query/effect.ts
      • packages/db/src/query/live/collection-subscriber.ts
      • packages/db/src/query/live/utils.ts
      • packages/db/tests/collection-subscriber-duplicate-inserts.test.ts

      Included review availability: Your plan provides up to 8 included reviews per hour; 7 remain after this review.


      📝 Walkthrough

      Walkthrough

      Live query and effect pipelines now track the exact rows contributed to D2. Updates and deletes retract those stored rows. Tests cover row reconciliation and updates to synchronously confirmed optimistic inserts. A patch changeset records the fix.

      Changes

      Live query reconciliation

      Layer / File(s)Summary
      Row reconciliation utility
      packages/db/src/query/live/utils.ts
      prepareChangesForD2 tracks sent rows in a map. It removes duplicate inserts and sets previousValue to the exact previously sent row for updates and deletes.
      Pipeline state integration
      packages/db/src/query/effect.ts, packages/db/src/query/live/collection-subscriber.ts
      Per-source state stores contributed rows. D2 preparation, ordered cursor tracking, truncation, and disposal use the row maps.
      Regression validation and release
      packages/db/tests/collection-subscriber-duplicate-inserts.test.ts, .changeset/sixty-nights-refuse.md
      Tests cover exact-row retraction and updates to synchronously confirmed optimistic inserts. The changeset declares a patch release.

      Estimated code review effort: 3 (Moderate) | ~20 minutes

      Merge Risk: 🔵 Low · up to 8abd3

      The PR correctly preserves exact rows during live-query reconciliation, preventing duplicate and update failures in normal operation. A bounded recovery risk remains if graph processing fails after bookkeeping is updated, which warrants owner awareness or follow-up but does not block merge under normal checks.

      Suggested reviewers: kevin-dp, kyleamathews

      Sequence Diagram(s)

      sequenceDiagram
        participant Collection
        participant CollectionSubscriber
        participant prepareChangesForD2
        participant D2Pipeline
        Collection->>CollectionSubscriber: update optimistically inserted row
        CollectionSubscriber->>prepareChangesForD2: provide change and sent row map
        prepareChangesForD2->>D2Pipeline: emit update with exact previousValue
        D2Pipeline-->>CollectionSubscriber: process reconciled live query change
      🚥 Pre-merge checks | ✅ 4 | ❌ 1

      ❌ Failed checks (1 warning)

      Check nameStatusExplanationResolution
      Docstring Coverage⚠️ WarningDocstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 4 files. (1 skipped: 1…Write docstrings for the functions missing them to satisfy the coverage threshold.
      ✅ Passed checks (4 passed)
      Check nameStatusExplanation
      Title check✅ PassedThe title clearly identifies the main change: preserving exact D2 rows during sync reconciliation.
      Description check✅ PassedThe description includes the required Changes, Checklist, and Release Impact sections. It explains the defect, fix, regression test, local test status, and changeset.
      Linked Issues check✅ PassedThe changes address issue #1783 by tracking exact rows for D2 reconciliation, preventing congruence errors after optimistic inserts and sync confirmation. The regression test covers the required optim…
      Out of Scope Changes check✅ PassedAll changed files support the linked issue. The implementation, changeset, and regression tests are directly related to D2 sync reconciliation and live-query consistency.
      Full details: Linked Issues check

      Explanation

      The changes address issue #1783 by tracking exact rows for D2 reconciliation, preventing congruence errors after optimistic inserts and sync confirmation. The regression test covers the required optimistic insert, synchronous confirmation, and later update flow.

      Full details: Docstring Coverage

      Explanation

      Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 4 files. (1 skipped: 1 unsupported.)

      • Fix all pre-merge checks with AI
      ✨ Finishing Touches
      🧪 Generate unit tests (beta)
      • Create PR with unit tests

      Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

      ❤️ Share
      Comment `@coderabbitai help` to get the list of available commands.
    2. We hit this in production on 0.8.7 through a path that does not involve an optimistic insert or writeInsert: plain collection.update calls on an existing key, confirmed by ordinary sync commits. The trigger is a new update on a key starting while a sync commit for that key is already deferred behind an earlier in-flight mutation. Claude reduced it to the script below against bare @tanstack/db@0.8.7 (main is at the 0.8.7 commit as of 2026-09-06). Applying this PR's diff to 0.8.7 makes it pass, so posting here in case the extra case is useful for the regression suite.

      import { createCollection, createLiveQueryCollection, eq } from '@tanstack/db'
      
      let hooks
      const releases = new Map()
      const col = createCollection({
        id: 'rows',
        getKey: (r) => r.id,
        startSync: true,
        sync: {
          rowUpdateMode: 'full',
          sync: (p) => {
            hooks = p
            p.begin()
            p.write({ type: 'insert', value: { id: 'c1', n: 0 } })
            p.commit()
            p.markReady()
          },
        },
        // Persist resolves when the test releases it, so confirmations can be ordered explicitly.
        onUpdate: ({ transaction }) =>
          new Promise((resolve) => releases.set(transaction.mutations[0].modified.n, resolve)),
      })
      const lq = createLiveQueryCollection({
        startSync: true,
        query: (q) => q.from({ c: col }).where(({ c }) => eq(c.id, 'c1')),
      })
      lq.subscribeChanges(() => {})
      col.subscribeChanges((changes) => {
        for (const ch of changes) console.log('   event', ch.type, ch.value.n, ch.value.$synced, 'prev', ch.previousValue?.n, ch.previousValue?.$synced)
      })
      
      const mutate = (n) => col.update('c1', (d) => { d.n = n }).isPersisted.promise.catch(() => {})
      const confirm = (n) => {
        hooks.begin()
        hooks.write({ type: 'update', value: { id: 'c1', n } })
        hooks.commit()
        releases.get(n)()
      }
      const tick = () => new Promise((r) => setTimeout(r, 0))
      
      // m = collection.update, c = server confirmation for that update.
      // c1 arrives while m2 is in flight, so its commit is deferred; m3 starts before it applies.
      for (const step of ['m1', 'm2', 'c1', 'm3', 'c2', 'c3', 'm4']) {
        const n = Number(step[1])
        if (step[0] === 'm') mutate(n)
        else confirm(n)
        await tick()
        console.log(step, 'collection.$synced =', col.get('c1').$synced, '| liveQuery.$synced =', lq.toArray[0]?.$synced, 'n =', lq.toArray[0]?.n)
      }

      Output on 0.8.7 (Node 24, last steps only):

         event update 3 false prev 2 false
      m3 collection.$synced = false | liveQuery.$synced = false n = 3
      c2 collection.$synced = false | liveQuery.$synced = false n = 3
      c3 collection.$synced = true | liveQuery.$synced = false n = 3      <- row flipped to synced, no change event
         event update 4 false prev 3 true                                   <- previousValue is a row the live query never received
      m4 Error: Query contributors with the same row key are not congruent

      After that, every further write on the key throws.

      Two sequences that do not throw, for contrast: m1 m2 c1 c2 m3 (the deferred commits apply before the next update starts, and the $synced flip is published as an event) and m1 m2 m3 c1 c2 c3 m4 (all three in flight, then all confirmations).

      So the collection-side defect in this path is that the last deferred commit updates the visible row without publishing the change, and the subscriber then retracts the collection's previousValue instead of the row it actually contributed. This PR's exact-row bookkeeping in CollectionSubscriber handles the second half, which is enough to keep the live query consistent. For completeness, #1795 alone does not change the outcome here. The missing publication on the state side may still be worth a cell in the publication oracle's overlap matrix mentioned on #1783.

      This was investigated and reduced with Claude Code, and the script and output above are from actual runs.

    3. Closing as superseded by merged #1797, which incorporates and tests exact D2 before-images during source reconciliation. The consolidated PR also includes direct Query DB optimistic write-back regression traces. This branch no longer needs a separate merge.

Sign in to join this conversation.