Skip to content
Source
Sign inSign up
TanStack/db

fix(db): drop stale direct optimistic rows after txid sync #1547

MergedKyleAMathews(GitHub profile, opens in a new tab)merged 6 commits intomain fromrefs/pull/1547/headmerged by samwillis(GitHub profile, opens in a new tab)

Summary

Fix stale optimistic rows persisting when sync confirms a different server-generated key (e.g., Postgres GENERATED ALWAYS AS IDENTITY). After a direct collection.insert() with a temporary client ID, the server-assigned row would appear alongside the stale optimistic row instead of replacing it.

Root Cause

CollectionStateManager maintained separate pendingOptimisticDirectUpserts / pendingOptimisticDirectDeletes sets that exempted direct transactions from stale-row cleanup. When sync confirmed a row under a different key than the client's temp key, the stale optimistic row could never be cleaned up — no sync transaction would ever reference the old client key, but the direct-transaction exemption kept it alive indefinitely.

Approach

Remove the direct-transaction exemption entirely. All completed optimistic mutations now go through the same stale-row check: kept only if pendingSyncKeys.has(key). This is safe because direct transactions stay in persisting state (keeping the optimistic row visible via the active-transaction loop) until the sync backend delivers the matching txid. By the time the transaction completes, the sync data is already available.

Also removed the now-dead DIRECT_TRANSACTION_METADATA_KEY constant and its usage in mutations.ts.

Key Invariants

  • A direct transaction's mutationFn (e.g., awaitTxId()) does not resolve until sync delivers the matching txid
  • While a transaction is in persisting state, its optimistic rows are kept alive by the active-transaction loop, not the pending-optimistic sets
  • Once completed, the pending-optimistic stale check is the sole gatekeeper

Non-goals

  • Adding observability/logging to state.ts (systemic gap, separate effort)
  • Covering optimistic delete with server-generated keys (deletes always target known keys)

Verification

pnpm vitest run --pool-options.threads.maxThreads=2 packages/db/tests packages/electric-db-collection/tests

All 2364 tests pass. 2 pre-existing failures in collection-subscribe-changes.test.ts (also fail on main).

Files Changed

  • packages/db/src/collection/state.ts — Remove pendingOptimisticDirect* sets, simplify stale-row check to pendingSyncKeys.has(key) only
  • packages/db/src/collection/mutations.ts — Remove DIRECT_TRANSACTION_METADATA_KEY import and metadata tagging
  • packages/db/src/collection/transaction-metadata.ts — Deleted (orphaned)
  • packages/db/tests/collection.test.ts — Two new tests: optimistic insert replaced by server key, live query reflects the key swap
  • packages/electric-db-collection/tests/electric.test.ts — One new test: txid sync confirms different server-generated key

Summary by CodeRabbit

  • Bug Fixes

    • Fixed stale optimistic rows persisting when sync confirms a different server-generated key during insert operations.
  • Tests

    • Added comprehensive tests for optimistic insert sync behavior, verifying proper handling when server-generated keys replace client-provided keys.
  • Chores

    • Updated CI to use Node.js 22.13.
  • Documentation

    • Added Git and PR hygiene guidelines.

Review Change Stack

    1. More templates

      @tanstack/angular-db

      npm i https://pkg.pr.new/@tanstack/angular-db@1547

      @tanstack/browser-db-sqlite-persistence

      npm i https://pkg.pr.new/@tanstack/browser-db-sqlite-persistence@1547

      @tanstack/capacitor-db-sqlite-persistence

      npm i https://pkg.pr.new/@tanstack/capacitor-db-sqlite-persistence@1547

      @tanstack/cloudflare-durable-objects-db-sqlite-persistence

      npm i https://pkg.pr.new/@tanstack/cloudflare-durable-objects-db-sqlite-persistence@1547

      @tanstack/db

      npm i https://pkg.pr.new/@tanstack/db@1547

      @tanstack/db-ivm

      npm i https://pkg.pr.new/@tanstack/db-ivm@1547

      @tanstack/db-sqlite-persistence-core

      npm i https://pkg.pr.new/@tanstack/db-sqlite-persistence-core@1547

      @tanstack/electric-db-collection

      npm i https://pkg.pr.new/@tanstack/electric-db-collection@1547

      @tanstack/electron-db-sqlite-persistence

      npm i https://pkg.pr.new/@tanstack/electron-db-sqlite-persistence@1547

      @tanstack/expo-db-sqlite-persistence

      npm i https://pkg.pr.new/@tanstack/expo-db-sqlite-persistence@1547

      @tanstack/node-db-sqlite-persistence

      npm i https://pkg.pr.new/@tanstack/node-db-sqlite-persistence@1547

      @tanstack/offline-transactions

      npm i https://pkg.pr.new/@tanstack/offline-transactions@1547

      @tanstack/powersync-db-collection

      npm i https://pkg.pr.new/@tanstack/powersync-db-collection@1547

      @tanstack/query-db-collection

      npm i https://pkg.pr.new/@tanstack/query-db-collection@1547

      @tanstack/react-db

      npm i https://pkg.pr.new/@tanstack/react-db@1547

      @tanstack/react-native-db-sqlite-persistence

      npm i https://pkg.pr.new/@tanstack/react-native-db-sqlite-persistence@1547

      @tanstack/rxdb-db-collection

      npm i https://pkg.pr.new/@tanstack/rxdb-db-collection@1547

      @tanstack/solid-db

      npm i https://pkg.pr.new/@tanstack/solid-db@1547

      @tanstack/svelte-db

      npm i https://pkg.pr.new/@tanstack/svelte-db@1547

      @tanstack/tauri-db-sqlite-persistence

      npm i https://pkg.pr.new/@tanstack/tauri-db-sqlite-persistence@1547

      @tanstack/trailbase-db-collection

      npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@1547

      @tanstack/vue-db

      npm i https://pkg.pr.new/@tanstack/vue-db@1547

      commit: 68992d5

    2. Size Change: +57 B (+0.05%)

      Total Size: 122 kB

      📦 View Changed
      FilenameSizeChange
      packages/db/dist/esm/collection/mutations.js2.46 kB-12 B (-0.49%)
      packages/db/dist/esm/collection/state.js5.33 kB+69 B (+1.31%)
      ℹ️ View Unchanged
      FilenameSize
      packages/db/dist/esm/collection/change-events.js1.39 kB
      packages/db/dist/esm/collection/changes.js1.38 kB
      packages/db/dist/esm/collection/cleanup-queue.js810 B
      packages/db/dist/esm/collection/events.js434 B
      packages/db/dist/esm/collection/index.js3.61 kB
      packages/db/dist/esm/collection/indexes.js1.99 kB
      packages/db/dist/esm/collection/lifecycle.js1.69 kB
      packages/db/dist/esm/collection/subscription.js3.74 kB
      packages/db/dist/esm/collection/sync.js2.88 kB
      packages/db/dist/esm/collection/transaction-metadata.js144 B
      packages/db/dist/esm/deferred.js207 B
      packages/db/dist/esm/errors.js5.01 kB
      packages/db/dist/esm/event-emitter.js748 B
      packages/db/dist/esm/index.js3.01 kB
      packages/db/dist/esm/indexes/auto-index.js829 B
      packages/db/dist/esm/indexes/base-index.js729 B
      packages/db/dist/esm/indexes/basic-index.js2.05 kB
      packages/db/dist/esm/indexes/btree-index.js2.17 kB
      packages/db/dist/esm/indexes/index-registry.js820 B
      packages/db/dist/esm/indexes/reverse-index.js538 B
      packages/db/dist/esm/local-only.js890 B
      packages/db/dist/esm/local-storage.js2.1 kB
      packages/db/dist/esm/optimistic-action.js359 B
      packages/db/dist/esm/paced-mutations.js496 B
      packages/db/dist/esm/proxy.js3.75 kB
      packages/db/dist/esm/query/builder/functions.js1.37 kB
      packages/db/dist/esm/query/builder/index.js5.79 kB
      packages/db/dist/esm/query/builder/ref-proxy.js1.23 kB
      packages/db/dist/esm/query/compiler/evaluators.js1.83 kB
      packages/db/dist/esm/query/compiler/expressions.js430 B
      packages/db/dist/esm/query/compiler/group-by.js3.56 kB
      packages/db/dist/esm/query/compiler/index.js6.67 kB
      packages/db/dist/esm/query/compiler/joins.js2.5 kB
      packages/db/dist/esm/query/compiler/lazy-targets.js918 B
      packages/db/dist/esm/query/compiler/order-by.js1.74 kB
      packages/db/dist/esm/query/compiler/select.js1.4 kB
      packages/db/dist/esm/query/effect.js4.77 kB
      packages/db/dist/esm/query/expression-helpers.js1.43 kB
      packages/db/dist/esm/query/ir.js1.25 kB
      packages/db/dist/esm/query/live-query-collection.js360 B
      packages/db/dist/esm/query/live/collection-config-builder.js8.34 kB
      packages/db/dist/esm/query/live/collection-registry.js264 B
      packages/db/dist/esm/query/live/collection-subscriber.js1.93 kB
      packages/db/dist/esm/query/live/internal.js145 B
      packages/db/dist/esm/query/live/utils.js1.81 kB
      packages/db/dist/esm/query/optimizer.js2.92 kB
      packages/db/dist/esm/query/predicate-utils.js2.97 kB
      packages/db/dist/esm/query/query-once.js359 B
      packages/db/dist/esm/query/subset-dedupe.js960 B
      packages/db/dist/esm/scheduler.js1.3 kB
      packages/db/dist/esm/SortedMap.js1.3 kB
      packages/db/dist/esm/strategies/debounceStrategy.js247 B
      packages/db/dist/esm/strategies/queueStrategy.js428 B
      packages/db/dist/esm/strategies/throttleStrategy.js246 B
      packages/db/dist/esm/transactions.js3.02 kB
      packages/db/dist/esm/utils.js927 B
      packages/db/dist/esm/utils/array-utils.js273 B
      packages/db/dist/esm/utils/browser-polyfills.js304 B
      packages/db/dist/esm/utils/btree.js5.61 kB
      packages/db/dist/esm/utils/comparison.js1.05 kB
      packages/db/dist/esm/utils/cursor.js457 B
      packages/db/dist/esm/utils/index-optimization.js1.54 kB
      packages/db/dist/esm/utils/type-guards.js157 B
      packages/db/dist/esm/virtual-props.js360 B
      compressed-size-action::db-package-size
    3. Caution

      Review failed

      Failed to post review comments

      Note

      Reviews paused

      It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

      Use the following commands to manage reviews:

      • @coderabbitai resume to resume automatic reviews.
      • @coderabbitai review to trigger a single review.

      Use the checkboxes below for quick actions:

      • ▶️ Resume reviews
      • 🔍 Trigger review
      📝 Walkthrough

      Walkthrough

      This PR fixes stale optimistic inserts by adding post-commit pruning logic for direct pending optimistic keys, refactors metadata formatting in transaction creation, and introduces comprehensive test coverage. It also updates CI to Node.js 22.13, adds a release changeset, and documents Git/PR hygiene practices.

      Changes

      Stale Optimistic Key Cleanup and Metadata Refactor

      Layer / File(s)Summary
      Post-commit pruning of stale optimistic keys
      packages/db/src/collection/state.ts
      commitPendingTransactions now prunes completed direct optimistic upserts and deletes not present in changedKeys, removing stale keys from pending maps, restoring visibility state from previousOptimisticUpserts, and clearing direct trackers.
      Direct transaction metadata formatting
      packages/db/src/collection/mutations.ts
      Insert, update, and delete transaction metadata objects are refactored from multi-line to single-line literals using DIRECT_TRANSACTION_METADATA_KEY.
      Test coverage for optimistic replacement
      packages/db/tests/collection.test.ts, packages/db/tests/utils.ts, packages/electric-db-collection/tests/electric.test.ts
      Live-query utilities imported; two collection tests verify optimistic insert removal and live-query updates when server returns a different key; mock config adds optional startSync field; Electric integration test validates txid-to-server-key replacement with metadata propagation.

      Release and Infrastructure Updates

      Layer / File(s)Summary
      Patch release changeset
      .changeset/fix-stale-optimistic-server-key.md
      Documents the fix for stale optimistic rows not being cleaned up when sync confirms a different server-generated key.
      CI Node.js version and Git hygiene
      .github/workflows/e2e-tests.yml, AGENTS.md
      E2E workflow updated to Node.js 22.13; AGENTS.md adds Git and PR Hygiene section covering branch history rewriting restrictions and best practices.

      Estimated code review effort

      🎯 3 (Moderate) | ⏱️ ~22 minutes

      • TanStack/db#1442: Implements the post-commit pruning of pendingOptimisticDirectUpserts/pendingOptimisticDirectDeletes that directly addresses the stale optimistic row retention bug.

      Poem

      🐰 A carrot was planted, then swept by the wind, The server said "No, use this key instead!" With pruning so swift, the old leaf is gone, Fresh metadata glows where optimism shone. 🌱

      🚥 Pre-merge checks | ✅ 5
      ✅ Passed checks (5 passed)
      Check nameStatusExplanation
      Title check✅ PassedThe PR title clearly summarizes the main change: fixing stale optimistic rows that persist after sync confirms a server-generated key.
      Description check✅ PassedThe PR description includes a comprehensive summary of changes, root cause analysis, approach, key invariants, verification steps, and files changed; it exceeds the template requirements by providing detailed technical context.
      Docstring Coverage✅ PassedNo functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
      Linked Issues check✅ PassedCheck skipped because no linked issues were found for this pull request.
      Out of Scope Changes check✅ PassedCheck skipped because no linked issues were found for this pull request.
      ✏️ Tip: You can configure your own custom pre-merge checks in the settings.
      ✨ Finishing Touches
      📝 Generate docstrings
      • Create stacked PR
      • Commit on current branch
      🧪 Generate unit tests (beta)
      • Create PR with unit tests
      • Commit unit tests in branch `fix/electric-txid-server-generated-key`

      Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

      ❤️ Share
      Comment `@coderabbitai help` to get the list of available commands and usage tips.
    4. Size Change: 0 B

      Total Size: 4.24 kB

      ℹ️ View Unchanged
      FilenameSize
      packages/react-db/dist/esm/index.js249 B
      packages/react-db/dist/esm/useLiveInfiniteQuery.js1.32 kB
      packages/react-db/dist/esm/useLiveQuery.js1.34 kB
      packages/react-db/dist/esm/useLiveQueryEffect.js355 B
      packages/react-db/dist/esm/useLiveSuspenseQuery.js567 B
      packages/react-db/dist/esm/usePacedMutations.js401 B
      compressed-size-action::react-db-package-size
    5. 🧹 Nitpick comments (1)
      packages/db/src/collection/state.ts (1)

      523-535: ⚡ Quick win

      Remove dead code: pendingSyncKeys is collected but never used.

      The pendingSyncKeys Set is populated at lines 524-529 but never read - the filtering that previously used it was removed in this refactor. The comment at line 523 ("only when a sync is pending") is also stale since the conditional seeding no longer occurs.

      ♻️ Proposed fix to remove dead code and stale comment
      -    // Seed optimistic state with pending optimistic mutations only when a sync is pending
      -    const pendingSyncKeys = new Set<TKey>()
      -    for (const transaction of this.pendingSyncedTransactions) {
      -      for (const operation of transaction.operations) {
      -        pendingSyncKeys.add(operation.key as TKey)
      -      }
      -    }
      +    // Seed optimistic state with pending optimistic mutations.
      +    // Stale keys are pruned later in commitPendingTransactions when sync confirms.
           for (const [key, value] of this.pendingOptimisticUpserts) {

      As per coding guidelines: "Unused code (dead code, commented-out code, debug artifacts)" should be removed, and "Update or remove comments when refactoring code to keep documentation in sync with implementation".

      🤖 Prompt for AI Agents
      Verify each finding against current code. Fix only still-valid issues, skip the
      rest with a brief reason, keep changes minimal, and validate.
      
      In `@packages/db/src/collection/state.ts` around lines 523 - 535, Remove the dead
      code and stale comment: delete the unused pendingSyncKeys Set declaration and
      its population loop (references to pendingSyncKeys and the for-loop iterating
      this.pendingSyncedTransactions), and update the comment that mentions "only when
      a sync is pending" to reflect the current behavior; leave the existing seeding
      of this.optimisticUpserts from pendingOptimisticUpserts and
      this.optimisticDeletes from pendingOptimisticDeletes intact (they are the active
      logic).
      🤖 Prompt for all review comments with AI agents
      Verify each finding against current code. Fix only still-valid issues, skip the
      rest with a brief reason, keep changes minimal, and validate.
      
      Nitpick comments:
      In `@packages/db/src/collection/state.ts`:
      - Around line 523-535: Remove the dead code and stale comment: delete the unused
      pendingSyncKeys Set declaration and its population loop (references to
      pendingSyncKeys and the for-loop iterating this.pendingSyncedTransactions), and
      update the comment that mentions "only when a sync is pending" to reflect the
      current behavior; leave the existing seeding of this.optimisticUpserts from
      pendingOptimisticUpserts and this.optimisticDeletes from
      pendingOptimisticDeletes intact (they are the active logic).

      ℹ️ Review info
      ⚙️ Run configuration

      Configuration used: defaults

      Review profile: CHILL

      Plan: Pro

      Run ID: cbdc375e-a1cf-4b61-a611-ddb7f54781e2

      📥 Commits

      Reviewing files that changed from the base of the PR and between 2735c3e5aa358900a4fa73f144be7f23a99cc38b and 8613e6526f01a9eaae1661f2a63a226affe8aa4c.

      📒 Files selected for processing (2)
      • .changeset/fix-stale-optimistic-server-key.md
      • packages/db/src/collection/state.ts
      ✅ Files skipped from review due to trivial changes (1)
      • .changeset/fix-stale-optimistic-server-key.md
    6. samwillis(GitHub profile, opens in a new tab) approved these changes

      LGTM, and makes perfect sense!

Sign in to join this conversation.