Skip to content
Source
Sign inSign up
TanStack/db

fix electric collection progressive mode #852

Mergedsamwillis(GitHub profile, opens in a new tab)merged 7 commits intomain fromrefs/pull/852/headmerged by samwillis(GitHub profile, opens in a new tab)

Depends on https://github.com/electric-sql/electric/pull/3463, must have deps bumped to new electric client version when that is released. done

Fix progressive mode to use fetchSnapshot and atomic swap

Progressive mode was broken because requestSnapshot() injected snapshots into the stream in causally correct position, which didn't work properly with the full mode stream. This release fixes progressive mode by:

Core Changes:

  • Use fetchSnapshot() during initial sync to fetch and apply snapshots immediately in sync transactions
  • Buffer all stream messages during initial sync (renamed flag to isBufferingInitialSync)
  • Perform atomic swap on first up-to-date: truncate snapshot data → apply buffered messages → mark ready
  • Track txids/snapshots only after atomic swap (enables correct optimistic transaction confirmation)

Test Infrastructure:

  • Added ELECTRIC_TEST_HOOKS symbol for test control (hidden from public API)
  • Added progressiveTestControl.releaseInitialSync() to E2E test config for explicit transition control
  • Created comprehensive progressive mode E2E test suite (8 tests):
    • Explicit snapshot phase and atomic swap validation
    • Txid tracking behavior (Electric-only)
    • Multiple concurrent snapshots with deduplication
    • Incremental updates after swap
    • Predicate handling and resilience tests

Bug Fixes:

  • Fixed type errors in test files
  • All 166 unit tests + 95 E2E tests passing
    1. 🦋 Changeset detected

      Latest commit: 2846a6ef5149930c4f784ebc267ea437163f5b74

      The changes in this PR will be included in the next version bump.

      This PR includes changesets to release 11 packages
      NameType
      @tanstack/electric-db-collectionPatch
      @tanstack/db-collection-e2ePatch
      @tanstack/dbPatch
      @tanstack/angular-dbPatch
      @tanstack/powersync-db-collectionPatch
      @tanstack/react-dbPatch
      @tanstack/rxdb-db-collectionPatch
      @tanstack/solid-dbPatch
      @tanstack/svelte-dbPatch
      @tanstack/trailbase-db-collectionPatch
      @tanstack/vue-dbPatch

      Not sure what this means? Click here to learn what changesets are.

      Click here if you're a maintainer who wants to add another changeset to this PR

    2. More templates

      @tanstack/angular-db

      npm i https://pkg.pr.new/@tanstack/angular-db@852

      @tanstack/db

      npm i https://pkg.pr.new/@tanstack/db@852

      @tanstack/db-ivm

      npm i https://pkg.pr.new/@tanstack/db-ivm@852

      @tanstack/electric-db-collection

      npm i https://pkg.pr.new/@tanstack/electric-db-collection@852

      @tanstack/offline-transactions

      npm i https://pkg.pr.new/@tanstack/offline-transactions@852

      @tanstack/powersync-db-collection

      npm i https://pkg.pr.new/@tanstack/powersync-db-collection@852

      @tanstack/query-db-collection

      npm i https://pkg.pr.new/@tanstack/query-db-collection@852

      @tanstack/react-db

      npm i https://pkg.pr.new/@tanstack/react-db@852

      @tanstack/rxdb-db-collection

      npm i https://pkg.pr.new/@tanstack/rxdb-db-collection@852

      @tanstack/solid-db

      npm i https://pkg.pr.new/@tanstack/solid-db@852

      @tanstack/svelte-db

      npm i https://pkg.pr.new/@tanstack/svelte-db@852

      @tanstack/trailbase-db-collection

      npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@852

      @tanstack/vue-db

      npm i https://pkg.pr.new/@tanstack/vue-db@852

      commit: 2846a6e

    3. Size Change: 0 B

      Total Size: 86.2 kB

      ℹ️ View Unchanged
      FilenameSize
      ./packages/db/dist/esm/collection/change-events.js1.38 kB
      ./packages/db/dist/esm/collection/changes.js977 B
      ./packages/db/dist/esm/collection/events.js388 B
      ./packages/db/dist/esm/collection/index.js3.24 kB
      ./packages/db/dist/esm/collection/indexes.js1.1 kB
      ./packages/db/dist/esm/collection/lifecycle.js1.67 kB
      ./packages/db/dist/esm/collection/mutations.js2.26 kB
      ./packages/db/dist/esm/collection/state.js3.43 kB
      ./packages/db/dist/esm/collection/subscription.js2.42 kB
      ./packages/db/dist/esm/collection/sync.js2.29 kB
      ./packages/db/dist/esm/deferred.js207 B
      ./packages/db/dist/esm/errors.js4.19 kB
      ./packages/db/dist/esm/event-emitter.js748 B
      ./packages/db/dist/esm/index.js2.64 kB
      ./packages/db/dist/esm/indexes/auto-index.js742 B
      ./packages/db/dist/esm/indexes/base-index.js766 B
      ./packages/db/dist/esm/indexes/btree-index.js1.87 kB
      ./packages/db/dist/esm/indexes/lazy-index.js1.1 kB
      ./packages/db/dist/esm/indexes/reverse-index.js513 B
      ./packages/db/dist/esm/local-only.js837 B
      ./packages/db/dist/esm/local-storage.js2.1 kB
      ./packages/db/dist/esm/optimistic-action.js359 B
      ./packages/db/dist/esm/paced-mutations.js496 B
      ./packages/db/dist/esm/proxy.js3.22 kB
      ./packages/db/dist/esm/query/builder/functions.js733 B
      ./packages/db/dist/esm/query/builder/index.js3.96 kB
      ./packages/db/dist/esm/query/builder/ref-proxy.js917 B
      ./packages/db/dist/esm/query/compiler/evaluators.js1.35 kB
      ./packages/db/dist/esm/query/compiler/expressions.js430 B
      ./packages/db/dist/esm/query/compiler/group-by.js1.8 kB
      ./packages/db/dist/esm/query/compiler/index.js1.96 kB
      ./packages/db/dist/esm/query/compiler/joins.js2 kB
      ./packages/db/dist/esm/query/compiler/order-by.js1.25 kB
      ./packages/db/dist/esm/query/compiler/select.js1.07 kB
      ./packages/db/dist/esm/query/expression-helpers.js1.43 kB
      ./packages/db/dist/esm/query/ir.js673 B
      ./packages/db/dist/esm/query/live-query-collection.js360 B
      ./packages/db/dist/esm/query/live/collection-config-builder.js5.26 kB
      ./packages/db/dist/esm/query/live/collection-registry.js264 B
      ./packages/db/dist/esm/query/live/collection-subscriber.js1.74 kB
      ./packages/db/dist/esm/query/live/internal.js130 B
      ./packages/db/dist/esm/query/optimizer.js2.56 kB
      ./packages/db/dist/esm/query/predicate-utils.js2.88 kB
      ./packages/db/dist/esm/query/subset-dedupe.js921 B
      ./packages/db/dist/esm/scheduler.js1.21 kB
      ./packages/db/dist/esm/SortedMap.js1.18 kB
      ./packages/db/dist/esm/strategies/debounceStrategy.js247 B
      ./packages/db/dist/esm/strategies/queueStrategy.js428 B
      ./packages/db/dist/esm/strategies/throttleStrategy.js246 B
      ./packages/db/dist/esm/transactions.js2.9 kB
      ./packages/db/dist/esm/utils.js881 B
      ./packages/db/dist/esm/utils/browser-polyfills.js304 B
      ./packages/db/dist/esm/utils/btree.js5.61 kB
      ./packages/db/dist/esm/utils/comparison.js852 B
      ./packages/db/dist/esm/utils/index-optimization.js1.51 kB
      ./packages/db/dist/esm/utils/type-guards.js157 B
      compressed-size-action::db-package-size
    4. Size Change: 0 B

      Total Size: 3.34 kB

      ℹ️ View Unchanged
      FilenameSize
      ./packages/react-db/dist/esm/index.js225 B
      ./packages/react-db/dist/esm/useLiveInfiniteQuery.js1.17 kB
      ./packages/react-db/dist/esm/useLiveQuery.js1.11 kB
      ./packages/react-db/dist/esm/useLiveSuspenseQuery.js431 B
      ./packages/react-db/dist/esm/usePacedMutations.js401 B
      compressed-size-action::react-db-package-size
    5. KyleAMathews(GitHub profile, opens in a new tab) approved these changes

      Looks great! All that work you did on truncate is coming in handy! Very nice semantics to be able to easily do this clean swap.

    6. Mostly note to self during review:

      Perform atomic swap on first up-to-date: truncate snapshot data → apply buffered messages → mark ready

      --> Need to check that we also cancel in-flight fetchSnapshot requests and/or ignore fetchSnapshot requests that resolve after the swap.

    7. This is great work, i think we're almost there but i believe there is a bug with fetchSnapshot requests that resolve after the up-to-date message. For the rest i only have minor comments/nits.

      OutdatedResolvedby samwillis(GitHub profile, opens in a new tab)
      OutdatedResolvedby samwillis(GitHub profile, opens in a new tab)
      OutdatedResolvedby samwillis(GitHub profile, opens in a new tab)
      OutdatedResolvedby samwillis(GitHub profile, opens in a new tab)
      kevin-dp(GitHub profile, opens in a new tab)replied on packages/electric-db-collection/src/electric.ts line 803

      I think there is a bug here. When we call fetchSnapshot we still didn't receive the up-to-date. But, when this call resolves it could be that we received the up-to-date message in the meantime. We should then ignore this snapshot.

      We should have a unit test for this corner case.

      kevin-dp(GitHub profile, opens in a new tab)replied on packages/electric-db-collection/src/electric.ts line 796

      Essentially, i think isBufferingInitialSync should be a function that evaluates syncMode === 'progressive' && !hasReceivedUpToDate

    8. @kevin-dp I've addressed your review. We should still come back to this file and tidy it up more.

Sign in to join this conversation.