Skip to content
Source
Sign inSign up
TanStack/db

feat: add persisted sync metadata support #1380

Mergedsamwillis(GitHub profile, opens in a new tab)merged 26 commits intomain fromrefs/pull/1380/headmerged by samwillis(GitHub profile, opens in a new tab)

Summary

  • add transactional persisted sync metadata support across @tanstack/db, db-sqlite-persisted-collection-core, query-db-collection, and electric-db-collection
  • persist row-scoped metadata with synced rows and collection-scoped metadata alongside persisted collection state
  • use that metadata to preserve query ownership/retention across restart and to persist Electric resume state

How To Use

Custom sync implementations

If a collection is running on a persistence layer that supports this feature, the sync function now receives an optional metadata API alongside begin, write, commit, markReady, and truncate.

sync: ({ begin, write, commit, metadata }) => {
  const startupValue = metadata?.collection.get('my-sync:startup')

  begin()
  write({
    type: 'update',
    value: row,
    metadata: { source: 'remote' },
  })
  metadata?.collection.set('my-sync:startup', { ready: true })
  commit()
}

Available operations:

  • metadata.row.get(key)
  • metadata.row.set(key, value)
  • metadata.row.delete(key)
  • metadata.collection.get(key)
  • metadata.collection.set(key, value)
  • metadata.collection.delete(key)
  • metadata.collection.list(prefix?)

Behavior:

  • startup reads through get / list are allowed before any transaction is opened
  • metadata writes are transactional and must happen inside begin() / commit()
  • write({ metadata }) and metadata.row.set() target the same row metadata store

SQLite persisted collections

persistedCollectionOptions(...) now persists both row metadata and collection metadata when used with db-sqlite-persisted-collection-core.

Typical usage looks like:

const collection = createCollection(
  persistedCollectionOptions({
    ...someCollectionOptions,
    persistence: {
      adapter: sqliteAdapter,
    },
  }),
)

What you get automatically:

  • row metadata hydrates with persisted rows
  • collection metadata is loaded during startup before persisted sync work begins
  • row data, row metadata, and collection metadata commit in the same persisted transaction

Query collections

Query collections can now retain persisted ownership metadata across restarts.

const collection = createCollection(
  persistedCollectionOptions({
    ...queryCollectionOptions({
      id: 'messages',
      queryClient,
      queryKey: ['messages', roomId],
      queryFn: fetchMessages,
      getKey: (row) => row.id,
      persistedGcTime: Number.POSITIVE_INFINITY,
    }),
    persistence: {
      adapter: sqliteAdapter,
    },
  }),
)

What this enables:

  • row ownership is persisted on rows, so warm starts do not incorrectly delete disjoint pre-hydrated rows
  • query retention is separate from in-memory TanStack Query gcTime
  • persistedGcTime can be finite or effectively indefinite for offline-first flows
  • when a retained query is requested again, reconciliation diffing uses that query's owned-row baseline instead of the whole collection

Electric collections

Electric collections now persist resume metadata at collection scope when wrapped with persisted collection options.

const collection = createCollection(
  persistedCollectionOptions({
    ...electricCollectionOptions({
      id: 'todos',
      getKey: (row) => row.id,
      shapeOptions: {
        url: electricUrl,
        params: { table: 'todos' },
      },
    }),
    persistence: {
      adapter: sqliteAdapter,
    },
  }),
)

What this enables:

  • persisted startup can reuse a saved Electric offset / handle resume point
  • resume metadata is committed transactionally with the batch that made it valid
  • must-refetch/reset paths write a reset marker before reloading so restart does not resume from stale stream state

Test plan

  • Run pnpm vitest --run packages/db/tests/collection.test.ts
  • Run pnpm --filter @tanstack/query-db-collection exec vitest run tests/query.test.ts --coverage.enabled false
  • Run pnpm --filter @tanstack/electric-db-collection exec vitest run tests/electric.test.ts --coverage.enabled false
  • Run pnpm --filter @tanstack/db-sqlite-persisted-collection-core exec vitest run tests/persisted.test.ts tests/sqlite-core-adapter.test.ts --coverage.enabled false

Notes

  • The persisted SQLite package tests are functionally green in package-local runs.
  • There is still noisy stderr in some query tests from expected error-path coverage and existing cleanupQueryIfIdle warnings.
    1. More templates

      @tanstack/angular-db

      npm i https://pkg.pr.new/TanStack/db/@tanstack/angular-db@1380

      @tanstack/db

      npm i https://pkg.pr.new/TanStack/db/@tanstack/db@1380

      @tanstack/db-browser-wa-sqlite-persisted-collection

      npm i https://pkg.pr.new/TanStack/db/@tanstack/db-browser-wa-sqlite-persisted-collection@1380

      @tanstack/db-capacitor-sqlite-persisted-collection

      npm i https://pkg.pr.new/TanStack/db/@tanstack/db-capacitor-sqlite-persisted-collection@1380

      @tanstack/db-cloudflare-do-sqlite-persisted-collection

      npm i https://pkg.pr.new/TanStack/db/@tanstack/db-cloudflare-do-sqlite-persisted-collection@1380

      @tanstack/db-electron-sqlite-persisted-collection

      npm i https://pkg.pr.new/TanStack/db/@tanstack/db-electron-sqlite-persisted-collection@1380

      @tanstack/db-expo-sqlite-persisted-collection

      npm i https://pkg.pr.new/TanStack/db/@tanstack/db-expo-sqlite-persisted-collection@1380

      @tanstack/db-ivm

      npm i https://pkg.pr.new/TanStack/db/@tanstack/db-ivm@1380

      @tanstack/db-node-sqlite-persisted-collection

      npm i https://pkg.pr.new/TanStack/db/@tanstack/db-node-sqlite-persisted-collection@1380

      @tanstack/db-react-native-sqlite-persisted-collection

      npm i https://pkg.pr.new/TanStack/db/@tanstack/db-react-native-sqlite-persisted-collection@1380

      @tanstack/db-sqlite-persisted-collection-core

      npm i https://pkg.pr.new/TanStack/db/@tanstack/db-sqlite-persisted-collection-core@1380

      @tanstack/db-tauri-sqlite-persisted-collection

      npm i https://pkg.pr.new/TanStack/db/@tanstack/db-tauri-sqlite-persisted-collection@1380

      @tanstack/electric-db-collection

      npm i https://pkg.pr.new/TanStack/db/@tanstack/electric-db-collection@1380

      @tanstack/offline-transactions

      npm i https://pkg.pr.new/TanStack/db/@tanstack/offline-transactions@1380

      @tanstack/powersync-db-collection

      npm i https://pkg.pr.new/TanStack/db/@tanstack/powersync-db-collection@1380

      @tanstack/query-db-collection

      npm i https://pkg.pr.new/TanStack/db/@tanstack/query-db-collection@1380

      @tanstack/react-db

      npm i https://pkg.pr.new/TanStack/db/@tanstack/react-db@1380

      @tanstack/rxdb-db-collection

      npm i https://pkg.pr.new/TanStack/db/@tanstack/rxdb-db-collection@1380

      @tanstack/solid-db

      npm i https://pkg.pr.new/TanStack/db/@tanstack/solid-db@1380

      @tanstack/svelte-db

      npm i https://pkg.pr.new/TanStack/db/@tanstack/svelte-db@1380

      @tanstack/trailbase-db-collection

      npm i https://pkg.pr.new/TanStack/db/@tanstack/trailbase-db-collection@1380

      @tanstack/vue-db

      npm i https://pkg.pr.new/TanStack/db/@tanstack/vue-db@1380

      commit: 4f3df54

    2. Size Change: +504 B (+0.46%)

      Total Size: 111 kB

      FilenameSizeChange
      ./packages/db/dist/esm/collection/state.js5.26 kB+54 B (+1.04%)
      ./packages/db/dist/esm/collection/sync.js2.88 kB+450 B (+18.5%)⚠️
      ℹ️ View Unchanged
      FilenameSize
      ./packages/db/dist/esm/collection/change-events.js1.39 kB
      ./packages/db/dist/esm/collection/changes.js1.38 kB
      ./packages/db/dist/esm/collection/cleanup-queue.js810 B
      ./packages/db/dist/esm/collection/events.js434 B
      ./packages/db/dist/esm/collection/index.js3.69 kB
      ./packages/db/dist/esm/collection/indexes.js2.35 kB
      ./packages/db/dist/esm/collection/lifecycle.js1.76 kB
      ./packages/db/dist/esm/collection/mutations.js2.47 kB
      ./packages/db/dist/esm/collection/subscription.js3.71 kB
      ./packages/db/dist/esm/collection/transaction-metadata.js144 B
      ./packages/db/dist/esm/deferred.js207 B
      ./packages/db/dist/esm/errors.js4.83 kB
      ./packages/db/dist/esm/event-emitter.js748 B
      ./packages/db/dist/esm/index.js2.85 kB
      ./packages/db/dist/esm/indexes/auto-index.js777 B
      ./packages/db/dist/esm/indexes/base-index.js766 B
      ./packages/db/dist/esm/indexes/btree-index.js2.17 kB
      ./packages/db/dist/esm/indexes/lazy-index.js1.24 kB
      ./packages/db/dist/esm/indexes/reverse-index.js538 B
      ./packages/db/dist/esm/local-only.js890 B
      ./packages/db/dist/esm/local-storage.js2.1 kB
      ./packages/db/dist/esm/optimistic-action.js359 B
      ./packages/db/dist/esm/paced-mutations.js496 B
      ./packages/db/dist/esm/proxy.js3.75 kB
      ./packages/db/dist/esm/query/builder/functions.js792 B
      ./packages/db/dist/esm/query/builder/index.js5.15 kB
      ./packages/db/dist/esm/query/builder/ref-proxy.js1.05 kB
      ./packages/db/dist/esm/query/compiler/evaluators.js1.62 kB
      ./packages/db/dist/esm/query/compiler/expressions.js430 B
      ./packages/db/dist/esm/query/compiler/group-by.js2.69 kB
      ./packages/db/dist/esm/query/compiler/index.js3.62 kB
      ./packages/db/dist/esm/query/compiler/joins.js2.11 kB
      ./packages/db/dist/esm/query/compiler/order-by.js1.5 kB
      ./packages/db/dist/esm/query/compiler/select.js1.11 kB
      ./packages/db/dist/esm/query/effect.js4.78 kB
      ./packages/db/dist/esm/query/expression-helpers.js1.43 kB
      ./packages/db/dist/esm/query/ir.js784 B
      ./packages/db/dist/esm/query/live-query-collection.js360 B
      ./packages/db/dist/esm/query/live/collection-config-builder.js7.63 kB
      ./packages/db/dist/esm/query/live/collection-registry.js264 B
      ./packages/db/dist/esm/query/live/collection-subscriber.js1.94 kB
      ./packages/db/dist/esm/query/live/internal.js145 B
      ./packages/db/dist/esm/query/live/utils.js1.57 kB
      ./packages/db/dist/esm/query/optimizer.js2.62 kB
      ./packages/db/dist/esm/query/predicate-utils.js2.97 kB
      ./packages/db/dist/esm/query/query-once.js359 B
      ./packages/db/dist/esm/query/subset-dedupe.js960 B
      ./packages/db/dist/esm/scheduler.js1.3 kB
      ./packages/db/dist/esm/SortedMap.js1.3 kB
      ./packages/db/dist/esm/strategies/debounceStrategy.js247 B
      ./packages/db/dist/esm/strategies/queueStrategy.js428 B
      ./packages/db/dist/esm/strategies/throttleStrategy.js246 B
      ./packages/db/dist/esm/transactions.js2.9 kB
      ./packages/db/dist/esm/utils.js927 B
      ./packages/db/dist/esm/utils/browser-polyfills.js304 B
      ./packages/db/dist/esm/utils/btree.js5.61 kB
      ./packages/db/dist/esm/utils/comparison.js1.05 kB
      ./packages/db/dist/esm/utils/cursor.js457 B
      ./packages/db/dist/esm/utils/index-optimization.js1.54 kB
      ./packages/db/dist/esm/utils/type-guards.js157 B
      ./packages/db/dist/esm/virtual-props.js360 B
      compressed-size-action::db-package-size
    3. Size Change: 0 B

      Total Size: 4.23 kB

      ℹ️ View Unchanged
      FilenameSize
      ./packages/react-db/dist/esm/index.js249 B
      ./packages/react-db/dist/esm/useLiveInfiniteQuery.js1.32 kB
      ./packages/react-db/dist/esm/useLiveQuery.js1.34 kB
      ./packages/react-db/dist/esm/useLiveQueryEffect.js355 B
      ./packages/react-db/dist/esm/useLiveSuspenseQuery.js559 B
      ./packages/react-db/dist/esm/usePacedMutations.js401 B
      compressed-size-action::react-db-package-size
    4. kevin-dp(GitHub profile, opens in a new tab) approved these changes

      Correctness: Does It Fix the Bug?

      Yes — the core fix is sound. The key change in query.ts is that applySuccessfulResult now computes previouslyOwnedRows from persisted ownership metadata rather than iterating collection._state.syncedData.entries():

      const previouslyOwnedRows = shouldUsePersistedBaseline
        ? new Set(persistedBaseline.keys())
        : getHydratedOwnedRowsForQueryBaseline(hashedQueryKey)

      This means an empty live query only removes rows it previously owned — not rows belonging to history. The ownership metadata (queryCollection.owners) is persisted alongside rows so it survives restarts.


      Issues Found

      MEDIUM Severity

      1. markReady() error handling removed (persisted.ts)

      The old code had a .catch() on runtime.ensureStarted() that called params.markReady() on failure:

      // Old:
      void runtime.ensureStarted()
        .then(() => { params.markReady() })
        .catch((error) => {
          console.warn(`Failed persisted sync startup before markReady:`, error)
          params.markReady()
        })
      
      // New:
      void (fullStartPromise ?? runtime.ensureStarted()).then(() => {
        params.markReady()
      })

      If ensureStarted() rejects (e.g., SQLite driver error during hydration or index bootstrap), markReady() is never called, leaving the collection stuck in a permanent loading state. The old code correctly degraded gracefully by marking ready anyway.

      2. truncate() preserves collectionMetadataWrites but clears rowMetadataWrites — undocumented asymmetry

      In both the core sync.ts and the persisted wrapper createWrappedSyncConfig, the truncate handler clears row-level state but preserves collection-level metadata writes:

      openTransaction.operations = []
      openTransaction.rowMetadataWrites.clear()
      openTransaction.truncate = true
      // collectionMetadataWrites NOT cleared

      This is intentional — the Electric resume metadata writes happen before truncate and must survive — and tests verify it explicitly. But the asymmetry is subtle and load-bearing. A comment explaining why collection metadata survives truncate would prevent future contributors from "fixing" this.

      LOW Severity

      3. ALTER TABLE ... ADD COLUMN migration wrapped in bare try {} catch {}

      try {
        await this.driver.exec(`ALTER TABLE applied_tx ADD COLUMN replay_json TEXT`)
      } catch {}

      This is a standard SQLite migration pattern (SQLite lacks ALTER TABLE ADD COLUMN IF NOT EXISTS), but the empty catch swallows all errors — not just "column already exists." If the driver has a permissions issue or the table is corrupt, the error is lost and the failure would surface later as a confusing "column not found" error.

      4. getStableShapeIdentity uses JSON.stringify for identity comparison

      function getStableShapeIdentity(shapeOptions) {
        return JSON.stringify({ url: shapeOptions.url, params: shapeOptions.params ?? null })
      }

      JSON.stringify iterates own enumerable properties in insertion order, so { table: 'x', where: 'y' } and { where: 'y', table: 'x' } produce different strings. If a user's params object is constructed with different key ordering between sessions, the persisted resume identity would mismatch and trigger a needless fresh sync. The codebase already has stableSerialize in the persisted package that handles this — consider using it here.

      5. cancelledLoads uses WeakSet<object> with reference identity

      const cancelledLoads = new WeakSet<object>()
      // In loadSubset:
      cancelledLoads.delete(options as object)
      // In unloadSubset:
      cancelledLoads.add(options as object)

      This only works if the exact same options object reference is passed to both loadSubset and unloadSubset. The collection's internal subscription management likely does pass the same reference, but the correctness depends on an undocumented invariant of the caller.


      Test Coverage Assessment

      Gaps:

      • No test for markReady when ensureStarted() rejects
      • No integration test that directly reproduces the original warm-start bug scenario (two disjoint queries with timing-dependent ordering)

Sign in to join this conversation.