feat: add persisted sync metadata support #1380
main fromrefs/pull/1380/head·merged by samwillis(GitHub profile, opens in a new tab) Summary#
- add transactional persisted sync metadata support across
@tanstack/db,db-sqlite-persisted-collection-core,query-db-collection, andelectric-db-collection - persist row-scoped metadata with synced rows and collection-scoped metadata alongside persisted collection state
- use that metadata to preserve query ownership/retention across restart and to persist Electric resume state
How To Use#
Custom sync implementations#
If a collection is running on a persistence layer that supports this feature, the sync function now receives an optional metadata API alongside begin, write, commit, markReady, and truncate.
sync: ({ begin, write, commit, metadata }) => {
const startupValue = metadata?.collection.get('my-sync:startup')
begin()
write({
type: 'update',
value: row,
metadata: { source: 'remote' },
})
metadata?.collection.set('my-sync:startup', { ready: true })
commit()
}
Available operations:
metadata.row.get(key)metadata.row.set(key, value)metadata.row.delete(key)metadata.collection.get(key)metadata.collection.set(key, value)metadata.collection.delete(key)metadata.collection.list(prefix?)
Behavior:
- startup reads through
get/listare allowed before any transaction is opened - metadata writes are transactional and must happen inside
begin()/commit() write({ metadata })andmetadata.row.set()target the same row metadata store
SQLite persisted collections#
persistedCollectionOptions(...) now persists both row metadata and collection metadata when used with db-sqlite-persisted-collection-core.
Typical usage looks like:
const collection = createCollection(
persistedCollectionOptions({
...someCollectionOptions,
persistence: {
adapter: sqliteAdapter,
},
}),
)
What you get automatically:
- row metadata hydrates with persisted rows
- collection metadata is loaded during startup before persisted sync work begins
- row data, row metadata, and collection metadata commit in the same persisted transaction
Query collections#
Query collections can now retain persisted ownership metadata across restarts.
const collection = createCollection(
persistedCollectionOptions({
...queryCollectionOptions({
id: 'messages',
queryClient,
queryKey: ['messages', roomId],
queryFn: fetchMessages,
getKey: (row) => row.id,
persistedGcTime: Number.POSITIVE_INFINITY,
}),
persistence: {
adapter: sqliteAdapter,
},
}),
)
What this enables:
- row ownership is persisted on rows, so warm starts do not incorrectly delete disjoint pre-hydrated rows
- query retention is separate from in-memory TanStack Query
gcTime persistedGcTimecan be finite or effectively indefinite for offline-first flows- when a retained query is requested again, reconciliation diffing uses that query's owned-row baseline instead of the whole collection
Electric collections#
Electric collections now persist resume metadata at collection scope when wrapped with persisted collection options.
const collection = createCollection(
persistedCollectionOptions({
...electricCollectionOptions({
id: 'todos',
getKey: (row) => row.id,
shapeOptions: {
url: electricUrl,
params: { table: 'todos' },
},
}),
persistence: {
adapter: sqliteAdapter,
},
}),
)
What this enables:
- persisted startup can reuse a saved Electric
offset/handleresume point - resume metadata is committed transactionally with the batch that made it valid
- must-refetch/reset paths write a reset marker before reloading so restart does not resume from stale stream state
Test plan#
- Run
pnpm vitest --run packages/db/tests/collection.test.ts - Run
pnpm --filter @tanstack/query-db-collection exec vitest run tests/query.test.ts --coverage.enabled false - Run
pnpm --filter @tanstack/electric-db-collection exec vitest run tests/electric.test.ts --coverage.enabled false - Run
pnpm --filter @tanstack/db-sqlite-persisted-collection-core exec vitest run tests/persisted.test.ts tests/sqlite-core-adapter.test.ts --coverage.enabled false
Notes#
- The persisted SQLite package tests are functionally green in package-local runs.
- There is still noisy stderr in some query tests from expected error-path coverage and existing
cleanupQueryIfIdlewarnings.
- 1
Size Change: +504 B (+0.46%)
Total Size: 111 kB
Filename Size Change ./packages/db/dist/esm/collection/state.js5.26 kB +54 B (+1.04%) ./packages/db/dist/esm/collection/sync.js2.88 kB +450 B (+18.5%) ⚠️ compressed-size-action::db-package-sizeℹ️ View Unchanged
Filename Size ./packages/db/dist/esm/collection/change-events.js1.39 kB ./packages/db/dist/esm/collection/changes.js1.38 kB ./packages/db/dist/esm/collection/cleanup-queue.js810 B ./packages/db/dist/esm/collection/events.js434 B ./packages/db/dist/esm/collection/index.js3.69 kB ./packages/db/dist/esm/collection/indexes.js2.35 kB ./packages/db/dist/esm/collection/lifecycle.js1.76 kB ./packages/db/dist/esm/collection/mutations.js2.47 kB ./packages/db/dist/esm/collection/subscription.js3.71 kB ./packages/db/dist/esm/collection/transaction-metadata.js144 B ./packages/db/dist/esm/deferred.js207 B ./packages/db/dist/esm/errors.js4.83 kB ./packages/db/dist/esm/event-emitter.js748 B ./packages/db/dist/esm/index.js2.85 kB ./packages/db/dist/esm/indexes/auto-index.js777 B ./packages/db/dist/esm/indexes/base-index.js766 B ./packages/db/dist/esm/indexes/btree-index.js2.17 kB ./packages/db/dist/esm/indexes/lazy-index.js1.24 kB ./packages/db/dist/esm/indexes/reverse-index.js538 B ./packages/db/dist/esm/local-only.js890 B ./packages/db/dist/esm/local-storage.js2.1 kB ./packages/db/dist/esm/optimistic-action.js359 B ./packages/db/dist/esm/paced-mutations.js496 B ./packages/db/dist/esm/proxy.js3.75 kB ./packages/db/dist/esm/query/builder/functions.js792 B ./packages/db/dist/esm/query/builder/index.js5.15 kB ./packages/db/dist/esm/query/builder/ref-proxy.js1.05 kB ./packages/db/dist/esm/query/compiler/evaluators.js1.62 kB ./packages/db/dist/esm/query/compiler/expressions.js430 B ./packages/db/dist/esm/query/compiler/group-by.js2.69 kB ./packages/db/dist/esm/query/compiler/index.js3.62 kB ./packages/db/dist/esm/query/compiler/joins.js2.11 kB ./packages/db/dist/esm/query/compiler/order-by.js1.5 kB ./packages/db/dist/esm/query/compiler/select.js1.11 kB ./packages/db/dist/esm/query/effect.js4.78 kB ./packages/db/dist/esm/query/expression-helpers.js1.43 kB ./packages/db/dist/esm/query/ir.js784 B ./packages/db/dist/esm/query/live-query-collection.js360 B ./packages/db/dist/esm/query/live/collection-config-builder.js7.63 kB ./packages/db/dist/esm/query/live/collection-registry.js264 B ./packages/db/dist/esm/query/live/collection-subscriber.js1.94 kB ./packages/db/dist/esm/query/live/internal.js145 B ./packages/db/dist/esm/query/live/utils.js1.57 kB ./packages/db/dist/esm/query/optimizer.js2.62 kB ./packages/db/dist/esm/query/predicate-utils.js2.97 kB ./packages/db/dist/esm/query/query-once.js359 B ./packages/db/dist/esm/query/subset-dedupe.js960 B ./packages/db/dist/esm/scheduler.js1.3 kB ./packages/db/dist/esm/SortedMap.js1.3 kB ./packages/db/dist/esm/strategies/debounceStrategy.js247 B ./packages/db/dist/esm/strategies/queueStrategy.js428 B ./packages/db/dist/esm/strategies/throttleStrategy.js246 B ./packages/db/dist/esm/transactions.js2.9 kB ./packages/db/dist/esm/utils.js927 B ./packages/db/dist/esm/utils/browser-polyfills.js304 B ./packages/db/dist/esm/utils/btree.js5.61 kB ./packages/db/dist/esm/utils/comparison.js1.05 kB ./packages/db/dist/esm/utils/cursor.js457 B ./packages/db/dist/esm/utils/index-optimization.js1.54 kB ./packages/db/dist/esm/utils/type-guards.js157 B ./packages/db/dist/esm/virtual-props.js360 B Size Change: 0 B
Total Size: 4.23 kB
compressed-size-action::react-db-package-sizeℹ️ View Unchanged
Filename Size ./packages/react-db/dist/esm/index.js249 B ./packages/react-db/dist/esm/useLiveInfiniteQuery.js1.32 kB ./packages/react-db/dist/esm/useLiveQuery.js1.34 kB ./packages/react-db/dist/esm/useLiveQueryEffect.js355 B ./packages/react-db/dist/esm/useLiveSuspenseQuery.js559 B ./packages/react-db/dist/esm/usePacedMutations.js401 B kevin-dp(GitHub profile, opens in a new tab) approved these changes
Correctness: Does It Fix the Bug?#
Yes — the core fix is sound. The key change in
query.tsis thatapplySuccessfulResultnow computespreviouslyOwnedRowsfrom persisted ownership metadata rather than iteratingcollection._state.syncedData.entries():const previouslyOwnedRows = shouldUsePersistedBaseline ? new Set(persistedBaseline.keys()) : getHydratedOwnedRowsForQueryBaseline(hashedQueryKey)This means an empty
livequery only removes rows it previously owned — not rows belonging tohistory. The ownership metadata (queryCollection.owners) is persisted alongside rows so it survives restarts.
Issues Found#
MEDIUM Severity#
1.
markReady()error handling removed (persisted.ts)The old code had a
.catch()onruntime.ensureStarted()that calledparams.markReady()on failure:// Old: void runtime.ensureStarted() .then(() => { params.markReady() }) .catch((error) => { console.warn(`Failed persisted sync startup before markReady:`, error) params.markReady() }) // New: void (fullStartPromise ?? runtime.ensureStarted()).then(() => { params.markReady() })If
ensureStarted()rejects (e.g., SQLite driver error during hydration or index bootstrap),markReady()is never called, leaving the collection stuck in a permanent loading state. The old code correctly degraded gracefully by marking ready anyway.2.
truncate()preservescollectionMetadataWritesbut clearsrowMetadataWrites— undocumented asymmetryIn both the core
sync.tsand the persisted wrappercreateWrappedSyncConfig, thetruncatehandler clears row-level state but preserves collection-level metadata writes:openTransaction.operations = [] openTransaction.rowMetadataWrites.clear() openTransaction.truncate = true // collectionMetadataWrites NOT clearedThis is intentional — the Electric resume metadata writes happen before truncate and must survive — and tests verify it explicitly. But the asymmetry is subtle and load-bearing. A comment explaining why collection metadata survives truncate would prevent future contributors from "fixing" this.
LOW Severity#
3.
ALTER TABLE ... ADD COLUMNmigration wrapped in baretry {} catch {}try { await this.driver.exec(`ALTER TABLE applied_tx ADD COLUMN replay_json TEXT`) } catch {}This is a standard SQLite migration pattern (SQLite lacks
ALTER TABLE ADD COLUMN IF NOT EXISTS), but the empty catch swallows all errors — not just "column already exists." If the driver has a permissions issue or the table is corrupt, the error is lost and the failure would surface later as a confusing "column not found" error.4.
getStableShapeIdentityusesJSON.stringifyfor identity comparisonfunction getStableShapeIdentity(shapeOptions) { return JSON.stringify({ url: shapeOptions.url, params: shapeOptions.params ?? null }) }JSON.stringifyiterates own enumerable properties in insertion order, so{ table: 'x', where: 'y' }and{ where: 'y', table: 'x' }produce different strings. If a user'sparamsobject is constructed with different key ordering between sessions, the persisted resume identity would mismatch and trigger a needless fresh sync. The codebase already hasstableSerializein the persisted package that handles this — consider using it here.5.
cancelledLoadsusesWeakSet<object>with reference identityconst cancelledLoads = new WeakSet<object>() // In loadSubset: cancelledLoads.delete(options as object) // In unloadSubset: cancelledLoads.add(options as object)This only works if the exact same
optionsobject reference is passed to bothloadSubsetandunloadSubset. The collection's internal subscription management likely does pass the same reference, but the correctness depends on an undocumented invariant of the caller.
Test Coverage Assessment#
Gaps:
- No test for
markReadywhenensureStarted()rejects - No integration test that directly reproduces the original warm-start bug scenario (two disjoint queries with timing-dependent ordering)
- No test for
samwillis(GitHub profile, opens in a new tab) closed this
Sign in to join this conversation.
More templates
@tanstack/angular-db
@tanstack/db
@tanstack/db-browser-wa-sqlite-persisted-collection
@tanstack/db-capacitor-sqlite-persisted-collection
@tanstack/db-cloudflare-do-sqlite-persisted-collection
@tanstack/db-electron-sqlite-persisted-collection
@tanstack/db-expo-sqlite-persisted-collection
@tanstack/db-ivm
@tanstack/db-node-sqlite-persisted-collection
@tanstack/db-react-native-sqlite-persisted-collection
@tanstack/db-sqlite-persisted-collection-core
@tanstack/db-tauri-sqlite-persisted-collection
@tanstack/electric-db-collection
@tanstack/offline-transactions
@tanstack/powersync-db-collection
@tanstack/query-db-collection
@tanstack/react-db
@tanstack/rxdb-db-collection
@tanstack/solid-db
@tanstack/svelte-db
@tanstack/trailbase-db-collection
@tanstack/vue-db
commit:
4f3df54