packages/db-sqlite-persistence-core
| Name | Last commit message | Last commit date |
|---|---|---|
| .. | ||
README.md
@tanstack/db-sqlite-persistence-core#
Shared SQLite persistence primitives for TanStack DB. Runtime packages provide the database drivers. Start with the SQLite Persistence guide if you are adding persistence to an application.
What this package provides#
- Generic persisted collection wrapper utilities
- Shared persistence/coordinator protocol types
- SQLite core persistence adapter (
createSQLiteCorePersistenceAdapter) - Validation and storage key helpers
- Shared error types
This package intentionally does not include a concrete SQLite engine
binding. Provide a runtime SQLiteDriver implementation from a wrapper package.
Exported API (complete)#
Persisted wrapper and protocol APIs#
PersistedMutationEnvelopeProtocolEnvelope<TPayload>LeaderHeartbeatTxCommittedEnsureRemoteSubsetRequestEnsureRemoteSubsetResponseReleaseRemoteSubsetRequestReleaseRemoteSubsetResponseRemoteSubsetOwnerDuplicateRemoteSubsetOwnerErrorTransportedLoadSubsetOptionsRemoteSubsetWirePrimitiveRemoteSubsetWireTypedArrayRemoteSubsetWireRecordRemoteSubsetWireValueRemoteSubsetWireExpressionRemoteSubsetWireCompareOptionsRemoteSubsetWireOrderByClauseRemoteSubsetWireCursorRemoteSubsetWireValueErrortoTransportedLoadSubsetOptions(...)ApplyLocalMutationsRequestApplyLocalMutationsResponseApplyCommittedTxRequestApplyCommittedTxResponsePullSinceRequestPullSinceResponseCollectionResetPersistedIndexSpecPersistedRowMetadataMutation<TKey>PersistedCollectionMetadataMutationReplayableTxDelta<T, TKey>PersistedScannedRow<T, TKey>PersistedRowScanOptionsPersistedKeySetEvidencePersistedTx<T, TKey>PersistenceAdapterSQLiteDriverPersistedCollectionCoordinatorPersistedCollectionPersistencePersistedCollectionModePersistedCollectionLeadershipStatePersistedCollectionUtilsPersistedSyncWrappedOptions<T, TKey, TSchema, TUtils>PersistedLocalOnlyOptions<T, TKey, TSchema, TUtils>SingleProcessCoordinatorvalidatePersistedCollectionCoordinator(...)persistedCollectionOptions(...)encodePersistedStorageKey(...)decodePersistedStorageKey(...)createPersistedTableName(...)
PersistedCollectionPersistence can optionally implement:
resolvePersistenceForMode(mode)(legacy mode-aware resolution)resolvePersistenceForCollection({ collectionId, mode, schemaVersion })(collection-aware resolution)
persistedCollectionOptions(...) now supports schemaVersion per collection
and resolves persistence using:
- collection id
- inferred mode (
sync-presentorsync-absent) - optional
schemaVersion
This lets runtime wrappers expose one shared persistence instance per database while still handling per-collection schema versions correctly.
Network-first initial rendering#
Set initialRender on every eager persisted source Collection in a query to
let React or Solid Suspense render restored SQLite rows when the network is
slow or unavailable:
const todos = createCollection(
persistedCollectionOptions({
id: 'todos',
getKey: (todo) => todo.id,
persistence,
sync,
initialRender: {
strategy: 'network-first',
networkTimeoutMs: 3_000,
},
}),
)
const result = useLiveQuery((q) => q.from({ todos }))
if (result.isPersistedReady) {
// The eager persisted restore completed, even if the network is pending.
}
The live-query observer and framework hooks expose persistedStatus
(unavailable, loading, ready, or error), isPersistedReady, and
persistedError. ready means every source Collection in the query opted in
and completed its current eager persisted restore; an empty database also
counts as ready. One unconfigured source makes the query's persisted status
unavailable. A failed restore reports its original error through
persistedError. An SSR seed does not count as persisted restore.
The initial render prefers Collection readiness. If the network has not made
the query ready after networkTimeoutMs, Suspense may use a completed
persisted restore. A network failure allows that fallback sooner. If the
restore is still loading, the query waits for it or for the network. The
deadline never cancels network sync. The default deadline is 3 seconds;
networkTimeoutMs: 0 allows fallback as soon as restore finishes. For a query
with multiple opted-in sources, the longest deadline applies. An unconfigured
source keeps the existing network-readiness gate. A query failure that did not
come from a source is still an error, not a network fallback. Ordinary
useLiveQuery row updates and Collection status, isReady, and preload() are unchanged;
status is a Collection lifecycle signal, not a pure network-health signal.
In React, a client query stream failure before the first render also allows a completed persisted restore to release Suspense. React waits if the stream fails before restore completes, including when it failed before the component mounted. The derived query result can be empty. A derived-query failure still reaches the Error Boundary.
Persisted readiness does not prove current authorization. A source error can
permit restored rows to render even when that error reports an authorization
failure. Applications with user- or tenant-specific data must isolate persisted
storage by identity and clear or replace it when the identity changes. If a
policy denial must hide cached rows, gate the view separately; initialRender
does not classify source errors.
initialRender is rejected with syncMode: 'on-demand', since an on-demand
Collection has no complete startup snapshot to certify.
Coordinator contract#
Every PersistedCollectionCoordinator must implement
requestApplyCommittedTx(collectionId, tx). A sync transaction with durable
effects is routed in full to the persistence adapter owned by that collection.
The PersistedTx includes any truncate, row mutations, row metadata mutations,
collection metadata mutations, and stream position. Multiprocess coordinators
must preserve the complete transaction when they forward it to the elected
writer. An effect-free source commit does not allocate a stream position,
publish an invalidation, or call persistence; the next durable transaction
retains the next coordinator-owned sequence.
This is a required invariant, not a capability to detect at commit time. The
TypeScript interface requires the method, and persistedCollectionOptions(...)
validates custom coordinators when it configures a collection. An untyped
integration that omits the method throws
InvalidPersistedCollectionCoordinatorError before the sync source starts or
publishes rows. Implementations must not fall back to a row-only route such as
requestApplyLocalMutations(...), because that would discard transaction
semantics.
After a mutating RPC transport failure, replay is allowed only while the
requester can prove the same non-null leader id and term still own the route.
If either value was unknown for the first attempt, or either value changes,
the direct request rejects with IndeterminateCommitError and requires
application reconciliation. Coordinators never retry that mutation against an
unknown or replacement leader.
A source sync transaction has a separate exact-ID reconciliation path. The
persisted wrapper holds its applied receipt while an elected coordinator and
SQLite adapter check the transaction under the writer lock. An existing ID
acknowledges the original durable position. An absent ID permits one application
only if the durable row version and reset epoch equal the wrapper's pre-send
snapshot; another durable write may carry newer row values or an opaque source
cursor and cannot be overwritten safely. On a certified result the wrapper
fulfills the receipt in the same sync run and the coordinator invalidates peers,
including a peer that missed the original notification. An uncertified result
retains IndeterminateCommitError and terminal source failure. Custom adapters
and coordinators without this optional operation keep that terminal behavior.
One generated txId identifies one immutable source transaction payload.
SingleProcessCoordinator also implements the complete route. It is always
leader and has no cross-process transport, but it applies the transaction to
the resolved adapter registered for the specific collection. The Browser and
Electron coordinators provide the corresponding elected-owner route in their
runtime packages.
An adapter/storage rejection is surfaced as
PersistedCollectionDurabilityError, never as CONFLICT. The error preserves
the original local cause and its code and path fields. A transported
PERSISTENCE_ERROR response carries the source code/path that can cross the
coordinator boundary. Source writes remain published before their applied
receipt settles; if persistence then fails, the receipt rejects with the named
error and the collection enters its existing error lifecycle state. The
collection does not continue, retry, or fall back to a partial writer.
Remote subset wire contract#
requestEnsureRemoteSubset(collectionId, options) accepts the full live
LoadSubsetOptions input. Before choosing a local owner or remote transport,
every built-in coordinator projects it once to the exported
TransportedLoadSubsetOptions contract. A registered remote-subset owner
receives that transported type, never live-only fields such as signal or
subscription.
Expression values may contain structured-clone-safe primitives, plain records,
arrays (including sparse arrays), Date, zero-position RegExp, fixed
ArrayBuffer, fixed DataView, the views listed by the exported
RemoteSubsetWireTypedArray union (Int8Array, Uint8Array,
Uint8ClampedArray, Int16Array, Uint16Array, Int32Array, Uint32Array,
Float32Array, Float64Array, BigInt64Array, and BigUint64Array), Map,
and Set. Cycles and aliases are preserved. Other view types; shared,
resizable, growable, or detached buffers; accessors; non-enumerable or symbol
keys; native expandos; custom prototypes; promises; weak collections;
functions; and symbols are outside the contract. A rejected value throws the exported
RemoteSubsetWireValueError before owner work or transport, with its exact
path in the request. The wire contract does not provide codecs, coercion, or
lossy normalization for unsupported values.
Remote subset ownership is lease-based. A coordinator gives each accepted
request object a stable acquisition identity scoped to its collection and
requester. Retrying the same acquisition coalesces with the original; a second
request object with equal values is an independent lease. Releasing a lease
calls the registered owner's required unloadSubset(...) method with the exact
transported options object that was loaded. Releasing it again is a no-op.
EnsureRemoteSubsetRequest.acquisitionId is the wire identity established by
acquisition; ReleaseRemoteSubsetRequest.acquisitionId must carry that same
identity so the elected owner releases the matching lease.
The owner also provides a required onError(error) callback. A load or unload
throw by the actual elected owner, or a runtime thenable rejection from either
operation, is reported through that callback with the original value before
the awaited request rejects. A follower transport or remote-owner admission
failure rejects the acquisition and remains eligible for the existing demand
retry; it is not a failure of that follower's local owner lifecycle. Retirement
and disposal observe and report actual owner failures while continuing cleanup
of sibling leases; they do not retry or silently recover the failed lifecycle.
Only one remote subset owner may be registered per collection in a coordinator.
A second live registration throws DuplicateRemoteSubsetOwnerError instead of
replacing the first, and startup reports the named failure before readiness
rather than leaving a rejected registration promise unobserved. Released
acquisition identities remain as dedupe tombstones for the coordinator's
existing RPC dedupe horizon, then expire during later subset traffic. When a
leader loses ownership or is disposed, it unloads every transferred lease. A
requester that still owns a lease replays that same acquisition when the next
leader is observed. There is no adapter fallback or direct-writer escape hatch
for missing ownership.
The named-error and exact-path guarantee covers values the boundary can
identify with standard JavaScript reflection. A fully transparent Proxy is
outside the public wire type and cannot be portably distinguished from its
target. Untyped callers must not rely on Proxy identity, traps, or a
Proxy-specific diagnostic; no non-standard detection is attempted.
Source replay ordering#
The persisted wrapper buffers authoritative source sync transactions and
replays them into the Collection in FIFO order, one transaction at a time.
Local optimistic mutations remain visible immediately; this queue governs
source publication and its durable suffix. A source may mark a transaction
immediate when it must update the authoritative base beneath an active
optimistic layer. The wrapper still gives that transaction one FIFO turn and
writes its durable suffix before admitting the next source turn.
Persisted rows establish a baseline for newer changes from the sync adapter.
Once an ordinary sync transaction is accepted at commit(), queued reads of
that baseline cannot revoke acceptance. This includes subset reads and rereads
requested by coordinator reset or full-reload notifications. This rule assumes
a compatible baseline; it does not authorize using an invalidated resume point.
A truncate replay replaces earlier source rows in the same sync run. Sync
transactions ordered after that replacement apply to the replacement.
An immediate sync transaction cannot enter while an earlier normal source
publication is waiting. The wrapper also currently rejects a sync transaction
that remains open across another persistence read. These admission restrictions
throw InvalidPersistedCollectionConfigError; an internal wait after accepted
commit() is not an admission failure. The open-transaction restriction is
separate from the accepted-transaction precedence rule.
Atomic resume snapshots#
Persistence adapters implement
loadResumeSnapshot(target, options) to let a sync source certify a
persisted resume baseline. One call must read rows, collection metadata, stream
position, reset epoch, and key-set evidence from the same atomic database
snapshot. includeRows: false requests the same certification data without
materializing rows; requiredIndexSignatures carries the indexes needed by a
row-bearing snapshot.
The adapter's data methods require an explicit PersistedStorageTarget. Use
{ kind: 'eager', collectionId } for an eager Collection, or
{ kind: 'managed', storageCollectionId, claimId } for a claimed on-demand
persisted cache generation. A bare string ID is rejected. The managed target
checks the claim even after its generation has been collected from SQLite;
choosing an eager target explicitly may reuse the same ID. Claim and
rotation methods take the logical Collection ID; renewal takes the physical
storage ID and release takes the claim ID.
PersistedKeySetEvidence.status has three states:
consistent: the persisted rows match the adapter's durable expected-key ledger.incompatible: row loss, substitution, or a reset-generation change makes the saved resume baseline unsafe.unknown: the adapter has no authoritative pre-migration key set and does not claim completeness.
A no-write election can reserve a greater durable term and reset its sequence to zero without changing persisted rows or the source resume cursor. The persisted wrapper keeps an already certified resume baseline across that transition only when the atomic snapshot has the same row version and reset epoch. A committed write or schema reset requires fresh certification and cannot inherit the earlier baseline merely because the key set still matches.
The method is required because the versioned metadata.persistence capability
always carries hydration, durable row scanning, certification, evidence, and
generation ownership as one complete bundle. Sync wrappers must forward the
capability object unchanged rather than copying individual methods. Adapter
methods are invoked with their receiver and may rely on instance state through
this.
SyncMetadataApi.persistence is always present. Core sync sources receive
null, which explicitly means that no persistence bridge is active. The
persisted wrapper replaces that sentinel with the complete versioned
capability. A wrapper that omits the property is invalid and fails before a
consumer can resume or query against uncertified durable state.
SQLite core adapter APIs#
SQLiteCoreAdapterOptionsSQLitePullSinceResult<TKey>SQLiteCorePersistenceAdapterDEFAULT_APPLIED_TX_PRUNE_MAX_ROWSDEFAULT_APPLIED_TX_PRUNE_MAX_AGE_SECONDScreateSQLiteCorePersistenceAdapter(...)
Error APIs#
PersistedCollectionCoreErrorInvalidPersistedCollectionConfigErrorInvalidSyncConfigErrorInvalidPersistedCollectionCoordinatorErrorInvalidPersistenceAdapterErrorPersistedCollectionDurabilityErrorOptionsPersistedCollectionDurabilityErrortoPersistedCollectionDurabilityError(...)IndeterminateCommitRequestTypeIndeterminateCommitErrorOptionsIndeterminateCommitErrorInvalidPersistedStorageKeyErrorInvalidPersistedStorageKeyEncodingErrorPersistenceUnavailableError
Adapter and coordinator commit failures reject the applied receipt and become
the collection's terminal sync error as PersistedCollectionDurabilityError.
That error preserves its original cause, code, and path. Hydration and
source errors remain distinct and are not wrapped as durability errors.
Typical usage (via runtime wrappers)#
In most applications, use a runtime package directly:
@tanstack/node-db-sqlite-persistence@tanstack/browser-db-sqlite-persistence@tanstack/electron-db-sqlite-persistence@tanstack/react-native-db-sqlite-persistence@tanstack/expo-db-sqlite-persistence@tanstack/capacitor-db-sqlite-persistence@tanstack/tauri-db-sqlite-persistence@tanstack/cloudflare-durable-objects-db-sqlite-persistence
Those packages provide concrete drivers and runtime wiring.