Skip to content
Source
Sign inSign up
TanStack/db

Persisted Query Collection direct writes validate against rows older than a queued refetch #2029

Send to my agent
ClosedKyleAMathews(GitHub profile, opens in a new tab) opened this issue 0 comments

Problem

In a persisted Query Collection, a Query direct write can validate against rows that are older than a refetch the source has already committed.

The persistence wrapper gives a source commit() to core only when the transaction holds the wrapper's apply lock (packages/db-sqlite-persistence-core/src/persisted.ts, applyHydrationBufferedTransaction → applyMutex.run → applyBufferedSyncTransactionUnsafe → transaction.applyToCollection()). The lock can be busy for any of these reasons:

  • A durable write is slow.
  • A mutation confirmation is in progress (persistAndConfirmCollectionMutations).
  • Startup is in progress.
  • A reload is in progress.

While the lock is busy, a committed refetch waits in the lock queue. Its result is already in the Query cache, but core has not accepted it.

A direct write (writeInsert, writeUpdate, writeDelete) validates the key against core's accepted rows when it is called (packages/query-db-collection/src/manual-sync.ts). Thus it cannot see the waiting refetch. The commit of the direct write then waits behind the refetch, so the application order stays correct. Only the validation is incorrect.

Observed

Key k exists only in the waiting refetch:

CallPersisted Query CollectionQuery Collection without persistence
writeInsert(k)Passes validation. The direct write wins.Throws a duplicate-key error.
writeUpdate(k)Throws UpdateOperationItemNotFoundError. k keeps the refetch value.Succeeds.
writeDelete(k)Throws DeleteOperationItemNotFoundError. k stays until the next refetch.Succeeds.

No variant enters an error state. No variant causes SQLite and the collection to disagree.

main before the settlement-drop change shows the same validation results. Also, after writeInsert(k), its Query cache does not include k.

Reachability

The window is narrow. All of these conditions are necessary:

  1. The collection has persistence.
  2. A slow storage write or a mutation confirmation overlaps a refetch.
  3. A direct write targets a key that only the waiting refetch contains.

That key is not yet visible, so code that writes only keys it shows cannot hit this.

Options

  • Accept a source commit into core at commit(), and keep only the durable write behind the lock. The accepted view then follows commit order. Mutation confirmations are also built inside the lock, so their acceptance must move to the time they reserve the lock. If it does not, an older confirmation could overwrite newer server rows. This change touches the hydration laws and reserveCommitTurn.
  • Validate a direct write when it applies, not when it is called. The direct write already waits behind the refetch, so it would validate against the correct rows. Direct-write errors would change from a synchronous throw to a rejected receipt. This is a public API change.

Test gap

The Query ownership-lifecycle oracle (packages/query-db-collection/tests/ownership-lifecycle.oracle.test.ts) does not interleave direct writes with a persisted refetch that waits on a busy apply lock. A fix should add that generator dimension for each kind of direct write and each holder of the lock.

Sign in to join this conversation.