Virtual props implementation #1213
main fromrefs/pull/1213/head·merged by samwillis(GitHub profile, opens in a new tab) 🎯 Changes#
This PR implements the virtual properties system as described in the RFC. This introduces computed, read-only metadata fields prefixed with $ to every row, providing information about its sync status, origin, key, and source collection ID.
Key aspects of this implementation:
$synced: Indicates whether a row reflects confirmed state from the backend (true) or has pending optimistic mutations (false).$origin: Specifies if the last confirmed change originated from the local client ('local') or was received via sync ('remote').$key: The primary identifier of the row.$collectionId: The ID of the collection the row originated from.- Pass-through semantics: Virtual properties are preserved and passed through in live queries and nested collections using an "add-if-missing" pattern.
- Queryability: Virtual properties are available for use in query builder expressions (e.g.,
where(({ item }) => eq(item.$synced, true))). - Type Integration: Updated
RefProxyandReftypes to include virtual properties, ensuring type safety and discoverability.
This feature enhances data visibility and enables more sophisticated UI logic based on the state and origin of data.
✅ Checklist#
- I have followed the steps in the Contributing guide.
- I have tested this code locally with
pnpm test:pr.
🚀 Release Impact#
- This change affects published code, and I have generated a changeset.
- This change is docs/CI/dev-only (no release).


- 1
🦋 Changeset detected#
Latest commit: 9e7dfb65a8c902f9edcec1998c58ae350efe63b7
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 12 packages
Name Type @tanstack/db Patch @tanstack/angular-db Patch @tanstack/electric-db-collection Patch @tanstack/offline-transactions Patch @tanstack/powersync-db-collection Patch @tanstack/query-db-collection Patch @tanstack/react-db Patch @tanstack/rxdb-db-collection Patch @tanstack/solid-db Patch @tanstack/svelte-db Patch @tanstack/trailbase-db-collection Patch @tanstack/vue-db Patch Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR
More templates
@tanstack/angular-db
npm i https://pkg.pr.new/@tanstack/angular-db@1213@tanstack/db
npm i https://pkg.pr.new/@tanstack/db@1213@tanstack/db-ivm
npm i https://pkg.pr.new/@tanstack/db-ivm@1213@tanstack/electric-db-collection
npm i https://pkg.pr.new/@tanstack/electric-db-collection@1213@tanstack/offline-transactions
npm i https://pkg.pr.new/@tanstack/offline-transactions@1213@tanstack/powersync-db-collection
npm i https://pkg.pr.new/@tanstack/powersync-db-collection@1213@tanstack/query-db-collection
npm i https://pkg.pr.new/@tanstack/query-db-collection@1213@tanstack/react-db
npm i https://pkg.pr.new/@tanstack/react-db@1213@tanstack/rxdb-db-collection
npm i https://pkg.pr.new/@tanstack/rxdb-db-collection@1213@tanstack/solid-db
npm i https://pkg.pr.new/@tanstack/solid-db@1213@tanstack/svelte-db
npm i https://pkg.pr.new/@tanstack/svelte-db@1213@tanstack/trailbase-db-collection
npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@1213@tanstack/vue-db
npm i https://pkg.pr.new/@tanstack/vue-db@1213commit:
9e7dfb6Size Change: +3.33 kB (+3.2%)
Total Size: 107 kB
Filename Size Change ./packages/db/dist/esm/collection/changes.js1.38 kB +157 B (+12.83%) ⚠️ ./packages/db/dist/esm/collection/index.js3.45 kB +126 B (+3.79%) ./packages/db/dist/esm/collection/mutations.js2.47 kB +135 B (+5.77%) 🔍 ./packages/db/dist/esm/collection/state.js5.2 kB +1.72 kB (+49.23%) 🚨 ./packages/db/dist/esm/collection/sync.js2.43 kB +24 B (+1%) ./packages/db/dist/esm/collection/transaction-metadata.js144 B +144 B (new file) 🆕 ./packages/db/dist/esm/index.js2.78 kB +29 B (+1.05%) ./packages/db/dist/esm/indexes/auto-index.js777 B +35 B (+4.72%) 🔍 ./packages/db/dist/esm/local-only.js890 B +82 B (+10.15%) ⚠️ ./packages/db/dist/esm/query/compiler/group-by.js2.69 kB +339 B (+14.41%) ⚠️ ./packages/db/dist/esm/query/compiler/index.js3.62 kB +122 B (+3.49%) ./packages/db/dist/esm/query/live/collection-config-builder.js7.63 kB +28 B (+0.37%) ./packages/db/dist/esm/utils/index-optimization.js1.54 kB +36 B (+2.39%) ./packages/db/dist/esm/virtual-props.js360 B +360 B (new file) 🆕 compressed-size-action::db-package-sizeℹ️ View Unchanged
Filename Size ./packages/db/dist/esm/collection/change-events.js1.39 kB ./packages/db/dist/esm/collection/events.js388 B ./packages/db/dist/esm/collection/indexes.js1.1 kB ./packages/db/dist/esm/collection/lifecycle.js1.75 kB ./packages/db/dist/esm/collection/subscription.js3.71 kB ./packages/db/dist/esm/deferred.js207 B ./packages/db/dist/esm/errors.js4.83 kB ./packages/db/dist/esm/event-emitter.js748 B ./packages/db/dist/esm/indexes/base-index.js766 B ./packages/db/dist/esm/indexes/btree-index.js2.17 kB ./packages/db/dist/esm/indexes/lazy-index.js1.1 kB ./packages/db/dist/esm/indexes/reverse-index.js538 B ./packages/db/dist/esm/local-storage.js2.1 kB ./packages/db/dist/esm/optimistic-action.js359 B ./packages/db/dist/esm/paced-mutations.js496 B ./packages/db/dist/esm/proxy.js3.75 kB ./packages/db/dist/esm/query/builder/functions.js792 B ./packages/db/dist/esm/query/builder/index.js5.15 kB ./packages/db/dist/esm/query/builder/ref-proxy.js1.05 kB ./packages/db/dist/esm/query/compiler/evaluators.js1.43 kB ./packages/db/dist/esm/query/compiler/expressions.js430 B ./packages/db/dist/esm/query/compiler/joins.js2.11 kB ./packages/db/dist/esm/query/compiler/order-by.js1.5 kB ./packages/db/dist/esm/query/compiler/select.js1.11 kB ./packages/db/dist/esm/query/effect.js4.78 kB ./packages/db/dist/esm/query/expression-helpers.js1.43 kB ./packages/db/dist/esm/query/ir.js784 B ./packages/db/dist/esm/query/live-query-collection.js360 B ./packages/db/dist/esm/query/live/collection-registry.js264 B ./packages/db/dist/esm/query/live/collection-subscriber.js1.94 kB ./packages/db/dist/esm/query/live/internal.js145 B ./packages/db/dist/esm/query/live/utils.js1.57 kB ./packages/db/dist/esm/query/optimizer.js2.62 kB ./packages/db/dist/esm/query/predicate-utils.js2.97 kB ./packages/db/dist/esm/query/query-once.js359 B ./packages/db/dist/esm/query/subset-dedupe.js960 B ./packages/db/dist/esm/scheduler.js1.3 kB ./packages/db/dist/esm/SortedMap.js1.3 kB ./packages/db/dist/esm/strategies/debounceStrategy.js247 B ./packages/db/dist/esm/strategies/queueStrategy.js428 B ./packages/db/dist/esm/strategies/throttleStrategy.js246 B ./packages/db/dist/esm/transactions.js2.9 kB ./packages/db/dist/esm/utils.js927 B ./packages/db/dist/esm/utils/browser-polyfills.js304 B ./packages/db/dist/esm/utils/btree.js5.61 kB ./packages/db/dist/esm/utils/comparison.js1.05 kB ./packages/db/dist/esm/utils/cursor.js457 B ./packages/db/dist/esm/utils/type-guards.js157 B Size Change: 0 B
Total Size: 4.23 kB
compressed-size-action::react-db-package-sizeℹ️ View Unchanged
Filename Size ./packages/react-db/dist/esm/index.js249 B ./packages/react-db/dist/esm/useLiveInfiniteQuery.js1.32 kB ./packages/react-db/dist/esm/useLiveQuery.js1.34 kB ./packages/react-db/dist/esm/useLiveQueryEffect.js355 B ./packages/react-db/dist/esm/useLiveSuspenseQuery.js559 B ./packages/react-db/dist/esm/usePacedMutations.js401 B samwillis(GitHub profile, opens in a new tab) changed the title
kevin-dp(GitHub profile, opens in a new tab) requested changes
We're adding a bunch of meta properties $key, $origin, etc. Each of these could conflict with user properties. I'd like to minimize potential conflicts by having a single
$metaproperty and having all these props nested under it, so$meta.synced,$meta.origin, etc.
I reviewed this big PR with Claude. Here's our feedback:
Semantic concerns with virtual props in joins and aggregations#
Nice work on the virtual props system — the core idea of
$synced,$origin,$key, and$collectionIdis really useful for single-collection queries. However, I think the "pass-through / add-if-missing" semantics break down in two important cases.Joins:
$collectionIdbecomes ambiguous#After a join, each result row is a composite of two (or more) collections:
q.from({ todo: todoCollection }) .join({ list: listCollection }, ({ todo, list }) => eq(list.id, todo.list_id), "inner") .select(({ todo, list }) => ({ id: todo.id, text: todo.text, listName: list.name, }))A single
$collectionIdstring can't represent this — it will just carry forward whichever value was set first (likely thefromsource), silently losing provenance from the joined side. If consumers use$collectionIdto route mutations back to the right collection, this is actively misleading rather than just incomplete.Options:
- Make
$collectionIdan array or a map keyed by alias after a join (e.g.{ todo: "todo-collection-id", list: "list-collection-id" }) - Strip/nullify
$collectionIdon join results since it's now ambiguous - Keep it but rename it to make clear it only refers to the primary source
Aggregations:
$synced,$origin, and$keylose their meaning#After a
groupBy+ aggregate, each result row represents a group of source rows, not a single entity:q.from({ user: usersCollection }) .groupBy(({ user }) => user.departmentId) .select(({ user }) => ({ departmentId: user.departmentId, userCount: count(user.id), avgAge: avg(user.age), }))For this output:
$key: The aggregate row doesn't correspond to any single source row. Whatever key is passed through is arbitrary and meaningless.$synced: A group may contain a mix of synced and unsynced rows. Is the aggregatetrueonly if all rows in the group are synced? Or is it just pass-through from an arbitrary row? The difference matters hugely for UI that conditionally shows "saving..." indicators.$origin: Same problem — the group may mix'local'and'remote'rows.
The
relax join and group-by type assertionscommit and the +20% size bump ingroup-by.jssuggest the implementation is patching around these issues by loosening types and carrying props through regardless, rather than addressing the semantic mismatch.Suggestion#
The pass-through pattern works well for operations that don't change row identity (
where,orderBy,limit,selectwithout aggregation). The problem is specifically joins (which merge identities) andgroupBy(which destroys individual identity).I'd suggest one of:
Strip or nullify virtual props after operations that break their semantics. Aggregation strips
$key/$synced/$origin; joins make$collectionIdan array or strip it. Reflect this in the types.Make virtual props conditional at the type level — a
from().where()result gets the full{ $synced, $origin, $key, $collectionId }, but agroupBy()result gets a narrower type that omits$key/$synced/$origin. More work, but more honest.At minimum, derive aggregate-level
$syncedmeaningfully — e.g.$synced = falseif any row in the group is unsynced. This would actually be useful ("some data in this group is still pending"). But$keyand$originstill don't have good aggregate semantics and should probably be omitted.
@@ -84,4 +133,20 @@133+ },134+ },135+ [VIRTUAL_HAS_LOCAL_KEY]: {136+ preMap: ([, row]: [string, NamespacedRow]) =>137+ getRowVirtualMetadata(row).hasLocal,138+ reduce: (values: Array<[boolean, number]>) => {139+ for (const [isLocal, multiplicity] of values) {140+ if (isLocal && multiplicity > 0) {141+ return true142+ }143+ }144+ return false145+ },146+ },147+ }148+84149 // Handle empty GROUP BY (single-group aggregation)85150 if (groupByClause.length === 0) {86151 // For single-group aggregation, create a single group with all data87- const aggregates: Record<string, any> = {}152+ const aggregates: Record<string, any> = { ...virtualAggregates }Do we really need to make a copy?
@@ -200,4 +277,4 @@200277 }201278202279 // Create aggregate functions for any aggregated columns in the SELECT clause203- const aggregates: Record<string, any> = {}280+ const aggregates: Record<string, any> = { ...virtualAggregates }Same, really needed to copy?
- Make
@kevin-dp
The aggregation concern is already addressed in the current implementation. We derive aggregate-level virtual props intentionally:
$syncedisfalseif any row in the group is unsynced$originislocalif any contributing row islocal$keyis a derived group key rather than a source-row key
On joins, I think the ambiguity is narrower than described. After a join, the virtual props remain attached per alias:
todo.$collectionId,list.$collectionId. The place where ambiguity does exist is flattened or spread select output, where top-level merges can overwrite one source's virtual props with another's, but I would see that as a users concern when projecting in theselect.Removed the object copes you flagged.
kevin-dp(GitHub profile, opens in a new tab) approved these changes
Code Review: Virtual Props Implementation#
Nice architecture overall — the "add-if-missing" enrichment pattern, WeakMap caching, and aggregate semantics (all-synced / any-local) are well-designed. A few issues worth addressing before merge:
Bug: Truncate doesn't clear origin tracking maps#
packages/db/src/collection/state.ts~line 839The truncate phase in
commitPendingTransactionsclearssyncedData,syncedMetadata, andsyncedKeys, but notrowOrigins,pendingLocalChanges, orpendingLocalOrigins. After truncate, if the server re-inserts the same keys,getRowOrigin(key)returns the stale value fromrowOriginsinstead of defaulting to'remote'.The full
cleanup()method at line 1257 does clear these maps, but truncate-during-sync doesn't.// After line 841 in the truncate block: this.rowOrigins.clear() this.pendingLocalChanges.clear() this.pendingLocalOrigins.clear()
Bug:
pendingLocalOriginsnot cleaned up on transaction failure#packages/db/src/collection/mutations.ts+state.tsmarkPendingLocalOrigins()is called beforecommit(), but if the transaction fails, the.catch(() => undefined)on the commit promise doesn't clean uppendingLocalOrigins. TherecomputeOptimisticStatefailedbranch clearspendingOptimistic*maps but doesn't touchpendingLocalOriginsfor non-optimistic mutations. Result:$originstays'local'for affected keys indefinitely after a failed mutation.Similarly,
scheduleTransactionCleanup's empty catch block (line 1198) keeps the transaction for reference but never cleans up origin tracking state.
Minor:
previousValuevirtual props in optimistic path use post-change state#packages/db/src/collection/state.ts—enrichChangeMessage(line 215)When enriching
previousValue, it callsenrichWithVirtualProps(change.previousValue, change.key)which reads currentisRowSynced/getRowOrigin— the post-mutation state. The sync commit path handles this correctly withgetPreviousVirtualProps, but the optimistic recompute path at line 533 (collectOptimisticChanges→emitEvents→enrichChangeMessage) doesn't capture pre-change virtual props.Impact is limited to
previousValue.$synced/previousValue.$originin change events during optimistic transitions.
Cleanup: Dead
VirtualRefPropstype#packages/db/src/virtual-props.ts:105—VirtualRefPropsis exported but never imported anywhere. The actual ref typing is handled byVirtualPropsRefintypes.tsandVirtualPropsRefProxyinref-proxy.ts. Should be removed.
Suggestion: Derive types from single source#
WithoutVirtualPropshardcodes the four property names as string literals rather than usingOmit<T, keyof VirtualRowProps>. Similarly,VirtualPropsRefandVirtualPropsRefProxyduplicate the same four properties independently. If a fifth virtual prop is added, three files need manual updates. Consider deriving fromVirtualRowProps:// WithoutVirtualProps export type WithoutVirtualProps<T> = Omit<T, keyof VirtualRowProps> // Ref types type VirtualRefShape<TKey extends string | number = string | number> = { readonly [K in keyof VirtualRowProps<TKey>]: RefLeaf<VirtualRowProps<TKey>[K]> }
Suggestion:
hasVirtualPropsshould check all 4 properties#packages/db/src/virtual-props.ts:161— Currently only checks for$syncedand$origin, but the type guard assertsVirtualRowPropswhich also requires$keyand$collectionId. Code relying on this guard could access$keyon an object that doesn't have it.samwillis(GitHub profile, opens in a new tab) closed this
Sign in to join this conversation.
Cursor Agent can help with this pull request. Just
@cursorin comments and I'll start working on changes in this branch. Learn more about Cursor Agents