Modify topK and orderBy to track by key instead of by object reference #405
main fromrefs/pull/405/head·merged by kevin-dp(GitHub profile, opens in a new tab) This PR modifies the topK and orderBy operators. Previoiusly, the topK was computing a topK per key and it tracked multiplicities for each value associated to that key. We were always keying by null in ts/db and the values were entire rows, i.e. objects. However, the objects are always different objects (even for the same row) so they were always treated as a new value.
As reported by community members, this led to a bug where the results of an orderBy query with e.g. limit: 10 would end up with just a single item in the result. This would happen in the following case:
- initial query run, results in 10 rows (because of the limit)
- update to a row in that top 10: ts/db splits it into a delete of the old row and an insert of the new row
- topK operator sees the delete for the old row, but the object reference is different from the object reference of the old row it has seen before, so instead of finding the multiplicity for the old row (which is 1), it doesn't find it and thinks it hasn't seen it before so multiplicity is 0. It decrements it by 1, yielding -1. It concludes that since it was 0 before, this row was not visible and is still not visible. Nothing happens. The row is in fact never deleted.
- topK operator sees the insert for the new (updated) row. It stores this new row with multiplicity 1.
- Now the old row is in the top 10 and the new row is in the top 10. But ts/db materializes the query results by key so you will only see it once (with the new value) in the results. However, since internally it's twice in the top K it kicked out another row, so there are only 9 distinct rows in the query result. If you repeat this update 8 more times, you end up with a single row in the query results
To solve this problem i modified the topK operator such that it takes a keyed stream, i.e. a stream of tuples: [Key, Value]. It computes the topK across all keys (i.e. not per key as it did before) and uses the key to identify values. As such, when a row is received we lookup its multiplicity based on the key instead of the object reference.
- 1
More templates
@tanstack/db
npm i https://pkg.pr.new/@tanstack/db@405@tanstack/db-ivm
npm i https://pkg.pr.new/@tanstack/db-ivm@405@tanstack/electric-db-collection
npm i https://pkg.pr.new/@tanstack/electric-db-collection@405@tanstack/query-db-collection
npm i https://pkg.pr.new/@tanstack/query-db-collection@405@tanstack/react-db
npm i https://pkg.pr.new/@tanstack/react-db@405@tanstack/solid-db
npm i https://pkg.pr.new/@tanstack/solid-db@405@tanstack/svelte-db
npm i https://pkg.pr.new/@tanstack/svelte-db@405@tanstack/trailbase-db-collection
npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@405@tanstack/vue-db
npm i https://pkg.pr.new/@tanstack/vue-db@405commit:
beace86Size Change: 0 B
Total Size: 58.5 kB
compressed-size-action::db-package-sizeℹ️ View Unchanged
Filename Size ./packages/db/dist/esm/change-events.js1.13 kB ./packages/db/dist/esm/collection.js9.85 kB ./packages/db/dist/esm/deferred.js230 B ./packages/db/dist/esm/errors.js2.98 kB ./packages/db/dist/esm/index.js1.51 kB ./packages/db/dist/esm/indexes/auto-index.js689 B ./packages/db/dist/esm/indexes/base-index.js605 B ./packages/db/dist/esm/indexes/btree-index.js1.47 kB ./packages/db/dist/esm/indexes/lazy-index.js1.25 kB ./packages/db/dist/esm/local-only.js827 B ./packages/db/dist/esm/local-storage.js2.03 kB ./packages/db/dist/esm/optimistic-action.js294 B ./packages/db/dist/esm/proxy.js4.19 kB ./packages/db/dist/esm/query/builder/functions.js575 B ./packages/db/dist/esm/query/builder/index.js3.79 kB ./packages/db/dist/esm/query/builder/ref-proxy.js890 B ./packages/db/dist/esm/query/compiler/evaluators.js1.48 kB ./packages/db/dist/esm/query/compiler/expressions.js631 B ./packages/db/dist/esm/query/compiler/group-by.js2.03 kB ./packages/db/dist/esm/query/compiler/index.js1.74 kB ./packages/db/dist/esm/query/compiler/joins.js1.56 kB ./packages/db/dist/esm/query/compiler/order-by.js677 B ./packages/db/dist/esm/query/compiler/select.js655 B ./packages/db/dist/esm/query/ir.js318 B ./packages/db/dist/esm/query/live-query-collection.js2.45 kB ./packages/db/dist/esm/query/optimizer.js2.44 kB ./packages/db/dist/esm/SortedMap.js1.24 kB ./packages/db/dist/esm/transactions.js2.29 kB ./packages/db/dist/esm/utils.js419 B ./packages/db/dist/esm/utils/btree.js5.93 kB ./packages/db/dist/esm/utils/comparison.js718 B ./packages/db/dist/esm/utils/index-optimization.js1.62 kB Size Change: 0 B
Total Size: 1.05 kB
compressed-size-action::react-db-package-sizeℹ️ View Unchanged
Filename Size ./packages/react-db/dist/esm/index.js152 B ./packages/react-db/dist/esm/useLiveQuery.js902 B There is a failed test in the react-db package - need to look at that.
@@ -353,7 +366,9 @@353366354367 graph.finalize()355368369+ const value1 = { id: 1, value: `a` }370+356371 // Initial data357372 input.sendData(358373 new MultiSet([359- [[`key1`, { id: 1, value: `a` }], 1],374+ [[`key1`, value1], 1],Test here is changed to use the same object ref, do we have a test where we explicitly check that it works with new ref?
Yep, this is a remnant of when we tracked object refs. I will change it to use different objects.
kevin-dp(GitHub profile, opens in a new tab)replied on
packages/db-ivm/tests/operators/orderByWithFractionalIndex.test.tsline 374Yep, this is a remnant of when we tracked object refs. I will change it to use different objects.
There is a failed test in the react-db package - need to look at that.
Seems to be a flake. CI is green now.
samwillis(GitHub profile, opens in a new tab) approved these changes
:shipit:
kevin-dp(GitHub profile, opens in a new tab) closed this
Sign in to join this conversation.
🦋 Changeset detected#
Latest commit: beace868b20aa37b71ad243ebf50d38a70f40893
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 9 packages
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR