Skip to content
Source
Sign inSign up
TanStack/db

Modify topK and orderBy to track by key instead of by object reference #405

Mergedkevin-dp(GitHub profile, opens in a new tab)merged 5 commits intomain fromrefs/pull/405/headmerged by kevin-dp(GitHub profile, opens in a new tab)

This PR modifies the topK and orderBy operators. Previoiusly, the topK was computing a topK per key and it tracked multiplicities for each value associated to that key. We were always keying by null in ts/db and the values were entire rows, i.e. objects. However, the objects are always different objects (even for the same row) so they were always treated as a new value.

As reported by community members, this led to a bug where the results of an orderBy query with e.g. limit: 10 would end up with just a single item in the result. This would happen in the following case:

  • initial query run, results in 10 rows (because of the limit)
  • update to a row in that top 10: ts/db splits it into a delete of the old row and an insert of the new row
  • topK operator sees the delete for the old row, but the object reference is different from the object reference of the old row it has seen before, so instead of finding the multiplicity for the old row (which is 1), it doesn't find it and thinks it hasn't seen it before so multiplicity is 0. It decrements it by 1, yielding -1. It concludes that since it was 0 before, this row was not visible and is still not visible. Nothing happens. The row is in fact never deleted.
  • topK operator sees the insert for the new (updated) row. It stores this new row with multiplicity 1.
  • Now the old row is in the top 10 and the new row is in the top 10. But ts/db materializes the query results by key so you will only see it once (with the new value) in the results. However, since internally it's twice in the top K it kicked out another row, so there are only 9 distinct rows in the query result. If you repeat this update 8 more times, you end up with a single row in the query results

To solve this problem i modified the topK operator such that it takes a keyed stream, i.e. a stream of tuples: [Key, Value]. It computes the topK across all keys (i.e. not per key as it did before) and uses the key to identify values. As such, when a row is received we lookup its multiplicity based on the key instead of the object reference.

    1. 🦋 Changeset detected

      Latest commit: beace868b20aa37b71ad243ebf50d38a70f40893

      The changes in this PR will be included in the next version bump.

      This PR includes changesets to release 9 packages
      NameType
      @tanstack/db-ivmPatch
      @tanstack/dbPatch
      @tanstack/electric-db-collectionPatch
      @tanstack/query-db-collectionPatch
      @tanstack/react-dbPatch
      @tanstack/solid-dbPatch
      @tanstack/svelte-dbPatch
      @tanstack/trailbase-db-collectionPatch
      @tanstack/vue-dbPatch

      Not sure what this means? Click here to learn what changesets are.

      Click here if you're a maintainer who wants to add another changeset to this PR

    2. More templates

      @tanstack/db

      npm i https://pkg.pr.new/@tanstack/db@405

      @tanstack/db-ivm

      npm i https://pkg.pr.new/@tanstack/db-ivm@405

      @tanstack/electric-db-collection

      npm i https://pkg.pr.new/@tanstack/electric-db-collection@405

      @tanstack/query-db-collection

      npm i https://pkg.pr.new/@tanstack/query-db-collection@405

      @tanstack/react-db

      npm i https://pkg.pr.new/@tanstack/react-db@405

      @tanstack/solid-db

      npm i https://pkg.pr.new/@tanstack/solid-db@405

      @tanstack/svelte-db

      npm i https://pkg.pr.new/@tanstack/svelte-db@405

      @tanstack/trailbase-db-collection

      npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@405

      @tanstack/vue-db

      npm i https://pkg.pr.new/@tanstack/vue-db@405

      commit: beace86

    3. Size Change: 0 B

      Total Size: 58.5 kB

      ℹ️ View Unchanged
      FilenameSize
      ./packages/db/dist/esm/change-events.js1.13 kB
      ./packages/db/dist/esm/collection.js9.85 kB
      ./packages/db/dist/esm/deferred.js230 B
      ./packages/db/dist/esm/errors.js2.98 kB
      ./packages/db/dist/esm/index.js1.51 kB
      ./packages/db/dist/esm/indexes/auto-index.js689 B
      ./packages/db/dist/esm/indexes/base-index.js605 B
      ./packages/db/dist/esm/indexes/btree-index.js1.47 kB
      ./packages/db/dist/esm/indexes/lazy-index.js1.25 kB
      ./packages/db/dist/esm/local-only.js827 B
      ./packages/db/dist/esm/local-storage.js2.03 kB
      ./packages/db/dist/esm/optimistic-action.js294 B
      ./packages/db/dist/esm/proxy.js4.19 kB
      ./packages/db/dist/esm/query/builder/functions.js575 B
      ./packages/db/dist/esm/query/builder/index.js3.79 kB
      ./packages/db/dist/esm/query/builder/ref-proxy.js890 B
      ./packages/db/dist/esm/query/compiler/evaluators.js1.48 kB
      ./packages/db/dist/esm/query/compiler/expressions.js631 B
      ./packages/db/dist/esm/query/compiler/group-by.js2.03 kB
      ./packages/db/dist/esm/query/compiler/index.js1.74 kB
      ./packages/db/dist/esm/query/compiler/joins.js1.56 kB
      ./packages/db/dist/esm/query/compiler/order-by.js677 B
      ./packages/db/dist/esm/query/compiler/select.js655 B
      ./packages/db/dist/esm/query/ir.js318 B
      ./packages/db/dist/esm/query/live-query-collection.js2.45 kB
      ./packages/db/dist/esm/query/optimizer.js2.44 kB
      ./packages/db/dist/esm/SortedMap.js1.24 kB
      ./packages/db/dist/esm/transactions.js2.29 kB
      ./packages/db/dist/esm/utils.js419 B
      ./packages/db/dist/esm/utils/btree.js5.93 kB
      ./packages/db/dist/esm/utils/comparison.js718 B
      ./packages/db/dist/esm/utils/index-optimization.js1.62 kB
      compressed-size-action::db-package-size
    4. Size Change: 0 B

      Total Size: 1.05 kB

      ℹ️ View Unchanged
      FilenameSize
      ./packages/react-db/dist/esm/index.js152 B
      ./packages/react-db/dist/esm/useLiveQuery.js902 B
      compressed-size-action::react-db-package-size
    5. There is a failed test in the react-db package - need to look at that.

      OutdatedResolvedby kevin-dp(GitHub profile, opens in a new tab)
    6. kevin-dp(GitHub profile, opens in a new tab)replied on packages/db-ivm/tests/operators/orderByWithFractionalIndex.test.ts line 374

      Yep, this is a remnant of when we tracked object refs. I will change it to use different objects.

    7. There is a failed test in the react-db package - need to look at that.

      Seems to be a flake. CI is green now.

Sign in to join this conversation.