Skip to content
Source
Sign inSign up
TanStack/db

`useCollection` proposal for dynamic/derived collections. #189

Send to my agent
Closedthruflo(GitHub profile, opens in a new tab) opened this issue 4 comments

I propose a useCollection style hook for dynamic/derived collections. Happy to implement if the design / approach seems right.

N.b.: This relates closely to https://github.com/TanStack/db/issues/187 (laziness may be a key building block of the solution) and https://github.com/TanStack/db/issues/58 (unions of multiple individual collections may be a smarter loading strategy for similar scenarios)

Context

I'm developing an app with a fairly straightforward data model that has auth and a many to many table. I have:

  • users
  • threads
  • memberships (a many to many table for users <> threads)
  • events (belong to threads)
  • facts (belong to threads)

Plus I have authentication state, either unauthenticated or authenticated with current user ID.

AUTH
 │
USERS -< MEMBERSHIPS >- THREADS
                        │     │
                    EVENTS -< FACTS

Now, what I want to do is:

  1. log the user in
  2. put the auth state in an authCollection
  3. base the other collections on this

So, for example:

  • I'd like to sync all the memberships where user_id = auth.currentUserId
  • then I'd like to sync the threads where id IN membershipsThreadIds
  • then all the events and facts where thread_id IN threadIds
  • then also sync all the memberships where thread_id IN membershipsThreadIds
  • in order to then sync all the users whose id IN allMembershipsUserIds

In this way, I would waterfall-sync in just the data that the user requires. Plus I would also immediately start syncing new data when optimistic state is added to the collections. For example, user joins a new thread, the threads, facts and events collections would all immediately adapt, without waiting on the server.

Proposal

I'd like to be able to derive the streamOptions of the collection definition from live queries. And I'd like the collection to automatically update when its dependencies change.

I think this would usually be done within the context of a reactivity framework, so something like a useCollection hook. This could, for example, take a function to rebuild the collection config when dependencies change:

function MyProvider() {
  const { data: auth } = useLiveQuery(query => query.from({ authCollection }))
  const currentUserId = auth.length === 1 ? auth[0].currentUserId : null
 
  const membershipCollection = useElectricCollection(() => ({
    url: '/sync/memberships',
    params: {
      where: `user_id = ${currentUserId}`
    }
  }, [currentUserId]))

  // etc.

Implementation

This requires the ability to stop collections and either replace or update them when the dependencies change. We have the abort signal in the shape options, so we can do this with the ElectricCollection. We can similarly interrupt and refetch with QueryCollections.

N.b.: we need to be careful not to cache the signal incorrectly as per https://github.com/electric-sql/electric/issues/2788

We could also perhaps look at tighter integration for derived collections. Such as passing in the parent collections:

function MyProvider() {
  const membershipCollection = useElectricCollection(({ authCollection }) => {
    const { data: auth } = useLiveQuery(query => query.from({ authCollection }))
   
    return {
      url: '/sync/memberships',
      params: {
        where: `user_id = ${auth.currentUserId}`
      }
    }
  }, { authCollection }))

  // etc.

This could then be smarter about:

  1. only starting sync on the derived collection when the passed in collections have synced
  2. updating atomically when changes come in in a single transaction across multiple collections

Questions

Does this seem like the right approach? Any feedback / suggestions on the design or implementation?

  1. thruflo(GitHub profile, opens in a new tab) added labelsquestion

  2. Makes sense! I wouldn't put this in the actual React component though — in general you'll want to define your collections outside of React.

    createCollection could take a function which returns an object e.g. createCollection(() => ...)

    That's easy. The tricky part is the returned collection would be stable even when the underlying collection keeps getting recreated. So some sort of special collection that can automatically handle switching atomically between the data from the old collection to the data from the new collection.

    So perhaps then createCollection(collectionProviderOptions((swap) => /* setup a collection & subscribe to changes and swap between collections whenever */))

  3. Interesting and yup, the ability to just define collections top level and have the same reference would be ideal. It's so simple for the apps where I've been able to just do that (like https://github.com/thruflo/tasks/blob/main/assets/js/client/db/collections.ts)

    Trying to wrap my head around how that would work. I can see a case where the collections respond to other collections. Then when you have something like say the currentUserId change, you update an authCollection and the others derive from it.

    That way in the top level module scope, you can just define the collections in order, passing in other collections (straight, or perhaps live queries) that they depend on? Then the function to specify the collection options can have the dependencies passed into. And the collection machinery then handles the swapping under the hood?

    const authCollection = createQueryCollection({
      queryFn: async () => // returns current auth state, trigger refetch when changed
      // ...
    })
    
    const currentUsersQuery = queryBuilder()
      .from({ authCollection })
      .where(`id`, `=`, `current`)
      .select(`@id`, `@name`)
    
    const membershipCollection = createElectricCollection(
      // this function gets the query results passed in and is called every time they change
      ({ currentUsers }) => {
        const currentUserId = currentUsers.length > 0 ? currentUsers[0].id : null
        
        return {
          url: '/sync/memberships',
          params: {
            where: `user_id = ${currentUserId}`
          }
        }
      }, 
      { currentUsers: currentUsersQuery } // pass in the query as dependency
    )

    The collection internals could then be responsible for subscribing to the query and re-running the function when they change. I wonder if this could be a wrapper around any collection, rather than being part of the collection internals. E.g. something like:

    const membershipCollection = dynamicCollection(
      ({ currentUsers }) => {
        const currentUserId = currentUsers.length > 0 ? currentUsers[0].id : null
        
        return createElectricCollection({
          url: '/sync/memberships',
          params: {
            where: `user_id = ${currentUserId}`
          }
        })
      }, 
      { currentUsers: currentUsersQuery }
    )

    So the dynamic wrapper could handle switching between actual collections and maybe the return value is a proxy to maintain the same top level reference?

  4. Just to add notes from discussion just now:

    1. dynamicCollection doesn't need to be a proxy, can just provide the collection API and manage the data internally
    2. the function that I currently had returning return createElectricCollection({ can just return the collection config (thereby naturally making it cross collection)
    3. @samwillis has a live query collection coming
  5. Let's just standardize on useLiveQuery

Sign in to join this conversation.