`useCollection` proposal for dynamic/derived collections. #189
Send to my agentI propose a useCollection style hook for dynamic/derived collections. Happy to implement if the design / approach seems right.
N.b.: This relates closely to https://github.com/TanStack/db/issues/187 (laziness may be a key building block of the solution) and https://github.com/TanStack/db/issues/58 (unions of multiple individual collections may be a smarter loading strategy for similar scenarios)
Context#
I'm developing an app with a fairly straightforward data model that has auth and a many to many table. I have:
- users
- threads
- memberships (a many to many table for
users <> threads) - events (belong to threads)
- facts (belong to threads)
Plus I have authentication state, either unauthenticated or authenticated with current user ID.
AUTH
│
USERS -< MEMBERSHIPS >- THREADS
│ │
EVENTS -< FACTS
Now, what I want to do is:
- log the user in
- put the auth state in an authCollection
- base the other collections on this
So, for example:
- I'd like to sync all the memberships where
user_id = auth.currentUserId - then I'd like to sync the threads where
id IN membershipsThreadIds - then all the events and facts where
thread_id IN threadIds - then also sync all the memberships where
thread_id IN membershipsThreadIds - in order to then sync all the users whose
id IN allMembershipsUserIds
In this way, I would waterfall-sync in just the data that the user requires. Plus I would also immediately start syncing new data when optimistic state is added to the collections. For example, user joins a new thread, the threads, facts and events collections would all immediately adapt, without waiting on the server.
Proposal#
I'd like to be able to derive the streamOptions of the collection definition from live queries. And I'd like the collection to automatically update when its dependencies change.
I think this would usually be done within the context of a reactivity framework, so something like a useCollection hook. This could, for example, take a function to rebuild the collection config when dependencies change:
function MyProvider() {
const { data: auth } = useLiveQuery(query => query.from({ authCollection }))
const currentUserId = auth.length === 1 ? auth[0].currentUserId : null
const membershipCollection = useElectricCollection(() => ({
url: '/sync/memberships',
params: {
where: `user_id = ${currentUserId}`
}
}, [currentUserId]))
// etc.
Implementation#
This requires the ability to stop collections and either replace or update them when the dependencies change. We have the abort signal in the shape options, so we can do this with the ElectricCollection. We can similarly interrupt and refetch with QueryCollections.
N.b.: we need to be careful not to cache the signal incorrectly as per https://github.com/electric-sql/electric/issues/2788
We could also perhaps look at tighter integration for derived collections. Such as passing in the parent collections:
function MyProvider() {
const membershipCollection = useElectricCollection(({ authCollection }) => {
const { data: auth } = useLiveQuery(query => query.from({ authCollection }))
return {
url: '/sync/memberships',
params: {
where: `user_id = ${auth.currentUserId}`
}
}
}, { authCollection }))
// etc.
This could then be smarter about:
- only starting sync on the derived collection when the passed in collections have synced
- updating atomically when changes come in in a single transaction across multiple collections
Questions#
Does this seem like the right approach? Any feedback / suggestions on the design or implementation?
thruflo(GitHub profile, opens in a new tab) added labelsquestion
Interesting and yup, the ability to just define collections top level and have the same reference would be ideal. It's so simple for the apps where I've been able to just do that (like https://github.com/thruflo/tasks/blob/main/assets/js/client/db/collections.ts)
Trying to wrap my head around how that would work. I can see a case where the collections respond to other collections. Then when you have something like say the currentUserId change, you update an authCollection and the others derive from it.
That way in the top level module scope, you can just define the collections in order, passing in other collections (straight, or perhaps live queries) that they depend on? Then the function to specify the collection options can have the dependencies passed into. And the collection machinery then handles the swapping under the hood?
const authCollection = createQueryCollection({ queryFn: async () => // returns current auth state, trigger refetch when changed // ... }) const currentUsersQuery = queryBuilder() .from({ authCollection }) .where(`id`, `=`, `current`) .select(`@id`, `@name`) const membershipCollection = createElectricCollection( // this function gets the query results passed in and is called every time they change ({ currentUsers }) => { const currentUserId = currentUsers.length > 0 ? currentUsers[0].id : null return { url: '/sync/memberships', params: { where: `user_id = ${currentUserId}` } } }, { currentUsers: currentUsersQuery } // pass in the query as dependency )The collection internals could then be responsible for subscribing to the query and re-running the function when they change. I wonder if this could be a wrapper around any collection, rather than being part of the collection internals. E.g. something like:
const membershipCollection = dynamicCollection( ({ currentUsers }) => { const currentUserId = currentUsers.length > 0 ? currentUsers[0].id : null return createElectricCollection({ url: '/sync/memberships', params: { where: `user_id = ${currentUserId}` } }) }, { currentUsers: currentUsersQuery } )So the dynamic wrapper could handle switching between actual collections and maybe the return value is a proxy to maintain the same top level reference?
Just to add notes from discussion just now:
- dynamicCollection doesn't need to be a proxy, can just provide the collection API and manage the data internally
- the function that I currently had returning
return createElectricCollection({can just return the collection config (thereby naturally making it cross collection) - @samwillis has a live query collection coming
Let's just standardize on useLiveQuery
KyleAMathews(GitHub profile, opens in a new tab) closed this
Sign in to join this conversation.
Makes sense! I wouldn't put this in the actual React component though — in general you'll want to define your collections outside of React.
createCollectioncould take a function which returns an object e.g.createCollection(() => ...)That's easy. The tricky part is the returned collection would be stable even when the underlying collection keeps getting recreated. So some sort of special collection that can automatically handle switching atomically between the data from the old collection to the data from the new collection.
So perhaps then
createCollection(collectionProviderOptions((swap) => /* setup a collection & subscribe to changes and swap between collections whenever */))