Request: SQLite IN-list chunking still exceeds the statement-wide parameter limit #1993
Send to my agentProblem#
The SQLite persistence compiler splits large IN arrays into smaller clauses joined with OR, but keeps all their bound parameters in one SQL statement.
SQLite's bound-parameter limit applies to the entire statement. Splitting the clauses therefore does not prevent the limit from being exceeded.
Multiple smaller IN predicates can also exceed the limit when composed into one query.
Versions#
@tanstack/db-sqlite-persistence-core:0.4.0@tanstack/browser-db-sqlite-persistence:0.2.25@tanstack/db:0.11.0
The same compilation strategy is present in upstream main inspected on October 2, 2026.
Reproduction#
Use a SQLite-backed adapter with the connection's variable limit set to 999. For example, our Node SQLite harness uses:
new DatabaseSync(':memory:', {
limits: { variableNumber: 999 },
})
After initializing a collection, request:
import { IR } from '@tanstack/db'
const ids = Array.from({ length: 1_000 }, (_, i) => `issue-${i}`)
await adapter.loadSubset(collectionId, {
where: new IR.Func('or', [
new IR.Func('in', [
new IR.PropRef(['issueId']),
new IR.Value(ids),
]),
new IR.Func('in', [
new IR.PropRef(['relatedIssueId']),
new IR.Value(ids),
]),
]),
})
Source inspection shows that this produces approximately 2,000 list parameters in one statement, despite the internal chunking. That exceeds the configured limit.
Expected behavior#
Bound the total number of parameters in each executed statement, including parameters from composed predicates.
Downstream workaround#
We bind each list as one JSON array and compile runtime predicates as:
field IN (SELECT value FROM json_each(?))
An alternative implementation would also be welcome. Important compatibility requirements include preserving typed comparisons and signed 64-bit bigint precision.
Index expressions and partial-index predicates need separate handling: SQLite does not permit these subqueries in index definitions.
Existing downstream tests cover 50,582 IDs across two fields, bigint values beyond JavaScript's safe-integer range, and literal arrays in index definitions. They were not rerun for this source-inspection report.
#810 concerns Electric HTTP request-size limits; this report concerns local SQLite statement parameters.
KyleAMathews(GitHub profile, opens in a new tab) closed this
Sign in to join this conversation.