Site administration
Load the batches that never got in
Some events never got into ClickHouse, because it refused them or a batch failed every retry, and the consumer wrote them to R2. Load each file into ClickHouse, then delete it.
2 min read
You need:
- The Source repository with wrangler signed in, on your computer.
- To be signed in to the server as root. See Sign in to the server.
1. Find the file
Each batch is one file in the bucket gitflare-analytics, at dead/<day>/<id>.jsonl. <day> is the UTC day of the write, and <id> is the id of the first queue message of the batch.
For each file, the Worker logs an error with its name, for example wrote it to R2 at dead/2026-11-04/<id>.jsonl.
2. Download the file
On your computer, run:
pnpm -C apps/web exec wrangler r2 object get gitflare-analytics/dead/<day>/<file> --remote --file <file>
3. Copy the file to the server
scp <file> root@<server IPv4>:/root/<file>
4. Insert its rows
On the server, insert the rows of the file. The copy on the server goes only when the insert works:
clickhouse-client --password "$(cat /root/.clickhouse-admin)" --query "INSERT INTO analytics.events FORMAT JSONEachRow" < /root/<file> && rm /root/<file>
Note
If ClickHouse refused the rows before, this insert can fail for the same reason. ClickHouse then prints the reason. Correct the file or the table, then insert again.
5. Delete the file
If the insert worked, delete the file in R2 and your own copy. Run this on your computer:
pnpm -C apps/web exec wrangler r2 object delete gitflare-analytics/dead/<day>/<file> --remote
rm <file>
Done
The events of the file are in analytics.events, and the file is gone from R2.
If you load a file two times, ClickHouse keeps one row for each uuid when it merges the parts of the table.
About these files
- The consumer writes a file when ClickHouse refused the rows (a 400 or 413 answer), or when the batch failed every retry. If ClickHouse is down explains the retries.
- Each line of the file is one row of
analytics.events, as JSON. - Nothing deletes these files. Delete each file after you load it.