Skip to content
Source
Sign inSign up
TanStack/db

DB tests: Istanbul coverage adds ~50% wall time / ~38% CPU to the packages/db suite #2274

Send to my agent
Openjp-knj(GitHub profile, opens in a new tab) opened this issue 1 comment

Summary

In packages/db, coverage is on by default (vite.config.ts: coverage: { enabled: true, provider: 'istanbul', include: ['src/**/*'] }), so every local pnpm test run pays for instrumentation. I measured the cost on the full suite:

Run (285 files, 4 workers)ElapsedTotal CPU
Istanbul coverage (default)144 s507 s
--coverage.enabled=false96 s367 s
Difference+48 s (+50%)+140 s (+38%)

Repro

From packages/db:

/usr/bin/time -p pnpm vitest --run --maxWorkers=4 /usr/bin/time -p pnpm vitest --run --maxWorkers=4 --coverage.enabled=false

Environment: Apple M3 Max (16 cores, 128 GB), Node v26.12.0, pnpm 11.1.0.

Goal

Reduce test overhead while keeping the same coverage and checks that CI enforces today. CI collects DB coverage per shard and merges it with vitest --mergeReports in .github/workflows/pr.yml, and that should keep working.

Possible directions (for discussion)

  • Turn coverage off by default for local runs and enable it explicitly in scripts/ci-tests.mjs / CI, so CI coverage doesn't change.
  • Compare provider: 'v8' with Istanbul for overhead and report parity before considering a switch.

I'm happy to do the follow-up PR, with before/after numbers, once there's agreement on a direction.

  1. Follow-up: here is where the rest of the cost goes, beyond coverage.

    Method. I ran packages/db with --maxWorkers=4, changing one condition per run. Times are wall time and total CPU (user+sys). I measured on commit 26d38a4cd with Node 26.12.0, on an M3 Max. A repeated run of the same condition matched within 0.02 s. On Node 24.18.0 with origin/main + #2275 (300 files), the coverage cost has the same shape: 122 s / 421 s without coverage, 165 s / 522 s with it.

    CostWallCPUKind
    Istanbul coverage+43–48 s+100–140 sInstrumentation
    tests/oracle-replay.test.ts+20 s+106 sProcess startup
    typecheck: { enabled: true }+26 s+35 stsc
    environment: 'jsdom' for all files+15 s+76 sEnvironment setup (~75 of 285 files touch the DOM)
    tests/indexed-db/retirement-oracle.test.ts+14 s+8 sIdle waiting, fixed in #2275 (64 s → 2.3 s)

    Notes:

    • oracle-replay.test.ts runs 38 tests. Each one calls spawnSync to start node --import tsx tests/oracle-replay.ts, which spawns vitest run. One replay costs about 1.7 s, and Vitest's own work is 0.55 s of that. The rest is process startup. The file runs serially in one worker for 77 s, which sets a floor on the suite's wall time.
    • Test time is concentrated. Before #2275, these two files accounted for 53% of summed per-file test time.
    • Coverage and the timeouts in #2238 are related but separate. Coverage made the cold in-body imports about 6× slower, and #2276 moves those imports out of the timed test body. Reducing coverage cost would hide that class of test defect, not fix it.

    Next I'm looking at reducing the replay test's process startup without weakening what it checks.

Sign in to join this conversation.