Skip to content
Source
Sign inSign up

Find the slow parts of a repository

Site administration

Find the slow parts of a repository

The forge times each request, each call of a Durable Object, each D1 query, each queue batch, cron task and Workflow step, each request to another service and each streamed response, and keeps the timings in ClickHouse for 30 days. The web Worker, the runner of CI and the factory, and the repo Worker of each repository's object all send them. Use them to find what makes a repository slow, and what to fix first as the forge grows.

14 min read

You need:

  • The Source repository on your computer, with wrangler signed in to the Cloudflare account.
  • To be signed in to the analytics server as root. See Sign in to the server.

1. Add the timings table

Do this once. Your server was set up before the schema had the table analytics.timings.

On your computer, copy the schema to the server:

scp apps/web/clickhouse/schema.sql root@<server IPv4>:/root/schema.sql

On the server, apply it as the admin user default. Each statement is safe to run again, so the other tables and their rows do not change:

clickhouse-client --password "$(cat /root/.clickhouse-admin)" --queries-file /root/schema.sql

This creates analytics.timings, lets ingest insert into it, and lets reader read it. Until the table exists, the forge drops the timings it sends, and nothing else changes.

2. Make sure that the timings arrive

A few minutes after the next deploy, count the rows of the last hour:

clickhouse-client --password "$(cat /root/.clickhouse-admin)" --query "SELECT kind, count() FROM analytics.timings WHERE at >= now() - INTERVAL 1 HOUR GROUP BY kind"

Each kind has rows: route, rpc, rpc_call, d1, queue, cron and boot, and the others when their work ran. For example, sql has rows only when a statement read or wrote 1,000 rows or more.

3. Read the report

On your computer, read the slowest parts of the forge:

pnpm -C apps/web perf --ssh root@<server IPv4>

Or of one repository, over the last day:

pnpm -C apps/web perf --ssh root@<server IPv4> --repo pingdotgg/t3code --hours 24

If your SSH key for the server is not one of the keys that ssh uses by default, give its file with --key:

pnpm -C apps/web perf --ssh root@<server IPv4> --key ~/.ssh/<key file>

The report runs its queries on the server over SSH, and finds the repository's id in D1 with wrangler. For each operation it shows how often it ran, its times in ms (p50, p95, p99 and the longest), its total time in seconds, the SQL rows of the object that it read and wrote, and how many failed.

Find the statement that reads the most rows

A repository's object is slow, and its calls read many SQL rows. To fix it, you need the statement that reads them, and the call that runs it.

  1. Run the report for the repository:

    pnpm -C apps/web perf --ssh root@<server IPv4> --repo pingdotgg/t3code --hours 24
  2. Read the section "SQL statements by rows read". The statement that read the most rows in total is first.

  3. For each statement, read its caller and how often it ran. The other columns give its total, average and largest rows read, and its total rows written.

To read the same rows on the server, run this query as the admin user. The forge report (without --repo) shows the repository's id in the repo_id column of the same section.

clickhouse-client --password "$(cat /root/.clickhouse-admin)" --query "SELECT name, sum(rows_read) AS total_rows_read, count() AS runs, max(rows_read) AS max_rows_read FROM analytics.timings WHERE kind = 'sql' AND repo_id = '<repo id>' AND at >= now() - INTERVAL 24 HOUR GROUP BY name ORDER BY total_rows_read DESC LIMIT 20"

Each name is <caller> | <SQL text>. The caller is before |, and the SQL text is after it.

You now have the statement that reads the most rows, and the call or step that runs it.

What each timing measures

  • route: a request to the web Worker, by its route pattern, such as GET /@:owner/:repo/blob/* or POST serverFn getRepoHome. It never holds an owner, a repository or a path.
  • rpc: a call of a Durable Object, timed inside the object, by <class>.<method>. A repository's object (RepoObject) adds its repository and the SQL rows of the call.
  • rpc_call: the same call, timed by the caller. It adds the wait in the object's queue and the transit. The report shows the difference as the wait for the object. A large wait means the object is busy.
  • d1: a D1 query, by its SQL text, and a batch by its first statement. Its values are never kept.
  • alarm: the alarm of a Durable Object.
  • boot: the start of a repository's object, as after a deploy, until its first call ends. Its SQL rows are those of the start. For 15 s after a boot, background calls such as RepoObject.githubSyncRead wait, so their times include that wait.
  • sync_step: a step of a GitHub sync pass, such as github.refs, with the SQL rows of the step. The call that runs the step still counts all of its rows.
  • stream: a clone, fetch or push, or the data sync, from its headers to its last byte. Its rows_read is the bytes it sent.
  • queue: a batch of a queue consumer, by its queue.
  • cron: a task of the cron, by its name.
  • workflow_step: a step of a job's Workflow.
  • fetch: a request to another service: GitHub's API by its path pattern (GET github /repos/:owner/:repo/issues/:n), a webhook delivery, an MCP server. It never holds a URL.
  • sql: one SQL statement of a repository's object that read or wrote 1,000 rows or more. Each run of the statement is one row. Its name is <caller> | <SQL text>.
    • The caller is the call or step that ran it, such as RepoObject.alarm, RepoObject.repoPage, github.refs, or RepoObject.boot for the start of the object.
    • The SQL text holds no values, because they are bound parameters. A name is cut at 300 characters.
    • Its ms is always 0, because the Workers clock does not move in synchronous SQL. Read rows_read and rows_written.

The times are wall times across I/O. The Workers clock moves only at I/O, so synchronous code takes 0 ms on it, and so does each SQL statement of an object. The SQL rows that a call read and wrote are what its SQL costs.

Check the home page

Use the existing timings report and focused tests to check home page work. Timings show observed request costs, not a guaranteed speed improvement.

  1. Open / while signed out, then open it with an approved browser session.
  2. Read the timings report without --repo, because the home page spans repositories.
  3. Compare the getDashboard route timings and D1 query counts for the same observation period.
  4. Inspect the home response in the browser network panel. Signed-out responses have empty dashboard lists and the request origin.
  5. With an approved session, check that the dashboard still shows recently updated public repositories and your personal sections.
  6. From the workspace root, run the focused regression tests:
pnpm -C apps/web exec vitest run test/home-data.test.ts test/session-read.test.ts --maxWorkers=2

The home data tests count public-list query preparations after fixture setup. They cover missing, unknown, expired, revoked and waiting sessions. These cases must not query the recently updated public list. The approved session must retain the newest 12 public repositories in order. The tests also check the empty response lists and request origin.

The session tests check the existing request-scoped session reads. Use these tests for session safety evidence. Aggregate production timings alone cannot prove access safety or identify one browser session. Do not record cookies or credentials in the evidence.

Check the home loading boundary

Use the deployed release to check the browser graph. Source import tests alone do not prove the production chunk layout.

Match the release

  1. Record the source commit, lockfile SHA-256, build and deployment identifiers, URL, and observation time.
  2. Save the HTML, asset URLs, and build manifest from that exact release in the evidence folder.
  3. Match the HTML asset hashes to the manifest before comparing releases. Do not mix local output with deployed HTML.
  4. Record the browser, cache state, session state, and theme for each observation. Keep credentials out of evidence.

Collect the complete mandatory graph

  1. Save every required module script from the anonymous Home HTML as a root.
  2. Add the rendered Home component chunk, even across a dynamic edge.
  3. Add every JavaScript modulepreload from the same HTML and manifest as a root.
  4. Parse every root body and follow every static import and reexport, including bare imports and shared chunks.
  5. Save each complete body, SHA-256, decoded byte length, resolved edges, and a root path to each included module.

The mandatory closure contains all roots and every module reached by these static edges. A visited set handles cycles. Optional runtime requests, including Three.js, have a separate list. Network requests alone do not define the static closure.

  1. Require zero missing, truncated, hash-mismatched, or wrong-release bodies.
  2. Require zero unresolved static edges and parser errors.
  3. Check for target bodies inside required entry and shared chunks, even when their old filenames disappear.
  4. Attribute bodies through the matching manifest, source maps when available, and emitted code. Record uncertain ownership as incomplete proof.
  5. Repeat the complete capture for the baseline and candidate. Compare only release-matched artifacts.

Check the route and UI targets

The anonymous mandatory closure must exclude these UI bodies:

AreaTargets
Repository codeCodeFilter and its controls; file-nesting patterns and arrangement code
Repository navigation and settingsSection icons; Toggle and notice controls
Apps and webhooksRepoApps, AccountAppsGrid, Consent, AppPermissions, AppForm, WebhooksPanel, and SuspendApp
App page assembliesMyApps, AppSettings, InstalledAppsPage, public AppPage, and AppDirectory
Personal Home and chromeDashboard, HeaderAccountControls, NotificationBell, SiteDock, and VerifyEmailBanner

Required data stays ready: filter parsers and loaders, open-row search, section redirects, app search, and permission descriptions. Existing role and action data can remain through a real consumer. Authorization remains on the server. Shared Base UI code can remain when another required component uses it. Its presence alone does not prove a target UI body remains.

  1. Check every mapped critical route and any tsr-shared compiler output for retained UI imports, including bare imports.
  2. Check that wrappers use their exact getRouteApi ids and have no route-definition import back into UI.
  3. Require the emitted closure check too. Source moves and compiler probes do not prove production chunk removal.
  4. Repeat with an approved session. Record optional chunk requests and when their controls become usable.

Landing stays eager. The home loader and shared data runtime remain shared. A dynamic edge alone does not prove less initial work.

Header keeps public navigation, search, theme controls, and sign-in links available. Account controls load for approved or waiting sessions. SiteDock preloads for an eligible approved identity. It renders only after the notification listener registers for that account period. The email banner retains its identity guard. The dashboard receives the existing loaded data through an account-keyed boundary.

Check the docs route boundary

The native /docs/$ loader awaits the existing docs registry. It resolves the sidebar title after the data client returns the page. The stored view keeps only the Markdown source, directory, and slug. Existing in-session data can be reused. Docs views have no durable offline persistence. The page assembly uses the native component split. The registry still owns navigation, page titles, Markdown imports, and image URLs.

  1. Check the eager route output for static and bare imports of the docs registry or its UI consumers.
  2. Check the component split and any shared compiler output. Follow imports and reexports into the registry.
  3. Check rebuilt entry and shared bodies in the complete mandatory Home graph. A different filename does not prove removal.
  4. Open /docs, a nested page, and a missing page through direct requests and client navigation.
  5. Check sidebar titles separately from Markdown H1 text, including pages without an H1 and the Docs title fallback.
  6. Check fresh loads, cached entry, preload, and available in-session offline data. The title must work when the fetch callback does not run.
  7. Check text for browser Find, search, sidebar and mobile navigation, child links, pager order, outline, relative links, images, and copied code.
  8. Check both themes, hydration, media controls, deferred home decoration, and the existing missing-chunk recovery path.

CI runs the docs SSR and hydration cases through production Redact aliases. Local source and compiler checks do not establish production savings. Keep server-function identities, authorization, and the existing data cache policy unchanged during this check.

Maintain the Redact serializer patch

Keep the exact @tanstack/redact@0.2.3 pnpm patch registered in pnpm-workspace.yaml and recorded in pnpm-lock.yaml. It keeps numeric CSS custom properties unitless during server serialization. Ordinary numeric lengths still receive px.

  1. Before changing or removing the patch, check the target package's source serializer, built JavaScript, and matching source map.
  2. Check zero, positive, negative, fractional, numeric-text, and explicit-unit custom properties. Keep ordinary width and opacity controls.
  3. Require CI to pass the actual SSR and hydration controls, both docs home themes, and the existing runtime files and production build.
  4. For each patch revision, regenerate its native pnpm lock identity and check a frozen install. Never leave a stale patch hash or embedded map source.

The docs home checks require grid steps 0, 1, and 2 before hydration, with no recoverable or console errors afterward. The content fixture skips animation timing and completes queued Motion work after unmount. It does not prove native WAAPI cancellation. Pure serializer checks do not prove application hydration, a deployed browser result, or a performance gain. Remove the patch only after the replacement package passes the same source, distribution, and runtime controls.

Check account changes and actions

  1. Hold optional chunk responses on a fresh visit. Check that public controls and page content remain usable while account controls wait.
  2. Change from account A to B, then sign out and sign in again while imports are pending.
  3. Check that old menus, organization results, notifications, dashboard state, and banners do not appear for the new identity.
  4. On the first usable dock, click Notifications once. Check that one panel opens at the dock anchor, even while unread data waits.
  5. Close the panel with Escape, then open it from the header Bell. Check its anchor and keyboard focus.

Also check waiting-account sign-out, failed sign-out retry, and unsent-change confirmation. Waiting and anonymous sessions must not get the signed-in dock.

Collect PR CI evidence

  1. Record the tested candidate commit and lockfile hash with the Source checks from src pr checks.
  2. Require the existing type, unit, production Redact runtime, and production build jobs to pass.
  3. Keep the matching client manifest, emitted JavaScript bodies, and private source maps when available with the review evidence.
  4. Record artifact hashes and availability. A build cache entry alone is not a complete exported graph artifact.
  5. Use the existing deployed-release capture when CI does not retain the full client output. Mark emitted proof pending until that capture exists.

The focused app-directory runtime case uses the real route, hooks, SSR, and hydration with a fake Start transport. Ordinary data-hook fakes do not prove production Redact hydration. CI runtime results also do not replace release browser checks.

Check the production runtime

Use the tested candidate preview or deployed release. Record its identity with each result.

JourneyRequired checks
Anonymous HomeSame SSR content, title, public navigation, search, theme, and sign-in readiness
Waiting and approved sessionsIdentity guards, dashboard and dock readiness, account changes, and sign-out
Repository root, tree, and blobDirect SSR and client navigation; refs, paths, only/hide/owner/open search, stored filters, cap fallback, icons, and keyboard behavior
Agents and SettingsRole redirects, hashes, GitHub callback search, archived access, admin checks, sidebar icons, errors, partial saves, and pending forms
App journeysDirectory paging; public, private, and suspended detail; registration and edit; personal, organization, and repository installs; consent, secrets, and suspend/lift
Redact and Base UIHydration, navigation, labels, focus return, dialogs, popovers, switches, checkboxes, and forms in both themes
  1. Record hydration errors, console errors, visible layout failures, and failed requests in both themes.
  2. Refuse a deferred UI chunk and check the existing route error path and accessible manual recovery.
  3. Require at most one automatic reload per missing chunk per tab, with no reload loop.
  4. Navigate from an already-open old page during deployment. Check stable server-function methods, ids, and payloads.
  5. Check old-chunk failure through the existing recovery path. Do not add another reload or cache system for this procedure.

Keep the account-change and notification-action checks above. Hold optional responses and leave a page while imports wait to expose stale controls.

The check is complete when matching evidence covers the full graph, deferred journeys, account changes, runtime behavior, and failure recovery. A source-only candidate can be ready for review while emitted and release checks remain pending. Do not claim fewer bytes, lower CPU use, or faster pages without separate measurements from matched releases.

Done

The forge keeps the timings of the last 30 days. To see them in another tool, read analytics.timings as reader.