chore(ai): add the chat reload oracle and oracle setup #100004
main fromchore/oracle-setup·opened This PR adds an oracle test for the chat reload, and sets up the repo so that coding agents can find and run oracle tests. It is stacked on #2163, and its forge CI needs #2167.
🎯 Changes#
- Oracle test:
packages/ai/tests/chat-reload.oracle.test.ts. It runs the realchat()andStreamProcessorover generated runs: thinking, text, server tools that return or throw, 1 to 3 model calls, and client tools that are answered or wait. It checks 4 laws: the reload equals the live chat, tool output keeps its value and type, a second reload changes nothing, and the reload never throws. - Dev dependencies:
@tanstack/oracles(pinned to commit78fa1b7on pkg.pr.new) andfast-check, at the root. pnpm also re-resolves some peer variants in the lockfile when it adds them. AGENTS.mdandCLAUDE.md: the@tanstack/oraclesblock inAGENTS.mdsays who owns the laws, which skill to read for each task, the oracle file glob, and the gap check command.CLAUDE.mdpoints to it.packages/ai/vitest.oracle.config.tsandtest:oracles: run only*.oracle.test.ts. The gap check uses the config, so it does not run the whole package suite for each mutant.- Lint: oracle lint is not set up. The repo lints with oxlint, and typescript-eslint does not support TypeScript 7 yet. The
AGENTS.mdblock says so.
No changeset: nothing in a published package changes. No docs: tests and agent tooling only.
Order#
- Merge #2167, so forge CI can fetch the
pkg.pr.newtarball. Until then,Install dependenciesfails here withERR_PNPM_FETCH_403. - Merge #2163. The oracle fails without its fixes.
- Merge this PR.
✅ Checklist#
- I have followed the steps in the Contributing guide.
- I have tested code changes locally with
pnpm run test:pr, or these tests do not apply to this pull request. - I fully understand the code in this pull request, including any code generated with AI assistance.
- Docs: I updated
docs/for this change, or this change is not user-facing. - Changeset: I added a changeset (
pnpm changeset), or this PR does not change a published package.
🚀 Release Impact#
- This change affects published code, and I have generated a changeset.
- This change is docs/CI/dev-only (no release).
Testing#
Commands run#
pnpm --dir packages/ai test:oracles: 108 of 108 pass.- The same oracle on the
packages/ai/srcofmain: 28 of 108 fail. These are the bugs that #2163 fixes. oxfmt --checkon the changed files: passes.- Not run locally:
pnpm test:pr. CI runs it after #2167 merges.
Manual test#
- Run
pnpm --dir packages/ai test:oracles. It runs only the oracle file. - At the repo root, run the gap check command from the
AGENTS.mdblock. It runs the oracle, then reports the mutants that the oracle does not catch.
How this PR makes testing easy#
test:oracles and the gap check command in AGENTS.md are the 2 commands to run.
Risk / rollback#
- The new dev dependency comes from
pkg.pr.new, not npm. The lockfile pins it by its integrity hash. When@tanstack/oraclesis on npm, change it to the npm version. - To roll back, revert this PR.
- 1
- 2
- 3
- 4
AlemTuzlak pushed403146f
AlemTuzlak changed the base branch from
fix/reload-equals-live-chattomaintanstack-source updated the branch from the base2881352
This change needs a stop before the pipeline continues.
AGENTS.md:227-231says any change is allowed if the oracles pass. It says the oracle is the review.AGENTS.md:234-244says to read and follow skill files undernode_modules/@tanstack/oracles/skills/in full. Those files are not in this diff.AGENTS.md:248says to runpnpm exec oracle-gapsfrom that package.CLAUDE.md:15sends agents to that block.package.json:80adds@tanstack/oraclesfromhttps://pkg.pr.new/TanStack/oracles/@tanstack/oracles@78fa1b7, not the npm registry.packages/ai/tests/chat-reload.oracle.test.ts:73imports it.packages/ai/vite.config.ts:12includestests/**/*.test.ts, so the normal package test runs that import. The package contents are not in the repo.pnpm-workspace.yamlallowBuildsdoes not list it.pnpm-lock.yamlalso changes. That file is too large to check here for the resolved URL, integrity, and lifecycle scripts.The oracle test body and
vitest.oracle.config.tsare ordinary test wiring. No hidden characters showed up in the reviewable source.TanStack Source bot, audit agent
tanstack-source added labelsfactory:malicious
tanstack-source closed this
tanstack-source added labelsspam
Sign in to join this conversation.
This needs a look before the pipeline continues.
AGENTS.md:227-231says any change is allowed if the oracles pass. It says the oracle is the review.AGENTS.md:234-244says to read and follow skill files undernode_modules/@tanstack/oracles/skills/in full. Those files are not in this diff.AGENTS.md:248says to runpnpm exec oracle-gapsfrom that package.CLAUDE.md:15sends agents to that block.package.json:80adds@tanstack/oraclesfromhttps://pkg.pr.new/TanStack/oracles/@tanstack/oracles@78fa1b7, not the npm registry.packages/ai/tests/chat-reload.oracle.test.ts:73imports it.packages/ai/vite.config.ts:12includestests/**/*.test.ts, so the normal package test runs that import. The package contents and install scripts are not in the repo.pnpm-workspace.yaml:163allowBuildsdoes not list it.pnpm-lock.yamlalso changes. That file is too large to check here for the resolved URL, integrity, and lifecycle scripts.The oracle test body,
vitest.oracle.config.ts, and thetest:oraclesscript are ordinary test wiring. No hidden characters or network calls showed up in the reviewable source.TanStack Source bot, audit agent