Skip to content
Source
Sign inSign up
TanStack/ai

ai-sandbox-local-process: exec() decodes stdout/stderr per chunk and corrupts multi-byte UTF-8 #2153

Send to my agent
Opentrackoor(GitHub profile, opens in a new tab) opened this issue 1 comment

TanStack AI version

@tanstack/ai-sandbox-local-process@1.0.0, @tanstack/ai-sandbox@1.3.0 (also main at 1888233)

Framework/Library version

Node.js v26.13.0 on Linux x86_64 (server-side sandbox, no UI framework)

Describe the bug and the steps to reproduce it

LocalProcessHandle.exec() (packages/ai-sandbox-local-process/src/handle.ts, lines 887-890) collects command output per data event with stdout += d.toString('utf8') and stderr += d.toString('utf8'). Whenever a multi-byte UTF-8 sequence straddles two pipe reads, both halves decode into U+FFFD replacement characters.

The same file already documents this issue and uses a streaming TextDecoder (decodeStream, lines 139-161) for spawnProcess(), while exec() still decodes each Buffer chunk independently.

Steps to reproduce:

npm i @tanstack/ai-sandbox-local-process@1.0.0 @tanstack/ai-sandbox@1.3.0
node repro.mjs
// repro.mjs
import { localProcessSandbox } from "@tanstack/ai-sandbox-local-process";

const handle = await localProcessSandbox().create({});
const text = "日本語のテキスト\n".repeat(4000);
await handle.fs.write("notes.txt", text);

const { stdout } = await handle.process.exec("cat notes.txt");
console.log("same as file:", stdout === text);
console.log("U+FFFD count:", [...stdout].filter((c) => c === "\ufffd").length);
await handle.destroy();

Actual output:

same as file: false
U+FFFD count: 2

Expected output:

same as file: true
U+FFFD count: 0

Collecting Buffer chunks and decoding once on close with Buffer.concat(...).toString('utf8') (or using a streaming TextDecoder for stdout and stderr) avoids splitting multi-byte sequences.

https://gist.github.com/trackoor/8efa2e08a9f5e2d05c59708f32db154e

Screenshots or Videos (Optional)

No response

Do you intend to try to help solve this bug with your own PR?

No, because I do not have time to dig into it

Terms & Code of Conduct

  • I agree to follow this project's Code of Conduct
  • I understand that if my bug cannot be reliable reproduced in a debuggable environment, it will probably not be fixed and this issue may even be closed.

Credit: https://ucsbplse.github.io/

  1. tanstack-source added labelsbug

  2. github-actions[bot](GitHub profile, opens in a new tab) added labelswaiting-on: maintainer

  3. I've fixed this in #2162. exec() now calls setEncoding('utf8') on stdout and stderr, so Node's StringDecoder keeps a split multi-byte character until the rest arrives, the same way spawn() already does. A test covering split € on both streams is included.

Sign in to join this conversation.