feat(ai-computer-sandbox, ai-sandbox-docker, ai-sandbox-cloudflare, examples): sign in on a sandbox desktop without the model seeing the password, and take over when it is stuck #1982
- docs/tutorials/computer-use-login.md lines 14 to 20Outdated@@ -0,0 +1,20 @@1+---2+title: Sign In Without the Model3+id: computer-use-login4+order: 95+description: "Let a computer use agent sign in to a site, and keep the password away from the model. The user types the password in a card, and the server fills the form in a Docker desktop."6+keywords:7+tanstack ai8+tutorial9+computer use10+sign in11+password12+requestLogin13+sandboxComputer14+docker15+harness16+---17+18+Give the model a `requestLogin` tool with `needsApproval: true`. When a page asks for a password, the run stops and your chat shows a login card. The server fills the form in the browser, so the model never sees the password.19+20+This tutorial uses React, TanStack Start, the harness, OpenAI, and a Docker desktop with Chromium. It takes about 30 minutes. Start Docker before you begin.examples/react/computer-use-login/package.json lines 5 to 9Outdated@@ -0,0 +1,9 @@1+{2+ "name": "computer-use-login",3+ "private": true,4+ "type": "module",5+ "scripts": {6+ "dev": "vite dev --port 3100",7+ "build": "vite build",8+ "build:desktop": "docker build -t computer-use-login-desktop sandbox",9+ "test:types": "tsc"examples/react/computer-use-login/README.md lines 37 to 50Outdated@@ -0,0 +30,21 @@30+31+`src/harness.ts`: the harness, the Docker provider, the `requestLogin` tool, and the two stores.32+`src/request-login.ts`: the tool definition that the server and the browser share.33+`src/routes/api.login.ts`: the route that takes the username and password.34+`src/routes/index.tsx`: the chat and the login card.35+`sandbox/`: the desktop image. `desktop.sh` starts the screen and Chromium. `login.py` fills the form.36+37+## Run it on a server38+39+The Docker setup runs on your own machine. To host the app, use a sandbox provider that pauses a sandbox and keeps its memory, like E2B:40+41+```ts42+import { e2bSandbox } from '@tanstack/ai-sandbox-e2b'43+44+const provider = e2bSandbox({45+ template: 'your-desktop-template',46+ onTimeout: 'pause',47+})48+```49+50+Build an E2B template from `sandbox/Dockerfile`, and start `desktop.sh` in it. Then move the two stores of `src/harness.ts` to your database: the sandbox id of each thread, and the logins (with a time limit).@@ -0,0 +26,21 @@26+# Fills the last text field before the password field, then submits the form.27+# The native value setter and the input events make frameworks like React28+# see the values.29+FILL = """(username, password) => {30+ const visible = (input) => input.offsetParent !== null31+ const password_ = [...document.querySelectorAll('input[type=password]')].find(visible)32+ if (!password_) return 'The page has no visible password field.'33+ if (!password_.form) return 'The password field is not in a form.'34+ const inputs = [...password_.form.querySelectorAll('input')]35+ const user = inputs36+ .slice(0, inputs.indexOf(password_))37+ .reverse()38+ .find((input) => ['text', 'email', 'tel'].includes(input.type) && visible(input))39+ if (!user) return 'The form has no username field before the password field.'40+ const setValue = Object.getOwnPropertyDescriptor(HTMLInputElement.prototype, 'value').set41+ for (const [input, value] of [[user, username], [password_, password]]) {42+ input.focus()43+ setValue.call(input, value)44+ input.dispatchEvent(new Event('input', { bubbles: true }))45+ input.dispatchEvent(new Event('change', { bubbles: true }))46+ }examples/react/computer-use-login/sandbox/login.py lines 63 to 64Outdated@@ -0,0 +44,21 @@44+ input.dispatchEvent(new Event('input', { bubbles: true }))45+ input.dispatchEvent(new Event('change', { bubbles: true }))46+ }47+ password_.form.requestSubmit()48+ return 'submitted'49+}"""50+51+TIMEOUT_S = 3052+53+54+class LoginError(Exception):55+ pass56+57+58+async def main():59+ site = os.environ["LOGIN_SITE"]60+ username = os.environ["LOGIN_USERNAME"]61+ password = os.environ["LOGIN_PASSWORD"]62+ parts = urlsplit(site)63+ origin = f"{parts.scheme}://{parts.netloc}"64+examples/react/computer-use-login/sandbox/login.py lines 67 to 68Outdated@@ -0,0 +48,21 @@48+ return 'submitted'49+}"""50+51+TIMEOUT_S = 3052+53+54+class LoginError(Exception):55+ pass56+57+58+async def main():59+ site = os.environ["LOGIN_SITE"]60+ username = os.environ["LOGIN_USERNAME"]61+ password = os.environ["LOGIN_PASSWORD"]62+ parts = urlsplit(site)63+ origin = f"{parts.scheme}://{parts.netloc}"64+65+ with urllib.request.urlopen("http://127.0.0.1:9222/json/list") as response:66+ targets = json.load(response)67+ page = next((target for target in targets if target["type"] == "page"), None)68+ if page is None:examples/react/computer-use-login/sandbox/login.py line 79Outdated@@ -0,0 +59,21 @@59+ site = os.environ["LOGIN_SITE"]60+ username = os.environ["LOGIN_USERNAME"]61+ password = os.environ["LOGIN_PASSWORD"]62+ parts = urlsplit(site)63+ origin = f"{parts.scheme}://{parts.netloc}"64+65+ with urllib.request.urlopen("http://127.0.0.1:9222/json/list") as response:66+ targets = json.load(response)67+ page = next((target for target in targets if target["type"] == "page"), None)68+ if page is None:69+ raise LoginError("Chromium has no open tab.")70+71+ async with websockets.connect(page["webSocketDebuggerUrl"], max_size=None) as socket:72+ next_id = 073+74+ async def send(method, **params):75+ nonlocal next_id76+ next_id += 177+ await socket.send(json.dumps({"id": next_id, "method": method, "params": params}))78+ while True:79+ message = json.loads(await socket.recv())examples/react/computer-use-login/sandbox/login.py lines 110 to 114Outdated@@ -0,0 +94,21 @@94+ async def wait_for(condition):95+ deadline = time.monotonic() + TIMEOUT_S96+ while time.monotonic() < deadline:97+ try:98+ if await evaluate(condition):99+ return True100+ except LoginError:101+ pass # The page is loading. Try again.102+ await asyncio.sleep(0.2)103+ return False104+105+ await send("Page.navigate", url=site)106+ if not await wait_for("document.readyState === 'complete'"):107+ raise LoginError(f"{site} did not load in {TIMEOUT_S} s. Nothing was filled.")108+ # A redirect can leave the site. Then the form belongs to someone else.109+ page_origin = await evaluate("location.origin")110+ if page_origin != origin:111+ raise LoginError(f"The page is on {page_origin}, not {origin}. Nothing was filled.")112+113+ before = await evaluate("location.href")114+ result = await evaluate(f"({FILL})({json.dumps(username)}, {json.dumps(password)})")examples/react/computer-use-login/src/harness.ts line 52Outdated@@ -0,0 +23,21 @@23+// ponytail: one file and one process. Use your database with more servers.24+const SANDBOX_IDS_FILE = resolve('.data/sandbox-ids.json')25+const sandboxIdsSchema = z.record(z.string(), z.string())26+let sandboxIdsWrite: Promise<void> = Promise.resolve()27+28+async function readSandboxIds() {29+ try {30+ return sandboxIdsSchema.parse(31+ JSON.parse(await readFile(SANDBOX_IDS_FILE, 'utf8')),32+ )33+ } catch (error) {34+ if (error instanceof Error && 'code' in error && error.code === 'ENOENT') {35+ return {}36+ }37+ throw error38+ }39+}40+41+const sandboxIds = {42+ get: async (threadId: string) => (await readSandboxIds())[threadId],43+ set: (threadId: string, sandboxId: string) => {examples/react/computer-use-login/src/harness.ts lines 56 to 59Outdated@@ -0,0 +30,21 @@30+ return sandboxIdsSchema.parse(31+ JSON.parse(await readFile(SANDBOX_IDS_FILE, 'utf8')),32+ )33+ } catch (error) {34+ if (error instanceof Error && 'code' in error && error.code === 'ENOENT') {35+ return {}36+ }37+ throw error38+ }39+}40+41+const sandboxIds = {42+ get: async (threadId: string) => (await readSandboxIds())[threadId],43+ set: (threadId: string, sandboxId: string) => {44+ // One write at a time, so two threads do not lose each other's id.45+ sandboxIdsWrite = sandboxIdsWrite.then(async () => {46+ const ids = await readSandboxIds()47+ await mkdir(dirname(SANDBOX_IDS_FILE), { recursive: true })48+ await writeFile(49+ SANDBOX_IDS_FILE,50+ JSON.stringify({ ...ids, [threadId]: sandboxId }),examples/react/computer-use-login/src/harness.ts lines 153 to 161Outdated@@ -0,0 +62,21 @@62+ string,63+ {64+ site: string65+ username: string66+ password: string67+ timer: ReturnType<typeof setTimeout>68+ }69+>()70+71+export const loginSchema = z.object({72+ threadId: z.string().min(1).max(200),73+ site: z.url({ protocol: /^https?$/ }).max(2000),74+ username: z.string().min(1).max(500),75+ password: z.string().min(1).max(500),76+})77+78+/** Keep a username and password for the next sign-in of a thread. */79+export function saveLogin({ threadId, ...login }: z.infer<typeof loginSchema>) {80+ clearTimeout(logins.get(threadId)?.timer)81+ const timer = setTimeout(() => logins.delete(threadId), LOGIN_TTL_MS)82+ timer.unref()examples/react/computer-use-login/src/request-login.ts lines 14 to 17Outdated@@ -0,0 +1,17 @@1+import { toolDefinition } from '@tanstack/ai'2+import { z } from 'zod'3+4+// The server and the browser both import this definition. The server gives5+// it the code (`src/harness.ts`). The browser gets the typed approval card.6+export const requestLoginTool = toolDefinition({7+ name: 'requestLogin',8+ description:9+ 'Ask the user to sign in to a site. The user types the username and password in a form that you never see, and the server fills the sign-in form in the browser. Call it when a page asks for a password. Pass the address of the sign-in page.',10+ // The run stops until the user answers the login card. The answer is only11+ // "yes", so the harness log and the persistence keep no secret.12+ needsApproval: true,13+ inputSchema: z.object({14+ site: z15+ .url({ protocol: /^https?$/ })16+ .describe(17+ 'The address of the sign-in page, for example https://example.com/login',packages/ai-computer-sandbox/src/driver.ts lines 293 to 295Outdated@@ -263,4 +290,6 @@263290 `The sandbox display is ${width}x${height}, not ${display.width}x${display.height}. Set display to the size of the running display.`,264291 )265292 }293+ // A kept sandbox outlives its thread. The provider timeout ends it.294+ if (sandboxIds) return driver266295 return { ...driver, close: () => sandbox.destroy() }packages/ai-computer-sandbox/tests/driver.test.ts lines 373 to 389Outdated@@ -283,3 +306,21 @@283306 )284307 error.mockRestore()285308 })309+310+ it('with sandboxIds, resumes the stored sandbox and never destroys it', async () => {311+ const kept = fakeSandbox()312+ const provider = fakeProvider([], [kept])313+ const sandboxIds = memoryIds({ a: kept.sandbox.id })314+ const factory = sandboxComputer({ provider, sandboxIds })315+ const driver = await factory.create({ threadId: 'a' })316+ expect(provider.create).not.toHaveBeenCalled()317+ expect(driver.close).toBeUndefined()318+ expect(kept.destroy).not.toHaveBeenCalled()319+ })320+321+ it('with sandboxIds, creates and stores a sandbox when the stored one is gone', async () => {322+ const created = fakeSandbox()323+ const provider = fakeProvider([created])324+ const sandboxIds = memoryIds({ a: 'gone' })325+ const factory = sandboxComputer({ provider, sandboxIds })326+ await factory.create({ threadId: 'a' })packages/ai-sandbox-cloudflare/tests/handle.test.ts lines 366 to 378Outdated@@ -358,3 +360,19 @@358360 expect(r.exitCode).toBe(0)359361 })360362363+ it('runs each command in a subshell, so exit cannot end the shared session', async () => {364+ const commands: Array<string> = []365+ const sandbox = sandboxFixture({366+ exec: (command: string) => {367+ commands.push(command)368+ return Promise.resolve({ stdout: '', stderr: '', exitCode: 3 })369+ },370+ setEnvVars: () => Promise.resolve(),371+ exposePort: (port: number) =>372+ Promise.resolve({ url: `https://${port}.example.workers.dev` }),373+ destroy: () => Promise.resolve(),374+ })375+ const handle = new CloudflareHandle('sbx-1', sandbox, '/workspace')376+ await handle.process.exec('exit 3 # done')377+ await (await handle.process.spawn('exit 3')).wait()378+ expect(commands).toEqual(['(\nexit 3 # done\n)', '(\nexit 3\n)'])
- docs/tutorials/computer-use-login.md lines 251 to 252Outdated@@ -0,0 +232,21 @@232+ PROFILE_FILES.some(233+ (file) =>234+ path === file ||235+ path.startsWith(`${file}/`) ||236+ file.startsWith(`${path}/`),237+ )238+239+async function browser(sandbox: SandboxHandle, action: 'stop' | 'start') {240+ const result = await sandbox.process.exec(`/opt/desktop/browser.sh ${action}`)241+ if (result.exitCode !== 0) {242+ throw new Error(`Could not ${action} Chromium: ${result.stderr.trim()}`)243+ }244+}245+246+const snapshots = {247+ store: await memorySandboxSnapshots(),248+ root: PROFILE,249+ policy: { exclude: (path: string) => !keepInProfile(path) },250+ // Chromium writes its profile all the time. Stop it, so the files are complete.251+ before: (sandbox: SandboxHandle) => browser(sandbox, 'stop'),252+ after: (sandbox: SandboxHandle) => browser(sandbox, 'start'),examples/react/computer-use-login/sandbox/desktop.sh line 22Outdated@@ -0,0 +1,14 @@1+#!/bin/sh2+# The main process of the sandbox. It starts the screen and Chromium. When3+# Chromium closes, it starts again. The profile folder keeps the cookies, so a4+# restarted container is still signed in.5+# Chromium listens for the debugging protocol on 127.0.0.1 inside the6+# container only. The sign-in script uses it. No port leaves the container.7+rm -f /tmp/.X0-lock8+Xvfb :0 -ac -screen 0 1024x768x24 -nolisten tcp &9+until DISPLAY=:0 xdpyinfo >/dev/null 2>&1; do sleep 0.2; done10+11+while true; do12+ # browser.sh stop pauses this loop while a snapshot saves or restores the profile.13+ # With no address, Chromium opens the tabs of its last session (RestoreOnStartup).14+ while [ -e /tmp/chromium-paused ]; do sleep 0.2; doneexamples/react/computer-use-login/src/harness.ts lines 104 to 111Outdated@@ -0,0 +57,21 @@57+ })58+ return sandboxIdsWrite59+ },60+}61+62+// The Chromium profile of each thread, saved after a sign-in and when the63+// thread goes idle. A thread whose container is gone gets a new container64+// with this profile, so it is still signed in.65+// ponytail: memory only, like the chat. Use durable stores to keep a sign-in66+// after a server restart. The profile holds session cookies: keep it secret.67+const PROFILE = '/root/chromium-profile'68+// Only the files that keep a sign-in. Caches stay out of the snapshot.69+const PROFILE_FILES = [70+ 'Local State',71+ 'Default/Cookies',72+ 'Default/Cookies-journal',73+ 'Default/Preferences',74+ 'Default/Sessions',75+ 'Default/Local Storage',76+ 'Default/Session Storage',77+]examples/react/computer-use-login/src/harness.ts lines 127 to 128Outdated@@ -0,0 +74,21 @@74+ 'Default/Sessions',75+ 'Default/Local Storage',76+ 'Default/Session Storage',77+]78+const keepInProfile = (path: string) =>79+ PROFILE_FILES.some(80+ (file) =>81+ path === file ||82+ path.startsWith(`${file}/`) ||83+ file.startsWith(`${path}/`),84+ )85+86+async function browser(sandbox: SandboxHandle, action: 'stop' | 'start') {87+ const result = await sandbox.process.exec(`/opt/desktop/browser.sh ${action}`)88+ if (result.exitCode !== 0) {89+ throw new Error(`Could not ${action} Chromium: ${result.stderr.trim()}`)90+ }91+}92+93+const snapshots = {94+ store: await memorySandboxSnapshots(),packages/ai-computer-sandbox/README.md lines 93 to 100Outdated@@ -68,0 +80,21 @@80+ },81+ }),82+})83+```84+85+A later call of the thread reconnects with `provider.resume`. If the sandbox is gone, the driver creates a new one and stores its id.86+The driver never destroys a kept sandbox. The provider timeout pauses or removes it.87+88+## Restore files into a new sandbox89+90+Give `sandboxComputer()` a `snapshots` option. The driver saves the files of one folder, like the browser profile, when a thread closes or goes idle. The next new sandbox of the thread gets those files back. This works with every provider.91+92+```typescript93+const computerUse = createComputerUse({94+ driver: sandboxComputer({95+ provider,96+ snapshots: {97+ store: await memorySandboxSnapshots(),98+ root: '/home/user/browser-profile',99+ before: (sandbox) => stopBrowser(sandbox),100+ after: (sandbox) => startBrowser(sandbox),packages/ai-computer-sandbox/README.md lines 96 to 98Outdated@@ -68,0 +78,21 @@78+ get: (threadId) => db.getSandboxId(threadId),79+ set: (threadId, sandboxId) => db.setSandboxId(threadId, sandboxId),80+ },81+ }),82+})83+```84+85+A later call of the thread reconnects with `provider.resume`. If the sandbox is gone, the driver creates a new one and stores its id.86+The driver never destroys a kept sandbox. The provider timeout pauses or removes it.87+88+## Restore files into a new sandbox89+90+Give `sandboxComputer()` a `snapshots` option. The driver saves the files of one folder, like the browser profile, when a thread closes or goes idle. The next new sandbox of the thread gets those files back. This works with every provider.91+92+```typescript93+const computerUse = createComputerUse({94+ driver: sandboxComputer({95+ provider,96+ snapshots: {97+ store: await memorySandboxSnapshots(),98+ root: '/home/user/browser-profile',packages/ai-computer-sandbox/src/driver.ts lines 40 to 41Outdated@@ -24,3 +30,12 @@2430 provider: SandboxProvider2531 /** The screen size in pixels. The default is `{ width: 1024, height: 768 }`. */2632 display?: { width: number; height: number }33+ /**34+ * Keep the sandbox of each thread, and reconnect to it later. Store the ids35+ * where your threads persist. With this option, the driver does not destroy36+ * a sandbox when its thread closes or goes idle. The provider timeout37+ * pauses or removes it, like E2B `onTimeout: 'pause'`.38+ */39+ sandboxIds?: {40+ get: (threadId: string) => Promise<string | undefined>41+ set: (threadId: string, sandboxId: string) => Promise<void>packages/ai-computer-sandbox/src/driver.ts lines 326 to 328Outdated@@ -263,4 +323,6 @@263323 `The sandbox display is ${width}x${height}, not ${display.width}x${display.height}. Set display to the size of the running display.`,264324 )265325 }266- return { ...driver, close: () => sandbox.destroy() }326+ if (isNew && snapshots) {327+ await restoreComputerSnapshot(sandbox, threadId, snapshots)328+ }packages/ai-computer-sandbox/src/driver.ts lines 374 to 375Outdated@@ -280,0 +355,21 @@355+async function destroyAfterFailure(sandbox: SandboxHandle) {356+ await sandbox.destroy().catch((destroyError: unknown) => {357+ console.error(358+ 'Failed to destroy the sandbox after setup failed.',359+ destroyError,360+ )361+ })362+}363+364+/**365+ * Save the files of `snapshots.root` as the newest snapshot of the thread.366+ * `sandboxComputer` calls it when a thread closes or goes idle. Call it367+ * yourself after a step that you must keep, like a sign-in.368+ */369+export async function saveComputerSnapshot(370+ sandbox: SandboxHandle,371+ threadId: string,372+ { store, root, policy, before, after }: SandboxComputerSnapshots,373+) {374+ await before?.(sandbox)375+ try {
github-actions[bot](GitHub profile, opens in a new tab) assignedjherr
github-actions[bot](GitHub profile, opens in a new tab) added labelsmerge-conflicts
github-actions[bot](GitHub profile, opens in a new tab) added labelswaiting-on: author
tombeckenham changed the title
- docs/tutorials/computer-use-login.md line 655Outdated@@ -0,0 +631,21 @@631+ ```632+633+In `src/routes/index.tsx`, add `handOffTool.client()` to the `tools` of `useChat`. Then show the live view in an `<iframe>`. Add `&view_only=1` to its address, except while a `handOff` approval waits:634+635+ ```tsx ignore636+ const handedOff = interrupts.some(637+ (interrupt) =>638+ interrupt.kind === 'tool-approval' && interrupt.toolName === 'handOff',639+ )640+641+ <iframe642+ src={handedOff ? url : `${url}&view_only=1`}643+ title="Live view of the desktop"644+ />645+ ```646+647+ For the `handOff` approval, show `interrupt.originalArgs.reason` with a Done button that calls `interrupt.resolveInterrupt(true)`. Copy the `LiveView` and `HandOffCard` components from the [example](https://github.com/TanStack/ai/blob/main/examples/react/computer-use-login/src/routes/index.tsx).648+649+Start the app, and send a task that needs a person, for example "Open https://www.google.com/recaptcha/api2/demo and send the form." The model calls `handOff`. Tick the captcha in the live view, then click Done. The model continues.650+651+The model sees only the screenshots that it takes after you click Done. If the user types a password in the live view, the model sees only dots.examples/react/computer-use-login/README.md line 42Outdated@@ -0,0 +20,21 @@20+## How the password stays away from the model21+22+The model calls `requestLogin({ site })`. The tool has `needsApproval: true`, so the run stops, and the login card shows.23+The card sends the username and password to `/api/login`. That route keeps them in memory for 5 minutes, and logs nothing.24+The card then approves the tool call. The approval is only "yes". The harness writes approvals to its log, so a password must never be in one.25+The tool takes the login from memory and deletes it. It runs `sandbox/login.py` in the container, with the password in the environment of that one process, not on its command line.26+The script opens the site in Chromium and checks its origin. It fills the username and the password field, then submits the form. Then it clears every password field.27+The tool saves a snapshot of the Chromium profile, with the cookies of the sign-in.28+The model gets "Signed in" and the address of the page.29+30+The model sees the screen, so Chromium must never show a password. `sandbox/policy.json` turns off the password manager and `javascript:` URLs. Chromium listens for its debugging protocol only inside the container. That port never leaves it.31+32+## Take over the desktop33+34+The live view is noVNC, a VNC viewer in the browser. It is view-only while the model works.35+36+The model calls `handOff({ reason })`, for example for a captcha or a two-factor code. The tool has `needsApproval: true`, so the run stops.37+A card shows the reason. The live view now takes your mouse and keyboard.38+Do the step on the desktop, then click Done. The model takes a screenshot and continues. Click Cancel to tell the model that you did not do it.39+40+The model sees only the screenshots that it takes after you click Done. If you type a password in the live view, the model sees only dots. The login card is still the better path for a password, because the server checks the origin of the page before it fills the form.examples/react/computer-use-login/sandbox/desktop.sh line 14Outdated@@ -0,0 +1,14 @@1+#!/bin/sh2+# The main process of the sandbox. It starts the screen and Chromium. When3+# Chromium closes, it starts again. The profile folder keeps the cookies, so a4+# restarted container is still signed in.5+# Chromium listens for the debugging protocol on 127.0.0.1 inside the6+# container only. The sign-in script uses it. That port never leaves the container.7+rm -f /tmp/.X0-lock8+Xvfb :0 -ac -screen 0 1024x768x24 -nolisten tcp &9+until DISPLAY=:0 xdpyinfo >/dev/null 2>&1; do sleep 0.2; done10+11+# The live view. x11vnc shares the screen on port 5900 inside the container,12+# with a new password at each start. websockify serves noVNC on port 6080,13+# which the app publishes on 127.0.0.1 only. The server reads the password.14+head -c 64 /dev/urandom | tr -dc 'A-Za-z0-9' | head -c 8 > /tmp/vnc-passwordexamples/react/computer-use-login/src/harness.ts lines 80 to 85Outdated@@ -0,0 +65,21 @@65+66+/**67+ * The container of a thread, or null before the model's first action. The68+ * container can be stopped, for example when the user answers a card hours69+ * later. `resume` starts it again.70+ */71+async function threadSandbox(threadId: string) {72+ const id = await sandboxIds.get(threadId)73+ return id === undefined ? null : provider.resume({ id })74+}75+76+/** The noVNC address of a thread's desktop, or null when it has none yet. */77+export async function liveViewUrl(threadId: string) {78+ const sandbox = await threadSandbox(threadId)79+ if (!sandbox) return null80+ const [{ url }, password] = await Promise.all([81+ sandbox.ports.connect(LIVE_VIEW_PORT),82+ sandbox.process.exec('cat /tmp/vnc-password'),83+ ])84+ if (password.exitCode !== 0) {85+ throw new Error(`No live view password: ${password.stderr.trim()}`)examples/react/computer-use-login/src/routes/index.tsx lines 250 to 271Outdated@@ -0,0 +246,21 @@246+ if (url) return247+ const load = async () => {248+ try {249+ const response = await fetch(250+ `/api/live?${new URLSearchParams({ threadId: THREAD_ID })}`,251+ )252+ if (!response.ok) {253+ throw new Error(`No live view (${response.status}).`)254+ }255+ setUrl(liveViewSchema.parse(await response.json()).url)256+ setError(undefined)257+ } catch (loadError) {258+ setError(259+ loadError instanceof Error ? loadError.message : String(loadError),260+ )261+ }262+ }263+ void load()264+ const timer = setInterval(load, 2000)265+ return () => clearInterval(timer)266+ }, [url])examples/react/computer-use-login/src/routes/index.tsx lines 287 to 292Outdated@@ -0,0 +267,21 @@267+268+ return (269+ <div className="flex w-1/2 flex-col gap-2 border-l border-orange-500/10 p-4">270+ <p className="text-sm text-gray-400">271+ {interactive272+ ? 'You have the desktop. Click Done in the chat when you finish.'273+ : 'Live view. The model has the desktop.'}274+ </p>275+ <div276+ className={`aspect-4/3 w-full overflow-hidden rounded border ${277+ interactive278+ ? 'border-sky-500 ring-2 ring-sky-500/50'279+ : 'border-gray-700'280+ }`}281+ >282+ {url ? (283+ <iframe284+ src={interactive ? url : `${url}&view_only=1`}285+ title="Live view of the desktop"286+ className="h-full w-full"287+ />
- examples/react/computer-use-login/src/harness.ts lines 217 to 219Outdated@@ -0,0 +199,21 @@199+ }200+201+ // The password goes in the environment of one process, not on its command202+ // line. The script opens the site, checks its origin, then fills the form.203+ const result = await sandbox.process.exec('python3 /opt/desktop/login.py', {204+ env: {205+ LOGIN_SITE: site,206+ LOGIN_USERNAME: login.username,207+ LOGIN_PASSWORD: login.password,208+ },209+ })210+ if (result.exitCode !== 0) {211+ throw new Error(`The sign-in failed: ${result.stderr.trim()}`)212+ }213+ const page = loginResultSchema.parse(JSON.parse(result.stdout))214+ // A wrong password. The error brings the login card back, and the card215+ // asks the user to try again.216+ if (page.hasPasswordField) {217+ throw new Error(218+ `The sign-in failed: ${page.url} still asks for a password. Call requestLogin again, so the user can try again.`,219+ )examples/react/computer-use-login/src/harness.ts line 244Outdated@@ -0,0 +224,21 @@224+})225+226+// The model is stuck. The user used the desktop in the live view and clicked227+// Done. A click on Cancel declines the call, and the model gets told so.228+const handOff = handOffTool.server(229+ () => 'The user is done. Take a screenshot to see the screen.',230+)231+232+// One time, outside `plugins`, so every session shares the containers.233+const computer = computerUse({234+ driver: sandboxComputer({ provider, sandboxIds, snapshots }),235+ // Pages load in a real browser. Give each one time to draw.236+ screenshotDelay: 1000,237+})238+239+const agent = defineHarness({240+ name: 'computer-use-login',241+ adapter: openaiText('gpt-6.1-sol'),242+ systemPrompts: [243+ 'You use a Linux desktop with Chromium on it. When a page asks you to sign in, call requestLogin with the address of the sign-in page. Never ask for a password in the chat, and never type one.',244+ 'When a sign-in fails, call requestLogin again. When you are stuck on anything else and cannot continue alone, call handOff. The user then uses the desktop and tells you when they are done.',
github-actions[bot](GitHub profile, opens in a new tab) added labelswaiting-on: maintainer
github-actions[bot](GitHub profile, opens in a new tab) removed labelswaiting-on: author
github-actions[bot](GitHub profile, opens in a new tab) removed labelsmerge-conflicts
github-actions[bot](GitHub profile, opens in a new tab) added labelsmerge-conflicts
github-actions[bot](GitHub profile, opens in a new tab) added labelswaiting-on: author
github-actions[bot](GitHub profile, opens in a new tab) removed labelswaiting-on: maintainer
Sign in to join this conversation.
🦋 Changeset detected#
Latest commit: e235c212d62815d3887bc35574d4ee0347aca096
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 4 packages
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR