[Bug]: 1Password failed-batch fallback exceeds the documented 45-second wait #35930

Summary#

When a batch secret read fails, T3 Code retries the references individually and can keep waiting beyond the limit promised in its setup guides. A reduced reproduction using the real driver, a fake subprocess boundary, and a virtual clock showed that the fallback extends the overall wait while the normal batch-timeout path stops on schedule. The server should honor its documented wait bound for the secret read, including fallback attempts.

Before submitting#

• Related issues: searched open and closed issues and PRs with narrow 1Password, secret, batch, timeout, and 45-second terms; no duplicate found. Related refresh history is listed under Additional context.
• Removed credentials, tokens, private paths, and sensitive data.

Area#

apps/server

Observed behavior#

A call to the 1Password driver's supported read API for two references remains pending after 45 seconds. With a failed batch at 40 seconds and two individually successful reads taking 40 seconds each, it finishes at 120 seconds. The output records one inject and two sequential read invocations. The control, a batch that never returns, stops at 45 seconds with no fallback.

Reproduction#

1. Check out commit 59e64eef881632a23f89c4138bcf37f23d92bf93 and install its pinned dependencies. A focused install is vp install --frozen-lockfile --ignore-scripts --filter '@t3tools/secret-provider-onepassword...'.
2. Save the script below outside the checkout as repro-timeout.mjs and run node repro-timeout.mjs /path/to/checkout.
3. Observe that control.finishedAt is 45000, while failed-batch.pendingAt45 is true and failed-batch.finishedAt is 120000.

The final script reproduced this deterministically in one run. During harness construction, an incomplete dependency install and clock jumps over asynchronous temporary-file cleanup produced inconclusive controls; the final script installs the dependency closure and allows cleanup to complete between clock advances. These were harness setup failures, not observations of the product contract.

The script requires Node.js with TypeScript support, Git, and the checkout's installed pinned packages. It uses the real driver with an in-process fake ChildProcessSpawner; no op process, account, network connection, or desktop prompt is used. All fixture state lives in a directory created by mkdtemp and is removed on exit. It prints runtime and dependency versions, runs the timeout control and failed-batch case, leaves checkout files unchanged, and exits 1 for the documented-bound violation, 0 when the read finishes within the bound, or 2 if setup or the control fails. Durations come from Effect's virtual clock, not wall-clock sleeps.

import * as fs from 'node:fs';
import * as path from 'node:path';
import { createRequire } from 'node:module';
import { pathToFileURL } from 'node:url';
import { execFileSync } from 'node:child_process';
import { setTimeout as realDelay } from 'node:timers/promises';

const repo = path.resolve(process.argv[2] ?? '.');
const scratch = fs.mkdtempSync(path.join(process.env.TMPDIR ?? '/tmp', 'resolver-repro-'));
process.env.HOME = path.join(scratch, 'home');
process.env.TMPDIR = scratch;
process.env.T3CODE_HOME = path.join(scratch, 't3');
process.env.OP_CONFIG_DIR = path.join(scratch, 'op');
for (const name of ['HOME', 'T3CODE_HOME', 'OP_CONFIG_DIR']) fs.mkdirSync(process.env[name], { recursive: true });
let status = 2;
try {
  const require = createRequire(path.join(repo, 'packages/secret-provider-onepassword/package.json'));
  const load = name => import(pathToFileURL(require.resolve(name)).href);
  const [Effect, Layer, Deferred, Fiber, Clock, TestClock, Sink, Stream, NodeCrypto, NodeFileSystem, { ChildProcessSpawner }] = await Promise.all([
    'effect/Effect', 'effect/Layer', 'effect/Deferred', 'effect/Fiber', 'effect/Clock', 'effect/testing/TestClock', 'effect/Sink', 'effect/Stream', '@effect/platform-node/NodeCrypto', '@effect/platform-node/NodeFileSystem', 'effect/process',
  ].map(load));
  const { driver } = await import(pathToFileURL(path.join(repo, 'packages/secret-provider-onepassword/src/server/driver.ts')).href);
  console.log('node=' + process.version);
  console.log(execFileSync('git', ['--version'], {encoding:'utf8'}).trim());
  console.log('effect=' + JSON.parse(fs.readFileSync(require.resolve('effect/package.json'), 'utf8')).version);
  console.log('commit=' + execFileSync('git', ['-C', repo, 'rev-parse', 'HEAD'], {encoding:'utf8'}).trim());
  const refs = ['op://Automation/First/credential', 'op://Automation/Second/credential'];
  async function probe(mode) {
    const calls = [];
    let finishedAt;
    let finished = false;
    const encoder = new TextEncoder();
    const spawned = Deferred.makeUnsafe();
    const spawner = Layer.succeed(ChildProcessSpawner.ChildProcessSpawner, ChildProcessSpawner.make(command => Effect.sync(() => {
      const args = command.args;
      calls.push(args.includes('inject') ? 'inject' : 'read');
      Deferred.doneUnsafe(spawned, Effect.void);
      const code = args.includes('inject') ? 1 : 0;
      const exitCode = mode === 'hang' ? Effect.never : Effect.sleep('40 seconds').pipe(Effect.as(ChildProcessSpawner.ExitCode(code)));
      return ChildProcessSpawner.makeHandle({
        pid: ChildProcessSpawner.ProcessId(1), exitCode,
        isRunning: Effect.succeed(true), kill: () => Effect.void, unref: Effect.succeed(Effect.void),
        stdin: Sink.drain, stdout: Stream.make(encoder.encode(code === 0 ? 'dummy-token' : '')), stderr: Stream.empty, all: Stream.empty,
        getInputFd: () => Sink.drain, getOutputFd: () => Stream.empty,
      });
    })));
    return Effect.runPromise(Effect.gen(function* () {
      const start = yield* Clock.currentTimeMillis;
      const fiber = yield* Effect.forkChild(driver.read({account: 'example.1password.com'}, refs).pipe(Effect.tap(() => Effect.gen(function* () {
        finishedAt = (yield* Clock.currentTimeMillis) - start;
        finished = true;
      }))));
      yield* Deferred.await(spawned);
      yield* TestClock.adjust(mode === 'hang' ? '45 seconds' : '40 seconds');
      if (mode !== 'hang') {
        yield* Effect.promise(() => realDelay(50));
        yield* TestClock.adjust('5 seconds');
      }
      yield* Effect.promise(() => realDelay(50));
      const pendingAt45 = !finished;
      if (mode !== 'hang') {
        yield* TestClock.adjust('35 seconds');
        yield* Effect.promise(() => realDelay(50));
        yield* TestClock.adjust('40 seconds');
      }
      const values = yield* Fiber.join(fiber);
      return {calls, pendingAt45, finishedAt, readable: values.map(x => x !== undefined)};
    }).pipe(Effect.provide(Layer.mergeAll(spawner, NodeFileSystem.layer, NodeCrypto.layer, TestClock.layer()))));
  }
  const control = await probe('hang');
  console.log('control=' + JSON.stringify(control));
  if (control.pendingAt45 || control.calls.join(',') !== 'inject' || control.finishedAt !== 45000) throw new Error('timeout control failed');
  const observed = await probe('slow-failure');
  console.log('failed-batch=' + JSON.stringify(observed));
  status = observed.pendingAt45 && observed.finishedAt > 45000 ? 1 : 0;
} catch (error) {
  console.error('setup/control failure: ' + error.message);
} finally {
  fs.rmSync(scratch, { recursive: true, force: true });
}
console.log(status === 1 ? 'RESULT: defect reproduced' : status === 0 ? 'RESULT: expected behavior' : 'RESULT: inconclusive');
process.exitCode = status;

Expected behavior#

T3's own service-account guide states that it waits up to 45 seconds for 1Password and then treats the secret as unreadable: docs/user/secrets-1password-service-account.md#L51-L52. The Connect guide makes the same promise: docs/user/secrets-1password-connect.md#L56-L57. That bound should include the driver's batch and fallback work for the read. This report does not challenge the choice of a 45-second limit or the intentional decision to skip fallback when the batch itself times out.

Impact and scope#

Impact: P2 - A slow failed batch can delay provider startup or refresh well beyond the documented bound. The reproduced two-reference read took 120 seconds of virtual time instead of completing within 45 seconds. A larger fallback is sequential, so additional slow references add waits. The failure is bounded and does not require a restart to finish; checking individual references separately before a provider startup also avoids this batch-plus-fallback sequence.

This was observed at the driver boundary with dummy references, not against a live 1Password account. NOT_ENOUGH_INFO: frequency under real network, locked-app, and rate-limit conditions; establishing it would require separate authorized real-account measurements. No exact quota cost or specific CLI exit-code taxonomy is assumed.

Version#

T3 Code commit 59e64eef881632a23f89c4138bcf37f23d92bf93. Project-pinned Effect 4.1.3; global Vite+ 1.6.0, local Vite+ 1.4.0, pnpm 12.14.0 for installation. The reproduction executes the pinned source directly with Node.js.

Runtime and platform#

Node.js v26.12.1, Git 2.56.0, Darwin arm64. No real 1Password CLI is involved. The fake returns subprocess exit code 1 for the batch and 0 for each individual read, with 40 seconds of virtual elapsed time per invocation.

Interface#

Library: packages/secret-provider-onepassword server driver, used by the server's SecretResolver for provider startup and credential refresh.

Earliest failure stage#

Core processing

Relevant output or evidence#

The final script emitted:

node=v26.12.1
git version 2.56.0
effect=4.1.3
commit=59e64eef881632a23f89c4138bcf37f23d92bf93
[16:00:45.000] WARN (#13): 1Password did not answer in time { stage: 'inject' }
control={"calls":["inject"],"pendingAt45":false,"finishedAt":45000,"readable":[false,false]}
[16:00:40.000] WARN (#47): Could not batch-read secrets from 1Password { stage: 'inject', exitCode: 1, references: 2 }
failed-batch={"calls":["inject","read","read"],"pendingAt45":true,"finishedAt":120000,"readable":[true,true]}
RESULT: defect reproduced

Source: packages/secret-provider-onepassword/src/server/driver.ts#L79-L110 bounds each subprocess invocation separately. After a non-timeout batch failure, it enters a sequential Effect.forEach whose individual reads each receive a new timeout. The existing batch-timeout test verifies the no-fallback path but does not exercise the cumulative fallback duration. The resolver's read waits for this driver result while holding its semaphore: packages/secret-provider-core/src/server/SecretResolver.ts#L110-L145.

Additional context#

• https://github.com/pingdotgg/t3code/pull/31226 — explains intentional retention of unreadable references until an explicit refresh or relevant configuration change; this report concerns the duration of one actual read.
• https://github.com/pingdotgg/t3code/pull/31723 — establishes fresh checking and dependent-provider refresh; this report concerns the driver's batch fallback rather than failed-check cache retention.