feat(ai-harness): keep routed agents' tool calls across turns, harnessAgent as a root agent, and routing.handoffTool #2145
OpenAlemTuzlakwants to merge 7 commits into
main fromt3/investigate-handoff-feedbackopened View on GitHub- .changeset/routed-agent-history.md line 9Outdated@@ -0,0 +1,9 @@1+---2+'@tanstack/ai-harness': minor3+---4+5+Routed agents now remember their own tool calls, you can pass your own handoff tool, and a `harnessAgent` can be a root agent.6+7+`routing.history` sets what each model call gets of the routed runs of earlier turns. With `'own'` (the default), a routed agent gets its own earlier runs in full, with tool calls and results. It gets the other agents and the main model as labeled text in a user message, for example `main:\n...`. The main model gets the routed answers the same way, so no model reads the words of another speaker as its own. `'all'` gives every speaker every routed run in full. `'text'` gives routed runs as text only, as before.8+`routing.handoffTool` makes the handoff tool of each speaker. It gets `from`, `targets`, `allowHuman`, and `handoff`. The tool can have any name and its own fields.9+A `harnessAgent` in `agents` with `routing` keeps one child session across turns. Each turn sends it what happened in the thread since its last turn. Its model gets the `handoff` tool, so it can hand back.docs/harness/handoffs.md line 113Outdated@@ -64,0 +93,21 @@93+ toolDefinition({94+ name: 'handoff',95+ description: 'Hand the user to another agent. Say what you did and what the user still needs.',96+ inputSchema: z.object({97+ to: z.enum(targets),98+ done: z.string().describe('What you did for the user'),99+ next: z.string().describe('What the user still needs'),100+ orderId: z.string().optional(),101+ }),102+ }).server(async ({ to, done, next, orderId }) => {103+ const order = orderId ? ` Order: ${orderId}.` : ''104+ await handoff({ to, note: `From ${from ?? 'main'}. Done: ${done}. Next: ${next}.${order}` })105+ return `Handed off to ${to}.`106+ }),107+ },108+})109+```110+111+The harness calls `handoffTool` once per turn for each speaker: the main model (`from: null`) and each root agent.112+`targets` holds the names that this speaker can hand off to. `allowHuman` is the value of `routing.allowHandoffToHuman`.113+`handoff({ to, note })` hands off. The tool call is the record in the transcript, so the harness adds no note messages.packages/ai-harness/src/session/agents.ts line 929Outdated@@ -899,5 +924,6 @@899924 })900925 signal.addEventListener('abort', onAbort, { once: true })901926 try {927+ const messages = convertMessagesToModelMessages(ctx.messages)902928 operation = session.send(903- lastUserInput(convertMessagesToModelMessages(ctx.messages)),929+ root ? await rootInput(messages, root.name) : lastUserInput(messages),packages/ai-harness/src/session/agents.ts lines 2852 to 2854Outdated@@ -2779,0 +2834,21 @@2834+ routedAgent(message) === name && !isHandoffNote(message)2835+ const lastAnswer = messages.findLastIndex(isAnswer)2836+ const latest = messages.findLastIndex((message) => message.role === 'user')2837+ const since = messages.filter(2838+ (_, index) => index > lastAnswer && index !== latest,2839+ )2840+ // No transcripts: other runs stay as their `name:\ntext` host message.2841+ const folded = await historyFor({2842+ messages: since,2843+ speaker: name,2844+ mode: 'own',2845+ childrenOf: () => [],2846+ transcriptOf: async () => [],2847+ })2848+ // A message of the user comes back as the same object. A folded message2849+ // of another speaker is a new user message that has its label already.2850+ const fromUser = new Set(since.filter((message) => message.role === 'user'))2851+ const lines = folded.flatMap((message) => {2852+ const line = textOf(message).trim()2853+ if (line === '' || message.role === 'tool') return []2854+ return [fromUser.has(message) ? `user:\n${line}` : line]packages/ai-harness/src/session/agents.ts line 2856Outdated@@ -2779,0 +2836,21 @@2836+ const latest = messages.findLastIndex((message) => message.role === 'user')2837+ const since = messages.filter(2838+ (_, index) => index > lastAnswer && index !== latest,2839+ )2840+ // No transcripts: other runs stay as their `name:\ntext` host message.2841+ const folded = await historyFor({2842+ messages: since,2843+ speaker: name,2844+ mode: 'own',2845+ childrenOf: () => [],2846+ transcriptOf: async () => [],2847+ })2848+ // A message of the user comes back as the same object. A folded message2849+ // of another speaker is a new user message that has its label already.2850+ const fromUser = new Set(since.filter((message) => message.role === 'user'))2851+ const lines = folded.flatMap((message) => {2852+ const line = textOf(message).trim()2853+ if (line === '' || message.role === 'tool') return []2854+ return [fromUser.has(message) ? `user:\n${line}` : line]2855+ })2856+ const input = lastUserInput(messages)packages/ai-harness/src/session/turn.ts lines 978 to 988Outdated@@ -959,0 +968,21 @@968+ const transcripts = new Map<string, Promise<Array<ModelMessage>>>()969+ const routedHistory =970+ handoffs && historyMode !== 'text'971+ ? routedHistoryMiddleware({972+ mode: historyMode,973+ ...(handoffTool ? { handoffToolName: handoffTool.name } : {}),974+ // ponytail: the log fold (the run store too) keeps the last975+ // `KEPT_ENDED` ended runs. An older routed run has no children976+ // there, so the model gets it as text. Keep the routed children977+ // in the fold if that matters.978+ childrenOf: (runId) =>979+ [...this.writer.state.thread.runs.values()].flatMap(980+ ({ parentRunId: parent, name, subagentRunId }) =>981+ parent === runId &&982+ name !== undefined &&983+ subagentRunId !== undefined &&984+ // Only routed root agents: a tool-mode child of the main985+ // model in the same run reports through its tool result.986+ rootAgents.some((agent) => agent.name === name)987+ ? [{ name, subagentRunId }]988+ : [],testing/e2e/tests/harness-handoff.spec.ts line 370Outdated@@ -352,3 +366,5 @@352366 await stop(request, started.runId)353367 }354368 })369+370+ test('a routed agent sees its own tool calls of an earlier turn', async ({
tanstack-source reviewed
packages/ai-harness/src/session/agents.ts line 2856Outdated@@ -2780,0 +2836,21 @@2836+ const latest = messages.findLastIndex((message) => message.role === 'user')2837+ const since = messages.filter(2838+ (_, index) => index > lastAnswer && index !== latest,2839+ )2840+ // No transcripts: other runs stay as their `name:\ntext` host message.2841+ const folded = await historyFor({2842+ messages: since,2843+ speaker: name,2844+ mode: 'own',2845+ childrenOf: () => [],2846+ transcriptOf: async () => [],2847+ })2848+ // A message of the user comes back as the same object. A folded message2849+ // of another speaker is a new user message that has its label already.2850+ const fromUser = new Set(since.filter((message) => message.role === 'user'))2851+ const lines = folded.flatMap((message) => {2852+ const line = textOf(message).trim()2853+ if (line === '' || message.role === 'tool') return []2854+ return [fromUser.has(message) ? `user:\n${line}` : line]2855+ })2856+ const input = lastUserInput(messages)
Applied a fix from CId07a002
- packages/ai-harness/src/handoff.ts line 499Outdated@@ -403,0 +479,21 @@479+ note: string,480+) {481+ const history = await store.loadThread(threadId)482+ for (let index = history.length - 1; index >= 0; index -= 1) {483+ const message = history[index]484+ if (485+ message?.role !== 'assistant' ||486+ isHandoffNote(message) ||487+ !message.toolCalls?.some((call) => call.function.name === toolName)488+ ) {489+ continue490+ }491+ const harness = isRecord(message.metadata?.harness)492+ ? message.metadata.harness493+ : {}494+ if (harness.handoffNote === note) return495+ const updated: ModelMessage = {496+ ...message,497+ metadata: {498+ ...message.metadata,499+ harness: { ...harness, handoffNote: note },
tanstack-source reviewed
packages/ai-harness/src/session/agents.ts line 2857Outdated@@ -2780,0 +2837,21 @@2837+ const since = messages.filter(2838+ (_, index) => index > lastAnswer && index !== latest,2839+ )2840+ // No transcripts: other runs stay as their `name:\ntext` host message.2841+ const folded = await historyFor({2842+ messages: since,2843+ speaker: name,2844+ mode: 'own',2845+ childrenOf: () => [],2846+ transcriptOf: async () => [],2847+ })2848+ // A message of the user comes back as the same object. A folded message2849+ // of another speaker is a new user message that has its label already.2850+ const fromUser = new Set(since.filter((message) => message.role === 'user'))2851+ const lines = folded.flatMap((message) => {2852+ const line = textOf(message).trim()2853+ if (line === '' || message.role === 'tool') return []2854+ return [fromUser.has(message) ? `user:\n${line}` : line]2855+ })2856+ const input = lastUserInput(messages)2857+ if (lines.length === 0) return inputpackages/ai-harness/src/session/turn.ts line 978Outdated@@ -957,3 +966,13 @@957966 const durableTools = this.durableTools(bindTool)958967 const signal = operation.abortController.signal959968 const { routing } = this.harness969+ // How a model call sees the routed runs of the thread. The transcripts970+ // do not change in a turn, and onConfig runs before each model call,971+ // so the turn reads each one once.972+ const historyMode = routing?.history ?? 'own'973+ const transcripts = new Map<string, Promise<Array<ModelMessage>>>()974+ const routedHistory =975+ handoffs && historyMode !== 'text'976+ ? routedHistoryMiddleware({977+ mode: historyMode,978+ ...(handoffTool ? { handoffToolName: handoffTool.name } : {}),
Applied a fix from CI8548c7b
tanstack-source reviewed
packages/ai-harness/src/routed-history.ts line 216Outdated@@ -0,0 +196,21 @@196+ * speaker becomes text in one user message, so the speaker does not read197+ * it as its own: the main model's text as `main:\ntext`, a routed answer198+ * as its `name:\ntext`, and each handoff call as one line. Other tool calls199+ * of another speaker go. `folded` collects the ids of the tool calls that200+ * this removes, so their results go too.201+ */202+function foldFor(203+ message: ModelMessage,204+ speaker: string | null,205+ handoffToolName: string,206+ folded: Set<string>,207+) {208+ if (message.role === 'tool') {209+ const answersFolded =210+ message.toolCallId !== undefined && folded.has(message.toolCallId)211+ return answersFolded ? [] : [message]212+ }213+ // A note names the agent that handed off. A router note of the main214+ // model names none.215+ const owner = routedAgent(message) ?? null216+ if (message.role !== 'assistant' || owner === speaker) return [message]packages/ai-harness/src/session/agents.ts line 2880Outdated@@ -2781,2 +2860,21 @@2860+ const line = textOf(message).trim()2861+ if (line === '' || message.role === 'tool') return []2862+ return [fromUser.has(message) ? `user:\n${line}` : line]2863+ })2864+ const input = hasNewUser ? lastUserInput(messages) : undefined2865+ if (lines.length === 0) return input ?? 'Continue.'2866+ const head =2867+ lastAnswer === -1 ? 'The conversation so far:' : 'Since your last turn:'2868+ const context: ContentPart = {2869+ type: 'text',2870+ content: [head, ...lines].join('\n\n'),2871+ }2872+ if (input === undefined) return [context]2873+ const sent: Array<ContentPart> =2874+ typeof input === 'string' ? [{ type: 'text', content: input }] : input2875+ return [context, ...sent]2876+}2877+2878+/**27812879 * The admission of agent runs: at most `max()` holders at once, no limit27822880 * when it is `undefined`. `busy` names more holders that the gate does not
Applied a fix from CI727c6c8
- packages/ai-harness/src/session/agents.ts line 2878Outdated@@ -2779,0 +2858,21 @@2858+ // No transcripts: other runs stay as their `name:\ntext` host message.2859+ const folded = await historyFor({2860+ messages: since,2861+ speaker: name,2862+ mode: 'own',2863+ childrenOf: () => [],2864+ transcriptOf: async () => [],2865+ })2866+ // A message of the user comes back as the same object. A folded message2867+ // of another speaker is a new user message that has its label already.2868+ const fromUser = new Set(since.filter((message) => message.role === 'user'))2869+ const lines: Array<string> = []2870+ const earlierMedia: Array<ContentPart> = []2871+ for (const message of folded) {2872+ if (message.role === 'tool') continue2873+ const line = textOf(message).trim()2874+ if (fromUser.has(message)) {2875+ if (line !== '') lines.push(`user:\n${line}`)2876+ if (Array.isArray(message.content)) {2877+ for (const part of message.content) {2878+ if (mediaIdOf(part) !== undefined) earlierMedia.push(part)packages/ai-harness/src/session/agents.ts line 2929Outdated@@ -2779,0 +2909,21 @@2909+ * child session can read them. Same copy as `adoptMessages`: each file gets2910+ * a new id on the child thread.2911+ */2912+async function copyThreadMedia(2913+ source: {2914+ getMedia: (id: string) => Promise<MediaRecord | null>2915+ open: (id: string) => Promise<BlobBody>2916+ },2917+ dest: {2918+ putMedia: (upload: {2919+ body: BlobBody2920+ mimeType: string2921+ name: string2922+ }) => Promise<MediaRecord>2923+ },2924+ input: UserInput,2925+): Promise<UserInput> {2926+ if (typeof input === 'string') return input2927+ const copies = new Map<string, MediaRecord | null>()2928+ const copy = async (id: string) => {2929+ if (!copies.has(id)) {packages/ai-harness/src/session/agents.ts line 2935Outdated@@ -2779,0 +2915,21 @@2915+ open: (id: string) => Promise<BlobBody>2916+ },2917+ dest: {2918+ putMedia: (upload: {2919+ body: BlobBody2920+ mimeType: string2921+ name: string2922+ }) => Promise<MediaRecord>2923+ },2924+ input: UserInput,2925+): Promise<UserInput> {2926+ if (typeof input === 'string') return input2927+ const copies = new Map<string, MediaRecord | null>()2928+ const copy = async (id: string) => {2929+ if (!copies.has(id)) {2930+ const record = await source.getMedia(id)2931+ copies.set(2932+ id,2933+ record &&2934+ (await dest.putMedia({2935+ body: await source.open(id),
tanstack-source reviewed
packages/ai-harness/src/routed-history.ts line 227Outdated@@ -0,0 +207,21 @@207+) {208+ if (message.role === 'tool') {209+ const answersFolded =210+ message.toolCallId !== undefined && folded.has(message.toolCallId)211+ return answersFolded ? [] : [message]212+ }213+ // A note names the agent that handed off. A router note of the main214+ // model names none.215+ const owner = routedAgent(message) ?? null216+ if (message.role !== 'assistant' || owner === speaker) return [message]217+ const isNote = isHandoffNote(message)218+ const from = owner ?? 'main'219+ const text = textOf(message).trim()220+ const lines: Array<string> = []221+ const calls = message.toolCalls ?? []222+ if (isNote && calls.length === 0 && text !== '') {223+ // A plain note (no handoff tool): its text says who answers next.224+ lines.push(text)225+ } else if (!isNote && text !== '') {226+ // A routed answer has its `name:` label already.227+ lines.push(owner === null ? `main:\n${text}` : text)
Applied a fix from CIf8c9ad1
tanstack-source reviewed
packages/ai-harness/src/routed-history.ts line 216Outdated@@ -0,0 +196,21 @@196+ * speaker becomes text in one user message, so the speaker does not read197+ * it as its own: the main model's text as `main:\ntext`, a routed answer198+ * as its `name:\ntext`, and each handoff call as one line. Other tool calls199+ * of another speaker go. `folded` collects the ids of the tool calls that200+ * this removes, so their results go too.201+ */202+function foldFor(203+ message: ModelMessage,204+ speaker: string | null,205+ handoffToolName: string,206+ folded: Set<string>,207+) {208+ if (message.role === 'tool') {209+ const answersFolded =210+ message.toolCallId !== undefined && folded.has(message.toolCallId)211+ return answersFolded ? [] : [message]212+ }213+ // A note names the agent that handed off. A router note of the main214+ // model names none.215+ const owner = routedAgent(message) ?? null216+ if (message.role !== 'assistant' || owner === speaker) return [message]
Applied a fix from CI97cc34a
tanstack-source reviewed
packages/ai-harness/src/session/agents.ts line 2935Outdated@@ -2780,0 +2915,21 @@2915+ getMedia: (id: string) => Promise<MediaRecord | null>2916+ open: (id: string) => Promise<BlobBody>2917+ },2918+ dest: {2919+ putMedia: (upload: {2920+ body: BlobBody2921+ mimeType: string2922+ name: string2923+ }) => Promise<MediaRecord>2924+ },2925+ input: UserInput,2926+): Promise<UserInput> {2927+ if (typeof input === 'string') return input2928+ const copies = new Map<string, MediaRecord | null>()2929+ const copy = async (id: string) => {2930+ if (!copies.has(id)) {2931+ const record = await source.getMedia(id)2932+ copies.set(2933+ id,2934+ record &&2935+ (await dest.putMedia({packages/ai-harness/src/session/agents.ts line 2876Outdated@@ -2780,0 +2856,21 @@2856+ const since = messages.filter(2857+ (_, index) => index > lastAnswer && index !== latest,2858+ )2859+ // No transcripts: other runs stay as their `name:\ntext` host message.2860+ const folded = await historyFor({2861+ messages: since,2862+ speaker: name,2863+ mode: 'own',2864+ childrenOf: () => [],2865+ transcriptOf: async () => [],2866+ })2867+ // A message of the user comes back as the same object. A folded message2868+ // of another speaker is a new user message that has its label already.2869+ const fromUser = new Set(since.filter((message) => message.role === 'user'))2870+ const lines: Array<string> = []2871+ const earlierMedia: Array<ContentPart> = []2872+ for (const message of folded) {2873+ if (message.role === 'tool') continue2874+ const line = textOf(message).trim()2875+ if (fromUser.has(message)) {2876+ if (line !== '') lines.push(`user:\n${line}`)
Applied a fix from CI16f1438
tanstack-source reviewed
packages/ai-harness/src/session/agents.ts line 2940@@ -2780,0 +2920,21 @@2920+ open: (id: string) => Promise<BlobBody>2921+ },2922+ dest: {2923+ putMedia: (upload: {2924+ body: BlobBody2925+ mimeType: string2926+ name: string2927+ }) => Promise<MediaRecord>2928+ },2929+ input: UserInput,2930+): Promise<UserInput> {2931+ if (typeof input === 'string') return input2932+ const copies = new Map<string, MediaRecord | null>()2933+ const copy = async (id: string) => {2934+ if (!copies.has(id)) {2935+ const record = await source.getMedia(id)2936+ copies.set(2937+ id,2938+ record &&2939+ (await dest.putMedia({2940+ body: await source.open(id),packages/ai-harness/src/session/agents.ts line 2883@@ -2780,0 +2863,21 @@2863+ // No transcripts: other runs stay as their `name:\ntext` host message.2864+ const folded = await historyFor({2865+ messages: since,2866+ speaker: name,2867+ mode: 'own',2868+ childrenOf: () => [],2869+ transcriptOf: async () => [],2870+ })2871+ // A message of the user comes back as the same object. A folded message2872+ // of another speaker is a new user message that has its label already.2873+ const fromUser = new Set(since.filter((message) => message.role === 'user'))2874+ const lines: Array<string> = []2875+ const earlierMedia: Array<ContentPart> = []2876+ for (const message of folded) {2877+ if (message.role === 'tool') continue2878+ const line = textOf(message).trim()2879+ if (fromUser.has(message)) {2880+ if (line !== '') lines.push(`user:\n${line}`)2881+ if (Array.isArray(message.content)) {2882+ for (const part of message.content) {2883+ if (mediaIdOf(part) !== undefined) earlierMedia.push(part)github-actions[bot](GitHub profile, opens in a new tab) assignedtombeckenham
github-actions[bot](GitHub profile, opens in a new tab) added labelswaiting-on: maintainer
Sign in to join this conversation.
🦋 Changeset detected#
Latest commit: 16f14384a454bccb69d034f87ccf71eaf8cde608
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 8 packages
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR