Software factory / Runs
Why a run failed
A run of an agent did not succeed, and you want to know if you must fix something, or only try again. Each run that failed names its cause and the next step.
9 min read
You need:
- Write access, or you are the author of the issue or the pull request. Everyone else reads a general reason. See Who sees what.
1. Read the badge and the reason
A run that failed, was skipped or was stopped shows three things:
- A badge that names the cause, for example Failed, can retry or Key refused.
- The reason: one sentence that says what happened.
- The next step: what to do now, with a link to the page where you do it.
You see all three under the row of the agent in the Factory panel, and on the page of the run. The runs list shows the badge and the reason.
Note
The reason never holds the text of the model provider. With write access, open the log of the run. Its notes at the top show the full error, with the model key hidden.
2. Look up the cause
Find the badge in one of these tables:
- Temporary causes can go away by themselves. Source retries the run once by itself.
- Causes that need a person wait until a person fixes the cause and clicks Retry.
3. Fix the cause and retry
Do what the table and the next step say. Then click Retry under the row of the agent. See Retry a run.
Done
You know why the run failed, and what fixes it. After the fix, Retry starts the agent again.
Temporary causes
A temporary cause can go away by itself. Source retries the run once by itself. See Source retries a temporary failure once.
| Badge | What happened | What to do |
|---|---|---|
| Failed, can retry | The model provider was busy or limited the requests. The model call failed 3 times. | Retry the run in a few minutes. |
| Failed, can retry | The run used all of its time: 15 minutes for Architecture and the steps of PR summary, 2 hours for every other step. | Retry the run. |
| Failed, can retry | The model sent nothing for 5 minutes. | Retry the run. |
| Failed, can retry | The job of the run did not start. | Retry the run. |
If the provider is busy for a short time, the forge tries the model call again, up to 2 times, before the run fails. A short busy time is a rate limit, an overload, a server error, or a stream that the provider cut before its end.
Causes that need a person
After one of these causes, the agent waits until a person fixes the cause and clicks Retry.
Money:
| Badge | What happened | What to do |
|---|---|---|
| Out of credits | The model account has no credits or no quota, or the run used the money left of the monthly spend cap. | See Out of credits. |
| Spend cap reached | The owner reached its monthly spend cap before the run started. The run is skipped. | Raise the cap in the Factory settings, then retry the run. |
| No AI left this month | Billing is on, and the plan, the credits and the overage of the owner have no AI left. The run is skipped, or it stops. | See When nothing is left. Then retry the run. |
| No container time left this month | Billing is on, and the owner has no CI minutes, credits or overage left. The run is skipped. | See When nothing is left. Then retry the run. |
The model:
| Badge | What happened | What to do |
|---|---|---|
| Key refused | The model provider refused the key (401 or 403). | Save a new key in Provider keys, then retry the run. |
| Model not found | The model provider does not have this model (404). | Pick another model in the Factory settings, then retry the run. |
| Model not ready | The provider of the model has no key, the model has no price, or it needs an AI Gateway that this forge does not have. | Fix the model in the Factory settings, then retry the run. |
Custom agents:
| Badge | What happened | What to do |
|---|---|---|
| Budget used | The agent used its own budget from its factory file, for example The agent stopped: it used its budget of 100000 tokens, from its factory file. | Raise the budget in the file, then retry the run. |
| Build missing | The build of the custom agent is gone. | Push to the default branch to build it again, then retry the run. |
| Agent missing | The factory file has no custom agent with this name. | Add the agent to .source/factory.ts, or turn it off. |
Other causes:
| Badge | What happened | What to do |
|---|---|---|
| Failed its checks | The answer of the agent did not have the expected shape, so nothing was stored. | Read the run log, then retry the run. |
| Daily limit reached | The author used all of their agent runs for today. The run is skipped. The limit starts again at midnight UTC. | Retry the run now. On a pull request, a push on a later day also starts the steps again. |
| Failed | Something else went wrong in Source. | Retry the run. If it fails again, read the run log. |
Out of credits
A run can stop because there is no money for its model. Your code and the agent are not broken. The run shows Out of credits as a warning, not as an error in red.
The reason says which of the two stopped the run:
The model account has no credits or no quota left.The provider answered 402 orinsufficient_quota, or its message names credits, billing or quota.The run used the money left of the monthly model spend cap.The run was on a model that the forge pays for, and the owner reached the cap while the run was going.
To fix it:
- If the model account has no credits, add credits to the account at the provider.
- If the cap stopped the run, open the Factory settings from the link of the next step. The page shows what the owner spent this month, and the cap.
- If the owner is an organization with a lower cap, its owners can raise the cap. See Set a lower cap for an organization.
- If the cap cannot go higher, wait for the next UTC month, or use a model with your own key.
- Retry the run.
The job of a run that is out of credits does not count as a failed job. The same is true for a run that stopped with No AI left this month.
The model key or account
The reason says what the provider did, for example The model provider refused the model key. To see the words of the provider:
- Open the page of the run. You need write access.
- Read the notes at the top of the log. The note of the error starts with
The error:, and the key is hidden. - Fix the key or the model in Provider keys or in the Factory settings.
- Retry the run.
Who sees what
| Who | Reason | Next step and Retry | Log |
|---|---|---|---|
| People with write access | The exact reason | Yes, for every run that can run again | Yes |
| The author of the issue or the pull request | The exact reason | Yes. Retry only for a temporary cause. | No |
| Everyone else who can read the item, also people who are not signed in | A general reason, for example "The run could not finish. A maintainer can retry it." | No | No |
The general reason never shows the billing, the model key or the settings of the owner. REST, MCP and src follow the same rule. See Retry from src, MCP or REST.
Source retries a temporary failure once
A run failed for a temporary cause. When the run ends, Source starts one new run of the same agent, on the same head. You do nothing.
- The new run does not count toward the daily runs of the author. The spend cap and the CI container time of the author count it as any run.
- An automatic retry gets no automatic retry of its own. If it fails too, click Retry.
- Only a step that a person can retry gets an automatic retry. See When a run can run again.
A lost container keeps its files
A long run can lose its container. The run then tries again in a new container. The agent goes on from its log, and its files come back.
- While the run works, Source saves the whole
/workspacefolder,.gitincluded. It saves after a tool call that runs a command or writes a file. It saves at most once every 5 minutes, and only when a file changed. - In the new container, the run gets its newest save. If the run has no save yet, the run checks out its commit again.
- If the run has a save but the restore fails, the run fails with the reason "its saved files could not be restored". It does not go on from a fresh checkout, because its log names changes that the new files do not have. Click Retry to start it again.
- A custom agent also gets its files back. Its own code starts again from the start, because it keeps no log.
- Source deletes the saves when the run ends, and when the repository is deleted.
A run whose tools only read has no container, so it has no saves.
Why an agent does not start again
After a temporary cause, new activity can start the agent again:
- On an issue, a new label starts the issue agents again.
- On a pull request, a push starts the steps on the new head.
After any other cause, a new label does not start the agent. Fix the cause, then click Retry.