The problem ReTrace targets
On 10 November 2026, researchers from Amazon and Columbia University posted ReTrace: Distilling Tool-Using Agents into Planner-Free Workflows to arXiv. The paper deals with a cost that any team pays once it hands a standard operating procedure (SOP) to an LLM agent. An SOP is a written, step-by-step business process, such as a refund check or a mobile network fault diagnosis. A ReAct-style agent asks the model to choose every next step. A procedure that runs thousands of times a day therefore pays for the same planning thousands of times over.
ReTrace takes the SOP and a few logged agent runs (traces) and turns them into an executable workflow where ordinary code picks each next step. The model does its work during a one-off build. Any workflow that still makes a planning call to a model at run time gets rejected during the build.
How the build works
The authors start from a simple observation. Agents usually write a short rationale beside each tool call, and that rationale often states the condition that triggered the action. A bare tool-call log records what the agent did. The rationale records why, and ReTrace mines that "why" to recover the branching logic. According to the paper's method section, the pipeline runs in four stages:
- Extraction. An LLM reads each trace step and writes a record holding the guard (the condition that must be true), the intended action, the tool, its typed inputs and where each input came from. Alternatives the agent mentioned but never took are kept as well.
- Aggregation. An LLM summarises each trace against the SOP, then merges the summaries into a rule document. The rule document holds a completion check plus an ordered list of guard-action rules, and each rule cites the trace steps or SOP lines that support it. Rules for alternatives no trace ever executed are flagged "unwitnessed".
- Synthesis. An LLM compiles the rule document into Python for the public benchmarks, or into a typed workflow graph for Amazon's production platform, which the paper doesn't name. At run time the workflow stops if the completion check passes. Otherwise it runs the first rule whose guard holds, and if no rule applies, it stops and marks the task incomplete.
- Repair. On the public benchmarks, failures on training inputs go back to an LLM, which returns a full replacement program. The loop runs for up to three rounds.
The paper gives one worked example. In a telecom troubleshooting task, an agent's reasoning for switching airplane mode off might also name disabled mobile data as a possible cause. That second cause becomes an unwitnessed rule even though no trace acted on it. Only the synthesis stage depends on the target platform, so one rule document could feed different orchestration tools. For the public benchmarks, Claude Sonnet 4.6 produced the traces and ran every build stage. For the production data, a Sonnet 4.0 agent produced the traces, and Opus 4.8 handled extraction and aggregation.
The results
The team tested on two public benchmarks. τ²-Bench, from Sierra, includes a telecom support domain where the agent and a simulated user both act on a shared system. SOP-Bench holds more than 2,000 tasks from SOPs across 12 business domains, and several of its authors also worked on ReTrace. The abstract and full paper report:
- Production, 79 procedures: 93.8% completion for ReTrace against 96.8% for the agent. Cost per task fell from US$0.127 to US$0.096, a 24% saving. Median run time dropped from 68 to 37 seconds, but mean time rose from 153 to 163 seconds.
- τ²-Bench telecom: 75.8% task success against the agent's 80.6%, in 1.3 seconds instead of 170. Workflows built from two traces rose from 66.7% to 81.2% after three repair rounds.
- SOP-Bench, eight procedures: Without repair, ReTrace scored 80.6% and trailed a few-shot baseline at 83.6%. With repair, ReTrace reached 87.8% at 0.40 seconds per task, against the agent's 85.5% at 46.6 seconds.
The few-shot baseline gives the same generator the SOP, the tool definitions and full traces, and skips extraction and aggregation. In production the few-shot baseline cost only US$0.033 per task, but it completed fewer tasks. The value of the agent's rationale varied by dataset. On the production data, most of an 8.8-point gain came from four procedures where the reasoning-free workflows failed every input at the same step. On SOP-Bench, removing the rationale slightly improved the score, to 82.3%.
Limits and costs
The authors are open about the weak points. ReTrace trusts the agent's rationale, and an agent's stated reasons can be wrong or unfaithful to what drove its actions. The build checks confirm that a workflow makes no planning calls, but they don't confirm that the recovered conditions are correct. Every evaluation used new inputs to procedures the system had already seen, so a change to an SOP or its tools means a rebuild. Production completion was reported by the platform itself, which may count some failed runs as complete. The traces were picked at random, and on one SOP-Bench procedure the score swung by up to 50 points depending on which traces were drawn. Feeding the generator more traces also backfired: on τ²-Bench the share of successful builds fell to 31% at 32 traces. The SOP-Bench repair loop used training labels, so its repaired numbers aren't directly comparable with the unrepaired methods. We found no public code release on the arXiv listing.
The build itself is cheap. A τ²-Bench build from four traces takes 11 model calls, and repair adds roughly seven more. SOP-Bench builds average 17.0 calls, or 35.8 with repair. The paper estimates that a build pays for itself after about 75 executions. The authors' example production workflows ran between roughly 297,000 and 528,000 times in three months.
What builders can do with it
A break-even of around 75 runs matters for New Zealand teams, whose volumes are usually far below Amazon's. A claims triage step, a telco support script or an onboarding check that runs a few hundred times a month would clear that bar easily, provided the procedure stays stable. The latency figures matter too for any customer-facing flow where a 170-second agent loop is too slow to use.
Teams running SOP agents today can start getting ready before any code ships. Have the agent write a rationale with every tool call, and keep the full traces, since ReTrace needs both. Pick one stable, high-volume procedure and keep the compiled workflow under version control with tests, as you would any generated code. The workflow's "no rule applies" exit gives a natural point to hand the task back to the full agent or to a person. Review unwitnessed rules by hand before trusting them, because no trace ever exercised them. Because the paper's results depended so heavily on which traces were sampled, build a few times from different trace sets and compare the outputs on held-out inputs before switching anything over.
If you reimplement the pipeline from the paper's description, we'd like to see your results on the ASAI projects page.
Sources
- ReTrace: Distilling Tool-Using Agents into Planner-Free Workflows (arXiv abstract)
- ReTrace: Distilling Tool-Using Agents into Planner-Free Workflows (full HTML paper)
- τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment
- SOP-Bench: Complex Industrial SOPs for Evaluating LLM Agents
- ChatPaper listing for ReTrace
Build with people working on the same things
Join the Auckland Society for AI: NZ$599 a year with a ForgeBox and the members' projects area. Our Auckland meetups are open to everyone.