What the paper tested
Plenty of people building coding agents have to decide how to spend extra inference budget on a hard problem. The most common answer is best-of-N sampling: run N independent agents on the same task and keep whichever result scores highest. The other answer is to put the agents in one big collaborating swarm. A paper submitted to arXiv on 6 November 2026 by Junlin Wang, Haozhe Chen, Ang Li, Rahul Chalamala and Zhuang Liu argues that both approaches waste compute. The authors are from Duke University, Modal Labs, Princeton University and New York University. Their proposal, which they call parallel agent teams, is to run many small teams at once and keep the best team's output.
The experiments ran on 12 long-horizon tasks: GPU kernel optimisation, a formal proof task in mathematics, a browser exploit, machine learning research problems and a structural topology optimisation. Each task had an automatic verifier, meaning a program that scores a submission from 0 to 100 without a human in the loop. According to the paper's full text, the main model was Qwen3.8-27B served with SGLang on H200 GPUs. Each agent ran inside OpenAI's Codex CLI in its own Modal sandbox for 12 hours (6 hours on the exploit task). The study covers more than 8,000 agent runs and roughly 95,000 agent-hours.
How a team works
The team design is deliberately plain. Agents in a team have no roles, no manager and no assigned work. Each agent keeps its own sandbox and context window and can use a shared forum through a small command-line script with post, list and read commands. Agents aren't required to post or read, and nothing gets pushed into their context. The prompt norms ask agents to post clear wins and failures with the exact config change and the measured score, and to check the forum list before each experiment. A team's score is the best verified submission from any of its members.
The authors also tried an orchestrated version, where one agent plans and assigns work to nine isolated workers. That variant scored worse at every budget tested, which is a useful data point for anyone who has defaulted to a planner and worker architecture.
The headline numbers
With a fixed budget of 100 agents, splitting them into ten teams of ten beat 100 independent agents on 10 of the 12 tasks. Averaged across all 12 tasks, the parallel teams scored about 10% higher than parallel solo agents at a budget of 10 agents and 27% higher at 100. The largest gap was on the formal proof task (Carleson Formalization), where the best of 100 solo agents reached 19.0 and the best of ten teams reached 54.8. The authors also ran single teams of 100 agents on four tasks, and ten teams of ten beat those single big teams by 8% to 30% (paper).
An independent write-up on DEV Community reports the same figures and adds one about diminishing returns: in the experiments, 50 solo agents captured 81% to 100% of the gain that 100 solo agents achieved. Past a certain point, extra independent samples buy very little.
Why small teams win: the statistics
The paper's explanation rests on a standard approximation for the expected best of N samples: the mean plus the standard deviation multiplied by the square root of 2 ln N. Independent agents give you many samples, but each sample has a low mean. One giant team has a higher mean, but it's a single sample, so you get no benefit from variance. Small teams sit between those extremes. On every task, a team of ten had a mean between 0.5 and 4.4 standard deviations above a single agent, and ten such teams still leave room for the best-of-N variance gain. The authors report that this simple model predicted whether teams would help on 9 of the 12 tasks (paper).
The forum also has a scaling cost. Team members used between 1.0 and 2.2 times the tokens of an independent agent. For the 64-agent and 100-agent teams, the forum index had to be limited to the newest 100 titles plus unread posts, so a big team ends up competing for attention on the board.
Costs and the frontier model comparison
Appendix J of the paper sets out the costs. The researchers priced the open-weight model by GPU time, with a four-GPU H200 engine at $4.54 per GPU-hour. That comes to $18.16 per 12-hour agent run on engines serving twelve agents, and $21.79 on the kernel engines serving ten. At those rates a full 100-agent budget costs roughly $1,800 per task, and that figure is our own multiplication. For comparison, the paper lists GPT-5.6-sol at $4 per million input tokens and $20 per million output tokens. A Qwen team of ten costs about the same as one or two GPT-5.6-sol agents.
At equal spend, the Qwen teams beat five GPT-5.6-sol agents on 4 of the 8 tasks where enough frontier runs existed. On the proof task, eight Qwen teams scored 52.0 against 47.9 for five GPT-5.6-sol agents at the same price. A mid-sized open model run in small teams can match a frontier API on half of these tasks, though the frontier model still wins the other half.
Infrastructure is the cost people tend to forget. A single 100-agent team on one kernel task needed 100 sandboxes, 100 H100 judge GPUs and 40 H200 GPUs running together for 12 hours. Few teams in New Zealand have that kind of local capacity, so anyone trying the full recipe here would be renting offshore GPUs. That brings latency, billing in US dollars and data residency questions for any proprietary code that goes into the sandboxes.
Limits worth knowing
The authors list several limitations themselves. The method depends on a verifier, and without one you're left with the separate problem of deciding which team's output is best. The statistical model is only qualitative when most attempts fail or scores hit a ceiling. Each task had a pool of only 10 or 11 team runs, and teams larger than ten were tested on just four tasks with one to five runs per size. Only one agent harness was used, and session length was varied on a single task. The paper hasn't been peer reviewed. The abstract links a code repository, but the link returned a 404 on 11 November 2026, so the forum and harness code isn't available to inspect yet.
The DEV Community post recommends pods of five to ten workers, but the paper itself only tested teams of ten against other sizes on a handful of tasks. Treat the ideal team size as an open question.
What builders can do with this
The pattern applies wherever a task has a cheap, trustworthy automatic score: kernel and query performance, test suites, fuzzing harnesses, optimisation problems with a measurable objective, or proof checkers such as Lean. If your agent pipeline already uses best-of-N on a task like that, the experiment is cheap to set up. Keep the total agent count fixed, group the agents into a few small teams that share an append-only notes file or a small HTTP board, and compare the best team score against your current best-of-N score. The paper's forum norms are easy to copy: post wins and failures with the exact diff and the measured score, and check the board before starting a new experiment.
Small agent teams also suit the open-weight models New Zealand builders can host on rented GPUs, because the paper's cost results come from a 27B model priced by GPU-hour. If you try parallel teams on your own verifiable tasks, we'd like to see the results on the ASAI projects page.
Sources
- arXiv 2611.09798: Test-Time Compute Scales Better with Parallel Agent Teams (abstract)
- Test-Time Compute Scales Better with Parallel Agent Teams (full HTML text)
- Why Parallel Agent Teams Outscale Solo Best-of-N, DEV Community
- modal-projects/parallel-agent-teams-scaling on GitHub (returned 404 on 11 November 2026)
Build with people working on the same things
Join the Auckland Society for AI: NZ$599 a year with a ForgeBox and the members' projects area. Our Auckland meetups are open to everyone.