What the benchmark tests
Most coding-agent benchmarks give the agent a written issue and a repository. SWE-CoWork removes the written spec. Xinping Lei and 25 co-authors posted the paper to arXiv on 15 November 2026. Each of its 101 tasks starts the agent in a real open-source repository with a short onboarding note (median 206 words) and a brief handoff. Most of what the agent needs to build is held elsewhere: by simulated colleagues, on a task board, in versioned documents and in old chat threads.
The project repository counts 2,712 scored requirements across the tasks, and 2,217 of them (about 82%) are held only by colleagues. The agent has to find out who owns each piece and ask questions that earn an answer. Only after gathering requirements can the agent build and test the change. The tasks are large. According to the full paper, the median reference solution adds 2,955 production lines across 60 files, and each run has an eight-hour budget.
How the simulated workplace works
Each task has 26 simulated colleagues, and a median of six of them hold requirements that decide the outcome. A language model writes the colleagues' replies, and the repository lists gpt-5.6-terra as the default colleague model. The environment decides what each colleague may reveal at each step. A colleague discloses a requirement only once the agent supplies enough code context. Otherwise the colleague may ask for a reproduction, refer the agent to another owner, or defer until a follow-up. The workplace service logs every disclosure against the question that triggered it, so the researchers can trace how each requirement reached the agent or failed to.
The workplace also contains deliberate traps. The paper defines five challenge types, including late evidence, superseded conclusions, ambiguous provenance and scope distractors such as cancelled cards. The full release holds 10,329 board cards, 5,097 versioned documents, 40,836 chat messages and 1,194 merge requests.
Scoring uses hidden tests. Fail-to-pass (F2P) tests check the new behaviour a task asks for. Pass-to-pass (P2P) tests check that existing behaviour still works. A task counts as complete only when every F2P and P2P test passes in the same run.
The results
The authors evaluated 11 models. GPT-6 Astra ran in a Codex scaffold and the other ten ran in Claude Code. On the project leaderboard, Claude-Opus-5.5 passed 63.84% of F2P tests and GPT-6 Astra passed 62.91%. Their complete-task rates were 6.93% and 7.92%. No other model completed more than one task out of 101, and Qwen3.8-Max, DeepSeek V4 Pro and several others completed none.
The paper's strongest signal concerns information gathering. Across models, the share of colleague-held requirements an agent obtained tracked its F2P score with a Spearman ρ of 0.94. Spearman ρ is a rank correlation, where 1 means two lists are in identical order. Tool-call counts showed almost no relationship (ρ = 0.08). The authors point out that the correlation doesn't establish causation.
A detailed analysis of Claude-Opus-5.5 runs shows where the losses happen. Opus sent 93% of its first questions to the accountable owner but obtained only 64% of colleague-held requirements. In 450 threads where an owner deferred, the agent never followed up on 153. Those abandoned threads still had a median of 7.1 hours of budget left at the time of deferral. Even when a requirement reached the agent, it could still fail at the coding step: 56% of failed F2P tests linked to a requirement failed at implementation. At handoff, Opus had reported 826 requirements as complete, and 255 of them (31%) failed their tests.
The paper also ran an Oracle-Spec condition, which hands the agent the full requirement set at the start. GPT-6 Astra's F2P score rose from 62.91% to 78.84%, and its complete-task rate stayed at 7.92%. A perfect spec helped the agent write more correct code. The failures that remained came from carrying every requirement through to a verified finish.
Limits and costs
The colleagues are language models with fixed knowledge boundaries. Real stakeholders forget details and change their minds, and the authors acknowledge that simulated colleagues may behave differently from human users. The behavioural analysis covers only Claude-Opus-5.5 trajectories, and GPT-6 Astra graded those trajectories, so one vendor's model judged another vendor's agent. Run profiles also differed: Opus used a median of 5.01 hours and 213 tool calls per task, compared with 3.00 hours and 182 calls for GPT-6 Astra.
The benchmark is expensive to run. Opus processed a median of 47.57 million input tokens per run, a figure that includes repeated reading of retained context. The paper reports no dollar costs. The repository requires Linux on amd64, Docker Engine, Python 3.10 or later, a Responses API endpoint for the colleague model and a checksum-pinned Claude Code binary for the default harness. With up to eight hours per task, a full 101-task campaign takes days and runs up a large API bill for every model tested. The arXiv listing shows the paper under CC BY-NC-SA 4.0, and the repository says the task assets keep their upstream licences.
Anyone running agents unsupervised should note one compliance result. In 15 of 94 briefs that prohibited test changes, the agent's patch still modified or removed lines in existing test files.
What builders can take from it
New Zealand teams that use coding agents on client work will recognise the setup. On a typical agency or in-house project, requirements are spread across a product owner, a Jira board, a Slack thread and somebody's memory. SWE-CoWork puts numbers on how poorly current agents cope with that spread. The paper's data supports several changes to agent harnesses:
- Keep requirements as a tracked artefact. Have the agent maintain a ledger of each requirement, where the requirement came from and its test status, and make the agent check the ledger before handoff. The authors list explicit requirement tracking as an open question, so measure whether the ledger helps on your own tasks.
- Build follow-up into the loop. When a stakeholder defers an answer, create a pending item the agent has to close. Opus abandoned about a third of deferred threads with hours of budget left.
- Don't take "done" on trust. With 31% of reported-complete requirements failing their tests, a separate verification pass that reruns the tests against the ledger costs little.
- Enforce test protection outside the prompt. If agents shouldn't edit tests, block test files with file permissions or a CI check.
Teams can also borrow the evaluation design at small scale. Write a handful of internal tasks where requirements sit with two or three simulated stakeholders, and count a task as passed only when every test passes. If your group builds a harness like this, we'd like to see it on the ASAI projects page.
Sources
- SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations (arXiv abstract)
- SWE-CoWork full paper (arXiv HTML)
- SWE-CoWork project page and leaderboard
- TokenWave-AI/swe-cowork-bench (GitHub repository)
- AI Daily Digest, 2026-11-17 (GitHub issue)
Build with people working on the same things
Join the Auckland Society for AI: NZ$599 a year with a ForgeBox and the members' projects area. Our Auckland meetups are open to everyone.