The claim

Model providers keep advertising bigger context windows, and plenty of agent builders treat them as a safety net: if the history fits, keep all of it. A technical report posted to arXiv on 3 November 2026 questions that habit. "You don't need long context" comes from twelve authors led by Peter Belcak and Pavlo Molchanov. The paper links to an NVIDIA lab and runs its local tests on NVIDIA DGX Spark hardware. The authors argue that frontier models can do long, multi-step work and read very long documents while keeping their working memory small, provided they regularly boil their history down into a recap. The abstract closes with a pointed line: "Our findings question the need for long-context language models."

It's a 31-page report with 8 figures. It hasn't been peer reviewed, and so far we haven't seen code or a project page released with it.

Why the authors care about cost

The economic argument comes before any benchmark. Each token a model generates has to attend to everything already in its context, so tokens produced late in a long session cost more than early ones. The team measured this on cloud-hosted open-weight models. Serving at a 256K context costs about 3.6 times as much per generated token as serving at 16K for GLM 5.2, about 5.6 times for Kimi K2.6 and about 7 times for Qwen 3 Coder Next. On a single DGX Spark running quantized local models, the time between tokens roughly doubled across the context lengths they tested.

The savings figures quoted later all build on this. They're modelled from a rounded throughput ratio and from list prices ($25 per million output tokens for Claude Opus 4.5, $20 for GPT 5.6 Sol). Nobody measured them from real invoices, so we'd treat them as directional.

How the experiments work

The baseline is what most agents do today: keep every message and tool result, which the paper calls full-history accumulation. Against that, the authors test two ways of throwing history away.

  • End-of-conversation compaction. The agent runs normally until its context crosses a threshold, then stops and writes a recap that replaces the old history. Budgets are given as a hard limit and a trigger point, such as 48K/36K or 16K/12K tokens.
  • Single-action compaction. This is the extreme version. Each step starts fresh with the original task plus a compact description of where things stand, the model takes exactly one action, and the cycle repeats.

The recap prompts matter, and the appendix spells them out. For the coding agent, the model is told to stop work, run no commands and write a handoff note with six parts: the objective, facts it has established, changes it has applied, how it verified them, dead ends and the remaining plan. The general agent's version asks for progress and the current best answer, supporting evidence with locations, what worked or failed, and next steps, and it tells the model not to invent observations.

Long documents get a different treatment. They're saved as files, and the agent explores them with ordinary shell tools like grep, sed and head, reading small windows of around 6,000 characters near each match. In one LongBench v2 example, a 36-line extract from an 18,830-line grammar was enough to answer a translation question.

Both harnesses are deliberately minimal ReAct loops with Bash: mini-swe-agent for software tasks and a general-purpose mini-agent. The main accuracy runs used Claude Opus 4.5 and GPT 5.6 Sol.

What they found

Long, multi-step tasks. Across SWE-bench Verified, SWE-bench Multilingual, Terminal-Bench 2.1, FRAMES, τ²-bench and BFCLv3, compaction shifted Opus 4.5 scores by between -3.7 and +5.6 percentage points and GPT 5.6 Sol by -3.3 to +4.4. For each model, no more than 5 of 22 threshold settings lost over 2 points. A few examples:

  • On SWE-bench Verified, Opus went from 70.0% to 74.8% at a 24K/18K budget.
  • On FRAMES, every setting cost something: 1.3 to 3.7 points for Opus, 1.5 to 3.3 for GPT.
  • Estimated savings ran from 2.2x to 8.8x for Opus and 2.4x to 5.0x for GPT.

One action per context. With single-action compaction, Opus scored 70.2% on SWE-bench Verified against its 70.0% baseline, while the largest context it ever held fell from 200K to 16K tokens. On Terminal-Bench 2.1 it rose from 47.2% to 51.8%. GPT 5.6 Sol slipped a little on both: 84.4% to 82.4% and 87.6% to 84.3%.

Long documents. Here searching beat reading everything, often by a lot. On RULER, mean recall climbed from 62.5% to 95.7% for Opus and from 65.5% to 96.4% for GPT. BABILong rose about 24 points for Opus and 23 for GPT. On AA-LCR Opus jumped from 65.0% to 84.0%. The estimated savings ranged from 6.5x to 54.6x for Opus and up to 233x for GPT. Agents often didn't need to compact at all, because they never pulled much text into memory.

Where it breaks down

The paper is upfront about several weaknesses, and we found a couple more.

  • Recaps lose information. Each summary drops detail, and repeated summaries compound the loss. The authors build a coin-toss puzzle that can't be solved under a fixed budget unless the full history is stored somewhere and read back on demand.
  • Open-weight models vary a lot. At a 16K/12K budget on SWE-bench Verified, Qwen 3 Coder Next collapsed from 53.0% to 3.6%. Qwen 3.6 27B (68.6% to 65.0%) and MiniMax M2.7 (72.6% to 67.8%) held up far better.
  • Some benchmarks flatter the method. The authors note Needle in a Haystack questions share words with the hidden answer, which makes grep unusually effective.
  • An internal mismatch. The text gives median peak context under single-action compaction as 77 to 99K tokens, while the matching table shows 7 to 9K.
  • The strongest claim is untested. The authors say they found no evidence that inherently long-context tasks exist. That's a claim about the absence of evidence, and they concede their results don't rule such tasks out.

They also propose "semantic state space models", which would carry an explicit natural-language state from turn to turn. It's an idea for now. The paper doesn't build or test one.

What builders can try

You don't need new models to test this. You can do it in an existing agent loop.

  • Add a trigger. Set a token threshold below your hard limit and, when it's crossed, ask the model for a handoff recap that replaces the transcript.
  • Write the recap prompt as a handoff. Ask for the goal, confirmed facts, changes made, checks run, dead ends and next steps, and forbid invented observations.
  • Move big inputs out of context. Save documents, logs and repositories as files and give the agent search tools. On this evidence that's often both cheaper and more accurate.
  • Keep the raw history. Store it on disk so the agent can look things up when a recap leaves something out.
  • Measure on your own model. Results ranged from small gains to near-total collapse depending on the model, so run your full-history setup next to a compacted one on your own tasks and compare accuracy and real spend before you switch.

For teams paying per token on long agent runs, this report gives good reason to see how small the context can get before quality drops.

Sources

Build with people working on the same things

Join the Auckland Society for AI: NZ$599 a year with a ForgeBox and the members' projects area. Our Auckland meetups are open to everyone.

Become a member