What the paper claims
A preprint posted to arXiv on 16 November and revised on 18 November describes a method its authors call self-rewriting. A language model inspects its own weights and hidden states, guesses why it fails on some problems, writes code to change specific tensors, and tests the change against its unedited self. The process uses no gradient descent, no optimiser and no new training data. The authors, Jiajun Fan and colleagues, list affiliations including UIUC, CMU, Princeton and Stanford in the paper's HTML version. GIGAZINE's write-up describes the work as led from UIUC and Carnegie Mellon.
The headline result: Qwen3.5-27B raised its accuracy on 490 held-out maths problems from 57.2% to 63.8%, a gain of 6.6 points, with a 95% interval of 5.0 to 8.1 points, according to the paper's appendix. The authors also gave the model a deliberately damaged copy of itself, and the model repaired that copy well enough to recover about 83% of the base model's accuracy across six image benchmarks, even though it found the fix using maths problems alone.
How a session works
Each session gives the model a sandboxed Python environment with read and write access to its own weight tensors. The model can generate answers to practice problems with or without its proposed edit, and it can read its hidden states (the internal activation vectors each layer produces). The paper caps each session at 20 forward passes, 200 code runs and 160 turns, with wall-clock limits of 2.5 or 3.5 hours. The sandbox refuses gradients, optimisers, model loaders, subprocesses and network access. The embedding layer, final normalisation and output layer are off-limits, and every saved tensor has to keep its original shape and contain only finite values.
The model works in turns, writing a short rationale and then one action. The evaluation method the authors lean on is a paired test: base and edited models answer the same problems with the same random seeds, and the score is the mean per-problem change in accuracy with a standard error. Pairing removes a large share of the noise that comes from sampling, which matters when the effect you're looking for is a few points. In the paper's second protocol, the framework also runs the candidate edit once on 256 reserved practice problems before the model decides whether to submit it.
The winning edit is surprisingly small. The 27B model noticed that many of its answers looped until they hit the token limit. Its fix raised a single time-step bias by 0.5 in all 48 of its linear-attention layers, which the model reasoned would make those layers forget older context faster. Qwen3.5 mixes Gated DeltaNet linear attention (a recurrent-style attention variant with a learned decay) with standard full attention, so that bias directly controls how quickly state fades. The share of answers that ran out of tokens fell from 33.6% to 26.1%, and 439 problems were solved only by the edited model against 182 solved only by the base model, per the paper's results appendix.
An earlier session's edit removed about 5% of one hidden-state direction from two output matrices in layer 8, using a projection of the form W − α·uuᵀW. That edit moved held-out accuracy from 57.2% to 58.3%, with p = 0.038, which is a much weaker result.
Scale decides whether the edits help
The authors ran the same setup on four Qwen3.5 sizes. The 2B model made no edit. The 4B and 9B models submitted edits that broke them, and the 9B model's edit dropped its held-out accuracy from 47.5% to 0.0%, according to the paper. Only the 27B model improved. The authors attribute the difference to how the larger model gathers evidence: it measured its hidden states before editing and acted on its own test results. The paper puts it this way: "What scale adds, in these sessions, is the experiment around the edit rather than the edit itself."
Limits worth weighing
The evidence base is thin. Each model size had one complete session, with two for the 27B model, and three further sessions were stopped early and left out of the analysis. Every result comes from one model family and from competition maths, apart from the image repair test. The damage in the repair test was a single, known fault: eight rows of one MLP output matrix in layer 30 overwritten with noise. On held-out maths, the repaired copy reached 33.0% against 58.5% for the base model, and the fix was a clipping rule since the model never saw the original weights.
The abstract says accuracy keeps rising over eight sessions, and the wording needs care. Every session started from the base weights. Across the eight sessions the model withdrew five edits and submitted three, so the deployed model only ever got better because bad edits weren't kept. The paper doesn't show improvements compounding on top of earlier improvements, a point GIGAZINE also raises. The authors frame self-rewriting as a possible step toward recursive self-improvement, and the experiments haven't yet tested that loop.
On cost, the authors say the sessions behind the largest gains each ran in a few hours on a single GPU, and that an edit's cost is paid once and adds nothing at inference time. The paper doesn't name the GPU. Generation runs through vLLM, and the full 16,384-token budget made 27B forward passes take about half an hour each, which is why one session used a 4,096-token budget. We found no code or data release on the arXiv listing or in the paper text.
What builders can take from it
Few teams will hand a production model write access to its own weights, and the paper gives good reasons for caution: a 9B model wiped out its own accuracy with one confident edit. Even so, three parts of the work are useful now.
The paired, same-seed evaluation is a cheap upgrade for anyone comparing prompts, quantisations, LoRA adapters or sampling settings. Scoring per-problem differences with a standard error tells you whether a two-point gain is real, which unpaired accuracy numbers on a few hundred items often can't.
The time-step bias finding is a concrete, testable lead for people running Qwen3.5-27B locally on reasoning tasks where answers run away to the token limit. It's a single scalar change per layer and costs nothing at inference. Anyone trying it should treat the paper's number as one result on maths with a 16,384-token cap, and check it against their own held-out set using the same paired method before shipping.
The safety setup also reads as a checklist for any agent that touches model artefacts: block network and subprocess access, freeze sensitive layers, validate tensor shapes and values, and gate submission on a held-out test. The paper notes that file access wasn't blocked in its sandbox, so locking down the filesystem is an obvious addition.
For New Zealand teams, the main appeal is that this work used an open-weight model and single-GPU budgets, which puts replication within reach of a university lab or a small company here without hyperscale compute. Until the authors release code, replicating the result means rebuilding the harness from the paper's description. If you do run a replication or apply the paired-test method to your own models, we'd like to see it on the ASAI projects page.
Sources
- Language Models Can Rewrite Their Own Weights (arXiv abstract, 2611.21885)
- Language Models Can Rewrite Their Own Weights (arXiv HTML, v2)
- The AI succeeded in 'making itself smarter' by rewriting its own weights without any additional training (GIGAZINE)
Build with people working on the same things
Join the Auckland Society for AI: NZ$599 a year with a ForgeBox and the members' projects area. Our Auckland meetups are open to everyone.