What the paper tested
Many teams pick a model by checking an arena leaderboard. These are sites where anonymous users compare two hidden models' answers to their own prompt and vote for the better one. A new paper, Leaderboard Hacking: Preference-Based Model Evaluations are Vulnerable to Manipulation by Eve Fleisig, Joachim Baumann, Dylan Hadfield-Menell and Dirk Hovy, was posted to arXiv on 10 November 2026. It shows those rankings can be moved a long way by people who know how the scoring works. The paper is a NeurIPS 2026 oral, and the authors are scheduled to present it in Sydney on 10 December.
The authors didn't touch any live leaderboard, for ethical reasons. They used LMArena's public arena-human-preference-100k dataset, which holds about 106,000 conversations collected between June and August 2024. According to the paper, their working set covers about 100,000 votes across 55 models. They rebuilt the rankings, simulated each kind of manipulation and measured how far a target model moved.
How the scoring works
Arena leaderboards turn votes into rankings with the Bradley-Terry model. Bradley-Terry is a statistical model that gives each competitor a strength score, so the chance that model A beats model B depends on the gap between their scores. The familiar Elo numbers are a linear rescaling of those scores. The authors argue that Bradley-Terry in practice behaves like a Borda count, the positional voting rule where every ballot carries equal weight. Voting theorists already know how Borda counts can be gamed, and the paper tests whether the same weaknesses carry over to model leaderboards.
The weakness comes from how ratings link together. A model's rank depends on how strong its opponents are as well as on its own wins. Votes that never involve the target can still move it up by weakening or strengthening the models around it.
The three manipulations
Strategic voting assumes the attacker can tell which model wrote each answer. The authors' simple rule is to vote for the target whenever it appears and otherwise vote for the weaker model in the pair. That rule pushed models up as far as 15 places, and 21 of the 55 models moved more than 10 places. A slower greedy method from earlier work reached 20 places. The paper reports that small gains of two or three places need under 2% of total votes, about 2,000 of 100,000. The assumption about spotting the model is realistic. Earlier work presented at ICML 2025 identified which model wrote a reply with over 95% accuracy in a simulated Chatbot Arena and shifted rankings for about 1,000 votes.
Strategic prompting means sending prompts from a category where the target does well. The authors grouped prompts by language. Reka Core rose from 33rd to 18th on Hungarian prompts, Llama 3.1 8B Instruct rose from 38th to 23rd on Croatian prompts, and Phi-3 Medium rose from 46th to 32nd on Scots prompts. Ranks moved with as few as 500 strategic prompts. The authors also grouped prompts using LMArena's own labels for complexity, creativity, domain knowledge and maths, and still saw gains of up to 14 places.
Strategic nomination means submitting extra models, up to 10 copies in the experiments, that happen to beat the target's main rivals. Those wins drag the rivals' scores down. Gains reached 15 places but were usually between one and seven. One model climbed from 19th to 4th. Combining nomination with prompting lifted models up to 45 places.
Limits of the result
The dataset is much smaller than the live service. The paper notes LMArena's text leaderboard has more than 5 million votes, so the exact boosts won't carry over directly. Even so, the authors found that the number of manipulations needed for a similar boost scales linearly with dataset size. A bigger leaderboard raises the cost of an attack in proportion to its size and doesn't remove the weakness. The voting attack also depends on telling models apart. The authors report over 80% precision for open-weight models but only about 40% to 50% for API-only models.
Models sitting just below a tight cluster of similar ratings were the easiest to move. In practice the middle of a crowded leaderboard is the least reliable part of it, and the middle is where many teams look for cheaper options.
The code isn't public yet. The paper's checklist says the authors plan to release it with the camera-ready version.
Proposed safeguards
For nomination, the authors suggest limiting submissions of many near-identical models. They also suggest reweighting Bradley-Terry with a harmonic scheme taken from Dowdall-style Borda variants. At one setting, that change cut the largest nomination boost from 15 places to 8 and kept the overall ranking close to the original, with a Kendall's tau of 0.946. Kendall's tau measures agreement between two orderings, where 1 means identical. For voting, they point to bot detection, verified logins and stopping users from viewing many comparisons without voting. The authors note these measures are hard to enforce against coordinated groups. For prompting, they suggest grouping prompts into clusters and weighting each cluster evenly. A smoothed Bradley-Terry variant appears in an appendix, but the paper doesn't test it.
What builders can do with the result
Use arena rankings to build a shortlist, and give gaps of a few places little weight. A model that jumps up a crowded mid-table may have earned the move, or it may have been helped by the effects the paper describes. The leaderboard itself can't tell you which.
The prompting result matters for New Zealand teams. A modest batch of prompts in one language moved Hungarian, Croatian and Scots rankings by 14 or 15 places. Small language and domain subsets therefore say little about how a model will handle your users. If your product serves te reo Māori speakers, Pacific languages or a narrow professional field, the leaderboard has few or no battles that look like your traffic.
A practical response is a small private evaluation built from real prompts your team handles, with the answers hidden from reviewers and scored against criteria you write down before testing. If you run internal pairwise comparisons with Bradley-Terry, the paper's advice applies to you too. Keep near-duplicate model variants from flooding the pool, balance prompts across task types and watch for reviewers who can recognise models by their style. If you put together an evaluation harness worth sharing, the ASAI projects page is a good place to post it.
NeurIPS is in Sydney this year, so the oral session is a short flight away for anyone in Auckland who wants to question the authors directly.
Sources
- Leaderboard Hacking: Preference-Based Model Evaluations are Vulnerable to Manipulation (arXiv abstract)
- Leaderboard Hacking (arXiv HTML full text)
- Leaderboard Hacking, NeurIPS 2026 oral listing
- lmarena-ai/arena-human-preference-100k (Hugging Face)
- Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboards (ICML 2025)
Build with people working on the same things
Join the Auckland Society for AI: NZ$599 a year with a ForgeBox and the members' projects area. Our Auckland meetups are open to everyone.