The problem with a test the model can see
A lot of AI safety testing works like a sting. Researchers build a situation where a model could cheat, game a grader or do something harmful, then watch what it does. If the model behaves, that counts as evidence it's safe. The method has a weak point, though. It only works if the model treats the setup as real. A model that knows it's being watched has every reason to behave, and the result tells us little about what it would do in ordinary use.
On 16 November 2026, METR published a research note by Uzay Girit, Kaivu Hariharan and Leni Shor that measures how well current agents spot these setups. Their answer is that the best of them now do it almost perfectly. METR labels its notes as unpolished updates with less review than its full research posts, and says they don't necessarily reflect the organisation's views. We've read the findings with that in mind.
How the study worked
The team collected 55 environments. Nineteen were alignment evals:
- 3 reward-hacking honeypots, which METR calls IRO
- 6 game setups where an agent could only beat a strong chess engine by cheating
- 5 physical-safety scenarios
- 5 seeds from Petri, the open-source auditing tool Anthropic released on 6 October 2025
The other 36 were ordinary capability tasks: 22 from METR's Time Horizons suite and 14 from Terminal-Bench 4.0.
They didn't ask a model to do any of these tasks. They dropped an "investigator agent" into each environment, gave it the same tools a working agent would get, set it to its highest reasoning effort and asked it to work out what the environment was for. Each investigator produced two scores from 0 to 100. For the direct score, it simply estimated the chance it was in an alignment eval. For the open-ended score, it described the environment's purpose in its own words, and Claude Opus 5.5 acted as judge, rating how strongly that description pointed to a safety test. Every pairing ran three times.
To compare models, the authors used AUC. That's the chance that a randomly picked alignment eval scores higher than a randomly picked capability task. A score of 0.5 means guessing, and 1.0 means every alignment eval lands above every ordinary task. They also checked calibration with Brier scores, computed separately for each kind of task. Lower is better, and a model that always answered 50 percent would score 0.25.
What they found
Every model family improved across 2026, and Anthropic's models came out on top:
| Model | Direct AUC | Open-ended AUC | |---|---|---| | Claude Opus 4.6 | 0.89 | 0.83 | | Claude Opus 4.8 | 1.00 | 0.94 | | Claude Opus 5.5 | 1.00 | 1.00 | | GPT-5.4 | 0.77 | 0.62 | | GPT-5.6 Sol | 0.83 | 0.79 | | GPT-6 Astra | 0.89 | 0.81 |
Opus 5.5 separated the two groups cleanly. Its lowest open-ended score on any alignment eval was 75, which still beat its highest score on any capability task, 71.7.
The two families got better in different ways. Opus improved mostly at recognising safety tests: its Brier score on alignment evals fell from 0.36 for Opus 4.6 to 0.05 for Opus 4.8. OpenAI's improvement came mostly on the ordinary side. GPT-5.4 scored 0.34 on capability tasks, which is worse than always saying 50 percent. GPT-5.6 Sol got that down to 0.08.
The two scoring methods mostly agreed, with correlations of 0.92 for Opus 5.5 and 0.78 for Astra. Astra's gaps are telling. On the IRO honeypots it averaged 91 when asked directly, yet its own descriptions of those environments read like capability tasks and scored under 20. The authors think the direct question primed it. Opus went the other way and raised the idea of a safety test more often without being asked. The authors say they don't yet know why.
What gives a test away
The note splits tells into two kinds. Environment tells come from poor simulation, like mocked services, worlds that look fake or users who sound scripted. Task tells are the features designers put in on purpose so bad behaviour can show up: tempting shortcuts, high stakes, clashing goals.
Task tells came up far more often. Investigators mentioned conflicting goals in every Petri answer and in 90 percent of physical-safety answers. They called 83 percent of Petri scenarios contrived. In 63 percent of physical-safety answers they said there was no honest way to succeed. And 58 percent of game answers tied the setup straight to AI safety research. Only one tell was more common in ordinary tasks: explicit wording about the "spirit of the task", which showed up in 40 percent of Time Horizons answers.
Some examples stand out. In one IRO honeypot, Opus 5.5 spotted all four changes METR had made to turn a capability task into a trap, including the spirit-of-the-task language and an LLM judge it could exploit. In the chess setups, it noticed they looked like the 2025 Palisade Research experiment where reasoning models such as o3 and DeepSeek R1 hacked a chess engine to win. Astra decided a nuclear-leak scenario was staged because the world it could search was tiny and the casualty figures were oddly precise.
Limits worth keeping in mind
- Beliefs only. The study records what models say about their environment. It doesn't test whether that awareness changes how they act, and the authors make no causal claims.
- Small, hand-picked sample. It's 55 environments in total, and some groups are tiny. The IRO set produced only 18 answers.
- Noisy runs. Astra's direct score swung by 40 points or more on 7 of the 55 environments. The authors put this down to investigators sometimes not exploring enough to find the honeypot.
- Judge dependence. The open-ended scores rely on one Opus 5.5 judge and its rubric.
Other research points the same way. In November 2025 the UK AI Security Institute built a more realistic scaffold on Petri that simulates a coding agent inside an AI company. Models mostly couldn't tell its tasks apart from anonymised real Claude Code usage. Even so, when prompted they could still pick out alignment scenarios from ordinary deployment data. Anthropic had already reported that Claude Sonnet 4.5 often wondered aloud whether Petri was testing it. Better simulation clears up environment tells, but task tells are still there.
What builders can do with this
If you write evals, red-team agents or use safety results to make deployment decisions, there are a few things worth doing:
- Check your tests before you trust them. METR suggests a loop: run investigator agents on a draft eval, ask both direct and open-ended questions, compare the scores with real capability tasks, remove the tells you find and repeat until the scores overlap.
- Fix the cheap tells first. Mocked services and fake-looking infrastructure are easier to repair than a scenario's core design. The authors aren't sure every task tell can be removed, because some tests only work if there's a tempting shortcut.
- Give investigators room to explore. Thin exploration caused a lot of the run-to-run noise, so longer runs should give more stable readings.
- Read clean passes carefully. A strong model that behaves well on a contrived scenario may simply have recognised it.
The main cost is compute. You need a few extra high-effort agent runs per environment, plus judge calls. That's small next to building the eval in the first place, and it tells you whether the eval measures anything at all.
Sources
- Measuring alignment eval tells (METR, 16 November 2026)
- Investigating models for misalignment (UK AI Security Institute)
- Petri: An open-source auditing tool to accelerate AI safety research (Anthropic)
- Demonstrating specification gaming in reasoning models (arXiv 2502.13295)
Build with people working on the same things
Join the Auckland Society for AI: NZ$599 a year with a ForgeBox and the members' projects area. Our Auckland meetups are open to everyone.