Level 5 · Flagship researchFlagship
Track B · Evaluation integrity
Do models behave differently when they detect they’re being evaluated, even when they never say so, and can you detect and remove that internally?
Build a realism-graded honeypot suite of paired eval-like and deployment-like scenarios, measure the behaviour gap across open models, then train an organism that games evals silently and build a probe that detects and suppresses it.
OpenAI (Nov 2026) studied gaps between agent behaviour in safety evals and in production. An ICML 2026 paper found near-zero verbalised eval awareness in most open models but steering evidence that unverbalised awareness is possible. Goodfire found models refuse harmful requests 3–18 points more often when they verbalise it, so measured safety may be inflated.
01Read the three resultsM1 · week 8
OpenAI’s eval–production gap work, the ICML 2026 paper and Goodfire’s finding.
02Build a realism-graded honeypot suiteM1 · week 8
Paired eval-like and deployment-like versions of each scenario. Have people rate realism blind – scenarios that look obviously fake are the main risk.
03Measure the gap across ~10 open models and publishM1 · week 8
The behaviour difference between the paired versions is your week-8 milestone.
04Train an organism that games evals silently
A model that behaves better when it detects an eval, without saying so.
05Build a probe with no verbal signal
Detect the eval awareness from internals alone.
06Suppress it and close the gap
Show that suppressing the probed direction closes the behaviour gap on held-out scenarios.
07Release
Open honeypot suite, organism, probe and results.