Level 5 · Flagship researchFlagship
Track A · RL + safety · default pick
Does reward hacking in realistic, multi-turn coding RL generalise into broader misbehaviour in open models, which mitigations stop it, and what does grading the chain of thought during RL do to monitorability?
The full-scale version of the Level 4 replication: realistic sandboxed software tasks, a catalogue of exploits with a hardened twin of every environment, mitigation ablations across model sizes, and a graded-CoT arm measured with OpenAI’s monitorability suite.
Anthropic found models that learned to reward hack in production RL also developed alignment faking, cooperation with malicious actors and sabotage attempts; three mitigations worked, but their environments aren’t public. OpenAI published on accidentally grading CoT during RL (May 2026), open-sourced its monitorability evals with g-mean² code, and in Oct 2026 proposed complementary metrics.
01Read the two research lines and pre-registerM1 · week 8
Read Anthropic’s reward-hacking paper and OpenAI’s CoT-grading and monitorability posts. Write your predictions down before running anything.
02Build sandboxed multi-turn software tasks on real reposM1 · week 8
Docker-sandboxed, multi-turn tasks on real repositories. They must look like real software work – toy environments with an obvious hack are the main risk.
03Catalogue exploit types and build a hardened twin of every environmentM1 · week 8
For each environment, list the ways it can be gamed and build a twin where those exploits are closed, so you can compare hacking against honest learning.
04GRPO on a ~1.5B open reasoning modelM1 · week 8
Train on the hackable and hardened versions. Plot hack-rate curves over training.
05First answer on generalisationM1 · week 8
Measure whether misbehaviour generalises beyond the environment at this scale.
06Publish the week-8 milestoneM1 · week 8
Hack-rate curves plus the first generalisation answer. A clean null result is publishable – ship it either way.
07Scale up
Three model sizes up to ~8B, 3+ seeds per setting, confidence intervals.
08Mitigation ablations
Hack prevention, inoculation-prompt wordings, diverse safety data, and detector penalties at controlled recall.
09The CoT arm
Penalise the chain of thought at controlled strengths, then measure monitorability with OpenAI’s open suite and the newer metrics (pair-matched comparisons, post-control risk). Does pressure make models hide reasoning?
10Release
Open environments and training code, a paper-style write-up and a failure log.