Level 4 · Frontier safety replications
If a small model learns to reward-hack in a coding environment, does misaligned behaviour generalise beyond the hack – and does inoculation prompting prevent it?
A scoped replication of Anthropic’s ‘Natural Emergent Misalignment from Reward Hacking in Production RL’: build a hackable coding environment, train a ~1.5B model with RL until it learns the hack, then measure misalignment on separate evals and test the paper’s mitigations.
It is one of the most discussed recent safety results. A careful small-scale replication – even a partial one – is exactly the kind of work the Anthropic Fellows program funds.
01Read the paper twice
Note exactly how reward hacking was induced, which misalignment evals were used, and the three mitigations: preventing hacking, more diverse safety training, and inoculation prompting.
02Write a one-page pre-registration
Your exact question, the model, what counts as a hack, which misalignment evals you’ll use, and what result would count as replication. Do this before training.
03Build a hackable coding environment
Coding tasks graded by tests with a deliberate loophole the model can exploit. Verify by hand that the hack is possible and detectable.
04Train until the hack appears
Use your GRPO setup on ~1.5B. Track the hack rate over training separately from genuine task success.
05Measure misalignment outside the environment
Run held-out evals that are unrelated to coding (for example, questions probing deception or harmful goals). Compare against the same model trained without the hack.
06Test inoculation prompting
Retrain with the hack framed as acceptable in the system prompt. Does generalised misalignment drop even though the hack is still learned?
07Scale up only to confirm
If the main result holds at 1.5B, confirm the key comparison once at a larger size. Ablations stay small.
08Write it up as a short paper
Replication status (full, partial, failed), effect sizes with seeds, limitations, and what you’d do with more compute.