Level 4 · Frontier safety replications
How monitorable is an open reasoning model’s chain of thought on OpenAI’s public evals – and does it change after you train it?
Run OpenAI’s open-sourced monitorability evaluations on an open-weights reasoning model, compute the g-mean² metric, then test whether a training intervention (for example, your GRPO run) makes the reasoning less monitorable.
Chain-of-thought monitoring is an active safety research area at several labs. OpenAI released the datasets and metric code so others can report results – a careful report on an open model is a genuinely new data point.
01Read the post and the paper
Understand the three eval types (intervention, process, outcome-property) and why g-mean² is undefined when the estimated effect is not positive.
02Clone the evals and run the reference code
The release includes most datasets, the g-mean² code and a cross-fit filtering method. The scaffold is unsupported, so expect to adapt it.
$ git clone https://github.com/openai/monitorability-evals.git03Choose your model and monitor
An open-weights reasoning model whose full chain of thought you can read, and a separate monitor model. Fix both and record versions.
04Run a small subset end to end
Start with one intervention eval (for example GPQA) and one outcome-property eval before scaling up.
05Apply the reporting rules
Follow the post’s rules: don’t report intervention results if fewer than 10% of instances have a positive effect, or non-intervention results with fewer than 3 positive instances.
06Run the full public subset
Report g-mean² per eval with uncertainty. Note which evals your model can’t be scored on and why.
07Test one training intervention
Compare the base model with a fine-tuned version (for example your GRPO model). Does training change monitorability?
08Write it up carefully
Results, caveats, and the known criticisms of these evals. Send it to people working on this area.