Level 2 · Small-model research on one GPU
How much does GRPO with a verifiable reward improve a 0.5–1.5B model on GSM8K – and what does the model learn to game?
Fine-tune a small instruct model with Hugging Face TRL’s GRPOTrainer using a correctness reward and a format reward, measure the gain on held-out GSM8K, and document any reward hacking you see.
RL with verifiable rewards is how today’s reasoning models are post-trained. Running it end to end – including catching the model gaming your reward – is the practical foundation for the reward-hacking replication on Level 4.
01Measure the baseline first
Evaluate your chosen model on a held-out GSM8K split with your Level 1 eval setup. This is the number GRPO must beat.
02Read the current TRL GRPO docs
Use the current GRPOTrainer docs and the built-in reward functions (trl.rewards) rather than old tutorials.
$ pip install trl03Write two reward functions
A correctness reward (extract the final number, compare to the answer) and a format reward (did it use your answer format). Unit-test both on hand-written good and bad outputs.
04Run a tiny smoke test
A few dozen steps on a handful of problems. Check the reward goes up and read the generations.
05Train for real and log everything
Log mean reward, each reward component, completion length and KL. Save samples every N steps.
06Evaluate on the held-out split
Compare against the baseline with confidence intervals. Never evaluate on training problems.
07Hunt for reward hacking
Read samples where reward is high but the answer is wrong or the reasoning is nonsense. Did it game your format or extraction? Write down every case.
08One ablation
Change one thing – KL coefficient, number of generations per prompt, or remove the format reward – and measure its effect.