Level 2 · Small-model research on one GPU
On MATH-500 with a 1B model, at what generation budget does PRM-guided beam search start to beat (weighted) best-of-N – and why?
Use Hugging Face’s search-and-learn toolkit to compare best-of-N, weighted best-of-N and beam search with a process reward model across compute budgets, then explain the crossover.
Spending compute at inference time is central to reasoning models. A clean compute-vs-accuracy curve with a real explanation of the crossover is exactly the narrow, measurable kind of result reviewers trust.
01Read the blog and cookbook
Hugging Face showed small Llama models gaining a lot on MATH-500 with search guided by a PRM. Read how they set up each strategy.
02Reproduce one point of their curve
With Llama-3.2-1B-Instruct as the generator and RLHFlow/Llama3.1-8B-PRM-Deepseek-Data as the PRM, reproduce best-of-N at one budget before anything else.
03Run the full grid
Best-of-N, weighted best-of-N and beam search at N = 4, 16, 64 (and 256 if budget allows). Log accuracy, wall-clock time and tokens generated per problem.
04Plot accuracy against compute
Put compute (tokens or FLOPs), not N, on the x-axis so strategies are compared fairly. Add confidence intervals.
05Break it down by difficulty
MATH problems have difficulty levels. Find where search helps (hard problems?) and where it hurts.
06Explain the crossover
Read samples where beam search wins and loses. Is the PRM misjudging partial solutions? Is diversity collapsing? Back the explanation with examples.
07Optional: swap the PRM
Published results suggest the best strategy depends on which PRM you use. Test one other PRM and see if your crossover moves.