Level 1 · Rebuild the foundations
When you run a published benchmark on an open model yourself, how close do you get to the model card – and which settings explain the difference?
Run EleutherAI’s lm-evaluation-harness on a small open model, try to match its published GSM8K and MMLU numbers, and attribute every point of difference to a setting.
Every lab and AI team argues about eval numbers. Being the person who can say exactly why two numbers differ – few-shot count, prompt template, answer extraction, harness version – is rare and immediately useful. It is also the core skill behind your P2 and P3 evals.
01Pick a model and two published numbers
Choose a 0.5–1.5B instruct model with a model card that reports GSM8K and MMLU and states its settings. Copy the exact numbers and settings into a table before you run anything.
02Install the harness and pin the version
Record the exact harness version or commit – version differences alone can move scores.
$ pip install lm-eval$ lm_eval --help03First run, with samples logged
Run GSM8K with the model card’s few-shot count and keep every sample so you can read failures.
$ lm_eval --model hf --model_args pretrained=<model-id> --tasks gsm8k --num_fewshot 5 --batch_size 8 --log_samples --output_path results/04Read 50 failures by hand
Open the logged samples. Classify each miss: wrong reasoning, right answer but wrong format, extraction failure, truncation. Compare GSM8K’s strict-match and flexible-extract metrics.
05Change one setting at a time
Few-shot count (0/5/8), chat template on/off, max generation length, dtype, harness version. One change per run, everything else fixed. Log each run’s config next to its score.
06Repeat for MMLU
Do the same for MMLU (log-likelihood scored, so the sources of difference are different). Note which settings mattered for which benchmark.
07Estimate the noise
Compute a bootstrap confidence interval for each score so you can say which differences are real and which are noise.
08Write the attribution table
Model card number → your baseline number → each setting change and its effect → final number. State plainly what you couldn’t explain.