Level 3 · Performance engineering with a public bar
Can you reproduce the current GPT-2 (124M) training speed record, then find a change that makes it measurably faster?
Keller Jordan’s speedrun times how fast 8×H100 can train a 124M model to 3.28 validation loss on FineWeb. Reproduce the record, then hunt for a statistically significant improvement – the path that led to Jordan’s own hiring.
It is the clearest public scoreboard in model training, and contributions are judged purely on measured speed with p<0.01. Even a careful negative result (‘I tried X, here is why it didn’t help’) shows the right habits.
01Read the record history
The README’s record table explains what each record changed. Read the logs of the last several records. Check the live table – the record moves fast.
02Reproduce the current record
On a rented 8×H100 node, run the official commands. The first torch.compile adds several minutes. Confirm you hit ≤3.28 validation loss in about the record time.
$ git clone https://github.com/KellerJordan/modded-nanogpt.git && cd modded-nanogpt$ pip install -r requirements.txt$ pip install torch==2.10 --index-url https://download.pytorch.org/whl/cu128$ python data/cached_fineweb10B.py 9$ ./run.sh03Measure the noise
Run the record 5–10 times. Measure the spread in time and final loss – you need this to know what counts as a real improvement.
04Pick one hypothesis from your own understanding
An optimiser setting, schedule, architecture tweak or systems change. Write down why you expect it to help before running it.
05Test it cheaply first
Consider the optimisation track (fixed architecture, data and batch size, unlimited wall-clock) or shorter runs to screen ideas before spending 8×H100 time.
06Follow the record rules
Don’t touch the train or validation data pipelines, don’t add extra torch.compile or inductor flags, beat the previous record on the same hardware, and include enough logs for p<0.01. Systems-only changes don’t need to re-prove the loss.
07Submit or write up
If you have a significant improvement, open a PR with logs. If not, publish a careful negative-result write-up – it still shows the method.