Level 1 · Rebuild the foundations
Can you pretrain a 124M-parameter GPT-2 yourself and match OpenAI’s GPT-2 124M on HellaSwag?
Follow Karpathy’s build-nanogpt end to end: write the model, the data pipeline and the training loop, debug on your Mac, then do the full run on a rented multi-GPU node.
Pretraining a real model end to end – data, distributed training, evaluation – is the foundation of every model-training role, and it is the starting line of the modded-nanogpt speedrun (Level 3). Karpathy’s own llm.c reproduction took about 90 minutes on 8×A100 for about $20, so the cost is real but small.
01Watch ‘Let’s reproduce GPT-2 (124M)’ once without coding
Get the shape of the whole build first: model, weight loading, optimisation tricks, data, HellaSwag eval.
02Write the model and load OpenAI’s weights
Write the GPT-2 module yourself, then load the official 124M weights from Hugging Face into it and check your outputs match the reference model’s logits.
03Training loop on a tiny dataset, on your Mac
Overfit a single batch first (loss should go near zero), then train briefly on a small text file on MPS. This is where you catch shape and masking bugs cheaply.
04Add the speed and stability tricks one at a time
Mixed precision, fused AdamW, learning-rate warmup and cosine decay, gradient clipping, weight decay, gradient accumulation. Measure tokens per second after each change.
05Prepare FineWeb-Edu shards
Download and tokenize the 10B-token sample into shards with the repo’s data script. Check a few decoded shards by eye.
06Add the HellaSwag evaluation
Implement the completion-scoring eval and confirm OpenAI’s GPT-2 124M weights score about 29.4% on it in your code before you trust your own model’s number.
07Rent a node and launch the full run
Use DDP across all GPUs. Run a 5-minute smoke test first; only then launch the full 10B-token run. Save checkpoints and the log.
$ torchrun --standalone --nproc_per_node=8 train_gpt2.py08Plot and compare
Plot training and validation loss and HellaSwag over steps, with OpenAI’s GPT-2 124M as a horizontal line. Note when you cross it.
09Write it up
What you changed from the video and why, cost per run, the bugs that cost you the most time, and the final table.