Level 1 · Rebuild the foundations
Can you build a byte-level BPE tokenizer that produces exactly the same token ids as GPT-4’s cl100k_base?
Implement byte-pair encoding yourself, add GPT-4’s regex pre-splitting and special tokens, and prove it matches tiktoken byte for byte.
Tokenization causes a surprising share of LLM bugs (odd spellings, numbers, non-English text). Matching a production tokenizer exactly shows you can read a spec and verify against ground truth – the habit every later rung depends on.
01Watch the lecture and read the exercise
Karpathy’s ‘Let’s build the GPT Tokenizer’ is in your Videos page under After Week 21. The minbpe repo’s exercise.md breaks the build into steps. Read it before writing any code.
02Set up a clean repo of your own
Don’t fork minbpe – write your own package and only use minbpe’s tests as the referee. Keep a NOTES.md of every surprise.
$ mkdir my-bpe && cd my-bpe && uv init --python 3.12$ uv add tiktoken pytest03Basic BPE on bytes
Encode text to UTF-8 bytes, count adjacent pairs, merge the most frequent pair into a new id, repeat for N merges. Write encode() and decode() and test that decode(encode(x)) == x on emoji, accents and Telugu text.
04Add GPT-4’s regex pre-splitting
Split text into chunks with the GPT-4 split pattern before applying merges, so merges never cross word or number boundaries. Explain in your notes why this matters for numbers and whitespace.
05Load GPT-4’s merges and match tiktoken
Recover the merge ranks from tiktoken’s cl100k_base and reproduce its byte shuffle. Compare your ids with tiktoken on 1,000+ varied strings (code, maths, multilingual, long whitespace).
$ uv run python -c "import tiktoken; print(tiktoken.get_encoding('cl100k_base').encode('hello world'))"06Special tokens
Support <|endoftext|> and friends with an explicit allow-list, like tiktoken. With special tokens allowed, ‘<|endoftext|>hello world’ should encode to [100257, 15339, 1917].
07Train your own vocabulary
Train a 4k–16k vocabulary on a text corpus you choose and compare compression (bytes per token) with cl100k_base on English, code and a non-English language. Note where yours is worse and why.
08Write it up
A README with: how BPE works in your words, the mismatches you hit and how you fixed them, and the compression table. Link it from P6.