Level 5 · Flagship researchFlagship
Track D · Low-precision training
Can end-to-end 4-bit training (including optimiser and attention) close the loss gap to BF16 at small scale, and which component actually matters?
Write fused Blackwell kernels for NVFP4 weight-gradient GEMMs, then train ~1B models end to end in FP4 with Muon and ablate which component drives the loss gap – upstreaming the result.
NVIDIA’s NVFP4 recipe trained a 12B model on 10T tokens with loss comparable to FP8. A 2026 MXFP4 study found quantising weight gradients is the main cause of divergence. ‘Full-Stack FP4’ (July 2026) extended NVFP4 to optimiser states and attention, including Newton–Schulz iterations for Muon, with a 1.47% loss gap at 3B / 64B tokens.
01Line up Blackwell GPU access firstM1 · week 8
Access to Blackwell GPUs is the main risk. Don’t start without it.
02Read the three FP4 papersM1 · week 8
The NVFP4 recipe, the MXFP4 divergence study and Full-Stack FP4.
03Fused kernels for NVFP4 weight-gradient GEMMsM1 · week 8
Triton or CUTLASS, with an unbiased estimator.
04Check numerics and speed, then publishM1 · week 8
Numerics against a reference implementation, speed against TransformerEngine. Publish as the week-8 milestone.
05Train ~1B on 20–50B tokens
Aim to beat the 1.47% loss gap end to end in FP4 with Muon.
06Clean ablations
Which component (weight gradients, optimiser states, attention, Newton–Schulz) actually matters.
07Upstream
Contribute to torchao or TransformerEngine.