Let's reproduce GPT-2 (124M)

Key Takeaways

• GPT-2 Architecture Specifics: The GPT-2 124M model features 12 Transformer layers with 768 channels, uses a vocabulary of 50,257 tokens, and has a maximum sequence length of 1024. Key architectural differences from the original Transformer include being decoder-only, employing pre-normalization (Layer Norm before attention/MLP), and adding an extra Layer Norm before the final classifier, with GELU as the activation function due to historical performance considerations. 1:38 14:05 20:53
• Efficient Training Infrastructure: Modern PyTorch allows significant speedups: TF32 (torch.set_float32_matmul_precision('high')) provides ~3x speedup, BFloat16 (torch.autocast(..., dtype=torch.bfloat16)) enables mixed precision without gradient scaling, torch.compile(model) offers ~2.3x speedup by reducing Python overhead and GPU I/O, and Flash Attention (F.scaled_dot_product_attention) gives ~27% further improvement by optimizing memory access in the attention mechanism. 1:23:57 1:39:41 1:48:23 2:00:14
• Optimization Hyperparameters: Following GPT-3 paper guidelines, use AdamW with betas=(0.9, 0.95) and eps=1e-8, clip global gradient norm at 1.0, and implement a cosine decay learning rate schedule with linear warmup (e.g., max LR of 6e-4 for the 124M model, warming up over 375 million tokens, then decaying to 10% of max). Apply a weight_decay of 0.1 exclusively to 2D parameters (embeddings and matrices), not biases or 1D Layer Norms. 2:16:46 2:17:17 2:19:05 2:28:47
• Scaling Training with Gradient Accumulation & DDP: Gradient accumulation enables simulating large batch sizes (e.g., 0.5 million tokens) on smaller GPUs by iteratively accumulating gradients over multiple micro-steps and dividing the loss by the accumulation steps. For multi-GPU training, torch.nn.parallel.DistributedDataParallel (DDP) distributes the workload, averages gradients across GPUs, and requires careful management of data loading (each GPU gets a unique data chunk) and gradient synchronization (disabling sync for intermediate accumulation steps). 2:36:37 2:41:30 2:46:57 3:01:31
• Data Set and Evaluation: The FineWeb Edu 10 billion token subset is a suitable, high-quality dataset for reproduction, allowing comparison to original GPT-2 performance. HellaSwag is a useful evaluation benchmark to track progress due to its "early signal" properties, where models typically start at random chance (25%) and gradually improve. 3:12:33 3:28:38
• Weight Tying & Numerical Padding: Tying the token embedding weights (input) with the language model head weights (output classifier) saves ~30% of parameters and generally improves performance. Padding the vocabulary size to a power-of-two friendly number (e.g., 50257 to 50304) can yield a ~4% speedup in PyTorch/CUDA kernels due to optimized block computations. 1:06:26 2:07:02
• Reproducibility and Performance Gains: A GPT-2 124M model trained from scratch on 10 billion tokens of FineWeb Edu can surpass the original GPT-2 124M HellaSwag accuracy (29.5%) with ~10x fewer tokens, indicating significant improvements in training efficiency due to higher data quality, advanced optimization techniques, and modern hardware capabilities. 3:44:54 3:47:04