Stanford CS229 I Machine Learning I Building Large Language Models (LLMs)

Key Takeaways

• Key LLM Components: Successful LLM training hinges on five critical components: architecture, training loss/algorithm, data, evaluation, and system components for running models on hardware. While academia often focuses on architecture and algorithms, industry prioritizes data, evaluation, and systems due to their practical impact. 0:52
• Pretraining Objective: Pretraining LLMs involves training an autoregressive language model to predict the next word or token based on context, essentially modeling the entire internet. This is achieved by maximizing the log-likelihood of text using a cross-entropy loss, aiming to model the probability distribution of token sequences. 3:37
• Tokenization is Crucial: Tokenizers are essential for efficiently breaking down text into common subsequences (around 3-4 letters per token) rather than whole words or individual characters, addressing issues like typos, non-Latin languages, and quadratic sequence length complexity in transformers. Byte Pair Encoding (BPE) is a common algorithm, and tokenizer design significantly impacts model performance, especially for domains like math or code. 10:43
• LLM Evaluation Methods: For pretraining, perplexity (exponentiated average per-token loss) measures model uncertainty, showing significant improvements from 70 to under 10 words between 2017-2023. For aligned models, academic benchmarks like MMLU (multi-choice QA across subjects) and aggregate leaderboards like HELM or Hugging Face's are used, though open-ended generation evaluation often requires human or LLM-based preference ratings. 19:13
• Data Curation for Pretraining: Training LLMs on the "dirty" internet involves extensive data processing: extracting text from HTML, filtering undesirable content (NSFW, PII), large-scale deduplication, heuristic filtering (e.g., outlier tokens, word length), model-based quality filtering (e.g., using Wikipedia references), and domain classification with weighting (e.g., upweighting code, books). 28:30
• Scaling Laws and Resource Allocation: Empirical scaling laws demonstrate that increasing compute, data, or parameters consistently improves LLM performance without observed plateaus. These laws enable prediction of future model performance and optimal resource allocation, with Chinchilla paper suggesting an optimal 20 tokens per parameter for training and 150 tokens per parameter when considering inference costs. 40:56
• Estimated Llama 3 400B Training Cost: Training Llama 3 400B involved 15.6 trillion tokens and 405 billion parameters, requiring an estimated 3.8e25 FLOPs. Using 16,000 H100 GPUs for approximately 70 days (26 million GPU hours) implies a compute cost of around $52 million, with total costs (including salaries) potentially reaching $75 million, highlighting the immense investment required. 55:08
• Post-training for AI Assistants: Post-training (alignment) transforms pretrained LLMs into helpful AI assistants. This process starts with Supervised Fine Tuning (SFT) on human-collected instruction-response pairs, which teaches the model to format desired answers using relatively small datasets. However, SFT can suffer from behavioral cloning, cost, and potential for hallucination. 59:50
• Reinforcement Learning from Human Feedback (RLHF): RLHF addresses SFT's limitations by maximizing human preference rather than cloning human behavior. It involves training a "reward model" (often using a Bradley-Terry model) on human preference data (e.g., comparing two model outputs) to provide a continuous reward signal, which is then used to fine-tune the LLM via reinforcement learning algorithms like PPO. 1:09:50
• Direct Preference Optimization (DPO): DPO simplifies RLHF by directly maximizing the likelihood of preferred responses and minimizing the likelihood of dispreferred responses, effectively achieving the same outcomes as PPO but with a much simpler maximum likelihood objective, making it the current standard for alignment due to its ease of implementation. 1:19:18
• System Optimization for LLMs: Hardware limitations mean GPUs are often idle if code is not optimized. Techniques like low-precision arithmetic (e.g., 16-bit floats for computations, 32-bit for weights) and operator fusion (combining multiple operations into a single "fused kernel" to reduce costly memory communication) are crucial for maximizing GPU utilization and speeding up training. 1:37:12