Summary
This lecture introduces the foundational concepts of Natural Language Processing (NLP), various text processing techniques, and the underlying Transformer architecture, highlighting its key self-attention mechanism that powers modern Large Language Models (LLMs).
Key Takeaways
- Course Scope: The Stanford CME 295 class, "Transformers and Large Language Models," aims to teach the underlying mechanisms, training, and application of LLMs, suitable for those pursuing ML careers, personal projects, or seeking to understand AI/GenAI in their field. 2:00
- Prerequisites & Logistics: Minimum prerequisites include foundational ML knowledge (model training, neural networks) and linear algebra basics (matrix multiplication). The class runs Fridays 3:30-5:20, is two units, offers recorded sessions, and is graded by a 50% midterm (Oct 24) and 50% final exam (week of Dec 8), focusing purely on concepts without coding. 3:21
- NLP Task Categories: Natural Language Processing (NLP) tasks are broadly classified into three buckets: Classification (e.g., sentiment analysis, intent detection), Multi-classification (e.g., Named Entity Recognition or NER, part-of-speech tagging), and Generation (e.g., machine translation, question answering, summarization). 11:12
- Tokenization & Embeddings: To process text numerically, tokenization segments text into "tokens" (subword tokenizers are preferred for balancing root leveraging and Out-Of-Vocabulary risk). These tokens are then converted into "embeddings," which are learned numerical representations (e.g., via Word2vec) that capture semantic similarity, overcoming the limitations of One-Hot Encoding. 23:38
- Limitations of RNNs: Recurrent Neural Networks (RNNs) capture sequential text nature by maintaining a hidden state, but struggle with "long-range dependencies" or "vanishing gradients," causing them to forget information from earlier in long sequences, and suffer from slow sequential computations. 59:14
- Transformer's Attention Mechanism: The Transformer architecture (introduced in "Attention Is All You Need," 2017) abandons sequential processing in favor of "self-attention," allowing direct connections between all parts of a text. This enables contextual word representations (e.g., distinguishing "bank" in different contexts) and faster parallel computations. 1:06:58
- Query, Key, Value Concept: The self-attention mechanism utilizes Query (Q), Key (K), and Value (V) projections to quantify similarity and weight importance. A query token is compared to all key tokens to determine similarity, and then the corresponding values are weighted and summed to form the new representation of the query. 1:09:15
- Encoder-Decoder Architecture: The Transformer consists of an encoder (processing source text by applying self-attention for rich representations) and a decoder (generating target text, using both masked self-attention on its own output so far, and cross-attention leveraging the encoder's output). Positional encodings are added to token embeddings to reintroduce word order information. 1:14:08
- Label Smoothing: This technique is applied during training, particularly in generation tasks like translation, to address the reality of multiple valid outputs. Instead of training the model to predict a single word with 100% certainty, it encourages less confident predictions (e.g., 90% target, 10% distributed among others), which empirically improves metrics like BLEU. 1:25:49
