Summary
The attention mechanism is a core component of transformers that allows models to update initial, context-free word embeddings with rich contextual meaning by determining the relevance of other words in the input sequence.
Key Takeaways
- Contextual Embeddings: A transformer's primary goal is to adjust initial, generic word embeddings—which represent only individual words—to incorporate much richer contextual meaning from the surrounding text, enabling tasks like next-word prediction. 1:29
- Query, Key, and Value Vectors: Attention works by computing three types of vectors for each word: a Query (what information it's looking for from other words), a Key (what information it offers to other words), and a Value (the actual information to be passed, living in the high-dimensional embedding space). 6:32 7:55 13:35
- Attention Pattern Calculation: The relevance between any two words is calculated by taking the dot product between a word's Query and another word's Key. These dot products are then normalized using a softmax function across columns to produce an "attention pattern," a grid of weights (0-1) indicating how much each word "attends" to every other word. 8:27 9:49
- Masking for Training: During training, a technique called masking is applied to prevent words from "seeing" or being influenced by subsequent words in the sequence. This is achieved by setting dot products for future tokens to negative infinity before softmax, effectively turning them into zero weights. 11:52
- Embedding Refinement: To update a word's embedding, the Value vectors of all other words are scaled by their corresponding weights from the attention pattern and summed together. This sum is then added to the original embedding, resulting in a new, contextually refined embedding. 14:40 15:09
- Multi-Headed Attention: A full attention block in a transformer, such as in GPT-3, employs multiple "attention heads" (e.g., 96 in GPT-3) running in parallel. Each head uses its own distinct set of Query, Key, and Value matrices to capture different types of contextual relationships (e.g., grammatical, semantic, sentiment), with all their proposed changes summed to create the final refined embedding. 20:31 21:14
- Scalability and Parameters: The attention mechanism's success is partly due to its extreme parallelizability on GPUs, enabling massive scaling in model size. A single multi-headed attention block can contain hundreds of millions of parameters (e.g., ~600 million in GPT-3 for 96 heads), with attention-related parameters totaling tens of billions across many layers in large models. 22:03 24:55
