Summary
This lecture introduces linear regression as a fundamental supervised learning algorithm for continuous output prediction, detailing its hypothesis representation, the squared error cost function, and two primary optimization methods: iterative gradient descent (batch and stochastic) and the direct normal equation solution.
Key Takeaways
- Supervised Regression: Linear regression is a supervised learning algorithm used to predict continuous output values (Y) from input features (X), such as estimating house prices from size and number of bedrooms, as opposed to classifying discrete categories. 1:14
- Hypothesis Representation: The model's prediction function, called the hypothesis (h), is represented as an affine function (often termed linear function in ML) of its input features: h(x) = Theta_0 + Theta_1X_1 + Theta_2X_2 + ... + Theta_n*X_n. This can be compactly written as a sum from j=0 to n of (Theta_j * X_j) by defining a dummy feature X_0 = 1. 5:39
- Cost Function (J): To learn the optimal parameters (Theta), linear regression uses a cost function, J(Theta), defined as one-half the sum of squared differences between the hypothesis's predictions and the true target values across all training examples: J(Theta) = 1/2 * Σ(h_Theta(x_i) - y_i)^2. Minimizing this function finds the best-fit line. 14:44
- Gradient Descent: This is an iterative algorithm that minimizes the cost function by repeatedly adjusting parameters (Theta) in the direction of the steepest descent. Each parameter Theta_j is updated by subtracting a step proportional to the learning rate (Alpha) multiplied by the partial derivative of J(Theta) with respect to Theta_j. 17:56
- Learning Rate (Alpha): The learning rate controls the step size in gradient descent. A large Alpha can cause overshooting the minimum, while a small Alpha leads to slow convergence. Empirically, typical starting values are around 0.01 (after feature scaling), and if the cost function increases, Alpha is likely too large. 24:55
- Batch Gradient Descent: This variant calculates the gradient using the entire training dataset for each parameter update. While it guarantees convergence to the global minimum for linear regression, it can be extremely slow for very large datasets (millions of examples) because each step requires scanning all data. 41:14
- Stochastic Gradient Descent (SGD): In contrast to batch GD, SGD updates parameters using the gradient calculated from just one randomly chosen training example at a time. This makes each step much faster, enabling quicker progress on large datasets, though its path to the minimum is noisy and it tends to oscillate around the minimum rather than perfectly converging. Decreasing the learning rate over time can reduce these oscillations. 44:55
- Normal Equation: For linear regression specifically, there is a direct, non-iterative method to solve for the optimal parameters Theta. This "normal equation" calculates Theta in one step using matrix operations: Theta = (X^T * X)^(-1) * X^T * y, where X is the design matrix of features and y is the vector of target values. 53:39
