Optimisation and learning
Introduced
Vocabulary
| English | Meaning |
|---|---|
| loss surface | The loss value viewed as a function of a model's parameter values. |
| gradient descent/ˈɡreɪdɪənt dɪˈsent/ | An iterative method that adjusts parameters in the direction opposite the gradient to seek a smaller value of an objective function. |
| learning rate/ˈlɜːnɪŋ reɪt/ | The step-size factor controlling the scale of a parameter update during training. |
Down the loss surface
- Gradient descent: $\theta_{t+1} = \theta_t - \eta \nabla L(\theta_t)$. The learning rate $\eta$ scales the step; the loss surface's curvature sets the difficulty.
{exercise:uni-cs1305-descent}
Practice
The update rule steps —
Difficulty 1/5 · 100 coins + a small bonus on first completion today
Downhill is the negative gradient, scaled by eta.
The quadratic case, exactly
- For $L(w) = (w-3)^2$: $w_{t+1} = w_t - 2\eta(w_t - 3)$. With $\eta = 0.1$ from 0: 0.6, 1.08, 1.464, 1.771, 2.017 — error shrinking geometrically to 3.
- $\eta = 1$: 0, 6, 0, 6 forever (pure oscillation); $\eta = 0.6$: 3.6, 2.88, 3.024 — damped. Stability: $|1 - 2\eta| < 1 \iff \eta < 1$.
{exercise:uni-cs1305-quadratic}
Practice
On L(w) = (w-3)^2, eta = 1 gives —
Difficulty 1/5 · 100 coins + a small bonus on first completion today
|1 - 2 eta| = 1: the boundary of stability.
Ravines and scaling
- On the least-squares line, two steps from $(0,0)$ with $\eta = 0.02$ reach $(0.84, 0.25)$; a thousand close in on the closed form $(0.9, 1.3)$.
- With $L = (x_1-1)^2 + 100(x_2-1)^2$ the surface is a ravine: one rate cannot serve both curvatures — $\eta = 0.009$ crawls; standardising features restores progress. Read the loss change, not the loss, for convergence.
{exercise:uni-cs1305-ravines}
Practice
Unscaled features with 100x curvature spread force —
Difficulty 1/5 · 100 coins + a small bonus on first completion today
One rate cannot serve both curvatures - standardise features.