Optimisation and learning
Introduit
Vocabulaire
| English | Sens |
|---|---|
| loss surface | The loss value viewed as a function of a model's parameter values. |
| gradient descent/ˈɡreɪdɪənt dɪˈsent/ | An iterative method that adjusts parameters in the direction opposite the gradient to seek a smaller value of an objective function. |
| learning rate/ˈlɜːnɪŋ reɪt/ | The step-size factor controlling the scale of a parameter update during training. |
English
Down the loss surface
- Gradient descent: $\theta_{t+1} = \theta_t - \eta \nabla L(\theta_t)$. The learning rate $\eta$ scales the step; the loss surface's curvature sets the difficulty.
{exercise:uni-cs1305-descent}
Pratique
The update rule steps —
Difficulté 1/5 · 100 pièces + un petit bonus lors de la première réussite aujourd'hui
Downhill is the negative gradient, scaled by eta.
English
The quadratic case, exactly
- For $L(w) = (w-3)^2$: $w_{t+1} = w_t - 2\eta(w_t - 3)$. With $\eta = 0.1$ from 0: 0.6, 1.08, 1.464, 1.771, 2.017 — error shrinking geometrically to 3.
- $\eta = 1$: 0, 6, 0, 6 forever (pure oscillation); $\eta = 0.6$: 3.6, 2.88, 3.024 — damped. Stability: $|1 - 2\eta| < 1 \iff \eta < 1$.
{exercise:uni-cs1305-quadratic}
Pratique
On L(w) = (w-3)^2, eta = 1 gives —
Difficulté 1/5 · 100 pièces + un petit bonus lors de la première réussite aujourd'hui
|1 - 2 eta| = 1: the boundary of stability.
English
Ravines and scaling
- On the least-squares line, two steps from $(0,0)$ with $\eta = 0.02$ reach $(0.84, 0.25)$; a thousand close in on the closed form $(0.9, 1.3)$.
- With $L = (x_1-1)^2 + 100(x_2-1)^2$ the surface is a ravine: one rate cannot serve both curvatures — $\eta = 0.009$ crawls; standardising features restores progress. Read the loss change, not the loss, for convergence.
{exercise:uni-cs1305-ravines}
Pratique
Unscaled features with 100x curvature spread force —
Difficulté 1/5 · 100 pièces + un petit bonus lors de la première réussite aujourd'hui
One rate cannot serve both curvatures - standardise features.