Least-Squares Regression
Introduit| English | Sens |
|---|---|
| least-squares line/liːst skweəz laɪn/ | droite des moindres carrés |
| coefficient of determination/ˌkəʊɪˈfɪʃənt ɒv dɪˌtɜːmɪˈneɪʃn/ | coefficient de détermination |
Why "least squares"?
- Many lines could pass through a cloud — which is best?
- The least-squares line 最小二乘线 is the one that makes the sum of squared residuals as small as possible.
- Squaring keeps errors positive and punishes big misses extra hard.
- Minimizing that total is what pins down the unique best-fit slope and intercept.

Residuals are observed minus predicted; least squares makes the sum of their squares as small as possible
r-squared: variation explained
- The coefficient of determination 决定系数 $r^2$ is the fraction of the variation in $y$ explained by the model.
- It's literally the square of the correlation, so $0 \le r^2 \le 1$.
- $r^2 = 0.64$ means "$64\%$ of the variation in $y$ is explained by the linear relationship with $x$."
- The rest ($36\%$) is due to other factors and scatter.
s: typical prediction error
- The standard deviation of the residuals $s$ is the typical size of a prediction error.
- It's in the units of $y$: "predictions are typically off by about $s$."
- For the same response, units, and comparable data, smaller $s$ means a tighter fit. Values on different response scales cannot be compared directly.
- $s$ answers "how far off, in real units?" while $r^2$ answers "what fraction explained?"
Slope from summary stats
- You can build the line from summary statistics without the raw data:
- Slope: $b = r\dfrac{s_y}{s_x}$ — correlation scaled by the ratio of spreads.
- Intercept: the line always passes through $(\bar{x}, \bar{y})$, so $a = \bar{y} - b\bar{x}$.
- These two formulas recover $\hat{y} = a + bx$ from $r$, the means, and the standard deviations.
Don't confuse the two fit numbers. $r^2$ is a unitless fraction ("$64\%$ of variation explained"); $s$ is a typical error in the units of $y$ ("off by about $s$"). And when you compute the slope, mind the order: $b = r\,s_y/s_x$ — it's $s_y$ (response spread) over $s_x$ (explanatory spread), not the reverse.
A fit has $r = 0.8$, $s_x = 2$, $s_y = 10$, $\bar{x}=5$, $\bar{y}=60$.
- Slope: $b = 0.8 \times \dfrac{10}{2} = 4$.
- Intercept: $a = 60 - 4(5) = 40$, so $\hat{y} = 40 + 4x$.
- $r^2 = 0.64$: $64\%$ of the variation in $y$ is explained by the model.
The least-squares line minimizes the sum of squared residuals. $r^2$ is the fraction of variation in $y$ explained (unitless, $0$–$1$); $s$ is the typical prediction error (units of $y$). From summary stats, $b = r\dfrac{s_y}{s_x}$ and the line passes through $(\bar{x}, \bar{y})$.
Write from the data given
- The correlation r is 0.9 for a linear model.
- The lesson r-squared of 0.64 is another model.
A least-squares linear model with an intercept has correlation r=0.9. What percentage of the variation in y does the model explain?
r squared is the fraction of variation explained.
For a least-squares linear model with an intercept and r=0.9, what percentage of variation in y is explained?
Difficulté 1/5 · 100 pièces + un petit bonus lors de la première réussite aujourd'hui
r²=0.9²=0.81. Convert the fraction to a percentage: 0.81×100=81% of the variation in y is explained by its linear relationship with x.
The best-fit least-squares line
The line that makes the total squared vertical gap smallest.
With r = 0.8, s_x = 2, s_y = 10, find the slope b = r·(s_y/s_x).
Difficulté 2/5 · 140 pièces + un petit bonus lors de la première réussite aujourd'hui
b = 0.8 × (10/2) = 0.8 × 5 = 4.
If the correlation is r = 0.8, what is r-squared?
Difficulté 2/5 · 140 pièces + un petit bonus lors de la première réussite aujourd'hui
r² = 0.8² = 0.64.
r² = 0.64 is best interpreted as...
Difficulté 1/5 · 100 pièces + un petit bonus lors de la première réussite aujourd'hui
r² is the fraction of y's variation explained.
The least-squares line minimizes the sum of the squared residuals.
Difficulté 1/5 · 100 pièces + un petit bonus lors de la première réussite aujourd'hui
That is exactly the criterion that defines it.
Which quantity is a typical prediction error, measured in the units of y?
Difficulté 1/5 · 100 pièces + un petit bonus lors de la première réussite aujourd'hui
s is in y-units; r² and r are unitless.