Least-Squares Regression
Introduced| English |
|---|
| least-squares line/liːst skweəz laɪn/ |
| coefficient of determination/ˌkəʊɪˈfɪʃənt ɒv dɪˌtɜːmɪˈneɪʃn/ |
Why "least squares"?
- Many lines could pass through a cloud — which is best?
- The least-squares line 最小二乘线 is the one that makes the sum of squared residuals as small as possible.
- Squaring keeps errors positive and punishes big misses extra hard.
- Minimizing that total is what pins down the unique best-fit slope and intercept.
r-squared: variation explained
- The coefficient of determination 决定系数 $r^2$ is the fraction of the variation in $y$ explained by the model.
- It's literally the square of the correlation, so $0 \le r^2 \le 1$.
- $r^2 = 0.64$ means "$64\%$ of the variation in $y$ is explained by the linear relationship with $x$."
- The rest ($36\%$) is due to other factors and scatter.
s: typical prediction error
- The standard deviation of the residuals $s$ is the typical size of a prediction error.
- It's in the units of $y$: "predictions are typically off by about $s$."
- Smaller $s$ = tighter fit; larger $s$ = looser predictions.
- $s$ answers "how far off, in real units?" while $r^2$ answers "what fraction explained?"
Slope from summary stats
- You can build the line from summary statistics without the raw data:
- Slope: $b = r\dfrac{s_y}{s_x}$ — correlation scaled by the ratio of spreads.
- Intercept: the line always passes through $(\bar{x}, \bar{y})$, so $a = \bar{y} - b\bar{x}$.
- These two formulas recover $\hat{y} = a + bx$ from $r$, the means, and the standard deviations.
Don't confuse the two fit numbers. $r^2$ is a unitless fraction ("$64\%$ of variation explained"); $s$ is a typical error in the units of $y$ ("off by about $s$"). And when you compute the slope, mind the order: $b = r\,s_y/s_x$ — it's $s_y$ (response spread) over $s_x$ (explanatory spread), not the reverse.
A fit has $r = 0.8$, $s_x = 2$, $s_y = 10$, $\bar{x}=5$, $\bar{y}=60$.
- Slope: $b = 0.8 \times \dfrac{10}{2} = 4$.
- Intercept: $a = 60 - 4(5) = 40$, so $\hat{y} = 40 + 4x$.
- $r^2 = 0.64$: $64\%$ of the variation in $y$ is explained by the model.
The least-squares line minimizes the sum of squared residuals. $r^2$ is the fraction of variation in $y$ explained (unitless, $0$–$1$); $s$ is the typical prediction error (units of $y$). From summary stats, $b = r\dfrac{s_y}{s_x}$ and the line passes through $(\bar{x}, \bar{y})$.
The best-fit least-squares line
The line that makes the total squared vertical gap smallest.
With r = 0.8, s_x = 2, s_y = 10, find the slope b = r·(s_y/s_x).
b = 0.8 × (10/2) = 0.8 × 5 = 4.
If the correlation is r = 0.8, what is r-squared?
r² = 0.8² = 0.64.
r² = 0.64 is best interpreted as...
r² is the fraction of y's variation explained.
The least-squares line minimizes the sum of the squared residuals.
That is exactly the criterion that defines it.
Which quantity is a typical prediction error, measured in the units of y?
s is in y-units; r² and r are unitless.