Least-Squares Regression

Why "least squares"?

• Many lines could pass through a cloud — which is best?
• The least-squares line 最小二乘线 is the one that makes the sum of squared residuals as small as possible.
• Squaring keeps errors positive and punishes big misses extra hard.
• Minimizing that total is what pins down the unique best-fit slope and intercept.

r-squared: variation explained

• The coefficient of determination 决定系数 $r^2$ is the fraction of the variation in $y$ explained by the model.
• It's literally the square of the correlation, so $0 \le r^2 \le 1$.
• $r^2 = 0.64$ means "$64\%$ of the variation in $y$ is explained by the linear relationship with $x$."
• The rest ($36\%$) is due to other factors and scatter.

Residuals are observed minus predicted; least squares makes the sum of their squares as small as possible

s: typical prediction error

• The standard deviation of the residuals $s$ is the typical size of a prediction error.
• It's in the units of $y$: "predictions are typically off by about $s$."
• Smaller $s$ = tighter fit; larger $s$ = looser predictions.
• $s$ answers "how far off, in real units?" while $r^2$ answers "what fraction explained?"

Slope from summary stats

• You can build the line from summary statistics without the raw data:
• Slope: $b = r\dfrac{s_y}{s_x}$ — correlation scaled by the ratio of spreads.
• Intercept: the line always passes through $(\bar{x}, \bar{y})$, so $a = \bar{y} - b\bar{x}$.
• These two formulas recover $\hat{y} = a + bx$ from $r$, the means, and the standard deviations.

Write from 81 percent of the variation

• The correlation r is 0.9 for a linear model.
• The lesson r-squared of 0.64 is another model.