Limitations: statistics that stays honest
Introduced| English | Meaning |
|---|---|
| p-value/piː ˈvæljuː/ | p-value |
| null hypothesis/nʌl haɪˈpɒθəsɪs/ | The default claim about a population parameter that a statistical test assesses against an alternative. |
Everything before this can be done correctly and still lie
- Ask twenty questions and one p-value p值 clears 0.05 by luck — each a vote against its null hypothesis 零假设 that nothing is going on. Peek at the data until it does. Report the metric that moved and file the rest. Forecast a single number with no band.
- This final lesson is the checklist that keeps statistics a discipline instead of a rhetoric.
The multiple-testing lottery
- Twenty independent tests at α = 0.05: $P(\text{at least one "finding"}) = 1 - 0.95^{20} = \mathbf{0.64}$ — a likely fluke, dressed as a discovery.
- The Bonferroni discipline: 20 tests at family α = 0.05 means each needs p < 0.0025. Harsh, and honest about the lottery it prevents.
Twenty independent tests at alpha 0.05. P(at least one false positive), to two decimals?
Difficulty 2/5 · 140 coins + a small bonus on first completion today
$1 - 0.95^{20} = 1 - 0.358 = 0.64$. A likely fluke - which is why families of tests need correction.
Bonferroni for 20 tests at family alpha 0.05: each test needs p below what value?
Difficulty 2/5 · 140 coins + a small bonus on first completion today
$0.05/20 = 0.0025$. Harsh - it trades a little power for an honest family error rate.
Stopping rules and peeking
- Keep testing as data arrive and stop the moment p < 0.05, and your true error rate multiplies — the threshold you claim is not the one you ran.
- The cure is declared, pre-registered rules: the hypothesis, the metric, the sample size, the stopping rule — written before the data speak.
Audit the headline
Route each reported result to the gate that challenges it - tests run, stopping rule, selection, or uncertainty band.
Checking p after every order and stopping at the first value under 0.05 keeps the true error rate at 5 percent.
Difficulty 1/5 · 100 coins + a small bonus on first completion today
Optional stopping inflates the false-positive rate far above the nominal level - the threshold you ran is not the threshold you claim.
Why does the file drawer matter to a reader of one significant result?
Difficulty 1/5 · 100 coins + a small bonus on first completion today
Selection happens across studies, not just within them. The question "how many cousins did this beat?" is always fair.
Match each report defect to its gate.
Difficulty 2/5 · 140 coins + a small bonus on first completion today
The four gates generalise: count, declare, show, band.
The file drawer
- The p = 0.09 the rider study found is "not significant" — and it is also ONE of the few tests you will ever hear about. Unreported nulls are data about the reporting, not the world.
- Ask of any headline: how many cousins did this result out-compete to reach me?
Forecast band for next Saturday's mean: point 27.43, SE 0.78, t about 2. The 95% half-width, to one decimal?
Difficulty 2/5 · 140 coins + a small bonus on first completion today
$2 \times 0.78 = 1.56 \approx 1.6$: report 27.4 +/- 1.6, and staff to about 29 - the band's upper end.
With sd 4.91 and n = 40, the SE of the mean, to two decimals?
Difficulty 2/5 · 140 coins + a small bonus on first completion today
$4.91/\sqrt{40} = 0.776 \approx 0.78$ - the mean's wobble across hypothetical Saturdays.
The most defensible improvement to a point-forecast culture?
Difficulty 1/5 · 100 coins + a small bonus on first completion today
Bands make forecasts testable; calibration scores them. Certainty never arrives - honesty about the band is the achievable standard.
Forecasts wear bands
- "Next Saturday's average delivery: 27.4 minutes" is a mood. The dataset's answer: SE ≈ $4.91/\sqrt{40} = 0.78$, so "27.4 ± 1.5, 95%" — and one Saturday in twenty lands outside even that.
- Staff to the band's upper end (28.9), not the point. A number without its uncertainty cannot be managed, only hoped at.
Three final traps
- One significant result among many is a finding. It is a lottery ticket that won.
- "We proved nothing changed." Fail-to-reject is not proof of the null — the mirror trap again.
- Precision theatre. More decimals than the interval supports; point forecasts without bands.
Recap
- Count the tests, declare the rules, show the file drawer, band every forecast.
- Statistics earns decisions by admitting what it does not know.