Limitations: statistics that stays honest
แนะนำใหม่| English | ความหมาย |
|---|---|
| p-value/piː ˈvæljuː/ | p-value |
| null hypothesis/nʌl haɪˈpɒθəsɪs/ | The default claim about a population parameter that a statistical test assesses against an alternative. |
Everything before this can be done correctly and still lie
- Ask twenty questions and one p-value p值 clears 0.05 by luck — each a vote against its null hypothesis 零假设 that nothing is going on. Peek at the data until it does. Report the metric that moved and file the rest. Forecast a single number with no band.
- This final lesson is the checklist that keeps statistics a discipline instead of a rhetoric.
The multiple-testing lottery
- Twenty independent tests at α = 0.05: $P(\text{at least one "finding"}) = 1 - 0.95^{20} = \mathbf{0.64}$ — a likely fluke, dressed as a discovery.
- The Bonferroni discipline: 20 tests at family α = 0.05 means each needs p < 0.0025. Harsh, and honest about the lottery it prevents.
Twenty independent tests at alpha 0.05. P(at least one false positive), to two decimals?
ความยาก 2/5 · 140 เหรียญ + บอนัสน้อยๆ เมื่อทำสำเร็จครั้งแรกในวันนี้
$1 - 0.95^{20} = 1 - 0.358 = 0.64$. A likely fluke - which is why families of tests need correction.
Bonferroni for 20 tests at family alpha 0.05: each test needs p below what value?
ความยาก 2/5 · 140 เหรียญ + บอนัสน้อยๆ เมื่อทำสำเร็จครั้งแรกในวันนี้
$0.05/20 = 0.0025$. Harsh - it trades a little power for an honest family error rate.
Stopping rules and peeking
- Keep testing as data arrive and stop the moment p < 0.05, and your true error rate multiplies — the threshold you claim is not the one you ran.
- The cure is declared, pre-registered rules: the hypothesis, the metric, the sample size, the stopping rule — written before the data speak.
Audit the headline
Route each reported result to the gate that challenges it - tests run, stopping rule, selection, or uncertainty band.
Checking p after every order and stopping at the first value under 0.05 keeps the true error rate at 5 percent.
ความยาก 1/5 · 100 เหรียญ + บอนัสน้อยๆ เมื่อทำสำเร็จครั้งแรกในวันนี้
Optional stopping inflates the false-positive rate far above the nominal level - the threshold you ran is not the threshold you claim.
Why does the file drawer matter to a reader of one significant result?
ความยาก 1/5 · 100 เหรียญ + บอนัสน้อยๆ เมื่อทำสำเร็จครั้งแรกในวันนี้
Selection happens across studies, not just within them. The question "how many cousins did this beat?" is always fair.
Match each report defect to its gate.
ความยาก 2/5 · 140 เหรียญ + บอนัสน้อยๆ เมื่อทำสำเร็จครั้งแรกในวันนี้
The four gates generalise: count, declare, show, band.
The file drawer
- The p = 0.09 the rider study found is "not significant" — and it is also ONE of the few tests you will ever hear about. Unreported nulls are data about the reporting, not the world.
- Ask of any headline: how many cousins did this result out-compete to reach me?
Forecast band for next Saturday's mean: point 27.43, SE 0.78, t about 2. The 95% half-width, to one decimal?
ความยาก 2/5 · 140 เหรียญ + บอนัสน้อยๆ เมื่อทำสำเร็จครั้งแรกในวันนี้
$2 \times 0.78 = 1.56 \approx 1.6$: report 27.4 +/- 1.6, and staff to about 29 - the band's upper end.
With sd 4.91 and n = 40, the SE of the mean, to two decimals?
ความยาก 2/5 · 140 เหรียญ + บอนัสน้อยๆ เมื่อทำสำเร็จครั้งแรกในวันนี้
$4.91/\sqrt{40} = 0.776 \approx 0.78$ - the mean's wobble across hypothetical Saturdays.
The most defensible improvement to a point-forecast culture?
ความยาก 1/5 · 100 เหรียญ + บอนัสน้อยๆ เมื่อทำสำเร็จครั้งแรกในวันนี้
Bands make forecasts testable; calibration scores them. Certainty never arrives - honesty about the band is the achievable standard.
Forecasts wear bands
- "Next Saturday's average delivery: 27.4 minutes" is a mood. The dataset's answer: SE ≈ $4.91/\sqrt{40} = 0.78$, so "27.4 ± 1.5, 95%" — and one Saturday in twenty lands outside even that.
- Staff to the band's upper end (28.9), not the point. A number without its uncertainty cannot be managed, only hoped at.
Three final traps
- One significant result among many is a finding. It is a lottery ticket that won.
- "We proved nothing changed." Fail-to-reject is not proof of the null — the mirror trap again.
- Precision theatre. More decimals than the interval supports; point forecasts without bands.
Recap
- Count the tests, declare the rules, show the file drawer, band every forecast.
- Statistics earns decisions by admitting what it does not know.