Hypothesis testing and contextual conclusions
Introduced| English | Meaning |
|---|---|
| test statistic/test stəˈtɪstɪk/ | A quantity calculated from sample data and used to decide how strongly the data challenge a null hypothesis. |
| significance level/sɪɡˈnɪfɪkəns ˈlevl/ | The chosen maximum probability of rejecting a true null hypothesis; a discrete test may have a smaller actual level. |
A lady who could taste the milk, 1920s
- In Cambridge in the 1920s, the biologist Muriel Bristol said she could taste whether milk or tea had gone into the cup first. The statistician Ronald Fisher designed a test: eight cups, four of each, in random order. If she were only guessing, the chance of getting all eight right was $\tfrac{1}{70}$. She got them all right. Fisher used the story in his 1935 book The Design of Experiments.
- A hypothesis test asks: if nothing special is happening, how likely is a result this extreme?
In Fisher's test, how many ways are there to choose which 4 of the 8 cups had milk first?
Difficulty 2/5 · 140 coins + a small bonus on first completion today
$\binom{8}{4} = 70$.
Two hypotheses
- The null hypothesis $H_0$ states the "no effect" value, for example $p = 0.5$; the alternative $H_1$ says what you suspect: $p > 0.5$, $p < 0.5$ or $p \ne 0.5$.
- The test statistic 检验统计量 is the observed count, modelled by $X \sim B(n, p)$ under $H_0$.
A drug company claims a cure rate above 40%. Which alternative hypothesis tests the claim?
Difficulty 1/5 · 100 coins + a small bonus on first completion today
The claim is "greater than".
Significance level and critical region
- The significance level 显著性水平 (for example 5%) is the largest probability of rejecting $H_0$ when it is true that you accept.
- The critical region is the set of values for which you reject $H_0$; the actual significance level is the probability of falling in it.
Hypothesis testing and contextual conclusions
State H₀ and H₁ in population parameters before inspecting the outcome
Compare the model with the worked case and explain one change.
Worked example: is the coin biased?
A coin is thrown 20 times and shows 15 heads. Test at the 5% level whether it is biased towards heads.
- Hypotheses: $H_0: p = 0.5$, $H_1: p > 0.5$. Under $H_0$, $X \sim B(20, 0.5)$.
- Probability: $P(X \ge 15) = 1 - P(X \le 14) = 0.0207$.
- Conclusion: $0.0207 < 0.05$, so reject $H_0$. There is sufficient evidence, at the 5% level, that the coin is biased towards heads.
Under $H_0$, $X \sim B(10, 0.5)$. Find $P(X \ge 9)$ to 4 d.p.
Difficulty 2/5 · 140 coins + a small bonus on first completion today
$\tfrac{10 + 1}{1024}$.
At the 5% level, a tail probability of 0.07 leads to rejecting $H_0$.
Difficulty 1/5 · 100 coins + a small bonus on first completion today
$0.07 > 0.05$.
One tail or two
- If $H_1$ is $p \ne 0.5$, the test is two-tailed: split the 5% into 2.5% in each tail.
- Choose the tails from the question before looking at the data.
In a two-tailed test at the 10% level, what probability goes in each tail?
Difficulty 1/5 · 100 coins + a small bonus on first completion today
Half of 10%.
Writing a conclusion
- Give a decision and then a sentence in context: "There is (in)sufficient evidence at the 5% level that…".
- Never write "$H_0$ is true": failing to reject means only that the evidence is not strong enough.
Which is the best conclusion when $H_0: p = 0.3$ is not rejected?
Difficulty 1/5 · 100 coins + a small bonus on first completion today
Failing to reject is not proof.
A test does not prove anything. If $P(X \ge 15) = 0.02$, it does not mean there is a 2% chance the coin is fair. It means that if the coin were fair, a result this extreme would happen about 2% of the time.
State $H_0$ and $H_1$ about the parameter $p$, model the count with $B(n, p)$ under $H_0$, and compare the tail probability (or the critical region) with the significance level. Halve the level for a two-tailed test, and end with a conclusion in context that never claims proof.
In the past 25% of customers bought dessert. After a menu change, 2 of 20 customers do. Test at the 5% level whether the proportion has decreased.
Lower-tail test.