Lesson 21: Hypothesis Tests — The Big Idea

(a) Warm-Up

In Lesson 19 you built a confidence interval to estimate an unknown parameter. Now we flip the question. Instead of "what is the value?" we ask "is a specific claimed value believable?"

Picture a snack company that prints "40% of buyers prefer the new flavor" on its marketing deck. You're skeptical. You survey a random sample of 200 buyers and find that only 34% prefer the new flavor — a sample proportion of p̂ = 0.34. Here's the tension at the heart of this whole unit: 0.34 is not 0.40, but samples always wobble. Maybe the true proportion really is 0.40 and you just happened to draw a sample on the low side. Or maybe the claim is too high and the data are telling you so.

A significance test (also called a hypothesis test) is the formal machinery for settling that argument. It starts by assuming the claim is true, then asks: if it really were true, how surprising would data like ours be? If the answer is "extremely surprising," we doubt the claim. If the answer is "not surprising at all," we have no case. This lesson is the logic. Lesson 22 plugs in the proportion arithmetic.

(b) Core Concept

The courtroom logic of a significance test

A significance test works like a criminal trial. The defendant is presumed innocent — that's our starting assumption. We only convict if the evidence against innocence is overwhelming. In statistics, the "presumption of innocence" is the null hypothesis, and the evidence is our data.

The four moving parts:

1. Assume the null hypothesis is true.
2. Measure how far our sample statistic falls from what the null predicts (the test statistic).
3. Compute how likely a result at least that extreme would be if the null were true (the p-value).
4. Decide: if that probability is small enough, the data are surprising under the null, so we reject the null. If not, we have no convincing case against it.

Stating the hypotheses

The null hypothesis H₀ is a statement of "no effect" or "no difference." It always contains an equality (=). It names the specific parameter value we will assume true.

The alternative hypothesis Hₐ is what we suspect instead — what we're gathering evidence for. It uses an inequality and comes in three flavors:

Two non-negotiable rules. First, hypotheses are always about a parameter, never a statistic. Write H₀: p = 0.40, never H₀: p̂ = 0.40. The sample proportion p̂ is just a number you computed; there's nothing to hypothesize about it. The unknown truth is the population parameter p. Second, decide one-sided vs. two-sided from the question, not from your data. If you only wonder "is it different?", use ≠. If the context points a direction ("is it less than claimed?"), use < or >.

For the snack example, we suspect the true preference is below 0.40, so:

H₀: p = 0.40   (the claim — buyers prefer the new flavor at the rate stated)
Hₐ: p < 0.40   (the true preference rate is lower than claimed)

where p = the true proportion of all buyers who prefer the new flavor. Always define the parameter in words.

The test statistic: counting standard errors

Once we assume H₀, we know where the sampling distribution of the statistic is centered (at the null value) and how spread out it is (its standard error). The test statistic measures how many standard errors our estimate sits from the null value:

test statistic = (estimate − null value) / (standard error of the estimate)

For proportions this is a z-statistic. A z of −2 means "our sample proportion is 2 standard errors below what the null claims." The bigger the test statistic in absolute value, the farther our data are from the null, and the more surprising they are. (You'll compute these standard errors in Lesson 22; here, focus on what the number means.)

The p-value — stated precisely

Here is the single most tested, most-often-misstated definition in the course. Read it slowly:

The p-value is the probability, assuming the null hypothesis H₀ is true, of obtaining a test statistic at least as extreme as the one actually observed.

Three load-bearing phrases:

• "Assuming H₀ is true" — the p-value is a conditional probability. We compute it inside the imaginary world where the null holds. It is not the probability that H₀ is true.
• "At least as extreme" — we include our observed result and everything more surprising in the direction(s) of Hₐ. For Hₐ: < it's a left tail; for > a right tail; for ≠ we count both tails.
• "Test statistic" — extremeness is measured on the standardized scale, as tail area under the (approximately Normal) sampling distribution.

A small p-value means: if H₀ were true, data like ours would rarely happen — so the data are evidence against H₀. A large p-value means: data like ours are perfectly ordinary under H₀ — no evidence against it.

[GRAPH: A standard Normal sampling distribution (bell curve) centered at 0, labeled "Sampling distribution of the test statistic, assuming H₀ is true." The horizontal axis is the z test statistic. A vertical line is drawn at the observed value z = −1.75 on the left side. The entire region to the LEFT of z = −1.75 (the tail) is shaded. A label points to the shaded tail: "p-value = P(z ≤ −1.75 | H₀ true) ≈ 0.040 — the probability of a result this extreme or more, in the direction of Hₐ: p < 0.40." Caption: "The p-value is tail area beyond the observed test statistic, computed assuming H₀ is true. For a two-sided Hₐ (≠), you would shade BOTH tails and double the area."]

A correct p-value interpretation sentence (memorize this template):

"Assuming the true proportion of buyers who prefer the new flavor is 0.40 (H₀ true), there is about a 4% probability of getting a sample proportion as low as 0.34 or lower — that is, a test statistic at least as extreme as the one we observed."

The significance level α and the decision rule

How small is "small enough"? We set a threshold before looking at the data: the significance level α (commonly α = 0.05, sometimes 0.01 or 0.10). The decision rule:

• If p ≤ α → reject H₀. The result is statistically significant. The data are too surprising to be comfortably explained by the null.
• If p > α → fail to reject H₀. The data are consistent with the null; we lack convincing evidence against it.

Notice the careful language. We reject H₀ or we fail to reject H₀. We never "accept H₀," and we never "prove Hₐ." Failing to reject is like a "not guilty" verdict — it means insufficient evidence to convict, not proven innocent. The truth might still differ from the null; our sample just wasn't strong enough to show it.

A correct conclusion sentence (the template that earns points):

"Because the p-value (0.040) ≤ α (0.05), we reject H₀. We have convincing evidence that the true proportion of buyers who prefer the new flavor is less than 0.40."

If instead p > α:

"Because the p-value (0.18) > α (0.05), we fail to reject H₀. We do not have convincing evidence that the true proportion is less than 0.40."

Every conclusion must (1) compare p to α, (2) state reject / fail to reject, and (3) describe the evidence about the parameter in context.

Two ways to be wrong: Type I and Type II errors

A test makes a decision from incomplete information, so it can be wrong in two ways:

• A Type I error is rejecting a true H₀ — a "false alarm." We conclude there's an effect when there really isn't. Its probability is exactly the significance level α.
• A Type II error is failing to reject a false H₀ — a "missed detection." A real effect exists, but our data didn't catch it. Its probability is denoted β.

These trade off. Lowering α (say from 0.05 to 0.01) makes you slower to reject, cutting Type I errors but raising the chance of a Type II error. The power of a test is the probability of correctly rejecting a false H₀, equal to 1 − β — its ability to detect a real effect when one exists. Power rises with a larger sample size, a larger true effect, and a larger α.

Naming a real-world consequence matters on the exam. For a drug trial with H₀: the drug is no better than placebo:

• Type I error: conclude the drug works when it doesn't → patients take a useless (possibly risky, costly) drug.
• Type II error: miss a drug that truly works → a beneficial treatment never reaches patients.

The duality of confidence intervals and two-sided tests

Confidence intervals (Lessons 19–20) and two-sided tests are two views of the same information. A two-sided test at significance level α and a (1 − α) confidence interval agree:

• If the null value falls outside a 95% CI → reject H₀: p = (null value) at α = 0.05.
• If the null value falls inside the 95% CI → fail to reject at α = 0.05.

Example: if a 95% CI for p is (0.28, 0.39), then 0.40 is not in the interval, so a two-sided test of H₀: p = 0.40 would reject at α = 0.05. The CI does double duty — it estimates the parameter and tells you which null values are plausible.

(c) Worked Examples

Example 1 (easy) — writing hypotheses, one- and two-sided

A factory's bottling line is supposed to fill bottles so that the true proportion of underfilled bottles is 0.05. Write H₀ and Hₐ for each scenario, defining the parameter.

(i) A quality manager wants to know whether the underfill rate has increased above 0.05.

(ii) An auditor only wants to know whether the underfill rate differs from 0.05 in either direction.

Solution. Let p = the true proportion of all bottles that are underfilled.

(i) The concern points a direction (increased), so the test is one-sided (right):

H₀: p = 0.05      Hₐ: p > 0.05

(ii) "Differs … in either direction" is two-sided:

H₀: p = 0.05      Hₐ: p ≠ 0.05

Interpretation. Same null both times — the null is always the equality claim. The alternative encodes what we suspect, and it is read from the question's wording, not from any data.

Example 2 (medium) — interpreting a p-value in context

A nutrition group claims 30% of teens skip breakfast. A researcher suspecting the rate is higher surveys a random sample and tests H₀: p = 0.30 vs. Hₐ: p > 0.30, obtaining a p-value of 0.012. Interpret this p-value in context.

Strategy. Use the precise template: "assuming H₀ true, probability of a result at least as extreme."

Solution.

"Assuming the true proportion of teens who skip breakfast is 0.30 (H₀ true), there is only about a 1.2% probability of obtaining a sample proportion as high as the one observed, or higher (a test statistic at least as extreme as ours, in the direction of Hₐ: p > 0.30)."

Interpretation. Because such a result would be rare if the null were true, this is fairly strong evidence against H₀. It is not "a 1.2% chance the null is true" — that misreads a conditional probability as the probability of the hypothesis.

Example 3 (AP-style) — making a decision and stating a conclusion

Using the breakfast study above (Hₐ: p > 0.30, p-value = 0.012), the researcher chose α = 0.05 in advance. Make a decision and write a conclusion in context.

Solution. Compare p to α: 0.012 ≤ 0.05, so we reject H₀.

"Because the p-value (0.012) ≤ α (0.05), we reject H₀. We have convincing evidence that the true proportion of teens who skip breakfast is greater than 0.30."

Interpretation. Note what we did not say: not "we proved more than 30% skip breakfast," and not "we accept Hₐ." We have convincing evidence, not proof. Had α been 0.01 instead, then 0.012 > 0.01 and we would fail to reject — the same data, a stricter threshold, a different decision. That's why α is fixed before seeing the data.

Example 4 (AP-style) — identifying error type and a consequence

A hospital screens for a rare infection. Define H₀: the patient does not have the infection vs. Hₐ: the patient has the infection. The screening test is the "hypothesis test."

(a) Describe a Type I error and one consequence.

(b) Describe a Type II error and one consequence.

(c) Which error would you rather guard against, and how does that affect α?

Solution.

(a) Type I error = rejecting a true H₀ = concluding the patient has the infection when they actually do not. Consequence: a healthy patient endures anxiety, follow-up tests, and possibly unnecessary treatment.

(b) Type II error = failing to reject a false H₀ = concluding no infection when the patient actually has it. Consequence: an infected patient goes untreated and may spread the infection or worsen.

(c) For a dangerous, contagious infection, a Type II error (missing a true case) is usually the graver danger. To reduce Type II errors we increase power, which we can do by raising α (e.g., to 0.10) so the test rejects more readily — at the cost of more Type I false alarms. The right balance depends on the relative costs of the two errors.

(d) Common Mistakes

1. Writing hypotheses about a statistic instead of a parameter. Students write H₀: p̂ = 0.40. Wrong — p̂ is the number you measured; there's nothing to hypothesize. Hypotheses are claims about the unknown parameter: H₀: p = 0.40. The same error appears as H₀: x̄ = ... for means. Always use the parameter symbol (p, μ) and define it in words.

2. "Accepting H₀." A large p-value means fail to reject H₀, not "accept H₀" or "H₀ is true." Insufficient evidence to convict is not proof of innocence. Likewise we never "prove Hₐ" — at most we have convincing evidence for it. Writing "accept H₀" or "the null is true" loses the conclusion point.

3. Misinterpreting the p-value as P(H₀ true). A p-value of 0.03 does not mean "a 3% chance the null is true." The p-value is a probability about the data (a test statistic at least as extreme), computed assuming H₀ is already true. It is a conditional probability about data, not a probability about the hypothesis.

4. Dropping context from the conclusion. "We reject H₀" by itself earns little. The exam wants the parameter, in words, in context: "…convincing evidence that the true proportion of buyers preferring the new flavor is less than 0.40." Compare p to α, state the decision, and describe the parameter in context — all three.

5. Choosing one-sided vs. two-sided from the data. Peeking at p̂ = 0.34 and then writing Hₐ: p < 0.40 to get a smaller p-value is cheating ("p-hacking"). The direction of Hₐ comes from the research question stated before data collection. If the question only asks "is it different?", use ≠.

(e) Practice Problems

(in context) A streaming service claims 60% of its users finish a show they start. A skeptical analyst believes the true completion rate is lower and surveys a random sample of users. (a) Define the parameter p in words. (b) Write H₀ and Hₐ.

(in context) In Problem 11, the analyst obtains a p-value of 0.08 and uses α = 0.05. (a) State the decision. (b) Write a conclusion in context. (c) The analyst then says, "This proves at least 60% finish their shows." Explain what is wrong with that statement.

A researcher writes: "The p-value was 0.03, so there is a 3% chance the null hypothesis is true." Identify the error and state the correct interpretation of a p-value.

(in context) A city claims that 25% of its bridges need repair. An engineer tests H₀: p = 0.25 vs. Hₐ: p ≠ 0.25, where p is the true proportion of all city bridges needing repair. Describe, in context, what a Type I error and a Type II error would mean here, and give one consequence of each.

A test of H₀: p = 0.30 vs. Hₐ: p > 0.30 gives a p-value of 0.27 at α = 0.05. A student concludes, "We accept H₀; exactly 30% have the trait." Give two distinct things wrong with this conclusion.

(f) FRQ Practice (10 points)

Statistical Practice: 1 — Formulate Questions; 4 — Interpret Results

A national education nonprofit reports that 70% of high school seniors at a certain large district complete the federal financial-aid application (the FAFSA). A counselor at one large high school in the district suspects that the true completion rate at her school is lower than the district-reported 70%. She takes a random sample of 150 seniors from her school, finds the sample proportion who completed the FAFSA, and carries out a significance test. Her test produces a p-value of 0.031. She uses a significance level of α = 0.05.

(a) State the null and alternative hypotheses for the counselor's test. Be sure to define the parameter in context. (3 points)

(b) Interpret the p-value of 0.031 in the context of this study. (3 points)

(c) Using α = 0.05, state the counselor's decision and write a conclusion in context. (2 points)

(d) Describe what a Type I error would be in the context of this study, and give one consequence of making that error. (2 points)

Model Response

(a) Let p = the true proportion of all seniors at the counselor's high school who complete the FAFSA.

H₀: p = 0.70
Hₐ: p < 0.70

The alternative is one-sided (left) because the counselor specifically suspects the completion rate at her school is lower than 0.70.

(b) Assuming the true FAFSA completion rate at the counselor's school is 0.70 (that is, assuming H₀ is true), there is only about a 3.1% probability of obtaining a sample proportion as low as the one she observed, or lower — a test statistic at least as extreme as the one she got in the direction of Hₐ: p < 0.70.

(c) Because the p-value (0.031) ≤ α (0.05), the counselor rejects H₀. There is convincing evidence that the true proportion of seniors at her school who complete the FAFSA is less than 0.70 (lower than the district-reported rate).

(d) A Type I error would be rejecting H₀ when H₀ is actually true — that is, concluding that her school's true FAFSA completion rate is below 0.70 when it is actually 0.70. Consequence: the counselor might spend limited time and money launching an unnecessary FAFSA-completion campaign (or alarm the administration) for a problem that does not actually exist.

Scoring Rubric (10 points)

Part (a) — 3 points

• 1 pt — Parameter p defined in context as the true proportion of seniors at her school who complete the FAFSA (must reference the population, not the sample).
• 1 pt — H₀: p = 0.70 stated with an equality and about the parameter (not p̂).
• 1 pt — Hₐ: p < 0.70 — one-sided in the correct (left) direction, matching the suspicion of a lower rate.

Part (b) — 3 points

• 1 pt — States the probability is computed assuming H₀ is true (the conditional "if the true proportion is 0.70").
• 1 pt — Refers to a result at least as extreme ("as low as observed, or lower" / in the direction of Hₐ).
• 1 pt — Correctly identifies the value as ≈ 0.031 (about 3.1%) as a probability about the data/test statistic (not about the hypothesis).

Part (c) — 2 points

• 1 pt — Compares p to α (0.031 ≤ 0.05) and states reject H₀.
• 1 pt — Conclusion in context: convincing evidence that the true completion rate at her school is less than 0.70.

Part (d) — 2 points

• 1 pt — Correctly defines Type I error in context: rejecting a true H₀ = concluding the rate is below 0.70 when it is actually 0.70.
• 1 pt — Gives a reasonable consequence of that error in context (e.g., wasted resources on an unneeded intervention).

Where students lose points:

• Defining the parameter as the sample proportion, or writing H₀: p̂ = 0.70 — hypotheses must be about the parameter p (loses part-a points).
• A two-sided Hₐ: p ≠ 0.70 when the context clearly points downward — loses the direction point.
• Interpreting the p-value as "a 3.1% chance H₀ is true" or "a 3.1% chance the result happened by chance" — both omit the "assuming H₀ is true / at least as extreme" structure (loses up to 2 of 3 in part b).
• Writing "accept H₀" or "we proved p < 0.70" anywhere — incorrect inference language.
• A bare "reject H₀" in (c) with no context about the FAFSA completion rate — loses the context point.
• In (d), describing a Type II error instead (failing to detect a truly low rate), or omitting a consequence.