Confidence intervals and hypothesis tests

An engineer never sees the whole population. We test twenty concrete cylinders, not the whole pour; eight rebar coupons, not the whole heat of steel; ten asphalt slabs, not the whole highway. Statistical inference is the reasoning that takes us from a sample to a justified statement about the population, with an honest measure of how sure we can be. This chapter covers the two main tools: confidence intervals, which estimate a quantity with a margin of error, and hypothesis tests, which decide whether data provide convincing evidence for a claim.

From sample to population

A population parameter, such as the true mean strength μ, is fixed but unknown. A sample statistic, such as the sample mean \(\bar{x}\), is known but varies from sample to sample. Lesson 3 showed how much: the sample mean has standard deviation \(\sigma/\sqrt{n}\) and is approximately normal. That standard deviation of a statistic is called its standard error. Because σ is usually unknown too, we estimate the standard error with the sample standard deviation s:

Inference needs a sample that represents the population. Random sampling (every unit has a chance of selection) is the foundation; a sample of the cylinders that "looked good", or only from the morning trucks, can be wrong in a way no formula can fix.

Confidence intervals for a mean

A confidence interval gives a range of plausible values for a parameter. For a population mean with σ unknown, it uses Student's t distribution:

The t distribution looks like the standard normal but has heavier tails, to allow for the extra uncertainty of estimating σ from the sample. It depends on the degrees of freedom n − 1; as n grows it approaches the normal. For 95% confidence, t is 2.776 with 4 degrees of freedom, 2.262 with 9, 2.093 with 19 and approaches 1.96 for large samples.

What does "95% confident" mean? The method, used over and over on new samples, produces intervals that contain the true mean 95% of the time. Any one interval either contains μ or does not; we just do not know which. It does not mean 95% of individual cylinders lie in the interval: the interval is about the mean, and it shrinks as n grows, while the spread of individual values does not.

Choosing the sample size

Solving the margin of error for n gives the sample size needed to estimate a mean to within ±E, using a planning value for σ from past data:

Confidence interval for a proportion

For a proportion, such as the fraction of defective items or of homes with a particular feature, with x successes in n trials and \(\hat{p} = x/n\), a large-sample 95% interval is

Hypothesis testing

A hypothesis test asks whether the data provide convincing evidence against a default claim. The procedure is the same for every test:

• State the null hypothesis H₀ (the default, "no effect" or "meets the claim", always an equality such as μ = 400) and the alternative H₁ (what we want evidence for: μ > 400, μ < 400, or μ ≠ 400).
• Choose a significance level α, the risk of wrongly rejecting H₀ you will accept; 0.05 is common.
• Compute a test statistic measuring how far the data are from H₀, in standard errors.
• Find the p-value: the probability, if H₀ were true, of a result at least as extreme as the one observed.
• If p < α, reject H₀: the data are inconsistent with it. Otherwise, fail to reject: the data are consistent with H₀, which is not the same as proving it.

For a mean, the test statistic is

Errors, power and the link to intervals

Two kinds of error are possible. A Type I error rejects a true H₀ (a false alarm) and has probability α. A Type II error fails to reject a false H₀ (a missed effect) and has probability β. The power of a test, 1 − β, is the probability of detecting a real effect of a given size. For a fixed α, power increases with the sample size and with the size of the effect, and decreases with the noise σ.

Which error matters more depends on the consequences. Accepting weak steel (failing to detect a strength deficit) is far more serious than rejecting good steel, so acceptance plans are designed to have high power against dangerous deficits. Engineering judgment sets α and the sample size, not convention alone.

Confidence intervals and two-sided tests carry the same information: a two-sided test at α = 0.05 rejects H₀: μ = μ₀ exactly when μ₀ lies outside the 95% confidence interval. Intervals are usually more informative, because they show the size of the effect and its uncertainty, not just a yes or no.

Comparing two means

Independent samples

To compare two groups, such as two mix designs or two suppliers, use the two-sample t test. The Welch version does not assume equal variances and is the safe default:

Its degrees of freedom come from a formula that software computes.

Paired samples

When the two measurements come in natural pairs, such as the same specimen measured two ways, or the same site before and after a change, analyze the differences d within each pair with a one-sample t test on the d values. Pairing removes the specimen-to-specimen variation, which can be far larger than the effect of interest.

Running tests in Python

from scipy import stats

rebar = [60.75, 62.70, 60.30, 63.15, 61.50, 59.85, 62.25, 63.90]
r = stats.ttest_1samp(rebar, popmean=60, alternative="greater")
print(f"Rebar: t = {r.statistic:.2f}, p = {r.pvalue:.4f}")

conventional = [6.8, 7.4, 5.9, 8.1, 7.0, 6.5, 7.7, 6.2, 7.3, 6.9]
modified     = [5.6, 6.1, 5.2, 6.8, 5.9, 5.0, 6.4, 5.7, 6.0, 5.5]
r = stats.ttest_ind(conventional, modified, equal_var=False)
print(f"Asphalt (Welch): t = {r.statistic:.2f}, p = {r.pvalue:.4f}")

lab1 = [6110, 5610, 6570, 5830, 5350, 6350, 5730, 6030]
lab2 = [6000, 5540, 6400, 5790, 5220, 6250, 5600, 5980]
r = stats.ttest_rel(lab1, lab2)
print(f"Labs (paired): t = {r.statistic:.2f}, p = {r.pvalue:.4f}")

Output: Rebar: t = 3.54, p = 0.0047; Asphalt (Welch): t = 4.24, p = 0.0005; Labs (paired): t = 6.37, p = 0.0004.

Other tests follow the same logic with different statistics: the chi-square test for counts in categories, the F test and analysis of variance (ANOVA) for comparing more than two means, and non-parametric tests when data are far from normal and samples are small.