⚖️Sample size for comparing two proportions

$$n = \frac{(z_{1-\alpha/2} + z_{1-\beta})^{2},\big[p_1(1-p_1) + p_2(1-p_2)\big]}{(p_1 - p_2)^{2}}$$

n is the number needed in each group.

• p₁: the proportion I expect in the control group (the baseline)
• p₂: the proportion in the treatment group if the smallest difference worth acting on is real
• z₁₋α/₂: 1.96 for a two-sided test at the 5 percent level
• z₁₋β: 0.84 for 80 percent power, 1.28 for 90 percent

With the usual 5 percent and 80 percent, the first bracket is (1.96 + 0.84)² ≈ 7.85, which is the only number in here worth memorising.

What it says in words: the sample grows with the square of how small a difference I want to see. Halve the difference, quadruple the test.

It is the standard normal approximation. It assumes independent observations, equal group sizes and one look at the data at the end. The material is in 🎓 Lecture 10 - Power, sample size and stopping early, which arrived six weeks after I needed it.

The arithmetic lives in my A-B test sample size sheet and, since the middle of April, in the Sample size calculator script, where the assumptions sit at the top and not in cell F14.

Worked example

From 📌 Design the triage A-B test, 2 April. For sizing I turned the outcome into a yes or no: was the exception triaged within thirty minutes.

• Baseline on the sheet: p₁ = 0.60
• Smallest difference worth acting on: ten points, so p₂ = 0.70
• Variance term: 0.60 × 0.40 + 0.70 × 0.30 = 0.24 + 0.21 = 0.45
• Difference squared: 0.10² = 0.01
• n = 7.85 × 0.45 / 0.01 ≈ 353.2, rounded up to 354 per group

Had I asked for five points (p₂ = 0.65), the same sum gives 7.85 × 0.4675 / 0.0025 ≈ 1,468 per group. Four times the test for half the difference, as promised.