Polson–Sokolov Lab · Sampling distributions

Sampling Distribution Lab

Pick a population, draw a sample of n values, and drop its average into the histogram below. Repeat a thousand times and the averages pile up around the population mean with spread σ/√n, close to a normal curve even when the population is far from normal.

← All labs

Based on Chapter 5, A/B Testing, of Polson and Sokolov, Bayes, Artificial Intelligence, and Deep Learning (vsokolov.org/html/_book).

Population One draw Sample mean x̄ Observed sample means N(μ, σ²/n), predicted
Newton at the Royal Mint

The Trial of the Pyx: a box of coins varies like σ√n, not nσ

The Mint tests a batch of n = 2,500 coins. Each should weigh 100 grams, with a tolerance of ±1 gram per coin. The Trial allowed the batch average the same ±1 gram, which means the whole box could be off by n × 1 = 2,500 grams. If each coin varies with SD σ = 1 gram, the box total varies with SD σ√n = 50 grams, so an honest box never comes close to that limit. A Master of the Mint can shave gold off every coin and still pass. These are illustrative numbers.

Mint rule: n × 1 g √n rule Box total, shaved coins A weighed box
0.50 g
1.00 g
2,500
√n rule: average within ± k · σ/√n
Mint rule√n rule
Allowed error on the total––
Allowed error on the average––
P(honest box passes)––
P(shaved box passes)––
Largest δ that passes 95% of the time––
Weighed boxes that passed––

Averages vary less than individuals

Each draw has variance σ². The mean of n independent draws has variance σ²/n, so its SD, the standard error, is σ/√n. To halve the standard error you need four times the data.

Var(x̄) = σ² / n, SE(x̄) = σ / √n
Var(total) = n σ², SD(total) = σ √n

The sampling distribution of x̄

Across many datasets of size n, x̄ is centered on μ and is close to normal once n is moderate. Skewed populations need a larger n: the skewness of x̄ shrinks like 1/√n. In practice σ is unknown and s replaces it.

x̄ ~ N(μ, σ² / n) approximately

Recipe 5: simulate it yourself

A classic exercise asks for 10,000 simulated A/B tests with 5,000 users per arm and 3% conversion. One arm's conversion rate is the mean of 5,000 Bernoulli(0.03) draws. Its histogram is the sampling distribution that the Z-test relies on.

SE(p̂) = √(0.03 × 0.97 / 5,000) = 0.00241

Try this in class

  1. Start with the exponential population and n = 1. The means histogram copies the population. Step n up to 2, 5, 10 and 30 and press "1,000 now" each time. Ask when the pile starts to look normal.
  2. Read the SD row of the table at n = 25 and n = 100. The standard error halves when n goes up four times, just as σ/√n predicts.
  3. Switch to "Bimodal" with n = 2. The means have three humps. By n = 10 they have one.
  4. Choose Bernoulli with p = 0.03 and n = 30. The means are badly skewed and lumpy. Run Recipe 5 (n = 5,000) and the same population gives a clean bell around 0.03.
  5. In the Pyx panel, ask students how much gold per coin a Master could shave and still pass the Mint's rule. Then switch on the √n rule and weigh 20 boxes.