Polson–Sokolov Lab · Sampling distributions

Sampling Distribution Lab

Pick a population, draw a sample of n values, and drop its average into the histogram below. Repeat a thousand times and the averages pile up around the population mean with spread \(\sigma/\sqrt{n}\), close to a normal curve even when the population is far from normal.

← All labs

Based on Chapter 5, A/B Testing, of Polson and Sokolov, Bayes, Artificial Intelligence, and Deep Learning (vsokolov.org/html/_book).

Population One draw Sample mean x̄ Observed sample means N(μ, σ²/n), predicted
Newton at the Royal Mint

The Trial of the Pyx: a box of coins varies like \(\sigma\sqrt{n}\), not \(n\sigma\)

The Mint tests a batch of n = 2,500 coins. Each should weigh 100 grams, with a tolerance of ±1 gram per coin. The Trial allowed the batch average the same ±1 gram, which means the whole box could be off by \(n \times 1 = 2{,}500\) grams. If each coin varies with SD \(\sigma = 1\) gram, the box total varies with SD \(\sigma\sqrt{n} = 50\) grams, so an honest box never comes close to that limit. A Master of the Mint can shave gold off every coin and still pass. These are illustrative numbers.

Mint rule: n × 1 g √n rule Box total, shaved coins A weighed box
0.50 g
1.00 g
2,500
√n rule: average within ± k · σ/√n
Mint rule√n rule
Allowed error on the total––
Allowed error on the average––
P(honest box passes)––
P(shaved box passes)––
Largest δ that passes 95% of the time––
Weighed boxes that passed––

Averages vary less than individuals

Each draw has variance \(\sigma^2\). The mean of n independent draws has variance \(\sigma^2/n\), so its SD, the standard error, is \(\sigma/\sqrt{n}\). To halve the standard error you need four times the data.

\[ \mathrm{Var}(\bar x) = \sigma^2 / n, \qquad \text{SE}(\bar x) = \sigma / \sqrt{n} \]
\[ \mathrm{Var}(\text{total}) = n \sigma^2, \qquad \text{SD}(\text{total}) = \sigma \sqrt{n} \]

The sampling distribution of \(\bar x\)

Across many datasets of size n, \(\bar x\) is centered on \(\mu\) and is close to normal once n is moderate. Skewed populations need a larger n: the skewness of \(\bar x\) shrinks like \(1/\sqrt{n}\). In practice \(\sigma\) is unknown and \(s\) replaces it.

\[ \bar x \sim N(\mu, \sigma^2 / n) \quad \text{approximately} \]

Recipe 5: simulate it yourself

A classic exercise asks for 10,000 simulated A/B tests with 5,000 users per arm and 3% conversion. One arm's conversion rate is the mean of 5,000 \(\text{Bernoulli}(0.03)\) draws. Its histogram is the sampling distribution that the Z-test relies on.

\[ \text{SE}(\hat p) = \sqrt{0.03 \times 0.97 / 5{,}000} = 0.00241 \]

Try this in class

  1. Start with the exponential population and n = 1. The means histogram copies the population. Step n up to 2, 5, 10 and 30 and press "1,000 now" each time. Ask when the pile starts to look normal.
  2. Read the SD row of the table at n = 25 and n = 100. The standard error halves when n goes up four times, just as \(\sigma/\sqrt{n}\) predicts.
  3. Switch to "Bimodal" with n = 2. The means have three humps. By n = 10 they have one.
  4. Choose Bernoulli with p = 0.03 and n = 30. The means are badly skewed and lumpy. Run Recipe 5 (n = 5,000) and the same population gives a clean bell around 0.03.
  5. In the Pyx panel, ask students how much gold per coin a Master could shave and still pass the Mint's rule. Then switch on the \(\sqrt{n}\) rule and weigh 20 boxes.