Polson–Sokolov Lab · Bayes' rule and base rates

Base-Rate Lab

One thousand people take a test. Red dots are sick and test positive, amber dots are healthy but test positive. Move the prevalence, sensitivity and specificity sliders and watch how many of the positives are real.

← All labs

Based on Chapter 2, Bayes' Rule, of Polson and Sokolov, Bayes, Artificial Intelligence, and Deep Learning (vsokolov.org/html/_book).

True positive (0) False positive (0) False negative (0) True negative (0)
Bayes' rule with the current numbers
Confusion matrix
Strong prior vs repeated evidence

The two-headed coin

P(two-headed | k heads in a row) Your coin after k flips
1,024

The jar also holds one two-headed coin. You draw one coin at random and flip it. Every flip so far came up heads.

Heads in a row, k0
Prior odds, two-headed : fair–
Likelihood ratio, 2k : 1–
Posterior odds–
P(two-headed | k heads)–

Each head is a "positive test" with sensitivity 1 (a two-headed coin always lands heads) and a false-positive rate of 1/2 (a fair coin lands heads half the time). The posterior reaches 50/50 when the evidence 2k matches the prior odds against, 1,024 : 1 at k = 10.

Bayes' rule with natural frequencies

Most positives come from the healthy majority

A test result is evidence, not a verdict. The posterior depends on three numbers, and the base rate is the one people forget. When a condition is rare, even a small false-positive rate applied to the large healthy group produces more false alarms than the test finds real cases.

  1. Start with 1,000 people

    With prevalence 1%, 10 women like Alice have cancer and 990 do not. This split is the prior, before anyone is tested.

  2. Apply the test to each group

    Sensitivity 80% finds 8 of the 10 cancers. A false-positive rate of 10% flags 99 of the 990 healthy women. That gives 107 positive mammograms.

  3. Condition on the positive result

    Keep only the 107 positives. Of these, 8 have cancer, so P(cancer | positive) = 8 / 107 ≈ 7.5%. A positive mammogram still leaves Alice about 92% likely to be healthy.

  4. Test again

    A second positive starts from 7.5% instead of 1% and moves the posterior to 39%. A third moves it to 84%. Repeated evidence can overcome a low prior, as long as the tests make independent errors.

Bayes' rule for a positive test
P(D | +) = P(+ | D)·P(D) / P(+)
P(+) = P(+ | D)·P(D) + P(+ | no D)·P(no D)
Confusion-matrix rates
sensitivity = P(+ | D) = 1 − β
specificity = P(− | no D) = 1 − α
Mammogram
0.80 × 0.01 / (0.80 × 0.01 + 0.10 × 0.99)
= 0.008 / 0.107 ≈ 0.075
Test marketing
0.70 × 0.08 / (0.70 × 0.08 + 0.15 × 0.92)
= 0.056 / 0.194 = 0.288

Takeaway for students. "98% accurate" describes P(test | truth). The decision needs P(truth | test). Bayes' rule converts one into the other, and the conversion runs through the base rate. Without the prior you cannot interpret the test.

Try this in class

  1. Load the mammogram case and ask students to guess P(cancer | positive) before you show the panel. Most guesses land near 80%. Then count the red and amber dots.
  2. Hold sensitivity and specificity fixed and drag prevalence from 1% to 30%. The amber cloud stays almost the same size while the red group grows, and the posterior climbs past 75%.
  3. Load the Apple Watch case. Raise specificity from 99% to 99.9% and watch the posterior jump from 67% to about 95%. With a rare condition, specificity matters more than sensitivity.
  4. Load the mammogram case and press "Positive again" twice. Ask why a second opinion from the same machine might not count as independent evidence.
  5. In the coin panel, set the jar to 1,048,576 fair coins and ask how many heads it takes to reach 50/50. Each extra head doubles the odds, so the answer is k = 20.