AII 600 midterm study guide

Explanations, worked examples, and practice. Fall 2026

The exam. Wednesday 14 October 2026, in class. 2 hours 30 minutes, pen and paper. One double-sided cheat sheet, printed or handwritten. No calculator, laptop, phone, watch, or AI assistant: the numbers on the exam are chosen so that the arithmetic is short, and fractions are fine as answers. The normal problems use one fact, which the exam restates where it is needed: a normal random variable is within 1.96 standard deviations of its mean with probability 0.95. The logistic regression problem gives a short table of values of the sigmoid and its logarithm.

What it covers. The Foundations deck, the Estimation and regression deck through the log transformation of the PGA earnings, and logistic regression from the 7 October class through the log loss. The exam concentrates on the core of that material: probability, Bayes rule, expectation and variance, the binomial and normal distributions, linear regression, and logistic regression. Estimation ideas from class appear where they support the core: the CDF and the link between least squares and the normal likelihood. The guide also reviews the standard deviation of an average and the central limit theorem, which explain why the normal model shows up so often.

How the exam is graded. As in class, a correct number is not enough. Most problems have three kinds of parts: compute a number, say what it means in the units of the problem, and say what would change the answer or the decision if an assumption changed. This guide practices all three.

The format. Seven problems.

A joint probability table: marginals, conditional distributions, conditional means, independence, and an expected value built from the table.
Bayes rule for a test: two people with different priors, the odds form, the prosecutor’s fallacy, and a second test.
A binomial random variable: the conditions, its distribution and CDF, its mean and variance, and an expected value built from it.
A normal model used to make a decision, and what would change the decision.
Linear regression: a least squares fit by hand, reading a fitted model, residual plots, and the normal model behind least squares.
Logistic regression: from a score to a probability, what a coefficient means, and the log loss.
True or false, each with a one-sentence justification.

Expectation and variance (Section 4) has no problem of its own: it runs through Problems 1, 3, and 7.

How to use this guide. Each section has the idea in plain words, a recipe, worked examples, and practice problems. Copy each worked example out by hand; reading it is not the same as doing it. Work each practice problem on paper before you open its solution. The exam allows about 1.5 minutes per point, roughly 20 minutes per problem, so time yourself on the practice problems too. Some examples below keep the book’s numbers, which need a calculator; for those, write the setup by hand and use a calculator only for the last step. The exam’s numbers are built for hand arithmetic, and the section Arithmetic without a calculator near the end shows the shortcuts.

Labs and the practice midterm. Each section has a See it move task in one of the interactive labs. Predict the answer on paper first, then use the lab to check it: a wrong prediction tells you which idea to reread. When you have worked through the guide, sit the practice midterm under exam conditions. Its seven problems mirror the exam’s, with the same kinds of parts and the same point values, and each one points back to the section of this guide that teaches it. For more problems on one topic, the problem bank has more from the textbook and the labs, with solutions, all on midterm material.

Section	You should be able to	Exam problem
1. Random variables and distributions	write a distribution and its CDF; use the complement and independence	3, 4, 7
2. Joint, marginal, and conditional distributions	read a joint table; condition on the right margin; test independence; compute a conditional mean	1
3. Bayes rule	use total probability; compute a posterior three ways; use odds and Bayes factors; spot the prosecutor’s fallacy	2, 7
4. Expectation and variance	compute means and variances; use covariance and correlation; combine two random variables	1, 3, 7
5. The binomial distribution	check the binomial conditions; compute probabilities, the mean, and the variance	3, 4
6. The normal distribution	standardize; compare $z$-values; use the 1.96 rule for ranges, tails, and guarantees; approximate sums; decide from tail probabilities	4, 5, 7
7. Least squares	fit a line by hand; interpret slope and intercept; predict; see why one point can move the line	5
8. Checking a regression	read residual plots; use the residual standard error; explain the normal model behind least squares	5
9. Logistic regression	turn a score into a probability; read a coefficient as a change in log-odds; compute and explain the log loss	6
10. Concept check	answer true or false with a reason	7

The book is Bayes, Artificial Intelligence, and Deep Learning. Most practice problems below come from its exercises in Chapter 1, Chapter 2, Chapter 12, and Chapter 13. Skip the Computing exercises: the exam is pen and paper.

Part I. Probability

1. Random variables and distributions

The idea

A random variable is a number whose value you do not know yet. Toss two fair coins. The outcomes TT, TH, HT, HH are equally likely. Let $X$ be the number of heads. $X$ is not the coins: it is a rule that turns each outcome into a number, here 0, 1, 1, 2.

The distribution of $X$ lists every value $X$ can take and its probability. Each probability is between 0 and 1, and together they sum to 1. For a discrete random variable this list is the probability mass function, $P(X=x)$.

The cumulative distribution function (CDF) accumulates the distribution from the left: \[F(x)=P(X\le x).\] It never decreases, starts at 0, and ends at 1. For a discrete $X$ it jumps at each possible value, and the size of the jump is $P(X=x)$. The CDF answers interval questions: \[P(X>a)=1-F(a), \qquad P(a<X\le b)=F(b)-F(a).\]

Four rules from class do most of the work.

Complement: $P(\text{not } A)=1-P(A)$. “At least one” is the complement of “none”.
Addition: $P(A \text{ or } B)=P(A)+P(B)-P(A \text{ and } B)$. The last term disappears only when $A$ and $B$ cannot happen together.
Multiplication for independent events: $P(A \text{ and } B)=P(A)P(B)$. This needs independence, and you have to say so.
Conditional probability: $P(A\mid B)=P(A \text{ and } B)/P(B)$. Section 2 is about this one.

Recipe

Name the random variable in words and list its possible values: “$X$ is the number of heads in two tosses, $X\in\{0,1,2\}$.”
List the outcomes that give each value and add their probabilities.
Check that the probabilities sum to 1.
For “at least one”, compute $1-P(\text{none})$.
For “all of these happen”, multiply only if the events are independent, and write down that assumption.

Worked example 1.1: two coins

Value $x$	0	1	2
Outcomes	TT	TH, HT	HH
$P(X=x)$	1/4	1/2	1/4
$F(x)=P(X\le x)$	1/4	3/4	1

$P(X\ge 1)=1-P(X=0)=3/4$. From the CDF, $P(X=2)=F(2)-F(1)=1/4$ and $P(X>1)=1-F(1)=1/4$. The two pictures below carry the same information: the bars are the jumps of the steps.

Two coins. Left: the distribution of X. Right: its CDF, which jumps by P(X = x) at each value.

Worked example 1.2: Pete Rose’s 44-game streak

Pete Rose hit safely in 44 consecutive games. Treat him as a 0.300 hitter with 4 independent at-bats per game. How rare is the streak?

One game. “At least one hit in four at-bats” is the complement of “four outs”: $P(\text{hit in a game})=1-0.7^4=1-0.2401=0.7599$.
44 games. If games are independent, multiply: $0.7599^{44}=5.67\times 10^{-6}$, about 1 in 176,000.

What it means: for one player of Rose’s quality starting a streak today, a 44-game run is very unlikely. What would change it: the answer assumes every at-bat is independent with the same 0.300. If hitting comes in hot and cold spells, long streaks are more likely than this. And with many players over many seasons, somebody will eventually do it: rare for one player is not rare for the league.

See it move

In the Streaks Lab, choose Hitting streaks. The defaults are Pete Rose’s: a .300 hitter, 4 at-bats a game, and a 44-game streak to beat. Predict first: Worked example 1.2 gives about 1 in 176,000 for one particular run of 44 games. A 162-game season has many games where a streak could start. Is the chance of a 44-game streak somewhere in a season larger or smaller than 1 in 176,000? Then read the exact answer in the lab, and raise the number of hitters to see why a streak this rare still happens to somebody.

Your turn

Problem 1.1 (delivery times, book Chapter 1). Deliveries take between 1 and 7 days. The CDF of the delivery time $X$ is:

$x$	1	2	3	4	5	6	7
$P(X\le x)$	0.10	0.20	0.70	0.75	0.80	0.90	1

Find $P(X=x)$ for each day.
A package has not arrived in the first three days. What is the probability that it arrives on day 4?
Find $P(2<X\le 5)$.
By which day have at least 90% of packages arrived?

Solution

The jumps of the CDF: $P(X=x)=0.10,\ 0.10,\ 0.50,\ 0.05,\ 0.05,\ 0.10,\ 0.10$. They sum to 1.
$P(X=4\mid X\ge 4)=\dfrac{P(X=4)}{P(X\ge 4)}=\dfrac{0.05}{1-0.70}=\dfrac{0.05}{0.30}=\dfrac{1}{6}$. Knowing it is late changes the denominator from 1 to 0.30.
$F(5)-F(2)=0.80-0.20=0.60$: days 3, 4, and 5.
Day 6, the first day with $F(x)\ge 0.90$. This is the 0.9 quantile.

Problem 1.2 (premium channels, book Chapter 1). A cable company has 10,000 subscribers. 2,750 receive HBO, 2,050 receive Showtime, and 6,200 receive neither.

What is the probability that a random subscriber receives both channels?
What is the probability that a random subscriber receives HBO but not Showtime?

Solution

$P(\text{HBO or Showtime})=1-0.62=0.38$. The addition rule gives $P(\text{both})=0.275+0.205-0.38=0.10$. The 1,000 subscribers with both were counted twice in $2{,}750+2{,}050$.
$P(\text{HBO})-P(\text{both})=0.275-0.10=0.175$.

Problem 1.3 (a streak for a classifier). An image classifier labels each image correctly with probability 0.98, independently of other images.

What is the probability that it labels 100 images in a row correctly?
What is the probability of at least one error in 100 images?
Errors in practice come in clusters: blurry batches, a new camera. With the same 2% error rate per image, would clustering make your answer to (a) larger or smaller?

Solution

$0.98^{100}=0.133$.
$1-0.98^{100}=0.867$. A 2% error rate per image makes at least one error in 100 the normal case.
Larger. When errors cluster, they arrive together in a few batches and the other batches are clean, so long error-free runs become more common than independence predicts. This is the same reasoning as the hot and cold spells in Worked example 1.2.

Traps

“At least one” is $1-P(\text{none})$, not the sum of the individual probabilities. The sum can exceed 1.
Multiplying probabilities needs independence. Write the assumption down.
$P(X\le 2)$ includes the value 2. $P(2<X\le 4)=F(4)-F(2)$ excludes it.

2. Joint, marginal, and conditional distributions

The idea

A joint table gives $P(X=x, Y=y)$ for every pair of values. The marginal distribution of $X$ is the row total, $P(X=x)=\sum_y P(X=x,Y=y)$. The marginal of $Y$ is the column total.

A conditional distribution keeps one row (or one column) and rescales it so that it sums to 1: \[P(Y=y\mid X=x)=\frac{P(X=x, Y=y)}{P(X=x)}.\] The denominator is the total of the row or column you condition on. The same cell divided by two different totals gives two different answers, which is why $P(x\mid y)$ and $P(y\mid x)$ are different numbers. In class: nearly everyone who plays in the NBA practiced hard, but almost nobody who practices hard plays in the NBA.

Rearranging gives the multiplication rule, $P(x,y)=P(x)P(y\mid x)=P(y)P(x\mid y)$.

$X$ and $Y$ are independent when $P(X=x,Y=y)=P(X=x)P(Y=y)$ for every cell, which is the same as $P(Y=y\mid X=x)=P(Y=y)$: knowing $X$ tells you nothing about $Y$. One cell that fails the product test proves dependence. For two yes/no variables (a 2 by 2 table), one cell that passes is enough, because the margins then force the other three cells. In a larger table, one cell that passes proves nothing: check every cell.

The conditional mean is the mean of a conditional distribution: \[E(X\mid Y=y)=\sum_x x\,P(X=x\mid Y=y).\] Averaging the conditional means with the weights $P(Y=y)$ gives back the overall mean. This is the law of total expectation, “averages within groups, then across groups”: \[E(X)=\sum_y E(X\mid Y=y)\,P(Y=y).\] In class: a quarter of houses are small with average price 2, three quarters are large with average price 6, so the overall average is $0.25(2)+0.75(6)=5$.

Expectation is linear: $E(X_1+\dots+X_n)=E(X_1)+\dots+E(X_n)$ for any random variables, dependent or not.

Recipe

Check the table: every entry between 0 and 1, total equal to 1. Add a total row and a total column.
Translate the question into probability notation with a bar: what is known goes to the right of the bar.
The denominator is the probability of what is known: the total of that row or column, or the sum of several of them.
For a conditional mean, write out the conditional distribution first, then average.
For independence, compare one cell with the product of its margins. If they differ, stop: dependent.

Worked example 2.1: drinks and TV (book Chapter 1)

The table relates the number of TV shows $Y$ a student watches in an evening to the number of drinks $X$.

	$Y=0$	$Y=1$	$Y=2$	$Y=3$	Total
$X=0$	0.07	0.09	0.06	0.01	0.23
$X=1$	0.07	0.06	0.07	0.01	0.21
$X=2$	0.06	0.07	0.14	0.03	0.30
$X=3$	0.02	0.04	0.16	0.04	0.26
Total	0.22	0.26	0.43	0.09	1.00

More than two drinks. $P(X>2)=P(X=3)=0.26$, the last row total.

More drinks than shows. Add the cells below the diagonal, where $x>y$: $0.07+0.06+0.07+0.02+0.04+0.16=0.42$.

Shows given three drinks. Keep the $X=3$ row and divide by its total 0.26: $P(Y=y\mid X=3)=0.02/0.26,\ 0.04/0.26,\ 0.16/0.26,\ 0.04/0.26$, which is $1/13,\ 2/13,\ 8/13,\ 2/13$. It sums to 1.

Drinks given no TV. Keep the $Y=0$ column and divide by 0.22: $P(X=x\mid Y=0)=7/22,\ 7/22,\ 6/22,\ 2/22$. Then \[E(X\mid Y=0)=\frac{0(7)+1(7)+2(6)+3(2)}{22}=\frac{25}{22}=1.14.\] Compare with $E(X)=0(0.23)+1(0.21)+2(0.30)+3(0.26)=1.59$: students who watch no TV drink less, on average.

Total expectation check. The four conditional means are $E(X\mid Y=y)=1.14,\ 1.23,\ 1.93,\ 2.11$. Weighted by the column totals: $1.14(0.22)+1.23(0.26)+1.93(0.43)+2.11(0.09)=1.59=E(X)$.

Independence. $P(X=2,Y=2)=0.14$, but $P(X=2)P(Y=2)=0.30\times 0.43=0.129$. Not equal, so drinking and TV are not independent. Note what this does not say: the table shows that the two go together, not that one causes the other.

Same cell, two denominators. In the class happiness example, 14% of clients were both top-salary and high-happiness. Among top-salary clients the share that is high-happiness was $0.14/0.24=0.58$; among high-happiness clients the share that is top-salary was $0.14/0.46=0.30$. Same cell, different questions. Always write which group you are inside of.

See it move

In the Joint Table Lab, load Happiness × Salary: the class table behind the last paragraph of Worked example 2.1, with salary in the rows and happiness in the columns. Predict first: the share of the top-salary row that is in the high-happiness column, and the share of the high-happiness column that is in the top-salary row. Then click the row heading and the column heading to condition on each, and check your two numbers (0.58 and 0.30). Last, press Make independent: the lab replaces every cell with its row total times its column total. Which cells change the most?

Your turn

Problem 2.1 (Cooper Realty, book Chapter 1). A real estate company sold 800 homes. Counts by asking price and days on the market:

Asking price	Under 30 days	31 to 90 days	Over 90 days	Total
Under $50K	50	40	10	100
$50K to $100K	20	150	80	250
$100K to $150K	20	280	100	400
Over $150K	10	30	10	50

Estimate the probability that a home is on the market for over 90 days, and the probability that its asking price is under $50K.
Estimate the probability of both. Are the two events independent?
A contract has just been signed for a home listed under $100K. What is the probability that it takes over 90 days to sell?
An agent says, “cheap homes sit on the market.” Compare $P(\text{over 90 days}\mid \text{under \$50K})$ with $P(\text{over 90 days})$, and compare it with $P(\text{under \$50K}\mid \text{over 90 days})$. Which number answers the agent?

Solution

$P(\text{over 90})=(10+80+100+10)/800=200/800=0.25$. $P(\text{under \$50K})=100/800=0.125$.
$P(\text{both})=10/800=0.0125$. Under independence it would be $0.25\times 0.125=0.03125$. Not equal, so not independent.
Condition on the first two rows, 350 homes: $P(\text{over 90}\mid \text{under \$100K})=(10+80)/350=0.257$.
$P(\text{over 90}\mid\text{under \$50K})=10/100=0.10$, well below the overall 0.25: cheap homes sell faster, so the agent is wrong. $P(\text{under \$50K}\mid\text{over 90})=10/200=0.05$ answers a different question: among slow sales, how many were cheap. The agent’s claim is about the first one.

Problem 2.2 (store brands, book Chapter 1). A supermarket survey gives this joint probability table for how often a customer visits and how often they buy generic products:

Visit	Often	Sometimes	Never
Frequent	0.10	0.50	0.17
Infrequent	0.03	0.05	0.15

What is the probability that a customer never buys generics? Often buys them?
Compute $P(\text{Frequent}\mid\text{Often})$ and compare it with $P(\text{Frequent})$. Does this suggest independence? Now check the other two columns. Are visiting and buying generics independent?
Find the conditional distribution of generic buying for frequent visitors.

Solution

$P(\text{Never})=0.17+0.15=0.32$. $P(\text{Often})=0.10+0.03=0.13$.
$P(\text{Frequent})=0.10+0.50+0.17=0.77$ and $P(\text{Frequent}\mid\text{Often})=0.10/0.13=0.769$. Almost equal, so this column alone looks independent. But $P(\text{Frequent}\mid\text{Sometimes})=0.50/0.55=0.909$ and $P(\text{Frequent}\mid\text{Never})=0.17/0.32=0.531$. They are not independent. This is why one matching cell proves nothing.
Divide the Frequent row by 0.77: Often $0.130$, Sometimes $0.649$, Never $0.221$.

Problem 2.3 (financial news, book Chapter 1). 28% of adults watch a financial news program, 17% read a financial publication, and 13% do both.

Complete the joint table (reads or not, watches or not), with totals.
What is the probability that someone who watches reads? That someone who reads watches?
Why are the two answers in (b) different?

Solution

Fill in the missing cells by subtraction: reads but doesn’t watch is $0.17-0.13$, watches but doesn’t read is $0.28-0.13$, and neither is $1-0.13-0.04-0.15$.

Watches Doesn’t watch Total

Reads 0.13 0.04 0.17

Doesn’t read 0.15 0.68 0.83

Total 0.28 0.72 1.00
$P(\text{reads}\mid\text{watches})=0.13/0.28=0.464$. $P(\text{watches}\mid\text{reads})=0.13/0.17=0.765$.
Same cell, different denominators. Readers are a smaller group than watchers, so the overlap is a larger share of the readers.

	Watches	Doesn’t watch	Total
Reads	0.13	0.04	0.17
Doesn’t read	0.15	0.68	0.83
Total	0.28	0.72	1.00

Problem 2.4 (ride-share fares). 70% of a driver’s rides are in the city, with mean fare $18. The other 30% are airport rides, with mean fare $45.

What is the mean fare of a ride?
The airport share rises to 40%, and the two group means stay the same. What is the new mean fare?
The driver keeps 75% of each fare and gives 30 rides a week. What are the expected weekly earnings, using the fares in (a)? Did you need to assume that the fares of different rides are independent?

Solution

Law of total expectation: $0.70(18)+0.30(45)=12.60+13.50=\$26.10$.
$0.60(18)+0.40(45)=10.80+18.00=\$28.80$. Neither group got more expensive; the mix changed.
$E(\text{earnings})=0.75\times 30\times 26.10=\$587.25$. No independence needed: the expected value of a sum is the sum of the expected values for any dependence. Independence would matter for the variance of weekly earnings.

Traps

Dividing by the wrong total. Ask “which group am I inside of?” and divide by that group’s probability.
Calling two variables independent because one cell matches. Outside a 2 by 2 table, every cell has to be checked.
Treating $P(A\mid B)$ as $P(B\mid A)$. The next section is about how different they can be.
Thinking you need independence to add expected values. You need it only for variances.

3. Bayes rule

The idea

You know how likely the evidence is under each explanation, $P(E\mid H)$, and you want how likely each explanation is given the evidence, $P(H\mid E)$. Bayes rule turns one into the other: \[P(H\mid E)=\frac{P(E\mid H)\,P(H)}{P(E)}.\] The denominator comes from the law of total probability: add up every way the evidence can happen. \[P(E)=P(E\mid H)P(H)+P(E\mid \bar H)P(\bar H).\] With more than two explanations, the sum has one term per explanation.

Three ways to compute a posterior. Use whichever you trust most under time pressure, and use a second one to check.

Counts. Imagine 1,000 people (or 100,000 if the base rate is tiny). Split them by the prior, then by the test. The posterior is “true positives over all positives”.
Formula. Bayes rule with the total-probability denominator.
Odds. Odds are $O(H)=P(H)/(1-P(H))$, and $P(H)=O(H)/(1+O(H))$. Then \[\underbrace{\frac{P(H\mid E)}{P(\bar H\mid E)}}_{\text{posterior odds}}=\underbrace{\frac{P(E\mid H)}{P(E\mid \bar H)}}_{\text{Bayes factor}}\times\underbrace{\frac{P(H)}{P(\bar H)}}_{\text{prior odds}}.\] The Bayes factor is the same for everyone who agrees about the test. People who disagree about the prior disagree about the posterior. The odds form also makes repeated evidence easy: if two pieces of evidence are independent given $H$ and given $\bar H$, multiply both Bayes factors into the prior odds.

Always say odds of what. Bookmakers quote odds against an event, which is the class Oddschecker slide’s $O(A)=(1-P(A))/P(A)$: “2 to 1” at a bookmaker means $P(A)=1/3$. In the island problem, “odds of innocence 1000 to 1” means $P(I)/P(G)=1000$. Odds in favor of $A$ of 2 to 7 mean $P(A)=2/(2+7)=2/9$.

Test vocabulary. For a disease $D$ and a positive test $T$: sensitivity is $P(T\mid D)$, specificity is $P(\bar T\mid \bar D)$, and the false positive rate is $1-\text{specificity}=P(T\mid\bar D)$. None of these is $P(D\mid T)$, the number a patient wants.

The prosecutor’s fallacy is treating $P(E\mid\text{innocent})$ as if it were $P(\text{innocent}\mid E)$, so that $1-P(E\mid\text{innocent})$ gets quoted as the probability of guilt. The same mistake runs the other way when $P(E\mid\text{guilty})$ is quoted as $P(\text{guilty}\mid E)$. The fix is always the same: bring in the base rate.

Recipe

Name the events: $D$ for the condition, $T$ for the evidence. Write every given number as a probability with a bar. Translate “detection rate” to $P(T\mid D)$ and “false positive rate” to $P(T\mid\bar D)$.
Write the prior $P(D)$. If two people disagree, it is the prior they disagree about.
Compute $P(T)$ by total probability.
Divide: $P(D\mid T)=P(T\mid D)P(D)/P(T)$.
Check with counts or odds. Then say what the number means and how it moves if the prior moves.

Worked example 3.1: Alice’s mammogram (class)

Alice is 40. For women like her, 1% have breast cancer. The mammogram detects 80% of cancers and has a 10% false positive rate. Her mammogram is positive. What is $P(\text{cancer}\mid\text{positive})$?

Counts. Of 1,000 women like Alice, 10 have cancer and 990 do not. Of the 10, 80% test positive: 8. Of the 990, 10% test positive: 99. There are $8+99=107$ positives, and 8 of them have cancer: $8/107=0.075$.

Formula. $P(T)=0.8(0.01)+0.1(0.99)=0.008+0.099=0.107$, so $P(D\mid T)=0.008/0.107=0.075$.

Odds. Prior odds $0.01/0.99=1/99$. Bayes factor $0.8/0.1=8$. Posterior odds $8/99$, so $P(D\mid T)=8/(99+8)=8/107=0.075$.

What it means: a positive mammogram raises Alice’s probability from 1% to 7.5%, but most positives (99 of 107) come from healthy women, because healthy women vastly outnumber sick ones. What would change it: a higher prior (a family history) or a more specific test. With a 1% false positive rate, the count of false positives drops from 99 to about 10 and the posterior rises to about 45%.

Worked example 3.2: an oil field with three possibilities (book Chapter 2)

Geologists assign $P(\text{high quality oil})=0.50$, $P(\text{medium})=0.20$, $P(\text{no oil})=0.30$. A soil test finds a particular soil type, with $P(\text{soil}\mid\text{high})=0.20$, $P(\text{soil}\mid\text{medium})=0.80$, $P(\text{soil}\mid\text{none})=0.20$.

Total probability, one term per explanation: $P(\text{soil})=0.50(0.20)+0.20(0.80)+0.30(0.20)=0.10+0.16+0.06=0.32$.

Posteriors: high $0.10/0.32=0.3125$, medium $0.16/0.32=0.50$, none $0.06/0.32=0.1875$. They sum to 1.

What it means: the soil type is four times as likely under medium oil as under the other two, so it moves probability toward medium and away from both high and none. The chance of some oil rises from 0.70 to 0.8125. A decision can flip on this: a firm that drills only when $P(\text{no oil})<0.20$ would not drill before the test (0.30) and would drill after it (0.1875).

A second test. Suppose a second soil sample, independent of the first given the oil type, shows the same soil. Use the posteriors as the new prior: $0.3125(0.2)+0.50(0.8)+0.1875(0.2)=0.0625+0.40+0.0375=0.50$, so the new posteriors are $0.125,\ 0.80,\ 0.075$. The independence assumption matters: if the second sample came from the same spot, it would mostly repeat the first one and should move you much less.

Worked example 3.3: the island and a DNA match (class)

A crime is committed on an island of 1,001 people. The criminal left a trait that occurs in 1 in 10,000 people, and a suspect has it. Before the evidence, every islander is equally likely: the prior odds that the suspect is innocent are $1000$ to 1. A match is certain if he is guilty and has probability $p=10^{-4}$ if he is innocent, so the Bayes factor for innocence is $p/1=10^{-4}$.

Posterior odds of innocence: $1000\times 10^{-4}=0.1$, so $P(\text{innocent}\mid\text{match})=0.1/1.1=0.091$ and $P(\text{guilty}\mid\text{match})=0.909$.

The prosecutor who says “the match probability is 1 in 10,000, so he is 99.99% likely to be guilty” reported $1-P(E\mid\text{innocent})$, not $P(\text{guilty}\mid E)$. The expected number of other islanders who also match is $1000\times 10^{-4}=0.1$, and that is exactly the posterior odds of innocence. On an island of 10,001 people the posterior odds would be 1 and the suspect a coin flip.

See it move

In the Base-Rate Lab, load the Mammogram case. Predict first: of the 1,000 dots, how many test positive, and how many of those have cancer? Count the red and amber dots to check. Press Positive again and predict the new posterior before you look: the posterior odds were $8/99$, and a second independent positive multiplies them by 8 again. Last, drag the prevalence slider and find the base rate at which one positive test gives a 50% posterior. (Answers: 107 positives, 8 with cancer; $64/163\approx 0.39$; about 11%, where the prior odds are $1/8$.)

Your turn

Problem 3.1 (light aircraft, book Chapter 2). 70% of light aircraft that disappear in flight are later found. Of the aircraft that are found, 60% have an emergency locator. Of the aircraft that are not found, 90% do not have a locator. An aircraft with a locator has disappeared. What is the probability that it will be found?

Solution

Let $F$ be “found” and $L$ “has a locator”. $P(F)=0.70$, $P(L\mid F)=0.60$, $P(L\mid\bar F)=1-0.90=0.10$. Then $P(L)=0.70(0.60)+0.30(0.10)=0.42+0.03=0.45$ and $P(F\mid L)=0.42/0.45=0.933$. Watch the translation: “90% of the aircraft not found do not have a locator” is $P(\bar L\mid\bar F)$, so you need its complement.

Problem 3.2 (blood pressure screening, book Chapter 2). A screening test for hypertension gives this joint probability table:

	Hypertension present	Absent
Test positive	0.09	0.03
Test negative	0.03	0.85

What is the probability that a random person has hypertension? That a random person tests positive?
Given a positive test, what is the probability of hypertension?
Your doctor already thought you had a 50% chance of hypertension before the test. Use the test’s sensitivity and false positive rate from the table to update that prior after a positive test.

Solution

$P(H)=0.09+0.03=0.12$. $P(+)=0.09+0.03=0.12$.
$P(H\mid +)=0.09/0.12=0.75$.
The test’s properties do not depend on your prior: $P(+\mid H)=0.09/0.12=0.75$ and $P(+\mid\bar H)=0.03/0.88=0.0341$. With $P(H)=0.5$: $P(+)=0.75(0.5)+0.0341(0.5)=0.392$ and $P(H\mid +)=0.375/0.392=0.957$. Odds check: the Bayes factor is $0.75/0.0341=22$, the prior odds are 1, so the posterior odds are 22 and the probability is $22/23=0.957$.

Problem 3.3 (random HIV testing, book Chapter 2). The ELISA test has sensitivity 0.993 and specificity 0.9999. In the general population, the probability that a random person has HIV is 0.000025.

Find the probability that a random person tests positive.
A random person tests positive. What is the probability that they have HIV?
The person is retested, and the second test, independent of the first given their status, is also positive. Update the probability using the odds form.
What problem does (b) create for a policy of testing everyone at random? What would change it?

Solution

$P(T)=0.993(0.000025)+0.0001(0.999975)=0.0000248+0.0001000=0.0001248$.
$P(H\mid T)=0.0000248/0.0001248=0.199$. About four of every five positives are false.
Bayes factor $0.993/0.0001=9{,}930$. Prior odds $0.000025/0.999975=0.000025$. Two independent positives: posterior odds $0.000025\times 9{,}930^2=2{,}465$, so the probability is $2{,}465/2{,}466=0.9996$.
Most people flagged by a single test are healthy, which brings anxiety, stigma, and follow-up costs. Two remedies follow from Bayes rule: test groups with a higher base rate, which raises the prior, or confirm every positive with a second independent test, which multiplies the Bayes factor.

Problem 3.4 (the jury, book Chapter 2). A jury reaches the correct verdict with probability 0.95, whether the defendant is innocent or guilty. 80% of defendants brought to trial are guilty.

Given an “innocent” verdict, what is the probability that the defendant is innocent?
Given a “guilty” verdict, what is the probability that the defendant is guilty?
Do the answers to (a) and (b) sum to one? Should they?

Solution

$P(\text{verdict innocent})=0.95(0.20)+0.05(0.80)=0.19+0.04=0.23$. $P(I\mid VI)=0.19/0.23=0.826$.
$P(G\mid VG)=0.95(0.80)/(1-0.23)=0.76/0.77=0.987$.
No, $0.826+0.987\ne 1$, and there is no reason they should: they condition on different events. What does sum to one is $P(I\mid VI)+P(G\mid VI)$. Even with a 95% accurate jury, about 1 in 6 “innocent” verdicts goes to a guilty defendant, because guilty defendants are the majority.

Problem 3.5 (a DNA match in a city). A DNA profile matches 1 in 10,000 people. The crime happened in a city with 500,000 adults, and before the DNA evidence any of them could have done it.

A suspect found by searching a database matches. Treat every adult as equally likely a priori. What are the prior odds of innocence, the Bayes factor, the posterior odds, and $P(\text{guilty}\mid\text{match})$?
Other evidence narrows the pool to 2,000 people before the DNA test. Recompute $P(\text{guilty}\mid\text{match})$.
A newspaper writes that “the chance an innocent person matches is 1 in 10,000, so the suspect is almost certainly guilty.” Name the error in one sentence.

Solution

Prior odds of innocence $499{,}999\approx 500{,}000$. Bayes factor $P(E\mid I)/P(E\mid G)=10^{-4}/1$. Posterior odds of innocence $500{,}000\times 10^{-4}=50$. $P(\text{guilty}\mid\text{match})=1/51=0.020$. About 50 other adults in the city match.
Posterior odds $2{,}000\times 10^{-4}=0.2$, so $P(\text{guilty}\mid\text{match})=1/1.2=0.833$. Same DNA evidence, very different conclusion: the prior does the rest.
It is the prosecutor’s fallacy: $P(\text{match}\mid\text{innocent})=10^{-4}$ is not $P(\text{innocent}\mid\text{match})$, which depends on how many people could have matched.

Traps

Reporting the sensitivity, or one minus the false positive rate, as the answer. The answer has the base rate in it.
Translating “90% of the not-found aircraft do not have a locator” into $P(L\mid\bar F)=0.90$. Read each sentence twice and write it with a bar.
Forgetting a term in $P(E)$ when there are three or more explanations.
Multiplying Bayes factors for repeated tests without saying the tests are independent given the truth.

4. Expectation and variance

The idea

The expected value is the probability-weighted average of the values: \[E(X)=\sum_x x\,P(X=x).\] It is the long-run average over many repetitions, not the value you expect to see next time. A $1 bet with expected value $-\$0.03$ never pays $-\$0.03$.

The variance measures spread around the mean, and the standard deviation is its square root, back in the units of $X$: \[Var(X)=E\big((X-E(X))^2\big)=E(X^2)-E(X)^2, \qquad sd(X)=\sqrt{Var(X)}.\]

Covariance measures whether two variables move together, and correlation rescales it to lie between $-1$ and $1$ with no units: \[Cov(X,Y)=E\big((X-E(X))(Y-E(Y))\big), \qquad Corr(X,Y)=\frac{Cov(X,Y)}{sd(X)\,sd(Y)}.\] Independent variables have zero covariance. The converse is false: if $X$ is $-1$, 0, or 1 with equal probability and $Y=X^2$, the covariance is zero but $Y$ is a function of $X$. Correlation sees only straight-line dependence.

Rules for combinations. For constants $a$ and $b$: \[E(aX+bY)=aE(X)+bE(Y), \qquad Var(aX+bY)=a^2Var(X)+b^2Var(Y)+2ab\,Cov(X,Y).\] Three consequences you will use:

Constants come out squared: $Var(3X)=9\,Var(X)$, and adding a constant does not change the variance.
For independent variables, variances add even when you subtract: $Var(X-Y)=Var(X)+Var(Y)$.
For $n$ independent copies with mean $\mu$ and standard deviation $\sigma$, the total has mean $n\mu$ and standard deviation $\sqrt{n}\,\sigma$, and the average has mean $\mu$ and standard deviation $\sigma/\sqrt{n}$.

Recipe

Write the distribution as a table of values and probabilities.
$E(X)$: multiply each value by its probability and add.
$Var(X)$: compute $E(X^2)$ the same way, then subtract $E(X)^2$. A variance can never be negative; if yours is, recheck.
For a portfolio or any combination $aX+bY$, write the formula first, then plug in. Do not forget the covariance term, and do not forget that the weights are squared.
Interpret: the mean is the long-run average; the standard deviation is the typical distance from it.

Worked example 4.1: two roulette bets (class)

European roulette has 37 pockets: 18 red, 18 black, and a zero. Compare two $1 bets.

Black pays even money: net $W=+1$ with probability $18/37$ and $-1$ with probability $19/37$. $E(W)=(18-19)/37=-0.027$. $E(W^2)=1$, so $Var(W)=1-0.027^2=0.999$ and $sd(W)=1.00$.
A single number pays 35 to 1: $W=+35$ with probability $1/37$ and $-1$ with probability $36/37$. $E(W)=(35-36)/37=-0.027$, the same. $E(W^2)=(1225+36)/37=34.08$, so $sd(W)=5.84$.

Same mean, very different risk. Both lose 2.7 cents per dollar in the long run. The single-number bet swings much more, so it ends an evening faster in either direction. If your goal is to stay at the table, bet on black.

In class we used the payout including the returned stake, $X=W+1$, so $E(X)=0.973$. The variances, 1 and 34, are the same as here, because adding a constant shifts a distribution without spreading it.

Worked example 4.2: Tortoise and Hare split the leads (class)

Tortoise sells 1 or 2 cars a day with equal probability; Hare sells 0 or 3. $E(T)=E(H)=1.5$, $Var(T)=0.25$, $Var(H)=2.25$. When one has a good day the other has a bad one: $Cov(T,H)=-0.75$. Split the leads, $P=\tfrac12 T+\tfrac12 H$.

$E(P)=\tfrac12(1.5)+\tfrac12(1.5)=1.5$.
$Var(P)=\tfrac14(0.25)+\tfrac14(2.25)+2\cdot\tfrac12\cdot\tfrac12(-0.75)=0.0625+0.5625-0.375=0.25$.

Without the covariance term you would get 0.625, worse than Tortoise alone. The negative covariance cancels Hare’s noise.

Worked example 4.3: a two-country portfolio (book Chapter 1)

Monthly returns: US mean 1.15%, sd 3.30%; Hong Kong mean 1.70%, sd 7.51%; correlation 0.3. Invest 75% in the US and 25% in Hong Kong.

$E(R_p)=0.75(0.0115)+0.25(0.0170)=0.012875$, about 1.29% a month.
$Var(R_p)=0.75^2(0.033)^2+0.25^2(0.0751)^2+2(0.75)(0.25)(0.3)(0.033)(0.0751)=0.000613+0.000353+0.000279=0.001244$, so $sd(R_p)=0.0353$, about 3.53%.

The mean is the weighted average of the means, but the standard deviation, 3.53%, is below the weighted average of the standard deviations, $0.75(3.30)+0.25(7.51)=4.35\%$. That gap is diversification. With correlation 1 there would be no gap: the sd would be exactly 4.35%. With correlation 0 the covariance term disappears and the sd falls to 3.11%.

See it move

In the Portfolio Lab, load Tortoise & Hare and keep the weight at 0.5. Predict first: $Var(P)$ with correlation 0, and with correlation $-1$. (Correlation $-1$ gives $Cov=-1\times 0.5\times 1.5=-0.75$, the covariance in Worked example 4.2.) Move the correlation slider to check both, then press Go to min variance and see what weight the lab puts on the Tortoise. (Answers: 0.625 and 0.25.)

Your turn

Problem 4.1 (two drilling strategies, book Chapter 1). An oil company estimates a 20% chance of a small amount of oil, 50% of a moderate amount, and 30% of a large amount. Payoffs in millions of dollars:

	Small	Moderate	Large
Standard drilling	20	30	40
Horizontal drilling	$-20$	40	80

Find the mean and variance of the payoff for each strategy.
Which strategy maximizes the expected payoff? Which would a risk-averse company choose, and why?
A geological survey would reveal the amount of oil before the company chooses. How much is that information worth?

Solution

Standard: $E=0.2(20)+0.5(30)+0.3(40)=31$; $E(X^2)=0.2(400)+0.5(900)+0.3(1600)=1010$; $Var=1010-31^2=49$, sd 7. Horizontal: $E=0.2(-20)+0.5(40)+0.3(80)=40$; $E(X^2)=0.2(400)+0.5(1600)+0.3(6400)=2800$; $Var=2800-1600=1200$, sd 34.6.
Horizontal has the higher mean, 40 against 31, but a far larger spread, including a loss in the small-oil case. A risk-averse company may give up 9 million in expected payoff for the much smaller variance of standard drilling.
With perfect information the company picks the better strategy in each state: 20, 40, 80, so the expected payoff is $0.2(20)+0.5(40)+0.3(80)=48$. The best it can do without the information is 40. The survey is worth $48-40=8$ million.

Problem 4.2 (diversifying with zero correlation, book Chapter 1). Investment $X$ has mean return 5% and standard deviation 5%. Investment $Y$ has mean 10% and standard deviation 6%. Their correlation is zero.

For the portfolio $0.5X+0.5Y$, compute the mean and the variance. Is there a portfolio with a higher mean and a lower variance than $X$?
Suppose the correlation is 0.9 instead. Recompute the variance of $0.5X+0.5Y$. Does the portfolio still beat $X$ on both counts?

Solution

Mean $0.5(5\%)+0.5(10\%)=7.5\%$. Variance $0.25(0.05^2)+0.25(0.06^2)=0.000625+0.0009=0.001525$, below $Var(X)=0.0025$. Yes: this portfolio has a higher mean and a lower variance than $X$.
Add the covariance term $2(0.5)(0.5)(0.9)(0.05)(0.06)=0.00135$: the variance is $0.002875$, now above $0.0025$. The portfolio still has the higher mean but no longer the lower variance, so the choice depends on how much risk you will take for the extra return. How much diversification helps depends on the correlation: the lower it is, the more of the noise cancels. Even at 0.9 the portfolio’s standard deviation, 5.36%, is below the weighted average of the two, 5.5%, but the gain is too small to beat $X$.

Problem 4.3 (quick rules). True or false, with a one-line reason.

LeBron James makes 85% of his free throws and 50% of his field goals. In a game he takes 20 field goals (2 points each) and 10 free throws (1 point each). His expected score is 28.5 points, even if his shots are not independent.
If $X$ and $Y$ are independent with $Var(X)=Var(Y)=6$, then $Var(X+Y)=Var(2X)$.
If the correlation between $X$ and $Y$ is $-0.81$ and their standard deviations are 20 and 25, then $Cov(X,Y)=-405$.
If $X$ takes the values 0, 1, 2 with equal probability, then its mean and variance are both 1.
A firm can spend $5,000 on a proposal and win an $80,000 contract with probability 0.5, or spend $10,000 and win with probability 0.6. To maximize expected value it should spend $10,000.

Solution

TRUE. $2(20)(0.5)+1(10)(0.85)=20+8.5=28.5$, and linearity of expectation does not need independence.
FALSE. $Var(X+Y)=6+6=12$, but $Var(2X)=4(6)=24$. Two independent draws partly cancel; one draw doubled does not.
TRUE. $Cov=r\,s_X s_Y=-0.81(20)(25)=-405$.
FALSE. The mean is 1, but $Var=\tfrac13(0-1)^2+\tfrac13(0)^2+\tfrac13(2-1)^2=2/3$.
TRUE. $0.5(80{,}000)-5{,}000=35{,}000$ against $0.6(80{,}000)-10{,}000=38{,}000$.

Traps

Forgetting to square the constants: $Var(2X)=4\,Var(X)$, not $2\,Var(X)$.
Subtracting variances: $Var(X-Y)=Var(X)+Var(Y)-2Cov(X,Y)$.
Mixing up variance and standard deviation. Variance is in squared units; quote the standard deviation when you interpret.
Dropping the covariance term from a portfolio variance.

Part II. Two distributions you must know

5. The binomial distribution

The idea

A Bernoulli trial has two outcomes, success ($X=1$) with probability $p$ and failure ($X=0$). Its mean is $p$ and, because $X^2=X$, its variance is $p-p^2=p(1-p)$. The variance is largest at $p=1/2$: a fair coin is the noisiest yes/no question, and at $p=0$ or $p=1$ there is nothing left to learn.

A binomial random variable counts the successes in $n$ Bernoulli trials. The conditions from class:

A fixed number of trials $n$.
Each trial is a success or a failure.
The probability of success $p$ is the same on every trial.
The trials are independent.

Then $X\sim\text{Binomial}(n,p)$ and \[P(X=x)=\binom{n}{x}p^x(1-p)^{n-x}, \qquad \binom{n}{x}=\frac{n!}{x!\,(n-x)!}.\] Each particular sequence with $x$ successes has probability $p^x(1-p)^{n-x}$, and $\binom{n}{x}$ counts how many such sequences there are.

Because $X$ is a sum of $n$ independent Bernoulli trials, Section 4 gives the mean and variance at once: means add, and for independent trials variances add. \[E(X)=np, \qquad Var(X)=np(1-p), \qquad sd(X)=\sqrt{np(1-p)}.\] The proportion of successes $\hat p=X/n$ has mean $p$ and standard deviation $\sqrt{p(1-p)/n}$. Four times as many trials halve that standard deviation.

When it is not binomial. No fixed $n$ (calls to 911 overnight: a count in a window, which is Poisson). A $p$ that changes from trial to trial. Trials that are not independent: cards drawn without replacement, a hitter who runs hot and cold, loans that default together in a recession.

Recipe

Check the four conditions and write $X\sim\text{Binomial}(n,p)$, saying what a success is.
Translate the event into values of $X$. “More than 2” is $\{3,4,\dots\}$; “at least 2” is $\{2,3,\dots\}$; “at least one” is $1-P(X=0)=1-(1-p)^n$.
Compute each term with the formula and add. If the complement has fewer terms, use it.
Compute $np$ and $\sqrt{np(1-p)}$. Say how many standard deviations the observed count is from $np$.
Interpret, and say which condition would fail first in the real setting.

Worked example 5.1: DiMaggio’s hits in a game (class)

Joe DiMaggio hit 0.325. Treat his four at-bats in a game as independent trials with $p=0.325$, so the number of hits is $X\sim\text{Binomial}(4,0.325)$.

$x$	0	1	2	3	4
$\binom{4}{x}$	1	4	6	4	1
$P(X=x)$	0.2076	0.3998	0.2888	0.0927	0.0112
$F(x)$	0.2076	0.6074	0.8962	0.9888	1

For example, $P(X=2)=6(0.325)^2(0.675)^2=6(0.1056)(0.4556)=0.2888$.

More than two hits: $P(X>2)=P(X=3)+P(X=4)=0.0927+0.0112=0.104$, or from the CDF, $1-F(2)=1-0.8962=0.104$.
At least one hit: $1-P(X=0)=1-0.675^4=0.7924$.
Mean $np=4(0.325)=1.3$ hits; standard deviation $\sqrt{4(0.325)(0.675)}=0.937$.

What it means: a three-hit game happens about one game in ten. The 56-game streak needs at least one hit in each of 56 games: $0.7924^{56}=2.2\times 10^{-6}$, about 1 in 456,000, if games are independent. Hot and cold spells would make the streak more likely than this calculation says.

Worked example 5.2: the Patriots’ coin tosses (class)

The Patriots won 19 of 25 coin tosses in the 2014 to 2015 season. With a fair coin, the number of wins is $Y\sim\text{Binomial}(25,\tfrac12)$.

Exactly 19: $\binom{25}{19}0.5^{25}=177{,}100/33{,}554{,}432=0.0053$.
At least 19, the more honest question: add the terms for 19 through 25. The counts are $177{,}100+53{,}130+12{,}650+2{,}300+300+25+1=245{,}506$, so $P(Y\ge 19)=245{,}506/33{,}554{,}432=0.0073$.
Mean $12.5$, standard deviation $\sqrt{25(0.5)(0.5)}=2.5$. Nineteen is $(19-12.5)/2.5=2.6$ standard deviations above the mean.

What it means, and what it does not: a fair coin rarely gives one team 19 or more wins. But the league has 32 teams. The chance that at least one of them does this in a season is $1-(1-0.0073)^{32}=0.21$. Rare for one team chosen in advance is not rare for somebody in the league. And $P(\text{data}\mid\text{fair})$ small is not $P(\text{fair}\mid\text{data})$ small: to get that you need a prior on cheating, exactly as in Section 3.

Worked example 5.3: one click and a hundred clicks (class)

A headline converts each visitor with probability $p=0.6$, independently.

One visitor is a Bernoulli trial with standard deviation $\sqrt{0.6\times 0.4}=0.49$. Whether the next visitor clicks is close to a coin flip.
The proportion who click among 100 visitors has standard deviation $\sqrt{0.6\times 0.4/100}=0.049$. Among 400 visitors, $0.0245$.

What it means: more data makes the estimate of $p$ precise; it never makes the next click predictable. These are the two standard deviations in the class slide, and they answer different questions.

Worked example 5.4: guessing on a true/false quiz, by hand

A student guesses on 5 true/false questions, so the number of right answers is $X\sim\text{Binomial}(5,\tfrac12)$. With $p=\tfrac12$ every sequence of right and wrong answers has the same probability, $(\tfrac12)^5=\tfrac{1}{32}$, so $P(X=x)=\binom5x/32$ and no calculator is needed.

$x$	0	1	2	3	4	5
$\binom{5}{x}$	1	5	10	10	5	1
$P(X=x)$	1/32	5/32	10/32	10/32	5/32	1/32
$F(x)$	1/32	6/32	16/32	26/32	31/32	1

At least 4 right: $P(X\ge 4)=1-F(3)=6/32=3/16\approx 0.19$.
Mean $5(\tfrac12)=2.5$ and variance $5(\tfrac12)(\tfrac12)=1.25$. For the standard deviation, bracket the root: $1.1^2=1.21$ and $1.12^2\approx 1.25$, so $sd(X)\approx 1.12$.
A scoring rule. Suppose a right answer earns 1 point and a wrong answer costs 1 point. One guess is worth $\tfrac12(1)+\tfrac12(-1)=0$ on average, so the expected score is 0 however many questions you guess on. Expected values add, so this needs no independence.

What it means: about one blind guesser in five gets at least 4 of 5 right, so a good score on a short quiz is weak evidence of knowledge. The penalty makes blind guessing worthless on average. Partial knowledge changes that, as Problem 5.6 shows.

See it move

In the Galton Board Lab, each ball bounces right or left at every row of pegs, and its bin counts the right bounces: a binomial count. Set the rows of pegs to 4 and the chance of bouncing right to 0.3, a .300 hitter’s four at-bats. Predict first: the share of balls in bin 0, and the mean and standard deviation of the bin. Turn on the exact expected counts, drop 1,000 balls, and compare. (Answers: $0.7^4=0.24$; mean 1.2; standard deviation $\sqrt{0.84}\approx 0.92$.) Then open the Streaks Lab in Coin tosses mode, which starts with the Patriots’ numbers. Predict from Worked example 5.2 how often at least one of 32 teams wins 19 or more of 25 tosses, then run 1,000 seasons. (About 21% of seasons.)

Your turn

Problem 5.1 (the World Series, book Chapter 1). In 2006 the St. Louis Cardinals led the World Series 3 games to 1. The first team to win four games wins the series. Games are independent.

If the Cardinals win each game with probability 0.55, what is the probability that they win the series?
In 2012 the Cardinals led another series 3 games to 1, but now with probability 0.45 of winning each game. What is the probability that they win?
The series stops as soon as one team has four wins, so the number of games is not fixed. Why can you still answer as if all three remaining games were played?
Which assumption is the weakest here?

Solution

The Cardinals lose the series only if they lose all three remaining games: $1-0.45^3=1-0.091=0.909$.
$1-0.55^3=1-0.166=0.834$. A 3 to 1 lead is worth a lot even for the weaker team.
Imagine the remaining three games are played no matter what. The Cardinals win the series exactly when they win at least one of the three, so the probability is the same. The games after the decision do not change who won.
Independence and a constant $p$. Starting pitchers rotate, home field changes, and a team down 1 to 3 may change its lineup.

Problem 5.2 (defective chips, book Chapter 1). Historically, 15% of the chips a company makes are defective. A quality check takes a random sample of 10 chips.

What is the probability that exactly one chip is defective?
That none is defective?
That at least two are defective?
Find the mean and standard deviation of the number of defective chips.
One morning the sample has 5 defective chips. How many standard deviations above the mean is that? $P(X\ge 5)=0.0099$. What would you conclude, and what would you check?

Solution

$\binom{10}{1}(0.15)(0.85)^9=10(0.15)(0.2316)=0.347$. Not 0.15: there are ten places the defect could be.
$0.85^{10}=0.197$.
$1-P(0)-P(1)=1-0.197-0.347=0.456$.
Mean $10(0.15)=1.5$; standard deviation $\sqrt{10(0.15)(0.85)}=\sqrt{1.275}=1.13$.
$(5-1.5)/1.13=3.1$ standard deviations. If the defect rate were still 15%, a sample this bad would happen about once in 100 mornings. More likely something changed: check the production line, and check whether the 10 chips really were a random sample rather than chips from one bad batch, which would break independence.

Problem 5.3 (quick checks, book Chapter 1). True or false, with a reason.

A mortgage bank knows that 2% of its loans default. If it makes 10 loans, the probability that at least one defaults is 95%.
40% of an amateur bowler’s throws are gutter balls. In 90 throws, the number of gutter balls has mean 36 and standard deviation 3.65.
The number of emails you receive in an hour is a binomial random variable.
You deal 5 cards from a shuffled deck. The number of aces is $\text{Binomial}(5,\tfrac{1}{13})$.
For a Bernoulli trial, $p=0.5$ gives the largest variance.

Solution

FALSE. $1-0.98^{10}=0.183$. And loans default together in a recession, so independence is doubtful, which changes the answer further.
FALSE. The mean is $90(0.4)=36$, but the standard deviation is $\sqrt{90(0.4)(0.6)}=\sqrt{21.6}=4.65$.
FALSE. There is no fixed number of trials. It is a count in a time window, which is what the Poisson distribution models.
FALSE. The cards are drawn without replacement, so the probability that the next card is an ace changes with what has been dealt. The trials are not independent.
TRUE. $p(1-p)$ is largest at $p=1/2$, where it equals $1/4$.

Problem 5.4 (Floyd Landis, book Chapter 2). Floyd Landis was disqualified after winning the 2006 Tour de France because a doping test came back positive. As a race leader he gave 8 samples, so there were 8 chances for a false positive.

Suppose the test has specificity 95%, and Landis is clean. Assume the 8 tests are independent. What is the probability that all 8 are negative? That at least one is a false positive?
Repeat (a) with specificity 99%.
How would you use Bayes rule to assess the probability that Landis doped? What else do you need?

Solution

$0.95^8=0.663$, so the chance of at least one false positive is $1-0.663=0.337$. A clean rider who is tested 8 times has about a one in three chance of a positive.
$0.99^8=0.923$, so $1-0.923=0.077$.
You need a prior, the share of riders like Landis who dope, and the probability that a doping rider tests positive. Then combine them with the evidence actually observed (one positive among 8 samples, not “a positive”) in Bayes rule. Testing a clean rider many times makes a false positive likely, so a single positive is weaker evidence than it sounds.

Problem 5.5 (the closer, class). You hired a closer. Half the people from this bootcamp convert 70% of qualified leads; the other half convert 30%. In the first week, the new closer converts 3 of 4 leads. Treat the leads as independent.

Compute the likelihood ratio $P(\text{3 of 4}\mid 70\%)/P(\text{3 of 4}\mid 30\%)$. Why do the binomial coefficients not matter?
Find $P(\text{70\% closer}\mid\text{3 of 4})$.
Repeat (b) for 4 of 4, and for 2 of 4.

Solution

$\dfrac{\binom43 0.7^3(0.3)}{\binom43 0.3^3(0.7)}=\dfrac{0.1029}{0.0189}=5.44$. The coefficient $\binom43=4$ is the same under both hypotheses, so it cancels.
Prior odds are 1, so the posterior odds are 5.44 and the probability is $5.44/6.44=0.845$. Four leads moved you from 50% to 85%. Four leads is a small sample, but not too small to learn from: Bayes rule already accounts for the size.
4 of 4: likelihood ratio $(0.7/0.3)^4=29.6$, probability $29.6/30.6=0.967$. 2 of 4: likelihood ratio $\dfrac{0.7^2 0.3^2}{0.3^2 0.7^2}=1$, so the probability stays at 0.5. Two of four is exactly as likely under either kind of closer, so it tells you nothing.

Problem 5.6 (a three-option quiz, by hand). A quiz has 3 questions, each with 3 options, exactly one of them correct. A student guesses at random on every question. Let $X$ be the number of correct answers.

Name the distribution of $X$ and check the conditions.
Make a table of $P(X=x)$ and $F(x)$. Use fractions.
What is the probability of at least 2 correct answers?
Find $E(X)$ and $sd(X)$.
A correct answer earns 1 point and a wrong answer costs $1/2$ point. What is the expected score of a student who guesses on every question? A second student can rule out one wrong option on every question and guesses between the other two. What is that student’s expected score?

Solution

$X\sim\text{Binomial}(3,\tfrac13)$. There is a fixed number of trials, $n=3$; each answer is right or wrong; $p=\tfrac13$ on every question because the pick is random; and picks on different questions are independent.
$P(X=x)=\binom3x(\tfrac13)^x(\tfrac23)^{3-x}=\binom3x 2^{3-x}/27$.

$x$ 0 1 2 3

$P(X=x)$ 8/27 12/27 6/27 1/27

$F(x)$ 8/27 20/27 26/27 1
$P(X\ge 2)=(6+1)/27=7/27\approx 0.26$, or $1-F(1)=1-20/27$.
$E(X)=3(\tfrac13)=1$. $Var(X)=3(\tfrac13)(\tfrac23)=\tfrac23$, so $sd(X)=\sqrt{2/3}\approx 0.82$ (check: $0.82^2\approx 0.67$).
One random guess is worth $\tfrac13(1)+\tfrac23(-\tfrac12)=\tfrac13-\tfrac13=0$, so the expected score is $3\times 0=0$. With one wrong option ruled out, $p=\tfrac12$ and one guess is worth $\tfrac12(1)+\tfrac12(-\tfrac12)=\tfrac14$, so the expected score is $\tfrac34$. The penalty of $\tfrac12$ is set so that blind guessing is worth nothing; any partial knowledge makes guessing pay on average.

\(x\)	0	1	2	3
\(P(X=x)\)	8/27	12/27	6/27	1/27
\(F(x)\)	8/27	20/27	26/27	1

Traps

Forgetting $\binom{n}{x}$: “exactly one defective in 10” has ten possible positions.
Mixing up “more than” and “at least”.
Writing the standard deviation as $np(1-p)$. That is the variance; take the square root.
Using the binomial when the trials are dependent or $p$ changes. Say which condition you are assuming.

6. The normal distribution

The idea

For a continuous random variable, probabilities are areas under a density curve. Any single exact value has probability zero, so $P(X<a)$ and $P(X\le a)$ are the same.

The normal distribution $X\sim N(\mu,\sigma^2)$ is the bell curve, centered at its mean $\mu$, with spread set by the standard deviation $\sigma$. The second number in $N(\mu,\sigma^2)$ is the variance: $N(190,100)$ has standard deviation 10, not 100. Class wrote the crash model as $N(0.012,\,0.043^2)$ to make this explicit.

All normals are the same bell, shifted and stretched. Standardizing turns any normal into the standard normal $Z\sim N(0,1)$: \[Z=\frac{X-\mu}{\sigma}.\] A $z$-value counts how many standard deviations $x$ is from the mean. The area to the left of $z$ is written $\Phi(z)=P(Z\le z)$; software computes it (pnorm in R).

The one fact you need. The exam restates this fact wherever you need it: \[P(-1.96\le Z\le 1.96)=0.95.\] Together with the symmetry of the bell, it gives everything else the exam asks for.

Two tails of 2.5%. The 5% outside $\pm 1.96$ splits equally: $P(Z>1.96)=P(Z<-1.96)=0.025$.
Half on each side. $P(Z<0)=P(Z>0)=0.5$, so half of any normal lies below its mean.
Ranges and guarantees. The middle 95% of $X\sim N(\mu,\sigma^2)$ is $\mu\pm 1.96\sigma$. The level that $X$ exceeds only 2.5% of the time is $\mu+1.96\sigma$, and the level it falls below only 2.5% of the time is $\mu-1.96\sigma$.
Comparing without computing. The farther a cutoff is from the mean in standard deviations, the less probability lies beyond it. So you can rank two tail probabilities by their $z$-values alone: $P(Z>1.5)<P(Z>1.33)$.
Bounding. If $z>1.96$, then $P(Z>z)<0.025$. If $0<z<1.96$, then $P(Z>z)$ is between 0.025 and 0.5.

In class you also met $P(-1\le Z\le 1)\approx 0.68$. It is worth knowing, but the exam does not need it.

Linear changes and sums. If $X\sim N(\mu,\sigma^2)$, then $aX+b\sim N(a\mu+b,\ a^2\sigma^2)$. A sum of independent normals is normal, with the means added and the variances added.

The central limit theorem. The sum (or the average) of many independent draws is approximately normal, even when a single draw is not. For $n$ independent draws with mean $\mu$ and standard deviation $\sigma$, the sum is approximately $N(n\mu,\ n\sigma^2)$ and the average is approximately $N(\mu,\ \sigma^2/n)$. The mean of the sum grows like $n$, but its standard deviation only like $\sqrt n$. The theorem needs independence: in the class Galton board with one coin per ball, it fails.

When the model is wrong. A $z$ of $-5$ is far beyond $-1.96$; under the normal model its probability is below one in a million. When it happens anyway, the likelier conclusion is that the tails of the real distribution are heavier than normal.

Recipe

Write $X\sim N(\mu,\sigma^2)$. If you were given a variance, take its square root now.
Sketch the bell. Mark $\mu$, the points $\mu\pm 1.96\sigma$, and the cutoff. Shade the area the question asks about.
Standardize: $z=(x-\mu)/\sigma$. Below the mean, $z$ is negative.
Place $z$. At 0 the tail beyond it is 50%. Beyond $\pm 1.96$ the tail is under 2.5%. In between, the tail is between 2.5% and 50%.
To compare two options, compare their $z$-values instead of computing probabilities.
For a range or a guarantee, work backward from 1.96: $\mu\pm 1.96\sigma$, or $\mu+1.96\sigma$ for the level exceeded only 2.5% of the time.
Interpret in the units of the problem, and say which parameter would have to change to change the decision.

Worked example 6.1: body weight (book Chapter 1)

Model weight as $X\sim N(190,100)$ in pounds.

The variance is 100, so $\sigma=10$.
The middle 95% of weights is $190\pm 1.96(10)=190\pm 19.6$, from 170.4 to 209.6 pounds.
So 2.5% of people weigh more than 209.6 pounds, and 50% weigh less than 190.
Is 200 pounds unusual? $z=(200-190)/10=1$, inside $\pm 1.96$, so no: more than 2.5% of people weigh more than 200 pounds. (Software: 15.9%.)

The trap: using 100 as the standard deviation gives a “95% range” of $190\pm 196$, from $-6$ to 386 pounds. A negative weight is the alarm that you used the variance.

Worked example 6.2: test scores, a range and a percentile

Scores on a test are $N(1000,\ 200^2)$.

The middle 95%: $1000\pm 1.96(200)=1000\pm 392$, from 608 to 1392.
The score that beats 97.5% of test takers: 1392.
Between 608 and 1000: half of the middle 95%, so 47.5%.
Is 1300 in the top 2.5%? $z=(1300-1000)/200=1.5<1.96$, so no: more than 2.5% of scores are above 1300.

Worked example 6.3: how extreme was the 1987 crash? (class)

Before October 1987, monthly S&P 500 returns were modeled as $N(0.012,\ 0.043^2)$: a mean of 1.2% and a standard deviation of 4.3%. The October return was $-21.76\%$.

$z=(-0.2176-0.012)/0.043=-5.34$. That is far beyond $-1.96$, so under the model a month this bad has probability far below 2.5%. Software puts it at $4.7\times 10^{-8}$: one month in about 21 million, or once in 1.8 million years.

What it means: either we saw a once-in-a-million-years event, or the normal model understates the tails of stock returns. The class histogram of S&P returns against the fitted normal settles it: real returns have far more extreme months than the bell allows. A model fitted to calm months is not safe to use for risk.

Worked example 6.4: is a 70% click rate surprising? (class, the central limit theorem)

The headline from Worked example 5.3 has true rate $p=0.6$. Among 100 visitors, would an observed rate of 70% be surprising?

The observed rate $\hat p$ is an average of 100 independent clicks, so by the central limit theorem it is approximately $N(0.6,\ 0.049^2)$. Then $z=(0.70-0.60)/0.049=2.04$, beyond 1.96, so a rate this high happens less than 2.5% of the time. (Software: 2.1%.)

What it means: a 70% rate from 100 visitors would be unusual if the truth were 60%. With only 25 visitors the standard deviation would be 0.098, $z$ would be about 1, well inside $\pm 1.96$, and 70% would be unremarkable. The same observed rate means more when it rests on more data.

See it move

In the Normal Tails Lab, set the mean to 0 and the SD to 1, choose $a<X<b$, and set $a=-1.96$ and $b=1.96$: the shaded area is 0.95. Switch to $X>b$ and read 0.025. Predict first, then check: with mean 190 and SD 10 (Worked example 6.1), the probability above 209.6 pounds, and the probability above 200 pounds. Then load the 1987 crash case, predict its $z$ from Worked example 6.3, and turn on Compare with Student-t to see how much a fat tail changes the answer. For the central limit theorem, the Sampling Distribution Lab draws averages from a population you choose: pick a skewed one, predict what happens to the spread of the averages when the sample size is multiplied by 4, then draw 1,000 samples. (Answers: 0.025; 0.159; $z=-5.34$; the spread halves.)

Your turn

Problem 6.1 (choosing a filling machine). Cereal boxes are labeled 500 grams, and a box that holds less than 500 grams is underfilled. Machine A fills $N(510,\ 4^2)$ grams. Machine B fills $N(515,\ 8^2)$ grams. Use only $P(-1.96\le Z\le 1.96)=0.95$ and symmetry.

Compute the $z$-score of 500 grams for each machine. Which machine underfills less often? Is each machine’s underfill rate above or below 2.5%?
Machine B’s mean stays at 515. How small must its standard deviation be to match machine A’s underfill rate? How small to bring B’s underfill rate down to 2.5%?
A new rule counts a box as underfilled below 505 grams. Compare the machines again. What do you notice?
Under a rule of 510 grams, what share of machine A’s boxes is underfilled? Which machine is better? Explain in one sentence why the answer flips.
Machine A’s maker promises a fill that A’s boxes fall below only 2.5% of the time. What fill is that? A store checks 12 boxes from A. What is the probability that at least one falls below the promised fill? What did you assume?

Solution

A: $z=(500-510)/4=-2.5$. B: $z=(500-515)/8=-1.875$. A’s cutoff sits further out in its lower tail, so A underfills less often. A’s $z$ is beyond $-1.96$, so A underfills fewer than 2.5% of boxes; B’s $z$ is inside, so B underfills more than 2.5%. (Software: 0.6% and 3.0%.) Choose A, even though B fills more on average.
Match the $z$: $(500-515)/\sigma_B=-2.5$ gives $\sigma_B=6$ grams. For a 2.5% rate: $(500-515)/\sigma_B=-1.96$ gives $\sigma_B=15/1.96=7.65$ grams.
A: $z=(505-510)/4=-1.25$. B: $z=(505-515)/8=-1.25$. The same $z$ means the same underfill rate: the machines tie at a 505 limit.
For A, 510 is the mean, so exactly 50% of its boxes are underfilled. For B, $z=(510-515)/8=-0.625$, below the mean, so less than 50%: B is better. Far below both means, the wider spread of B puts more of its boxes in the tail; close to the means, B’s higher mean matters more. Which machine wins depends on where the limit sits relative to the means and spreads.
$510-1.96(4)=502.16$ grams. Each box falls below it with probability 0.025, so $P(\text{at least one of }12)=1-0.975^{12}=0.262$, assuming the boxes are independent. Boxes filled in the same run may share a problem, which would break that assumption.

Problem 6.2 (snowboards, book Chapter 1). Snowboard life is normal with mean 10 years and standard deviation 3 years. A board is defective if its life is less than 5 years.

Is the share of defective boards above or below 2.5%? Explain with a $z$-score.
The maker wants a warranty period that only 2.5% of boards fail to reach. How long should the warranty be?
With that warranty, let $N$ be the number of warranty claims in a shipment of 120 boards. Name the distribution of $N$, and find its mean and standard deviation. What is the probability of no claims?
One shipment produces 9 claims. How many standard deviations above the mean is that? Software gives $P(N\ge 9)=0.003$ for this binomial. What would you conclude, and what would you check?

Solution

$z=(5-10)/3=-1.67$, inside $\pm 1.96$, so more than 2.5% of boards are defective. (Software: 4.8%.)
$10-1.96(3)=4.12$ years. Only 2.5% of boards wear out before then.
$N\sim\text{Binomial}(120,\ 0.025)$: mean $120(0.025)=3$, standard deviation $\sqrt{120(0.025)(0.975)}=1.71$. $P(N=0)=0.975^{120}=0.048$.
$(9-3)/1.71=3.5$ standard deviations above the mean. If boards fail independently, each with probability 0.025, a shipment this bad happens about 3 times in 1,000. More likely something changed: check whether the shipment came from one bad production run, which would break independence, or whether the life model itself is off.

Problem 6.3 (Black Monday in China, book Chapter 1). On 24 August 2015, Chinese stocks fell 8.5% in one day. Over the previous 25 years, daily returns averaged 0.09% with a standard deviation of 2.6%, and 56% of days closed within one standard deviation of the mean. For the S&P 500 that share was 74%. Under a normal model, about 68% of days would fall within one standard deviation.

Compute the $z$-score of the 8.5% drop. Under a normal model, is a drop at least this large rarer than 2.5% of days?
Compare the normal model’s 68% with the 56% for China and the 74% for the S&P 500.
What do (a) and (b) say about the normal model for each market?

Solution

$z=(-8.5-0.09)/2.6=-3.30$, far beyond $-1.96$, so under the normal model a drop this large happens on far fewer than 2.5% of days. (Software: 0.0005, about one trading day in 2,100, roughly once in 8 years.) Notice that 25 years is about 6,300 trading days, so even the normal model expects about three such days in the sample. Rare on a given day is not rare over 25 years.
China has too few days near the mean (56% against 68%); the S&P 500 has too many (74%).
Neither market is normal, and China is further off. For the S&P 500, too many quiet days with the same standard deviation means that the rest of the variance must come from a few wild days: heavier tails than the bell. For China, only 56% of days near the mean says the bell shape itself is wrong, so a mean and a standard deviation do not describe a typical Chinese trading day. In both markets, a normal model fitted with the historical standard deviation misjudges how often extreme days happen.

Problem 6.4 (a hundred spins of roulette). You bet $1 on black 100 times at a European wheel. From Worked example 4.1, each bet has mean $-0.027$ and standard deviation 1.00.

Use the central limit theorem to describe your total winnings $S$ after 100 bets.
Give a range that contains $S$ with probability about 95%. Is being ahead after 100 bets unusual?
Now play 10,000 bets. Repeat (a) and (b). What does the comparison teach about the house edge?

Solution

Mean $100(-0.027)=-2.70$, standard deviation $\sqrt{100}(1.00)=10.0$. So $S\approx N(-2.70,\ 10^2)$.
$-2.70\pm 1.96(10.0)=-2.70\pm 19.6$, from $-\$22.30$ to $\$16.90$. Zero is well inside the range, only 0.27 standard deviations above the mean, so being ahead after 100 bets is common. (Software: about 39%.)
Mean $10{,}000(-0.027)=-270$, standard deviation $\sqrt{10{,}000}(1.00)=100$. The 95% range is $-270\pm 196$, from $-\$466$ to $-\$74$. Zero is above the whole range, so being ahead is rarer than 2.5%. (Software: 0.3%.) The expected loss grows like $n$ and the spread only like $\sqrt n$, so in the long run the edge swamps the luck. A short session is a gamble; a long one is a loss.

Problem 6.5 (earnings day, book Chapter 1). On an earnings announcement, a stock beats expectations 75% of the time. When it beats, its return is $N(10\%,\ 5\%^2)$. When it misses, its return is $N(-5\%,\ 8\%^2)$. Use the 1.96 fact and symmetry, and give bounds where you cannot give exact values.

What is the probability that the return is below $-5\%$?
What is the probability that the return is above 10%?

Solution

Total probability over the two cases. If the stock beats, $z=(-5-10)/5=-3$, beyond $-1.96$, so the chance is below 0.025. If it misses, $-5\%$ is the mean, so the chance is exactly 0.5. The total is $0.75\times(\text{less than }0.025)+0.25(0.5)$, between 0.125 and $0.125+0.019=0.144$. Almost all of it comes from the misses. (Software: 0.126.)
If it beats, 10% is the mean, so the chance is 0.5. If it misses, $z=(10+5)/8=1.875$, just inside 1.96, so the chance is a little more than 0.025. The total is $0.75(0.5)+0.25\times(\text{a little more than }0.025)$, about $0.375+0.006=0.38$. (Software: 0.383.)

Problem 6.6 (quick checks, adapted from book Chapter 1). True or false, with a one-line reason.

Steaks advertised as 16 ounces have weights $N(16,\ 2^2)$. About 2.5% of steaks weigh less than 12.08 ounces.
Spending per visit at a theme park is $N(\$60,\ \$15^2)$. Half of visitors spend less than $60, and about 95% spend between $30.60 and $89.40.
The cost of a 30-second commercial is $N(10{,}000,\ 1{,}000^2)$. The probability that a commercial costs between 9,000 and 10,000 is 50%.
A stock’s returns on earnings day are $N(5\%,\ 5\%^2)$. The probability of losing money that day is below 2.5%.
If $X\sim N(50,25)$, then about 95% of values of $X$ lie between 40.2 and 59.8.
If $X$ and $Y$ are independent and both $N(0,1)$, then about 95% of values of $X+Y$ lie between $-1.96$ and $1.96$.
If $X\sim N(3,9)$, then $P(X>1)=0.254$.

Solution

TRUE. $16-1.96(2)=12.08$ is the level that only 2.5% of steaks fall below.
TRUE. The mean splits the bell in half, and $60\pm 1.96(15)=60\pm 29.4$.
FALSE. Half of all commercials cost less than 10,000, and part of that half costs less than 9,000. Since 9,000 is within $1.96$ standard deviations of the mean, more than 2.5% lies below it, so the interval holds less than 47.5%. (Software: 34%.)
FALSE. $z=(0-5)/5=-1$, inside $\pm 1.96$, so the chance of a loss is more than 2.5%. (Software: 16%.)
TRUE. The variance is 25, so $\sigma=5$ and $50\pm 1.96(5)=50\pm 9.8$.
FALSE. Variances of independent variables add, so $X+Y\sim N(0,2)$, with standard deviation $\sqrt 2=1.41$. The middle 95% is $\pm 1.96(1.41)=\pm 2.77$.
FALSE. The value 1 is below the mean 3, so more than half of the distribution lies above it: $P(X>1)>0.5$. (Software: 0.75.)

Traps

Using the variance as the standard deviation. Check the units: $\sigma$ has the units of $X$. A 95% range with impossible values is a warning sign.
Putting all 5% in one tail. The 5% outside $\mu\pm 1.96\sigma$ splits: 2.5% in each tail.
Dropping the sign of $z$ for values below the mean.
Writing the standard deviation of a sum of $n$ draws as $n\sigma$. It is $\sqrt n\,\sigma$.
Treating a 5-sigma event as a miracle instead of as evidence against the model.

Part III. Linear regression

7. Least squares

The idea

Regression predicts a response $y$ from a predictor $x$ with a line, $\hat y=a+bx$. The line is a conditional mean: the average of $y$ among cases with that value of $x$. In class, with two groups of houses, you reported 2 for a small house and 6 for a large one; the regression line does the same thing with living area in place of the two groups.

A residual is the vertical distance from a point to the line, in the units of $y$: \[e_i=y_i-\hat y_i.\] A positive residual means the point is above the line: the observed value is larger than predicted.

Least squares chooses $a$ and $b$ to make the sum of squared residuals as small as possible: \[SSE(a,b)=\sum_i (y_i-a-bx_i)^2.\] In class we wrote the same objective as the loss $J(a,b)=\frac{1}{2n}\sum_i(a+bx_i-y_i)^2$; the factor $\frac{1}{2n}$ does not change which line wins. Squaring stops positive and negative errors from canceling, and it punishes a big miss much more than several small ones.

The minimizing line has a formula: \[\hat b=\frac{\sum_i (x_i-\bar x)(y_i-\bar y)}{\sum_i (x_i-\bar x)^2}, \qquad \hat a=\bar y-\hat b\,\bar x.\] Two consequences you can use as checks: the line passes through the point of means $(\bar x,\bar y)$, and the residuals sum to zero.

Reading the coefficients.

The slope $\hat b$ is the difference in predicted $y$ between two cases whose $x$ differs by one unit. Its units are units of $y$ per unit of $x$. It describes how $y$ and $x$ go together across the cases in the data, not what would happen if you changed $x$ for one case.
The intercept $\hat a$ is the predicted $y$ at $x=0$. It means something only if $x=0$ is inside the range of the data. Otherwise it is an extrapolation whose job is to pin down the line.

Predicting inside the range of the data is interpolation. Predicting far outside it is extrapolation: the data say nothing about whether the line continues.

Influence. Write $S_{xx}=\sum_i(x_i-\bar x)^2$. Because $\sum_i(x_i-\bar x)\bar y=0$, the slope is $\hat b=\sum_i (x_i-\bar x)\,y_i/S_{xx}$, a weighted sum of the responses with weights proportional to $x_i-\bar x$. So raising one response $y_i$ by $\Delta$ changes the slope by \[\Delta\,\frac{x_i-\bar x}{S_{xx}}.\] A point far from $\bar x$ is a long lever and can tilt the line. A point at $\bar x$ cannot change the slope at all; it only lifts the whole line by $\Delta/n$.

Predictions as a matrix product (class). Put a column of ones next to the column of $x$ values to form the design matrix $X$, and let $\theta=(a,b)$. Then the vector of predictions is $X\theta$: each row produces one prediction, and the column of ones supplies the intercept. If two columns of $X$ are identical, only the sum of their coefficients matters, so the coefficients are not unique. The predictions, and the loss, still are.

Recipe: a line by hand

Compute $\bar x$ and $\bar y$.
Make a table with columns $x-\bar x$, $y-\bar y$, their product, and $(x-\bar x)^2$. The first two columns must each sum to zero.
$\hat b=$ (sum of products) / (sum of squares). Then $\hat a=\bar y-\hat b\,\bar x$.
Fitted values $\hat y_i=\hat a+\hat b x_i$ and residuals $e_i=y_i-\hat y_i$. Check that the residuals sum to zero.
$SSE=\sum_i e_i^2$.
Interpret the slope and the intercept in one sentence each, in units. Flag any extrapolation.

Worked example 7.1: three points (class)

The class data are $(0,1)$, $(1,3)$, $(2,3)$, so $\bar x=1$ and $\bar y=7/3$.

$x$	$y$	$x-\bar x$	$y-\bar y$	product	$(x-\bar x)^2$
0	1	$-1$	$-4/3$	$4/3$	1
1	3	0	$2/3$	0	0
2	3	1	$2/3$	$2/3$	1
Sum		0	0	2	2

$\hat b=2/2=1$ and $\hat a=7/3-1(1)=4/3$, so $\hat y=4/3+x$. The fitted values are $4/3,\ 7/3,\ 10/3$, the residuals are $-1/3,\ 2/3,\ -1/3$ (they sum to zero), and $SSE=1/9+4/9+1/9=2/3$.

The class candidate line $\hat y=1+x$ passes exactly through two of the points. Its residuals are $0,\ 1,\ 0$ and its $SSE$ is 1. The least squares line misses all three points and still wins: $2/3<1$. In terms of the class loss, $J=SSE/(2n)$ is $1/9$ against $1/6$.

Worked example 7.2: one point moves the line

Same three points, with $S_{xx}=2$.

Raise the last response by 3, to $(2,6)$. That point has $x-\bar x=1$, so the slope rises by $3(1)/2=1.5$, to $\hat b=2.5$. The new mean of $y$ is $10/3$, so $\hat a=10/3-2.5(1)=0.83$.
Instead raise the middle response by 3, to $(1,6)$. That point sits at $\bar x$, so the slope does not move: $\hat b=1$. The whole line lifts by $3/3=1$: $\hat a=4/3+1=2.33$.

The same change in one response has completely different effects. The point at the edge has leverage; the point in the middle has none. In class, raising the rightmost response by 15 visibly tilted the whole line.

Worked example 7.3: reading a fitted line, Saratoga houses (class)

For 1,728 house sales, $\hat y=13.44+113.12x$, with $x$ in thousands of square feet and $y$ in thousands of dollars. Living areas run from 616 to 5,228 square feet.

Slope. Two houses that differ by 1,000 square feet differ in predicted price by $113,120, about $113 per square foot. This compares different houses. It does not promise that a 1,000 square foot addition would raise the price of a given house by $113,120.
Prediction. At 2,200 square feet: $13.44+113.12(2.2)=262.3$, about $262,300.
Intercept. 13.44 is the predicted price of a house with no living area, about $13,000. No house in the data is smaller than 616 square feet, so this is an extrapolation that anchors the line, not the value of the land.

See it move

In the Least Squares Lab, load the Clean line data. Turn on Draw squared residuals and Show my line and its handles. Predict first: can any line you draw have a smaller SSE than the least squares line? Try, then press Snap to least squares. Next, turn on The point $(\bar x,\bar y)$ and drag the rightmost point straight up. Predict which way the slope moves, and whether the line still passes through $(\bar x,\bar y)$. Then drag a point near the middle by the same amount and compare, as in Worked example 7.2.

Your turn

Problem 7.1 (salary and experience). Five employees:

Experience $x$ (years)	2	4	6	8	10
Salary $y$ ($1,000s)	60	68	74	80	88

Fit the least squares line by hand. Interpret the slope and the intercept in the units of the problem.
Compute the fitted values and the residuals. Check that the residuals sum to zero, and compute $SSE$.
Predict the salary at 7 years and at 30 years. Which prediction do you trust, and why?
The 10-year employee’s salary is corrected from 88 to 98. Recompute $\hat b$ and $\hat a$. Then go back to the original data and suppose instead that the 6-year employee earns 84 rather than 74. Recompute. Explain the difference.

Solution

$\bar x=6$ and $\bar y=74$. The deviations are $x-\bar x=-4,-2,0,2,4$ and $y-\bar y=-14,-6,0,6,14$. Products: $56,12,0,12,56$, sum 136. Squares: $16,4,0,4,16$, sum 40. So $\hat b=136/40=3.4$ and $\hat a=74-3.4(6)=53.6$. Each extra year of experience goes with $3,400 more salary, on average. The intercept, $53,600, is the predicted salary with no experience; nobody in the data has less than 2 years, so it is a mild extrapolation.
Fitted values $60.4,\ 67.2,\ 74.0,\ 80.8,\ 87.6$. Residuals $-0.4,\ 0.8,\ 0,\ -0.8,\ 0.4$, which sum to zero. $SSE=0.16+0.64+0+0.64+0.16=1.6$.
At 7 years: $53.6+3.4(7)=77.4$, about $77,400. Seven is inside the data, so this is interpolation and is reasonable. At 30 years: $53.6+3.4(30)=155.6$. Thirty is far outside 2 to 10 years. Salaries usually flatten with seniority, and five data points cannot say whether the line continues.
Changing $y$ at $x=10$ by $\Delta=10$: the slope changes by $10(10-6)/40=1$, so $\hat b=4.4$. The new $\bar y$ is 76, so $\hat a=76-4.4(6)=49.6$. Changing $y$ at $x=6$ by 10: $x-\bar x=0$, so $\hat b=3.4$ is unchanged, and $\hat a=76-3.4(6)=55.6$, a lift of $10/5=2$. A point at the mean of $x$ can only lift the line; a point at the edge tilts it.

Problem 7.2 (reading a fitted model). A regression of weekly sales (thousands of units) on advertising spend (thousands of dollars) over 30 weeks gave $\hat y=4.20+0.35x$. Weekly ad spend ranged from $5,000 to $40,000.

Interpret the slope in the units of the problem.
Predict sales in a week with $20,000 of advertising.
Interpret the intercept. Should the marketing team report it as “the sales we would get with no advertising”?
The manager says: “At $20,000 we sell 11,200 units, so doubling the budget to $40,000 will double sales.” Check the claim against the model. Then give one reason, beyond the arithmetic, to doubt that the slope measures the effect of advertising.

Solution

Weeks with $1,000 more advertising sold 350 more units, on average.
$4.20+0.35(20)=11.2$, about 11,200 units.
The model predicts 4,200 units with no advertising. But no week in the data spent less than $5,000, so zero is outside the data and the intercept is an extrapolation. It may be roughly right; the data cannot confirm it.
The model gives $4.20+0.35(40)=18.2$ at $40,000, not $2(11.2)=22.4$: doubling $x$ does not double $\hat y$ when the intercept is not zero. Beyond the arithmetic: the slope compares weeks, and weeks with big budgets may be holiday weeks, when sales are high anyway. Then the slope mixes the effect of advertising with the effect of the season.

Problem 7.3 (predictions as a matrix product, class). Let \[X=\begin{pmatrix}1&2\\1&3\\1&5\end{pmatrix},\qquad \theta=\begin{pmatrix}10\\2\end{pmatrix}.\]

Compute $X\theta$. What does each row of the result represent? Which column of $X$ carries the intercept?
Add a third column to $X$, identical to the second, so the model is $\hat y=\theta_0+\theta_1x+\theta_2x$. Show that $\theta=(10,2,0)$, $(10,0,2)$, and $(10,5,-3)$ give the same predictions. What can the fit still tell you, and what can it not?

Solution

$X\theta=(10+2\cdot 2,\ 10+2\cdot 3,\ 10+2\cdot 5)=(14,\ 16,\ 20)$. Each entry is the prediction for one observation (one row of $X$). The column of ones carries the intercept.
The predictions depend only on $\theta_1+\theta_2$, which is 2 in all three cases, so all three give $(14,16,20)$. The predictions and the loss are determined by the data; the separate coefficients are not. You cannot say how much of the effect belongs to each of two identical columns.

Traps

Mixing up $\sum(x_i-\bar x)(y_i-\bar y)$ and $\sum(x_i-\bar x)^2$: the slope has the products on top.
Computing $\hat a=\bar y-\hat b\,\bar y$. It is $\bar x$ that goes with $\hat b$.
Interpreting a slope as what would happen if you changed $x$ for one case.
Treating the intercept as meaningful when $x=0$ is outside the data.
Trusting predictions far outside the range of $x$.

8. Checking a regression

The idea

A fitted line is a model. In class we wrote it as \[Y_i\mid x_i\sim N(a+bx_i,\ \sigma^2),\quad\text{independently}.\] That one line makes four claims: the mean of $y$ is a straight line in $x$; the errors are normal; they have the same spread $\sigma$ at every $x$; and they are independent.

Least squares is maximum likelihood under this model. The log-likelihood is \[\ell(a,b)=-\frac{1}{2\sigma^2}\sum_i (y_i-a-bx_i)^2+C,\] where $C$ does not depend on $a$ or $b$. Maximizing $\ell$ is the same as minimizing the sum of squares, so the least squares line is the maximum likelihood line when the errors are normal with constant variance. The normal density makes a large error extremely unlikely, so the fit will bend a lot to avoid one. That is why a single wild point can pull a least squares line: a residual of 10 costs 100 in the sum of squares, as much as a hundred residuals of 1.

The residual standard error $\hat\sigma$ estimates $\sigma$, the typical distance of a point from the line. Software reports it as “Residual standard error” and computes it as $\sqrt{SSE/(n-2)}$; on the exam you will be given it. It turns the model into statements about individual cases:

About 95% of cases with a given $x$ fall within $\hat y\pm 1.96\,\hat\sigma$. (This ignores the uncertainty in the line itself, which is small when $n$ is large.)
The standardized residual $e_i/\hat\sigma$ counts how many $\hat\sigma$ a case is from the line. Beyond about $\pm 3$ is unusual under the normal model: look at that case.

Residual plots check the four claims. Plot the residuals, or the standardized residuals, against $x$ (or against $\hat y$).

What you see	What it means	What to do
A random cloud around 0 with constant spread	The line is adequate	Nothing
A U shape or an inverted U	The mean curves; the line is wrong in a systematic way	Add a curved term such as $x^2$
One point far from all the others	An outlier	Check the record; refit with and without it; report both
Spread that grows with $x$ (a fan)	The errors do not have one $\sigma$; common with positive, skewed responses like money	Model $\log y$

Curvature and an outlier are different clues, and they have different remedies. An outlier far from $\bar x$ is also influential: as in Section 7, it can tilt the line.

Recipe

Write the model, $y\mid x\sim N(a+bx,\ \sigma^2)$, with $\hat\sigma$ from the output.
For one case: compute $\hat y$, the residual $e=y-\hat y$, and the standardized residual $e/\hat\sigma$. Beyond $\pm 3$, investigate.
For a range of individual outcomes: $\hat y\pm 1.96\,\hat\sigma$.
For a residual plot: look for shape (a curve), single points (outliers), and changing spread (a fan). Name the problem and one remedy.

Worked example 8.1: Saratoga, continued (class)

The Saratoga fit has $\hat\sigma=69.10$, that is $69,100.

A 95% range at 2,200 square feet: $262.3\pm 1.96(69.10)=262.3\pm 135.4$, from 126.9 to 397.7, so $127,000 to $398,000. The line pins down the average price of such houses, but individual houses vary a lot around it.
One unusual sale. A 2,200 square foot house sold for $500,000. Its residual is $500-262.3=237.7$, so it sold $237,700 above the line. Its standardized residual is $237.7/69.10=3.44$, which is unusual under the model. Check that the sale is recorded correctly, then ask what the house has that size does not capture: waterfront, land, condition. Do not delete it just because it is far from the line.

Worked example 8.2: Tiger Woods and the log scale (class)

In class we regressed the 2000 earnings of 195 golfers on four statistics: events played, driving distance, greens in regulation, and putts per round.

Dollar scale. Tiger Woods earned $9.19 million; the model predicts $3.58 million. His residual is $5.6 million, or $e/\hat\sigma=8.3$ residual standard errors ($\hat\sigma\approx\$0.67$ million). The class histogram puts him near 8.8, because R’s standardized residuals also adjust for how unusual a golfer’s statistics are. Under a normal model either number is essentially impossible. He alone contributes 37% of the sum of squared residuals, so one golfer shapes the whole fit.
Log scale. Regress $\log(\text{earnings})$ on the same statistics. Tiger’s standardized residual becomes $-0.57$ ($e/\hat\sigma=-0.54$): the log model predicts about $12.5 million for him, a bit more than he earned. The largest standardized residual on the log scale, 3.40, belongs to Notah Begay III.

What it means: on the dollar scale the model adds dollars for each statistic, and it cannot reach Tiger. On the log scale each statistic multiplies earnings, and Tiger’s exceptional statistics multiply into a large prediction. An outlier on one scale can be ordinary on another. The transformation changes which errors matter; it does not prove a causal story.

See it move

In the Regression Model Lab, every point is drawn from a normal distribution centered on the line, with the same spread $\sigma$ at every $x$. Turn on the $\pm 2\sigma$ band. Predict first: what share of the points falls inside the band? Read Points within $\pm 2\sigma$, then move the noise slider and watch the band and the points stretch together. Then in the Least Squares Lab, load Mammals on the raw scale and look at the residuals: a few giant animals dominate, and the spread fans out. Predict what a log-log scale will do, then load Mammals on the log-log scale. Worked example 8.2 is the same idea with golfers.

Your turn

Problem 8.1 (advertising, continued). For the advertising model of Problem 7.2, the residual standard error is 1.10 thousand units.

Give a range that should contain about 95% of weekly sales in weeks with $20,000 of advertising.
One such week sold 15,000 units. Compute its residual and standardized residual. What would you do about this week?
If the residual standard error were 3.0 instead, how would your answer to (b) change?

Solution

$11.2\pm 1.96(1.10)=11.2\pm 2.16$, from 9.04 to 13.36: about 9,000 to 13,400 units.
$e=15-11.2=3.8$ thousand units, and $3.8/1.10=3.45$. Unusual under the model. Look for a cause: a holiday, a promotion, a competitor out of stock, or a data entry error. If there is a real reason, record it or model it; do not just delete the week.
Then $3.8/3.0=1.27$, an ordinary week. Whether a residual is large depends on the typical scatter around the line.

Problem 8.2 (reading residual plots). Figure 1 shows standardized residuals from four straight-line fits. In all four, the response is a positive amount, such as weekly sales in dollars. For each panel, say whether the straight line is adequate. If not, name the problem and one remedy.

Figure 1: Standardized residuals plotted against x for four different fitted lines.

Solution

Panel 1. An inverted U: negative residuals at both ends, positive in the middle. The relationship bends downward, and the line overpredicts at both ends. Remedy: add an $x^2$ term.
Panel 2. A random cloud around zero with constant spread, everything within $\pm 3$. The line is adequate.
Panel 3. One point near $-6$; the rest is random. An outlier far below the line. Check that record, refit without it to see how much the line moves, and report both fits.
Panel 4. A fan: the spread grows with $x$. There is no single $\sigma$, so $\hat y\pm 1.96\hat\sigma$ is too wide at small $x$ and too narrow at large $x$. Remedy: model $\log y$, which turns errors that grow in proportion to $y$ into errors of constant size. The point near $-3.7$ is part of the fan, not a separate outlier: with one $\hat\sigma$ for every $x$, residuals at large $x$ look extreme because the spread there is larger. Fix the fan first, then look again.

Problem 8.3 (least squares and the normal model).

Write the log-likelihood for the model $y_i\mid x_i\sim N(a+bx_i,\sigma^2)$, independently for $i=1,\dots,n$. Show that maximizing it over $a$ and $b$ gives the least squares line.
Suppose the real errors have much heavier tails than the normal: wild errors are not rare. Would you still trust a line fitted by least squares? Why?
A colleague proposes to choose the line that minimizes the sum of the absolute residuals instead. Compared with least squares, how much would that line react to one wild point?

Solution

Each $y_i$ has density $\frac{1}{\sqrt{2\pi}\sigma}\exp\left(-\frac{(y_i-a-bx_i)^2}{2\sigma^2}\right)$. Independence makes the likelihood a product, so the log-likelihood is a sum: \[\ell(a,b)=-n\log\left(\sqrt{2\pi}\,\sigma\right)-\frac{1}{2\sigma^2}\sum_i (y_i-a-bx_i)^2.\] The first term does not involve $a$ or $b$, and the second is $-SSE/(2\sigma^2)$. Maximizing $\ell$ means minimizing $SSE$.
Less. Least squares is the right choice when large errors are rare, as under the normal. When wild errors are common, each one pulls the line hard, because its squared residual is huge. Look at the residual plots, and consider a transformation or a method less sensitive to outliers.
Much less. With absolute residuals, a residual of 10 costs 10, the same as ten residuals of 1, instead of 100. One wild point has far less pull. The price: this line is no longer the maximum likelihood line under normal errors.

Traps

Calling a point an outlier without computing its standardized residual.
Using $\hat\sigma$ as if it were the uncertainty of the slope. It is the scatter of individual cases around the line.
Deleting an unusual case without checking it, or without reporting what the fit looks like with it.
Reading a U-shaped residual plot as “a few outliers”. A systematic pattern means the mean is wrong.

Part IV. Logistic regression

9. Logistic regression

The idea

Many questions have a yes-or-no answer: will a customer cancel, is an email spam, will the favorite win. Code the answer as $y=1$ for yes and $y=0$ for no. The thing to predict is a probability, $p=P(y=1\mid x)$. A straight line is the wrong tool for it: in class, a line through NBA wins and losses predicted values above 1 for large point spreads, and a probability has to stay between 0 and 1.

Logistic regression keeps the straight line but uses it on a different scale. It computes a linear score, which can be any number, \[z=\beta_0+\beta_1x_1+\dots+\beta_px_p,\] and turns the score into a probability with the sigmoid (logistic) function \[p=\sigma(z)=\frac{1}{1+e^{-z}}.\] The sigmoid is an S-curve. At $z=0$ it gives $p=0.5$; large positive scores give $p$ near 1, large negative scores give $p$ near 0, and $p$ never leaves the interval from 0 to 1. It is symmetric about the middle, $\sigma(-z)=1-\sigma(z)$, so the values for negative $z$ follow from those for positive $z$. These values are worth knowing (log is the natural logarithm):

$z$	$-3$	$-2$	$-1$	$0$	$1$	$2$	$3$
$\sigma(z)$	0.05	0.12	0.27	0.50	0.73	0.88	0.95
$\log\sigma(z)$	$-3.0$	$-2.1$	$-1.3$	$-0.69$	$-0.31$	$-0.13$	$-0.05$

Figure 2: Left: the sigmoid turns any score z into a probability; the dots are the table values. Right: the log loss of one prediction, for y = 1 and for y = 0.

Log-odds. Solving $p=\sigma(z)$ for $z$ gives \[z=\log\frac{p}{1-p}.\] The ratio $p/(1-p)$ is the odds in favor of $y=1$, so the score is the log-odds of $y=1$. That is what the model makes linear: the log-odds are a straight line in $x$, and the probability is an S-curve in $x$ (Figure 2, left).

What a coefficient means. $\beta_1$ is the change in the log-odds when $x_1$ rises by one unit and the other predictors stay fixed. It is not a change in probability. How much $p$ moves depends on where the case starts on the S-curve: one unit of $z$ moves $p$ from 0.50 to 0.73 in the middle, but only from 0.88 to 0.95 near the top. Equivalently, one more unit of $x_1$ multiplies the odds by $e^{\beta_1}$. Coefficients can be traded against each other on the log-odds scale. In the class LinkedIn model of who becomes an executive, one more job function had coefficient 0.26 and one more year of experience had 0.09, so one extra function was worth about three years of experience. And because the sigmoid is not linear, probability changes from separate predictors do not add: in class, six changes to person B, each worth 3 to 15 points on its own, took the probability from 15% to about 81%, not to the 52% that adding the separate lifts would give.

Log loss. Judge a probability by the probability it gave to what actually happened: $p$ if $y=1$, and $1-p$ if $y=0$. The log loss of one prediction is minus the log of that probability, \[J=-y\log p-(1-y)\log(1-p),\] which is $-\log p$ when $y=1$ and $-\log(1-p)$ when $y=0$ (Figure 2, right). A high probability for what happened costs almost nothing, and a confident mistake costs a lot. In class, two models predicted an event that did not happen, one with $p=0.60$ and one with $p=0.99$. The first lost $-\log(0.40)=0.92$, the second $-\log(0.01)=4.61$. A count of errors at a cutoff of 0.5 calls both of them one mistake; the log loss charges the model that was sure and wrong.

Log loss is maximum likelihood. For one observation, the probability of the observed outcome is \[P(Y=y\mid p)=p^y(1-p)^{1-y},\] which is $p$ when $y=1$ and $1-p$ when $y=0$. Its negative log is exactly the log loss. For independent observations the likelihood is a product, so the total log loss is minus the log-likelihood, and fitting the model by minimizing the total log loss is maximum likelihood. It is the same story as Section 8, where least squares is maximum likelihood under normal errors. Unlike least squares, there is no formula for the best coefficients: software climbs the log-likelihood in steps.

Recipe

Compute the score $z=\beta_0+\beta_1x_1+\dots$ for the case. Watch the signs and the 0/1 coding of yes-or-no predictors.
Turn the score into a probability, $p=\sigma(z)$. For a negative score use $\sigma(-z)=1-\sigma(z)$.
Interpret a coefficient on the log-odds scale: “one more unit of $x$ raises the log-odds of $y=1$ by $\beta$, holding the other predictors fixed.” Never “raises the probability by $\beta$.”
To trade two predictors, divide their coefficients: a coefficient of $-2$ offsets four units of a predictor with coefficient 0.5.
For the log loss, find the probability the model gave to the outcome that happened, $p$ if $y=1$ and $1-p$ if $y=0$, and take minus its log.

Worked example 9.1: NBA point spreads (class)

The class model for the probability that the favorite wins an NBA game uses the point spread $s$ and no intercept: \[\log\frac{p}{1-p}=0.156\,s.\]

Spread 0: $z=0$ and $p=0.50$. Without an intercept, a game with no favorite is a coin flip.
Spread 4: $z=0.156(4)=0.62$ and $p=\sigma(0.62)=0.65$.
Spread 8: $z=1.25$ and $p=0.78$.
Each extra point of spread adds 0.156 to the log-odds, which multiplies the odds of a win by $e^{0.156}=1.17$.

What it means: doubling the spread from 4 to 8 does not double the probability, which goes from 0.65 to 0.78. The same step in log-odds buys less and less probability as the favorite gets stronger, and the probability never reaches 1. What would change it: the model has no intercept, so it says that a spread of 0 is exactly 50%. If home teams won more often than the spread says, an intercept would shift the whole curve.

Worked example 9.2: a hospital model by hand

A hospital models whether a patient is readmitted within 30 days of discharge ($y=1$) with \[z=-3+0.5\,\text{visits}+1\,\text{alone},\] where $\text{visits}$ is the number of emergency room visits in the past year, and $\text{alone}$ is 1 if the patient lives alone and 0 otherwise.

Probabilities. A patient with no visits who does not live alone has $z=-3$ and $p=0.05$. A patient with 4 visits who lives alone has $z=-3+2+1=0$ and $p=0.50$.
Coefficients. Each extra visit raises the log-odds of readmission by 0.5, holding living arrangements fixed. Living alone raises them by 1, as much as two extra visits.
Equal steps in $z$, unequal steps in $p$. For patients who live alone, going from 4 to 6 visits moves $z$ from 0 to 1 and $p$ from 0.50 to 0.73, up 0.23. Going from 8 to 10 visits moves $z$ from 2 to 3 and $p$ from 0.88 to 0.95, up only 0.07.
Log loss. A patient who lives alone with 8 visits ($p=0.88$) was not readmitted, so $y=0$ and the loss is $-\log(1-0.88)=-\log(0.12)=-\log\sigma(-2)=2.1$. A patient who lives alone with 4 visits ($p=0.50$) was readmitted: the loss is $-\log(0.50)=0.69$. A patient with no visits who does not live alone ($p=0.05$) was readmitted: the loss is $-\log(0.05)=3.0$.

What it means: the costliest prediction is the one that gave the outcome only a 5% chance. A prediction of 0.50 costs 0.69 whatever happens: it is the price of saying “I don’t know.” What would change it: the coefficients describe the patients in the hospital’s data. They say how readmission goes together with visits and living alone, not how one patient’s risk would change if that patient moved in with someone.

See it move

In the Logistic Regression Lab, the red S-curve is the class NBA model, $\log\frac{p}{1-p}=0.156\times\text{spread}$. Predict first: the win probability at a spread of 0, and whether 4 more points of spread add more probability starting from a spread of 0 or from a spread of 12. Then use Predict at a spread to check 0, 4, 8, 12, and 16. Press New season a few times: the maximum likelihood fit to each simulated season lands near 0.156, but not on it. (Answers: 0.50; from 0, about 15 percentage points, against about 6 from 12.)

Your turn

Problem 9.1 (the sigmoid, book Chapter 13).

Show that $\sigma(-z)=1-\sigma(z)$.
Use the table and part (a) to find $\sigma(0)$, $\sigma(2)$, and $\sigma(-2)$.
The slope of the sigmoid is $\sigma'(z)=\sigma(z)\big(1-\sigma(z)\big)$. Compute it at $z=0$ and at $z=2$. What does the difference say about how much a coefficient moves the probability?

Solution

$\sigma(-z)=\dfrac{1}{1+e^{z}}$. Multiply the top and the bottom by $e^{-z}$: $\sigma(-z)=\dfrac{e^{-z}}{e^{-z}+1}$. And $1-\sigma(z)=\dfrac{1+e^{-z}-1}{1+e^{-z}}=\dfrac{e^{-z}}{1+e^{-z}}$. The two are equal.
$\sigma(0)=1/(1+1)=0.5$. $\sigma(2)=0.88$ from the table, so $\sigma(-2)=1-0.88=0.12$.
At $z=0$: $0.5\times 0.5=0.25$. At $z=2$: $0.88\times 0.12\approx 0.11$. Near the middle, a small change $\Delta$ in the log-odds changes $p$ by about $0.25\,\Delta$, so a coefficient $\beta$ moves $p$ by about $\beta/4$ per unit of $x$. At $z=2$ the same change moves $p$ by less than half as much. In the NBA model, one more point of spread in an even game adds about $0.156/4\approx 0.04$, four percentage points.

Problem 9.2 (accepting an offer). A university models whether an admitted student accepts its offer ($y=1$) with \[z=-2+1\,\text{visit}+0.5\,\text{aid},\] where $\text{visit}$ is 1 if the student came to a campus visit day and $\text{aid}$ is the scholarship in thousands of dollars.

Find the probability of accepting for a student with no visit and no aid, for a visitor with $2,000 of aid, and for a visitor with $4,000 of aid.
Interpret the coefficient on aid. How much aid is a campus visit worth on the log-odds scale?
A dean says: “Doubling a visitor’s aid from $2,000 to $4,000 doubles the chance that the student accepts.” Is the dean right?
For visitors, compare the effect of an extra $2,000 of aid on a student who already has $2,000 with its effect on a student who already has $6,000. Both add 1 to the log-odds. Why do they move the probability by different amounts?

Solution

No visit, no aid: $z=-2$ and $p=0.12$. Visitor with $2,000: $z=-2+1+0.5(2)=0$ and $p=0.50$. Visitor with $4,000: $z=-2+1+2=1$ and $p=0.73$.
Each extra $1,000 of aid raises the log-odds of accepting by 0.5, holding the visit fixed. A visit, with coefficient 1, is worth $1/0.5=2$ units of aid, that is $2,000.
No. Going from $2,000 to $4,000 adds 1 to the log-odds, and the probability goes from 0.50 to 0.73, not to 1. A coefficient adds to the log-odds; it does not multiply the probability.
From $2,000 to $4,000, $z$ goes from 0 to 1 and $p$ from 0.50 to 0.73, up 0.23. From $6,000 to $8,000, $z$ goes from 2 to 3 and $p$ from 0.88 to 0.95, up 0.07. The S-curve is steep in the middle and flat near 1: a student who is already likely to accept has little room left to move.

Problem 9.3 (scoring a forecaster). A weather app gives the chance of rain each morning. On four days:

Day	1	2	3	4
Chance of rain $p$	0.88	0.88	0.50	0.05
Rained? $y$	1	0	1	1

Compute the log loss for each day, and the total.
Which day costs the most, and why?
A lazy app says 50% every day. What is its log loss each day, and its total over these four days? Which app scores better here, and what does that say about confidence?
Write the probability of one day’s outcome as a function of $p$. Explain why the app with the smallest total log loss is the one under which the observed weather is most likely.

Solution

Day 1: $y=1$, loss $-\log 0.88=0.13$. Day 2: $y=0$, loss $-\log(1-0.88)=-\log 0.12=2.1$. Day 3: $-\log 0.5=0.69$. Day 4: $y=1$, loss $-\log 0.05=3.0$. Total $0.13+2.1+0.69+3.0=5.92$.
Day 4. The app gave rain a 5% chance and it rained: the most confident mistake costs the most. Day 2 is also a confident mistake, 88% for rain that did not come.
$-\log 0.5=0.69$ every day, whatever happens, so the total is $4\times 0.69=2.76$. The lazy app scores better on these four days, because the first app made two confident mistakes in four days. Confidence pays only when it is earned: an app that says 88% should see rain on about 88% of those days. Over many days, an informative app that is honest about its uncertainty beats the lazy one.
$P(Y=y\mid p)=p^y(1-p)^{1-y}$. Treating the days as independent given the forecasts, the probability of the whole record is the product over days, and the total log loss is minus its log. The smallest total log loss is therefore the largest probability of the observed weather: maximum likelihood.

Problem 9.4 (true or false, book Chapter 13 and class). Answer TRUE or FALSE, with a one-sentence justification.

Logistic regression coefficients represent the change in probability for a one-unit increase in the predictor.
The odds ratio $e^{\beta}$ for a predictor is always positive.
If the score is $z=0$, the model’s probability is 0.5.
In the NBA model, doubling the spread doubles the probability that the favorite wins.
Two models predict $p=0.60$ and $p=0.99$ for an event that does not happen. Both are wrong at a cutoff of 0.5, so they receive the same log loss.
Minimizing the total log loss gives the maximum likelihood estimates of the coefficients.

Solution

FALSE. They are changes in the log-odds. The change in probability depends on where the case starts on the S-curve.
TRUE. $e^{x}>0$ for every $x$. A negative $\beta$ gives an odds ratio between 0 and 1: the odds go down.
TRUE. $\sigma(0)=1/(1+e^0)=1/2$.
FALSE. From a spread of 4 to 8 the probability goes from 0.65 to 0.78. The log-odds double; the probability does not.
FALSE. The losses are $-\log 0.40=0.92$ and $-\log 0.01=4.61$. The log loss charges the confident mistake more.
TRUE. The total log loss is minus the log-likelihood of independent Bernoulli outcomes, so minimizing one maximizes the other.

Traps

Reading a coefficient as a change in probability. It is a change in log-odds.
Using $p$ in the log loss when $y=0$. The loss uses the probability of what happened, $1-p$.
Reading $\sigma(-2)$ as 0.88. A negative score gives a probability below 0.5: $\sigma(-2)=1-0.88=0.12$.
Adding probability changes from several predictors. Add on the log-odds scale, then convert once.
Forgetting that log is the natural logarithm.

Part V. Putting it together

10. Concept check

Problem 10.1 (true or false). Answer TRUE or FALSE, with a one-sentence justification. On the exam the justification earns the point.

You roll two fair three-sided dice. The probability that they show the same number is 1/4.
If $A$ and $B$ are independent, $P(A)>0$, and $P(B)>0$, then $P(A\text{ and }B)>0$.
If $P(A\mid B)=0.5$ and $P(B)=0.5$, then $A$ and $B$ are independent.
A box has three drawers: one holds two gold coins, one two silver coins, and one a gold and a silver coin. You pick a drawer at random and a coin from it at random, and the coin is gold. The probability that the drawer holds two gold coins is 50%.
45% of students play golf, 55% play tennis, and 70% play at least one of the two. Then 15% play golf but not tennis.
550 of 100,000 emails are spam. Half of all spam contains the word “bigger”, and 2% of other emails do. An email containing “bigger” is spam with probability about 12%.
Of 140 night club customers, 77 of the 91 under 30 are single and 28 of the 49 over 30 are single. Age and being single are independent.
If two random variables have nonzero correlation, they must be dependent.
If $X\sim\text{Binomial}(10,\ 0.3)$, then $Var(X)=3$.
The average of 400 independent measurements has one-twentieth the standard deviation of a single measurement.
A $z$-score of $-2$ means the value is 2 units below the mean.
In a regression with an intercept, the least squares residuals sum to zero and the line passes through $(\bar x,\bar y)$.
In the Saratoga fit, the slope 113.12 means that adding 1,000 square feet to a house will raise its price by $113,120.
If two columns of the design matrix are identical, the least squares predictions are not unique.
In the Saratoga fit, a house with a standardized residual of 3.4 sold for $3,400 more than predicted.
Converting prices from thousands of dollars to dollars multiplies the slope, the intercept, and $\hat\sigma$ by 1,000 and leaves the standardized residuals unchanged.
In a logistic regression, a coefficient of 0.5 means that one more unit of $x$ raises the probability that $y=1$ by 0.5.
A prediction of $p=0.5$ has the same log loss whether $y=0$ or $y=1$.

Solution

FALSE. There are 9 equally likely pairs and 3 of them match: $3/9=1/3$.
TRUE. $P(A\text{ and }B)=P(A)P(B)>0$.
FALSE. Independence needs $P(A\mid B)=P(A)$, and nothing says $P(A)=0.5$.
FALSE. Of the three gold coins, two are in the gold-gold drawer, so the probability is $2/3$. Bayes: $\frac{1\cdot\frac13}{1\cdot\frac13+\frac12\cdot\frac13+0}=\frac23$.
TRUE. $P(\text{both})=0.45+0.55-0.70=0.30$, so golf but not tennis is $0.45-0.30=0.15$.
TRUE. $\frac{0.5(0.0055)}{0.5(0.0055)+0.02(0.9945)}=\frac{0.00275}{0.02264}=0.121$.
FALSE. Among those under 30, $77/91=0.85$ are single; among those over 30, $28/49=0.57$.
TRUE. Independent variables have zero correlation, so nonzero correlation rules out independence.
FALSE. $Var(X)=10(0.3)(0.7)=2.1$; 3 is the mean.
TRUE. $\sigma/\sqrt{400}=\sigma/20$.
FALSE. It means 2 standard deviations below the mean.
TRUE. Both follow from the least squares formulas: $\hat a=\bar y-\hat b\bar x$ puts $(\bar x,\bar y)$ on the line.
FALSE. The slope compares different houses in the data; it is not the effect of an addition to one house.
FALSE. The coefficients are not unique, but the predictions are.
FALSE. It sold 3.4 residual standard errors above the line: $3.4\times\$69{,}100$, about $235,000.
TRUE. Every price, prediction, and residual is multiplied by 1,000, and the ratio $e/\hat\sigma$ does not change.
FALSE. It raises the log-odds by 0.5. The change in probability depends on where the case starts on the S-curve, and it is never more than about $0.5/4=0.125$.
TRUE. Either way the model gave the outcome probability 0.5, and the loss is $-\log 0.5=0.69$.

How to sit the exam

First five minutes. Read every problem. Start with the one you are surest about.
Budget. About 1.5 minutes per point. If a part takes twice its share, write down your setup and move on; come back at the end.
Set up before you compute. Name the events or the random variable, write the formula in symbols, then substitute numbers. Most partial credit is in the setup.
Arithmetic. There is no calculator. Write the formula in symbols, keep fractions as fractions, and cancel before you divide. A fraction such as $3/8$ is a complete answer; for a decimal, two or three places are plenty. The next section shows the shortcuts.
Interpret. One sentence, in the units of the problem, about the people or things in the problem.
What would change. Name the assumption (independence, the prior, normality, a straight line, a constant $p$) and say which way the answer would move.
Normal problems. Standardize, then compare $z$ with 0 and $\pm 1.96$. When the 1.96 fact does not give an exact probability, the question is asking you to compare or bound instead.
Logistic problems. Compute the score $z$ first, then read $p$ from the table. For a negative score, or for $1-p$, use $\sigma(-z)=1-\sigma(z)$. Interpret a coefficient as a change in log-odds.
Sanity checks. Probabilities are between 0 and 1; a conditional distribution sums to 1; a variance is positive; residuals sum to zero; a shaded area smaller than half the bell gives a number below 0.5.
True or false. The justification earns the point. A one-line calculation or a counterexample is the best justification.

Arithmetic without a calculator

The exam’s numbers are chosen so that every calculation is short. A few habits make hand arithmetic fast and safe.

Fractions first. Keep fractions until the end and cancel before you divide: $0.42/0.45=42/45=14/15\approx 0.93$.
Clear the decimals. Multiply the top and the bottom by the same power of 10: $0.008/0.107=8/107$, a little less than $8/100$, so about $0.075$.
Products of decimals. Multiply the digits, then count the decimal places: $0.03\times 0.7=0.021$ and $0.25\times 0.36=0.09$.
Dividing by a simple decimal. Dividing by 0.5 is multiplying by 2; by 0.25, by 4; by 0.2, by 5; by 0.4, by 2.5.
Powers. Square, then square again: $0.9^2=0.81$ and $0.9^4=0.81^2=0.6561$; $0.7^2=0.49$ and $0.7^3=0.343$. A power of $\tfrac12$ is $1/2^n$.
Square roots. Look for a perfect square first: $\sqrt{2.25}=1.5$, $\sqrt{0.0016}=0.04$. Otherwise bracket the root between two squares you know: $1.1^2=1.21$ and $1.2^2=1.44$, so $\sqrt{1.3}$ is about 1.14.
Multiplying by 1.96. $1.96\,\sigma=2\sigma-0.04\,\sigma$. For example, $1.96\times 15=30-0.6=29.4$.
Logs and the sigmoid. The logistic problem gives a table of $\sigma(z)$ and $\log\sigma(z)$. For $1-p$, flip the sign of $z$: if $p=\sigma(2)=0.88$, then $1-p=\sigma(-2)=0.12$, and $-\log(1-p)=-\log\sigma(-2)=2.1$.
Values the exam gives you. If a part needs a power that is slow by hand, such as $0.97^{20}$, the problem states its value. Use it.

Arithmetic warm-up. Compute by hand, then check with the solution.

$0.0045/0.09$
$0.6(0.2)+0.4(0.7)$
$0.9^3$
$\sqrt{1.69}$
$1.96\times 25$
$\binom52(0.1)^2(0.9)^3$
The probability that goes with odds of 1 to 4 in favor.
$\dfrac{0.02}{0.02+0.18}$

Solution

$45/900=0.05$.
$0.12+0.28=0.40$.
$0.81\times 0.9=0.729$.
$1.3$, because $13^2=169$.
$50-1=49$.
$10(0.01)(0.729)=0.0729$.
$1/(1+4)=0.2$.
$2/(2+18)=2/20=0.1$.

What to put on your cheat sheet

Write the sheet yourself: making it is studying. A suggested list, one block per section:

Probability. Complement; addition rule; $P(A\mid B)=P(A\text{ and }B)/P(B)$; multiplication rule; the independence test; total probability; Bayes rule; posterior odds $=$ Bayes factor $\times$ prior odds; $P=O/(1+O)$ for odds in favor.
Expectation and variance. $E(X)=\sum x\,p(x)$; $Var(X)=E(X^2)-E(X)^2$; $Cov$ and $Corr$; $E(aX+bY)$ and $Var(aX+bY)$ with the covariance term; a sum of $n$ independent draws has mean $n\mu$ and standard deviation $\sqrt n\,\sigma$, and an average has $\mu$ and $\sigma/\sqrt n$; total expectation.
Binomial. The four conditions; $P(X=x)=\binom nx p^x(1-p)^{n-x}$; $np$; $np(1-p)$; $sd(\hat p)=\sqrt{p(1-p)/n}$.
Normal. $z=(x-\mu)/\sigma$; 95% lies within $\mu\pm 1.96\sigma$, so 2.5% in each tail and 50% on each side of $\mu$; rank tail probabilities by comparing $z$-values; the level exceeded only 2.5% of the time is $\mu+1.96\sigma$; the central limit theorem.
Regression. $\hat b$ and $\hat a$; $e=y-\hat y$; $SSE$ and $J=SSE/(2n)$; the influence $\Delta(x_i-\bar x)/S_{xx}$; $\hat y\pm 1.96\hat\sigma$; $e/\hat\sigma$; least squares equals maximum likelihood under normal errors; the residual-plot table.
Logistic regression. $z=\beta_0+\beta_1x$; $p=\sigma(z)=1/(1+e^{-z})$; $\sigma(-z)=1-\sigma(z)$; $z=\log\frac{p}{1-p}$; a coefficient changes the log-odds, and $e^{\beta}$ multiplies the odds; log loss $-y\log p-(1-y)\log(1-p)$; $P(Y=y\mid p)=p^y(1-p)^{1-y}$, so the total log loss is minus the log-likelihood.
Arithmetic. The squares from $1^2$ to $20^2$, so you recognize a perfect square such as $2.25=1.5^2$, and $1.96\,\sigma=2\sigma-0.04\,\sigma$.
One worked example per section, in your own handwriting.

Mistakes that cost points

Dividing by the wrong total in a conditional probability.
Confusing $P(A\mid B)$ with $P(B\mid A)$, including the prosecutor’s fallacy.
Leaving the base rate out of Bayes rule.
Multiplying probabilities without independence, or adding probabilities of events that overlap.
Dropping the covariance term, or forgetting to square the weights, in a variance.
Using the variance where the standard deviation belongs, in a $z$-score or in $N(\mu,\sigma^2)$.
Putting all 5% in one tail: 95% within $\pm 1.96\sigma$ leaves 2.5% on each side.
Writing the standard deviation of a sum of $n$ draws as $n\sigma$ instead of $\sqrt n\,\sigma$.
Reading a slope as a causal effect, or an intercept as meaningful outside the data.
Calling a case an outlier without its standardized residual.
Reading a logistic coefficient as a change in probability, or using $p$ instead of $1-p$ in the log loss when $y=0$.
A number with no sentence. Every exam problem asks what the number means.

\(x\)	\(y\)	\(x-\bar x\)	\(y-\bar y\)	product	\((x-\bar x)^2\)
0	1	\(-1\)	\(-4/3\)	\(4/3\)	1
1	3	0	\(2/3\)	0	0
2	3	1	\(2/3\)	\(2/3\)	1
Sum		0	0	2	2

\(z\)	\(-3\)	\(-2\)	\(-1\)	\(0\)	\(1\)	\(2\)	\(3\)
\(\sigma(z)\)	0.05	0.12	0.27	0.50	0.73	0.88	0.95
\(\log\sigma(z)\)	\(-3.0\)	\(-2.1\)	\(-1.3\)	\(-0.69\)	\(-0.31\)	\(-0.13\)	\(-0.05\)