Three situations

Map for the hour

• The trap is mixing 2 and 3. The machine sounds like 2.

1. No data

• No sample from the process you will decide for.
• Probability is a language for uncertainty. Usually lack of information, not a property of the coin.
• Operational answer: a price you would bet at.
• The number has to cohere. Otherwise someone books you.

How many coins in my pocket?

Probability is a property of what you know.

Probability as a bet

The coin is risk

• Most people agree on $P(\text{Heads})=1/2$. We elicit that number as a willingness to play.
• Will you pay $1 to receive $2 if the coin lands tails, and $0 if it lands heads? If you are indifferent, your $P(\text{Tails})$ is $1/2$.
• Probability and payoff arrive together. You cannot separate them.
• Litowitz asks the same question, then raises the payoff. Pay $1 for a fair coin that pays $2 on heads. Most people pause. Make it $3 on heads. Almost everyone says yes.
• We know the possibilities and we know the odds, fifty-fifty. That is risk.
• The mudslide tree was the same object: $0.01$ from historical seasons, $0.95$ treated as known.

Dutch book

• If you set $P(A)=0.7$ and $P(\bar A)=0.2$, someone can lock in a loss against you.
• Pay $0.70 on $A$ and $0.20 on $\bar A$. You pay $0.90 and receive $1 either way.

• Coherence means no Dutch book exists. In finance that is no-arbitrage.
• The leak is $P(A)+P(\bar A)=0.9\neq 1$.

Kolmogorov’s axioms

How do we estimate a probability?

2. Good data

• A sample from the process you will decide for.
• Then you estimate: n/N, a mean, a Poisson $\lambda$.
• Trust in the usual sense: more good data, smaller variance around the right number.
• Frequency, then a sampling distribution. Alpha particles after the warning.

Frequency

• Event $A$ is heads.
• Each grey path is one sequence of tosses. The blue path is one of them. The red line is $1/2$.

As $N$ grows, $n/N$ settles near $1/2$.

• That is the law of large numbers.
• von Mises wrote the limit as the definition: $$P(\text{Heads}) = \lim_{N \to \infty} \frac{n}{N}.$$

Fitting a probability law

• Fit a Poisson or a Normal to observed data.
• Most laws depend on a few parameters: $\lambda$, or $\mu$ and $\sigma$.
• Estimate those from the sample. Then the distribution is specified.
• Then ask whether the fit is any good. Later.

3. Bad data

• Bias, not variance. We name this after sampling distributions: Literary Digest, WHI.
• Wrong law is the same trap: a tight Normal on the sample, a crash the model called impossible.

The 1987 crash

• October 1987: S&P 500 dropped $-21.76\%$.
• Prior monthly returns: mean $1.2\%$, volatility $4.3\%$.
• The model fitted to earlier months: $X \sim N(0.012, 0.043^2)$.
• Standardize: $$Z = \frac{-0.2176 - 0.012}{0.043} = -5.34.$$
• A 5-sigma month. Under that Normal, $P(Z < -5.34) \approx 4.6 \times 10^{-8}$.

The 1987 crash: the model

• We assumed returns are Normal. The middle of the histogram looks like 2.
• The tails are not. Extreme months happen. The crash is one of them.
• Same picture as the orange curve: a law fitted to the sample, a target the law cannot see.
• Week 1 computed the z-score. The lesson here is the assumption.

The limit is not always a method

• Same plot as 2: $n/N$ settling toward a number. That is what makes 3 dangerous.
• The von Mises limit needs an infinite sequence of repeatable trials. You never observe it.
• Ville pointed this out. Bernoulli’s workable version is the finite ratio $n/N$ in the experiment you actually ran.
• It requires the same conditions. Change the coin, and the number changes.
• Blue is fair. Orange is $P(\text{Heads})=0.7$. Same $N$. Two different samples. Two different limits.

Data-driven AI

• Modern AI is data-driven. The rest of the course is this approach: estimate from samples, then predict.
• A model trained on a limited sample is a frequency from that sample. The interval can be tiny around the wrong number.
• Very common in modern AI: confident when it should not be. The machine sounds sure. The sample was not the population.

Data is a currency

• Self-driving is sold as a data race: the fleet that logs the most miles trains the better driver. Tesla’s pitch. Waymo’s counter: unsupervised miles are not the same as a human still in the seat.
• Data sells. Spirit Airlines, in bankruptcy in 2026, auctioned an operational archive. Google’s bid was $10 million: emails, chats, operations, for AI.
• The named passenger list was carved out of that Google deal. The estate kept the right to sell it to travel buyers.
• Ars Technica, September 2026.

From posting a price to estimating one

• Whenever we have a sample we should estimate from it.
• First a pattern with no named law: correlation. Two series, one number $r$.
• Then a law with a named parameter. The alpha counts: we estimate $\lambda$ in the open.
• Good data if the sample is from the process you will decide for. Bad data is the standing risk: same formulas, wrong population.
• The mudslide $0.01$ was a frequency from historical data, then treated as known.
• Working mechanism after the pattern: frequency. $n/N$, likelihood, sampling distributions.

Why Pattern Matching?

• Traditional approach: estimating parameters when functional form is known
  • Linear relationships (Earth-shape example)
  • Multiplicative relationships (Newton’s $F=ma$)
• Modern challenge: relationship itself is unknown
  • Human behavior, natural language
  • Complex, non-mathematical relationships
• Pattern matching: identifying recurring sequences, relationships, or structures
  • Like finding puzzle pieces that complete a picture

Early Example: Mammography

Feynman on Pattern Matching

• Experts develop pattern recognition abilities
• Chess players don’t calculate every move
• They recognize advantageous patterns
• Skill transfers to other domains

Machine vs Human Intelligence

Evolution in Chess:

• 1996: Computers stronger than grandmasters
  
• AlphaZero: Self-learning through competition
• Neural networks learn features vs human heuristics
• Preference for initiative, speed, momentum over material

Correlations: Simplest Pattern Matching

Correlation coefficient: measures strength of linear relationship

$$r = \frac{\text{cov}(X,Y)}{\sqrt{\text{var}(X)} \sqrt{\text{var}(Y)}}$$

Range: $-1 \leq r \leq 1$

Applications:

• Financial market relationships
• Portfolio diversification
• Risk assessment

Google Stock Example

Unsupervised Learning: Election Forecasting

2012 Obama Election Prediction

• Goal: predict probability of Obama winning
• Data: polling data across 50 states + DC
• Method: Bayesian simulation with Dirichlet distribution
• Technique: Monte Carlo simulation (10,000 runs)

Loading the data and aggregating it by state.

Display the data

Run the simulation and plot probabilities by state.

Probabilities of Obama winning by state

Simulate the probability of Obama winning the election.

Calculate the probability of Obama having 270 EV or more

[1] 0.9631

Plot the distribution of the simulated electoral votes for Obama.

Beta distribution

Dirichlet Distribution

• Same idea for $k$ shares that sum to 1. Poll: Republican, Democrat, other.
• $k=2$ is Beta: $\mathrm{Dirichlet}(\alpha,\beta)=\mathrm{Beta}(\alpha,\beta)$.
• Parameters $\alpha=(\alpha_1,\ldots,\alpha_k)$, each $\alpha_i>0$.
• Support: $x_i\ge 0$ and $\sum_i x_i=1$. $$f(x\mid\alpha)=\frac{\Gamma(\sum_i\alpha_i)}{\prod_i\Gamma(\alpha_i)}\prod_{i=1}^k x_i^{\alpha_i-1}.$$

Dirichlet Distribution

Application in Election Forecasting:

• Models uncertainty in polling percentages
• Three components: Republican, Democrat, Undecided
• Parameters reflect poll sample sizes and percentages
• Generates probability distributions for each candidate’s support

Why Use Dirichlet for Polling?

• Naturally handles the constraint that percentages sum to 100%
• Incorporates polling uncertainty
• Allows for Monte Carlo simulation of election outcomes
• Conjugate prior for multinomial likelihood (Bayesian convenience)

2008 Election Example

Results of recent state polls in the 2008 United States Presidential Election between Barack Obama and John McCain.

Case Study: Emissions of Alpha Particles

• Berkson (1966) conducted a careful analysis of data obtained from the National Bureau of Standards.
• The source of the alpha particles was americium 241.
• n is the number of counts that were observed in 1207 intervals, each of length 10 sec. 
• In 18 of the 1207 intervals, there were 0, 1, or 2 counts; in 28 of the intervals there were 3 counts, etc.

  n Observed Expected
1 1       18     12.2
2 3       28     27.0
3 4       56     56.5
4 5      105     94.9
5 6      126    132.7
6 7      146    159.1

[1] 0.8354598

Case Study: Emissions of Alpha Particles

To fit a Poisson distribution, we need to estimate its single parameter, λ.

$$P(X=k) = \frac{\lambda^k e^{-\lambda}}{k!}$$

A natural choice for the estimate, $\hat{\lambda}$, is the sample mean of the observed counts.

• Estimate: A value for an unknown parameter based on data.
• Sampling Distribution: The probability distribution of an estimate if the experiment were repeated. The spread of this distribution indicates the estimate’s variability.

We will explore the rationale for this choice and how to assess its reliability.

Fitting a Poisson Distribution

• In order to fit the Poisson distribution, we must estimate a value for λ from the observed data.
• Since the average count in a 10-second interval was 8.352, we take this as an estimate of λ (recall that the E(X) = λ) and denote it by $\hat{\lambda}$.

Let’s plot the observed and expected frequencies.

Sampling Distribution of Estimates

• If the experiment were to be repeated, the counts would be different and the estimate of λ would be different
• It is thus appropriate to regard the estimate of λ as a random variable which has a probability distribution referred to as its sampling distribution
• The situation is entirely analogous to tossing a coin 10 times and regarding the number of heads as a binomially distributed random variable

Understanding Variability of Estimates

• Observing 6 heads in 10 coin tosses generates one realization of a random variable
• In the same sense, 8.392 is a realization of a random variable
• Key Question: What is the sampling distribution?
• This is of practical interest, since the spread of the sampling distribution reflects the variability of the estimate
• We could ask: to what decimal place is the estimate 8.392 accurate?
• Later we will discuss the rationale for choosing to estimate λ as we have done

Normal Distribution

Normal Distribution

The density of normal is $$f(x) = \frac{1}{\sqrt{2\pi\sigma^2}} e^{-\frac{(x-\mu)^2}{2\sigma^2}}$$

The mean and variance of the normal distribution are: $$E(X) = \mu$$

Fit Normal Distribution

Now Plot the histogram and the fitted normal curve.

Parameter Estimation: The Core Idea

• We treat observed data ($x_1, x_2, \dots, x_n$) as realizations of a random variable ($X_1, X_2, \dots, X_n$).
• The joint distribution of these random variables depends on an unknown parameter, $\theta$.
• Often, we assume the $X_i$ are independent and identically distributed (i.i.d.) from a distribution $f(x|\theta)$.
• An estimate of $\theta$ is a function of the data, and thus is itself a random variable with a sampling distribution.

We will look at two general methods for creating estimates: 1. The Method of Moments 2. The Method of Maximum Likelihood

The Method of Moments

This method equates sample moments with the theoretical moments of the distribution, which are functions of the parameters.

Procedure:

1. Calculate the first few theoretical moments of the distribution, $E(X^k)$, in terms of the parameters.
2. Express the parameters as functions of these moments.
3. Replace the theoretical moments with the sample moments, $\hat{\mu}_k = \frac{1}{n}\sum_{i=1}^n X_i^k$, to get the estimates.

Method of Moments: Poisson Example

For a Poisson distribution, the first theoretical moment is the mean: $$E(X) = \lambda$$

The first sample moment is the sample mean: $$\hat{\mu}_1 = \bar{X}$$

Equating them gives the method of moments estimate for λ: $$\hat{\lambda} = \bar{X}$$

This is precisely what was done in the alpha particle example.

Method of Moments: Body Temperature Example

Let’s fit a normal distribution to human body temperature data. We’ll use the method of moments to estimate μ and σ and visualize the fit.

[1] "Estimated mu: 98.25"

[1] "Estimated sigma: 0.73"

Error = bias + variance

Courses teach variance. Life is bias.

• Variance dies like $1/\sqrt{n}$. Bias does not.
• Ten times the sample, a third the noise. The systematic miss is still there.

Internet-scale $n$ makes the blue line vanish.

• The orange line is the error that is left.
• A tight interval around the wrong number is not a good estimate.
• Week 10 is the other bias-variance: model complexity. This one is the data.

Biased data

• The sample is not the population you will decide for. More rows of the same kind do not fix that.
• Who answered, who clicked, who left a review, who got scraped into the training set.
• Literary Digest, 1936: two million replies, Landon by a landslide. Roosevelt won. The list was phone books and car registrations.
• App-store stars, Twitter as “the public,” click-through as “what people want.”
• The instrument is off. The labeler is tired. You measured the easy cases. Wald: you only see the planes that came back.
• Write the sentence: how did a row get here? If you cannot, you do not know what $\hat\theta$ is estimating.

How to detect it

• Bias does not announce itself as a wide interval. The interval is often tiny.
• Compare margins to something you trust: a census, an administrative file, a small paid panel, last year’s audited count.
• Split by source, week, device, geography. If the answer moves, a group is missing or over-weighted.
• A number that cannot be: turnout over 100%, a mean that the mechanism cannot produce.
• Add more of the same data. If the estimate does not move and you still doubt the story, that is bias, not precision.
• Ask who is absent. The missing rows are the bias.
• Variance you see by resampling. Bias you see by changing who is in the sample.

How to treat it

• You cannot average it away. Do not buy a larger pull from the same hose.
• Get the missing group, or narrow the claim to the population you actually have.
• Reweight only when you know who is missing and by how much. Otherwise you are inventing a census.
• Measure the same thing another way: a different instrument, a different window, a designed sample.
• If the decision needs an unbiased number, design the data. A survey with a frame. An experiment. A smaller honest sample beats a large biased one.
• Report the selection story with the number. A correct figure with the wrong population is still a wrong answer.

Case: Women’s Health Initiative

• Hormone replacement was supposed to protect the heart.
• Observational cohorts, including the Nurses’ Health Study, found that women on estrogen had about a third the coronary disease of women who were not.
• Guidelines followed. By 2001 about 15 million women a year were filling prescriptions.
• The rows were women who had chosen the pill: they also exercised, kept appointments, had money. A tight interval around the wrong $\hat\theta$.

WHI: how they treated it

• They treated the bias by changing who is in the sample.
• They did not buy a larger observational pull, and they did not reweight it until they knew who starts and who stays on the drug.
• Randomized 16,608 postmenopausal women to estrogen plus progestin or placebo. Stopped early.
• Coronary disease went up, about 29 percent, not down.
• Later new-user analyses of the observational sample move toward the trial, but only after the mechanism is named.
• Read Shrank, Patrick, and Brookhart, JGIM 2011 (open). The trial is Rossouw et al., JAMA 2002. Catalog of Bias: healthy users.