Polson–Sokolov Lab · Logistic regression

Logistic Regression Lab

Each dot is a simulated NBA game: the Vegas point spread on the x-axis, and whether the favorite won (1) or lost (0). The red S-curve is the reference model with \(\beta = 0.156\). Fit the model by maximum likelihood, read off win probabilities, and move the cutoff to see the confusion matrix and the ROC curve change.

← All labs

Based on Chapter 13, Logistic Regression and Generalized Linear Models, of Polson and Sokolov, Bayes, Artificial Intelligence, and Deep Learning (vsokolov.org/html/_book).

Simulated game (jittered) Observed win rate, 2-point bins Reference model, β = 0.156 MLE fit to these games Cutoff t Chosen spread
ROC curve · fitted p̂ as the score
Confusion matrix · predict a win if p̂ > 0.50
Predicted 1Predicted 0
Actual 1
(fav. won)		–TP–FN
Actual 0
(fav. lost)		–FP–TN
Accuracy (TP + TN) / n–
Sensitivity TP / (TP + FN)–
Specificity TN / (TN + FP)–
Precision TP / (TP + FP)–
Area under the ROC curve–
From log-odds to a decision

One coefficient, three scales: log-odds, odds, probability

The model is linear on the log-odds scale. Exponentiate to get odds, then divide odds by one plus odds to get a probability. These buttons load the worked cases.

  1. Linear regression fails for a yes/no outcome

    A straight line through 0/1 data predicts values below 0 and above 1 once the spread is large enough. We want \(P(Y = 1 \mid X)\), which must stay in \((0, 1)\). The logistic curve does that for any value of \(\beta_0 + \beta_1 X\).

  2. Read \(\beta\) on the log-odds scale

    Each extra point of spread adds 0.156 to the log-odds that the favorite wins. Equivalently, it multiplies the odds by \(e^{0.156} = 1.17\). The change in probability per point is not constant: the curve is steepest at 50% and flattens near 100%.

  3. Maximum likelihood picks \(\beta\)

    There is no least-squares formula here. R's glm, and this page, climb the log-likelihood with Newton–Raphson steps (iteratively reweighted least squares). The standard error comes from the curvature of the log-likelihood at the top.

  4. A cutoff turns probabilities into decisions

    With no intercept and a positive spread, every \(\hat p\) is above 0.5, so the 0.5 cutoff always picks the favorite. Sensitivity is 100% and specificity is 0%. Raise the cutoff and the model starts calling upsets in close games. The ROC curve shows every cutoff at once, and its area (AUC) is the chance that a random win gets a higher \(\hat p\) than a random loss.

The logistic model
\(\displaystyle \log\left(\frac{p}{1 - p}\right) = \beta_0 + \beta_1 X_1 + \dots + \beta_p X_p\)
\(\displaystyle p = \frac{e^{\beta_0 + \beta_1 X_1 + \dots}}{1 + e^{\beta_0 + \beta_1 X_1 + \dots}}\)
NBA, glm(favwin ~ spread − 1)
\(\displaystyle \text{Spread 4: } 0.156 \times 4 = 0.62 \to \frac{e^{0.62}}{1 + e^{0.62}} = 65\%\)
\(\displaystyle \text{Spread 8: } 0.156 \times 8 = 1.25 \to \frac{e^{1.25}}{1 + e^{1.25}} = 77\%\)
Newton–Raphson step, with p = logit⁻¹(Xβ), W = diag(p(1 − p))
\(\displaystyle \beta \leftarrow \beta + (X^\top W X)^{-1} X^\top (y - p)\)
\(\displaystyle \text{SE}(\hat\beta) = \sqrt{\operatorname{diag}\,(X^\top W X)^{-1}}\)
Confusion-matrix metrics
\(\displaystyle \text{Accuracy} = (\text{TP} + \text{TN}) / n\)
\(\displaystyle \text{Sensitivity} = \text{TP} / (\text{TP} + \text{FN})\)
\(\displaystyle \text{Specificity} = \text{TN} / (\text{TN} + \text{FP})\)
\(\displaystyle \text{Precision} = \text{TP} / (\text{TP} + \text{FP})\)

Takeaway for students. The reference 65% and 77% come straight from plugging the spread into the fitted curve. The market looks calibrated because the curve passes through the observed win rates. Whether to call a game for the underdog is a separate business decision, set by the cutoff and the cost of each kind of mistake.

Two predictors, a straight-line boundary

Tennis: break points won by each player

Each dot is one simulated Grand Slam match, placed at the break points won by player 1 (x) and player 2 (y). The reference fit glm(Result ~ BPW.1 + BPW.2 − 1) on 943 real matches from 2013. With two predictors, the set of matches where \(\hat p = 0.5\) is a straight line.

Player 1 won (simulated) Player 2 won (simulated) Reference boundary, p̂ = 0.5 p̂ = 0.25 and 0.75 MLE boundary Chosen match

Symmetry in the tennis coefficients

\(\hat\beta_1 = 0.402\) and \(\hat\beta_2 = -0.418\) are close to equal and opposite. Winning a break point helps player 1 about as much as losing one hurts. So the boundary is close to the 45° line \(\text{BPW.2} = \text{BPW.1}\), tilted slightly by the ratio \(0.402 / 0.418 = 0.96\).

\[ \text{boundary: } \text{BPW.2} = (0.402 / 0.418) \times \text{BPW.1} \]

Effect size: a three-break gap

The reference fit approximates both coefficients by 0.40. Then a gap of three break points gives \(0.40 \times 3 = 1.2\) on the log-odds scale, and \(\hat p = 77\%\), up from 50% at parity. That is 27 percentage points.

\[ \operatorname{logit}^{-1}(1.2) = \frac{e^{1.2}}{1 + e^{1.2}} = 0.77 \]

LinkedIn: coefficients as odds multipliers

In the LinkedIn example a top MBA has \(\hat\beta = +1.16\), so it multiplies the odds of becoming VP, CXO or partner by \(e^{1.16} = 3.19\). A year of experience multiplies them by \(e^{0.09} = 1.09\). The ratio \(1.16 / 0.09 \approx 13\) is the "worth 13 years" result.

Featureβ̂eβ̂
Top MBA+1.163.19
Function count+0.261.30
Years of experience+0.091.09
Industry count−0.220.80

Try this in class

  1. Load "Spread = 4" and walk the chain in the panel: 0.624 on the log-odds scale, odds of 1.87, probability 65%. Ask students to do spread 8 by hand before pressing the second button.
  2. Set \(n = 100\) and press "New season" a few times. Watch \(\hat\beta\) bounce around 0.156 and its SE grow. Set \(n = 2{,}000\) and the SE shrinks to about \(\sqrt{100/2000} \approx 0.22\) of its \(n = 100\) size.
  3. At the 0.5 cutoff the model always picks the favorite. Ask why specificity is 0%. Then slide the cutoff to 0.75 and read the new sensitivity and specificity off the matrix and the ROC point.
  4. Tick "Fit an intercept too". \(\hat\beta_0\) lands near 0, which is what a calibrated market implies: a spread of 0 should mean a 50/50 game.
  5. In the tennis plot, click a match at 8 vs 5 and another at 3 vs 0. Both have a three-break gap. Compare their probabilities and explain the difference using \(\hat\beta_1 \ne -\hat\beta_2\).