3. Classification and generalization

Predicting whether the favorite wins from the point spread. No intercept: a spread of zero is a probability of one half.

• Every 1-point increase in spread raises the log-odds by 0.156.
• 4-point spread: about 65 percent. 8-point spread: about 78 percent.
• Doubling the spread does not double the probability. That is the curve on the next slide.

The Prediction Problem:

$$y = f(x)$$

%%| fig-width: 6.5
%%| fig-align: center
graph LR
    x --> f((f))
    f --> y

Key properties:

• Maps any real number to $(0,1)$
• $\sigma(0) = 0.5$
• Symmetric around 0
• Smooth S-shaped curve

Probit model uses normal CDF: $$\Phi^{-1}(p_i) = \beta^Tx_i$$

Comparison:

• Both give similar results in practice
• Logistic more popular (easier computation, multi-class extension)
• Probit common in economics/psychology

Predicted probabilities: $\hat{p}_i = \sigma(\hat{\beta}^Tx_i)$

Binary prediction: $$\hat{y}_i = \begin{cases} 1 & \text{if } \hat{p}_i > \alpha \\ 0 & \text{if } \hat{p}_i \leq \alpha \end{cases}$$

Common choice: $\alpha = 0.5$

Accuracy: Proportion of correct predictions

Let’s start by loading and exploring the tennis match data:

d = read.csv("data/tennis.csv")
dim(d)

Let’s look at a few columns of randomly selected five rows:

d[sample(1:943,size = 5),c("Player1","Player2","Round","Result","gender","surf")]

• 943 matches with 44 columns of data
• Player names, gender, surface type, and match statistics
• Focus on Break Points Won (BPW) as predictor
• Result: Binary outcome (Player 1 wins/loses)

• Clear pattern: Number of break points won strongly predicts match outcome
• This makes intuitive sense: winning break points is fundamental to winning tennis matches
• Green dots (Player 1 wins) cluster where Player 1 has more break points
• Red dots (Player 1 loses) cluster where Player 2 has more break points

The logistic regression model: $$\log\left(\dfrac{p}{1-p}\right) = \beta_1 \cdot \text{BPW.1} + \beta_2 \cdot \text{BPW.2}$$

Where:

• $p$ = probability of Player 1 winning
• $\beta_1$ ≈ 0.40 (positive effect of Player 1’s break points)
• $\beta_2$ ≈ -0.40 (negative effect of Player 2’s break points)
• Symmetric effects: Winning/losing break points have equal but opposite impacts

• TPR: True Positive Rate
• FPR: False Positive Rate
  
• TNR: True Negative Rate
• FNR: False Negative Rate

The decision boundary occurs where $p = 0.5$, meaning log-odds = 0:

$$\beta_1 x_1 + \beta_2 x_2 = 0$$

Therefore: $$x_2 = \dfrac{-\beta_1}{\beta_2}x_1$$

Points above the line: Player 2 more likely to win
Points below the line: Player 1 more likely to win

What happens when Player 1 wins one additional break point?

# Baseline probability (both players at 0 break points)
predict.glm(m, newdata = data.frame(BPW.1 = c(0), BPW.2 = c(0)), type="response")

# After Player 1 wins 1 break point
predict.glm(m, newdata = data.frame(BPW.1 = c(1), BPW.2 = c(0)), type="response")

Effect: Probability increases from 50% to 60%

# Odds ratio for one additional break point
exp(0.4019)

Interpretation: Each additional break point won increases the odds of winning by a factor of 1.5

Mathematical derivation: $$\dfrac{p}{1-p} = 1.5 \Rightarrow p = 0.6$$

What about winning 3 more break points than the opponent?

predict.glm(m, newdata = data.frame(BPW.1 = c(0), BPW.2 = c(0)), type="response")
predict.glm(m, newdata = data.frame(BPW.1 = c(3), BPW.2 = c(0)), type="response")

Result: Probability increases by 27 percentage points (from 50% to 77%)

“Are women’s matches less predictable than men’s matches?”

This is a common claim in tennis commentary. We can test this using model residuals.

library(dplyr)
d$res = abs(m$residuals)
outlind = which(d$res<2)  # Remove extreme outliers
boxplot(d$res[outlind] ~ d$gender[outlind], col=c(2,3), xlab="Gender", ylab="Residual")

Larger residuals = Less accurate predictions = More unpredictable matches

• Mean difference: 0.07 (women have higher residuals)
• t-statistic: -4.7 (highly significant)
• p-value: < 0.001

Conclusion: The conventional wisdom is statistically supported - women’s tennis matches are indeed less predictable based on break points won.

1. Strong Predictor: Break points won is highly predictive of match outcomes
2. Symmetric Effects: Winning/losing break points have equal magnitude effects
3. Quantified Impact: Each break point increases winning odds by 50%
4. High Accuracy: Model achieves ~88% prediction accuracy
5. Gender Difference: Women’s matches are significantly less predictable

• Logistic regression effectively models binary sports outcomes
• Confusion matrices provide comprehensive performance evaluation
• Residual analysis can reveal interesting domain-specific patterns
• Statistical testing validates common beliefs with data

Let’s create a circular dataset to demonstrate the kernel trick:

The data in the circle-data plot is clearly not linearly separable. However, if we map it to a three-dimensional space using the transformation: $$\begin{aligned} \phi: R^{2} & \longrightarrow R^{3} \\ \left(x_{1}, x_{2}\right) & \longmapsto\left(z_{1}, z_{2}, z_{3}\right)=\left(x_{1}^{2}, \sqrt{2} x_{1} x_{2}, x_{2}^{2}\right), \end{aligned}$$ and attempt to linearly separate the transformed data, the decision boundaries become hyperplanes in $R^{3}$, expressed as $\omega^{T} z + b = 0$. In terms of the original variables $x$, these boundaries take the form: $$\omega_{1} x_{1}^{2} + \omega_{2} \sqrt{2} x_{1} x_{2} + \omega_{3} x_{2}^{2} = 0,$$ which corresponds to the equation of an ellipse. This demonstrates that we can apply a linear algorithm to transformed data to achieve a non-linear decision boundary with minimal effort.

Now, consider what the algorithm is actually doing. It relies solely on the Gram matrix $K$ of the data. Once $K$ is computed, the original data can be discarded: $$\begin{aligned} K & = \left[\begin{array}{ccc} x_{1}^{T} x_{1} & x_{1}^{T} x_{2} & \cdots \\ x_{2}^{T} x_{1} & \ddots & \\ \vdots & & \end{array}\right]_{n \times n} = X X^{T}, \\ \text{where} \quad X & = \left[\begin{array}{c} x_{1}^{T} \\ \vdots \\ x_{n}^{T} \end{array}\right]_{n \times d}. \end{aligned}$$ Here, $X$, which contains all the data, is referred to as the design matrix.

When we map the data using $\phi$, the Gram matrix becomes: $$K = \left[\begin{array}{ccc} \phi\left(x_{1}\right)^{T} \phi\left(x_{1}\right) & \phi\left(x_{1}\right)^{T} \phi\left(x_{2}\right) & \cdots \\ \phi\left(x_{2}\right)^{T} \phi\left(x_{1}\right) & \ddots & \\ \vdots & & \end{array}\right].$$ Let us compute these inner products explicitly. For vectors $r$ and $s$ in $R^{3}$ corresponding to $a$ and $b$, respectively: $$\begin{aligned} \langle r, s \rangle & = r_{1} s_{1} + r_{2} s_{2} + r_{3} s_{3} \\ & = a_{1}^{2} b_{1}^{2} + 2 a_{1} a_{2} b_{1} b_{2} + a_{2}^{2} b_{2}^{2} \\ & = (a_1 b_1 + a_2 b_2)^2 \\ & = \langle a, b \rangle^{2}. \end{aligned}$$

where $a = (a_1, a_2)$ and $b = (b_1, b_2)$ are the original 2D vectors, and $r = \phi(a)$ and $s = \phi(b)$ are their transformations in $R^3$.

Let’s implement and visualize the transformation:

Three main components:

1. Splitting: Divide data based on feature values to create homogeneous subsets
2. Stopping: Decide when to stop splitting (prevent overfitting)
   
3. Pruning: Remove nodes that don’t improve accuracy

Goal: Find regions $R_1, R_2, \ldots, R_J$ that minimize: $$\text{RSS} = \sum_{j=1}^J\sum_{i \in R_j}(y_i - \bar{y}_j)^2$$

Algorithm: Top-down, greedy approach

At each step, find optimal split:

• Choose feature $j$ and split point $s$
• Create regions: $R_1(j,s) = \{x \mid x_j < s\}$ and $R_2(j,s) = \{x \mid x_j \geq s\}$

Optimization problem: $$\min_{j,s}\left[ \sum_{i:x_i\in R_1}(y_i - \bar{y}_1)^2 + \sum_{i:x_i \in R_2}(y_i - \bar{y}_2)^2\right]$$

Prediction in region $R_j$: $$f(x) = \bar{y}_j = \text{Average}(y_i \mid x_i \in R_j)$$

• Imagine you’re a jewelry appraiser determining a diamond’s value
• You follow a series of questions: Is the carat weight above 1.0?
• If yes, is the clarity VS1 or better?
• Each question leads to another, creating a decision path
• This mirrors our natural decision-making process

Decision trees create a flowchart of if-then rules that:

• Mirror human decision-making processes
• Create interpretable prediction rules
• Handle non-linear relationships naturally
• Capture feature interactions automatically

We’ll explore tree-based models using the classic diamonds dataset:

• 53,940 diamonds with prices and attributes
• Start with simple decision trees
• Progress to ensemble methods (random forests, gradient boosting)
• Develop insights into when and how to use these algorithms
• Learn to avoid common pitfalls

Although carat is the most important factor, it’s not the only one:

Let’s start simple with just two predictors:

Observation: Clarity becomes most important around 1 carat level

The visualization shows several key patterns:

• Vertical bands indicate carat as strongest predictor
• Non-linear patterns that linear models would miss
• Rectangular regions with sharp boundaries
• Rate of price increase varies across clarity levels

From here we’ll explore:

• Tree construction algorithms (recursive binary splitting)
• Pruning techniques to avoid overfitting
• Ensemble methods (Random Forests, Gradient Boosting)
• Hyperparameter tuning for optimal performance
• Model interpretation and feature importance

Key insights:

• Primary split on carat (most important predictor)
• Secondary splits on clarity
• Rectangular prediction regions
• Natural handling of feature interactions

Results: Step function fit with 7 regions, clear visualization of decision boundaries

Variable importance: Total decrease in node impurity when splitting on each variable, averaged over all trees

Build logistic regression model

Define the ROC curve function

Test Setup:

• Pink background: 40% purchase rate, 500 visitors
• Black background: 30% purchase rate, 550 visitors

Result: Pink background significantly outperforms!

October 19, 1987: S&P 500 dropped -21.76%

Historical Context: - Prior monthly returns: 1.2% mean, 4.3% volatility - Distribution: $X \sim N(1.2, 4.3^2)$

Z-Score Calculation: $$Z = \frac{X-\mu}{\sigma} = \frac{-0.2176 - 0.012}{0.043} = -5.34$$

Interpretation:

• This is a 5-sigma event
• Under normal model: $P(X < -0.2176) =$

An extremely rare occurrence!

Conclusion: Since 0.5 is outside the confidence interval, we have evidence that coin tosses were not fair.

2006 Discovery Channel Experiment

• 50 participants total
• 34 in experimental group (interviewer yawned)
• 16 in control group (no yawn)

Results:

• Experimental: 10/34 yawned (29.4%)
• Control: 4/16 yawned (25.0%)
• Difference: 4.4%

Mythbusters Conclusion: Yawning is contagious!

But is this statistically significant?

Statistical Conclusion: Large overlap, CI contains 0 - no evidence of difference!

The Experiment:

• Test effectiveness of new search algorithm
• Measure: Click-through rate on search results
• Random assignment to Algo 1 or Algo 2
• 2500 requests per algorithm

Estimated Success Rates:

• Algo 1: $\hat{p}_1 = 0.702$
• Algo 2: $\hat{p}_2 = 0.727$

Result: Intervals don’t overlap, difference CI doesn’t contain 0 - Algo 2 is significantly better!

Pepsi’s Claim: “More than half of Diet Coke drinkers prefer Diet Pepsi”

• The Test: Double-blind experiment with 100 drinkers

• Result: 56 favor Pepsi

• Hypothesis Setup:

  • $H_0: p = 0.5$ (no preference)
  • $H_1: p > 0.5$ (Pepsi preferred)

Conclusion: $p=0.5$ is in the CI - Pepsi’s claim is not supported!

Biogen’s Claim: “95% patient satisfaction rate” FDA Investigation: 60% satisfaction in random sample of 75 patients Question: Who is right?

Hypothesis Test: - $H_0: p = 0.95$ (Biogen innocent) - $H_1: p < 0.95$ (FDA right) - $\alpha = 0.01$ (1% significance level)

Result: Z = -14, p-value ≈ 0 - FDA is right, Biogen is wrong!

1998 Data:

• 6 million Viagra users: 77 deaths from coronary problems
• 1.5 million non-users: 11 deaths from coronary problems Pfizer’s Claim: Death rate no higher than general population

Statistical Analysis:

• Viagra rate: $\hat{p}_1 = 77/6,000,000 = 0.00001283$
• Control rate: $\hat{p}_2 = 11/1,500,000 = 0.00000733$

Result: CI doesn’t contain 0 - evidence of higher death rate with Viagra!

The Experiment:

• Method 1: Remove oxygen from air, measure remaining gas density
• Method 2: Prepare pure nitrogen by chemical decomposition
• Question: Are the densities the same?

The Data:

The Discovery: - Z-score ≈ 20 (20-sigma event!) - Massive statistical significance - Argon was discovered - the “inert” gas in air!

Approximating a challenging function: $$f(x) = \frac{1}{1+25x^2}$$

• Missing values: Systematic vs random patterns
• Measurement errors: Sensor noise, human error
• Sampling bias: Non-representative samples
• Label noise: Incorrect classifications
• Temporal drift: Changing data distributions

Solutions:

• Data cleaning pipelines
• Robust estimation methods
• Uncertainty quantification
• Professional annotation services (Scale AI, Toloka)

1. Quick Context (5 min) - Introduction to Bayesian Health and their approach
2. Small Group Brainstorm (10 min) - Analyze a clinical scenario
3. Round Table Reporting (15 min) - Share insights and debate approaches
   
4. Instructor Wrap-Up (15 min) - Technical deep dive and reflection

How can Bayesian methods transform healthcare from reactive to proactive?

From MYCIN’s rule-based approach in the 1970s to modern Bayesian AI

Healthcare generates massive amounts of data:

• Electronic Health Records (EHRs)
• Lab results updated continuously
• Vital signs monitored in real-time
• Imaging studies
• Clinical notes

But clinicians are overwhelmed and critical events are missed

How do we transform raw healthcare data into actionable insights that save lives?

Traditional AI (Rules-Based):

IF (Temperature > 38°C AND Heart_Rate > 100) 
   THEN Alert = "Possible Sepsis"

Problems: - Brittle rules - No uncertainty quantification - Can’t learn from data - High false positive rates

Key Advantage: Updates beliefs as new evidence arrives

$$P(\text{Disease} \mid \text{Symptoms, History, Labs}) = \frac{P(\text{Symptoms, Labs} \mid \text{Disease}) \cdot P(\text{Disease} \mid \text{History})}{P(\text{Symptoms, Labs})}$$

• Prior: $P(\text{Disease} \mid \text{History})$ - Based on patient history
• Likelihood: $P(\text{Symptoms, Labs} \mid \text{Disease})$ - Clinical evidence
• Posterior: Updated probability of disease

Combines prior medical knowledge with real-time patient data to provide probabilistic risk scores

Sepsis: Life-threatening organ dysfunction caused by infection

• Prevalence: 1.7 million cases/year in US
• Mortality: 270,000 deaths/year (15-30% mortality rate)
• Time-sensitive: Every hour delay in treatment increases mortality by 4-8%
• Cost: $62 billion/year in US healthcare costs

The Challenge: Early detection is critical but difficult

SIRS Criteria (Systemic Inflammatory Response Syndrome):

At least 2 of: - Temperature < 36°C or > 38°C - Heart rate > 90 bpm - Respiratory rate > 20/min - WBC < 4,000 or > 12,000 cells/mm³

Problems: - Low specificity (many false positives) - Detected too late - Doesn’t account for patient-specific factors - Rule-based, not probabilistic

Patient: 68-year-old male, post-surgery day 3

Current Status: - Temperature: 37.8°C (slightly elevated) - Heart Rate: 95 bpm (slightly elevated) - Blood Pressure: 110/70 (normal) - Recent trend: HR increasing from 85 to 95 over 6 hours - Labs: WBC 11,500 (borderline), Lactate 1.8 mmol/L

Question for Groups:

1. What additional information would help assess sepsis risk?
2. How would a Bayesian model differ from a rule-based system here?
3. What are the trade-offs between sensitivity and specificity?
4. How should prior patient history influence the decision?

Assign roles: Facilitator, Note-taker, Timekeeper, Reporter

Consider these perspectives:

Clinical Perspective: - What subtle patterns might indicate early sepsis? - How do trends matter vs. single measurements?

Statistical Perspective: - How do we quantify uncertainty? - What’s the cost of false positives vs. false negatives?

ML/AI Perspective: - What features should the model learn? - How do we update predictions in real-time?

Each group shares (2-3 min each):

1. Key insights from your discussion
2. Most important factors you identified
3. Biggest challenges in implementation
4. One question that emerged

After presentations, class discussion:

Ethics & Trust: - Should algorithms override clinical judgment? - How transparent should the model be?

Implementation: - How to reduce alert fatigue? - Integration with existing workflows?

Equity: - Do Bayesian priors perpetuate bias? - Performance across diverse populations?

TREWS: Targeted Real-time Early Warning Score

Published in Nature Medicine (2022)

Key Innovation: Patient-specific dynamic risk assessment

$$P(\text{Sepsis}_t \mid \text{Data}_{1:t}, \text{Patient History}) = f_{\text{Bayesian}}(\text{all available information})$$

• Updates every hour
• Uses 65+ features from EHR
• Incorporates temporal trends
• Patient-specific baselines

%%| fig-width: 6
%%| fig-align: center
graph TB
    PH[Patient History] --> Prior[Prior Risk]
    VS[Vital Signs] --> Likelihood[Likelihood]
    Labs[Lab Results] --> Likelihood
    Prior --> Post[Posterior Risk]
    Likelihood --> Post
    Post --> Alert{Alert Threshold}
    Alert -->|High Risk| Notify[Notify Clinician]
    Alert -->|Low Risk| Monitor[Continue Monitoring]

Advantages over simple ML: - Incorporates domain knowledge - Quantifies uncertainty - Handles missing data naturally - Interpretable reasoning chain

Hierarchical Bayesian Model:

Level 1 - Patient-specific parameters: $$\theta_i \sim P(\theta \mid \alpha)$$

Level 2 - Population parameters: $$\alpha \sim P(\alpha)$$

Level 3 - Observations: $$y_{it} \sim P(y \mid \theta_i, x_{it})$$

where: - $\theta_i$ = patient-specific parameters - $\alpha$ = population-level hyperparameters
- $y_{it}$ = outcome at time $t$ for patient $i$ - $x_{it}$ = observed features

Prior Probability (before new labs): $$P(\text{Sepsis}) = 0.03 \text{ (3$$

New Evidence: Lactate = 2.5 mmol/L

$$P(\text{Lactate} \geq 2.5 \mid \text{Sepsis}) = 0.70$$ $$P(\text{Lactate} \geq 2.5 \mid \text{No Sepsis}) = 0.10$$

Posterior Probability: $$P(\text{Sepsis} \mid \text{Lactate} \geq 2.5) = \frac{0.70 \times 0.03}{0.70 \times 0.03 + 0.10 \times 0.97} = \frac{0.021}{0.118} = 0.178$$

Risk increases from 3% to 17.8% with new evidence!

Hour 1: Prior risk = 3%

Hour 2: New HR trend → Posterior = 8% (becomes new prior)

Hour 3: New lab results → Posterior = 18%

Hour 4: BP drops → Posterior = 45% → ALERT!

$$P(\text{Sepsis}_t \mid \text{All data up to time } t) = \frac{P(\text{New data}_t \mid \text{Sepsis}) \cdot P(\text{Sepsis}_{t-1} \mid \text{Previous data})}{P(\text{New data}_t)}$$

Each new piece of evidence updates the probability, incorporating the full history

Unlike point predictions, Bayesian methods provide credible intervals:

$$P(0.35 \leq \text{Sepsis Risk} \leq 0.55) = 0.95$$

“We are 95% confident the true risk is between 35% and 55%”

Clinical Value: - High uncertainty → gather more data before acting - Low uncertainty → confident in recommendation - Transparent about what model doesn’t know

Study: Randomized controlled trial at Johns Hopkins

TREWS vs. Standard Care:

Key Finding: 1.3 hours faster treatment, 4.1% absolute mortality reduction

Problem: Pressure ulcers affect 2.5 million patients/year in US

Bayesian Approach: - Real-time risk assessment every 30 minutes - Incorporates: mobility, nutrition, tissue perfusion, medical devices - Predicts risk 12-24 hours in advance

Model: $$P(\text{Ulcer}_{t+12} \mid \text{Data}_t) = \text{Bayesian Model}(\text{Patient factors, Environmental factors, Time trends})$$

Results: - 40% reduction in hospital-acquired pressure ulcers - Proactive interventions (repositioning, support surfaces)

Top 10 Predictive Features in TREWS:

1. Lactate trend (Δ over 6h)
2. Heart rate variability
   
3. Temperature trajectory
4. Respiratory rate trend
5. White blood cell trend
6. Blood pressure variability
7. Recent procedures
8. Chronic conditions
9. Age-adjusted baselines
10. Time of day patterns

Bayesian Model automatically weights features based on: - Information gain - Uncertainty reduction - Interaction effects

In Healthcare: 30-40% of EHR data is missing!

Traditional ML: - Delete rows (lose data) - Impute with mean (lose variance) - Create missing indicators

Bayesian Approach: - Treat missing data as latent variables - Marginalize over possible values weighted by probability

$$P(\text{Sepsis} \mid \text{Observed data}) = \int P(\text{Sepsis} \mid \text{Observed, Missing}) \cdot P(\text{Missing} \mid \text{Observed}) \, d(\text{Missing})$$

Result: Uses partial information optimally without discarding data

Calibration Plot: Do predicted probabilities match observed frequencies?

TREWS Calibration: - When model says 30% risk → observed frequency is 28-32% - Well-calibrated across risk spectrum

Static Model: Single risk score

Dynamic Bayesian Model: Risk evolves over time

%%| fig-width: 8
%%| fig-align: center
graph LR
    R0[Risk t=0] --> R1[Risk t=1]
    R1 --> R2[Risk t=2]
    R2 --> R3[Risk t=3]
    D1[Data t=1] --> R1
    D2[Data t=2] --> R2
    D3[Data t=3] --> R3

Dynamic Model captures: - Trends (increasing/decreasing) - Rates of change - Temporal patterns - Recovery trajectories

Problem: Too many alerts → clinicians ignore them

Bayesian Solution: - Threshold optimization: $P(\text{Sepsis}) > \tau$ - Multi-level alerts based on uncertainty - Personalized to unit/clinician

Alert Strategy:

Result: 60% reduction in alert fatigue vs. SIRS criteria

Critical Question: Do Bayesian priors perpetuate healthcare disparities?

Potential Issues: - Historical data reflects biased treatment patterns - Different baseline risks across populations - Access to care differences

Bayesian Health’s Approach: - Stratified validation across demographics - Audit for differential performance - Adjust priors based on social determinants of health - Transparency in feature importance

$$P(\text{Sepsis} \mid \text{Race, Gender}) \stackrel{?}{=} P(\text{Sepsis} \mid \text{Clinical factors only})$$

Scale of Implementation:

• 11 hospitals
• 6,000+ beds
  
• 200,000+ patient encounters/year

Technical Infrastructure: - Real-time EHR integration (Epic) - <5 minute latency for predictions - Dashboard integration - Mobile alerts

Key to Success: - Clinician buy-in through co-design - Workflow integration - Continuous learning from feedback - Performance monitoring

Clinicians need to understand WHY the model alerts

TREWS provides:

SEPSIS RISK: 52% (High)

Top Contributing Factors:
1. Lactate increasing: 1.2 → 2.5 mmol/L (+35% risk)
2. Heart rate trend: 85 → 105 bpm (+12% risk)  
3. Recent surgery (2 days ago) (+8% risk)
4. Age-adjusted baseline (+5% risk)

Alternative explanations considered:
- Post-surgical inflammation (22% probability)
- Pain response (15% probability)

Bayesian framework naturally supports explainability through probabilistic reasoning chains

From Lecture 1-6:

1. Bayes Rule: Core of the update mechanism
2. Prior/Posterior: Patient history → Real-time updates
3. Likelihood: Clinical evidence model
4. Decision Theory: Optimal alert thresholds
5. Sequential Analysis: Time-series of patient data
6. Hierarchical Models: Patient-level + Population-level parameters

This case study demonstrates how foundational Bayesian concepts directly translate to life-saving applications

Research Challenges:

1. Causality: Association vs. causation in predictions
2. Generalization: Transfer across hospitals/populations
3. Multi-task learning: Simultaneous prediction of multiple conditions
4. Human-AI collaboration: Optimal division of labor

Emerging Applications: - ICU deterioration prediction - Post-discharge monitoring - Medication response prediction - Surgical complication forecasting

1. Bayesian methods provide principled uncertainty quantification - critical for medical decisions

2. Sequential updating enables real-time risk assessment - catches deterioration early

3. Probabilistic framework handles missing data naturally - common in healthcare

4. Calibrated probabilities enable better decision-making - transparent about confidence

5. Implementation requires careful attention to clinical workflow - technology alone insufficient

Consider the trade-offs:

A Bayesian early warning system has: - 85% sensitivity (catches 85% of sepsis cases) - 88% specificity (12% false positive rate)

Questions:

1. What false positive rate is acceptable given the stakes?
2. How should the system handle uncertainty at the decision boundary (40-60% risk)?
3. Should all patients be monitored equally, or risk-stratified?
4. How do we balance algorithmic recommendations with clinical autonomy?

Form pairs and discuss for 5 minutes, then share with class

FDA Regulation: - Bayesian Health’s platform is FDA-cleared - Classified as Software as Medical Device (SaMD) - Requires continuous monitoring and updates

Ethical Framework: - Beneficence: Does it improve outcomes? - Non-maleficence: Potential for harm from false alerts? - Autonomy: Preserves clinician decision-making? - Justice: Fair across patient populations?

Ongoing Debate: Should AI systems be held to higher standards than human clinicians?

Cost-Benefit Analysis (per 1000 patients):

Costs: - Platform license: $500K/year - Implementation: $200K - Training: $50K - Maintenance: $100K/year

Benefits: - Lives saved: 40 patients (@ $10M VSL) = $400M - Reduced ICU days: 1000 days × $3K = $3M - Reduced readmissions: 50 × $15K = $750K - Efficiency gains: $500K

Net benefit: Substantial even without valuing lives

Bayesian Health vs. Other Approaches:

Bayesian Advantage: Best combination of performance and interpretability

What Made Bayesian Health Successful:

1. Solve a real clinical problem with clear value proposition
2. Co-design with clinicians from day one
3. Rigorous validation through RCTs
4. Probabilistic approach provides interpretability
5. Continuous learning from deployment
6. Workflow integration not just technology

Technology + Clinical workflow + Organizational change = Impact

Take 3 minutes to write:

1. One concept from Bayesian statistics you now understand better through this case study

2. One question you still have about implementation

3. One other domain where similar Bayesian methods could be applied

Share with a partner, then volunteers share with class

Mini-Project Ideas:

1. Build a simple Bayesian network for diabetes risk prediction
2. Compare Bayesian vs. frequentist approaches to clinical trial analysis
   
3. Implement sequential Bayesian updating for a time-series medical dataset
4. Analyze fairness/bias in an existing clinical prediction model
5. Design an explainable AI system for a healthcare application

Datasets: - MIMIC-III (ICU data) - eICU Collaborative Research Database - UK Biobank

The horseshoe prior provides adaptive shrinkage through a hierarchical Bayesian framework. For a linear model with response (y_i) and covariates (x_i ^p), the horseshoe prior is specified as:

$$\begin{aligned} y_i &= x_i^T \beta + \varepsilon_i, \quad \varepsilon_i \sim N(0, \sigma^2) \\ \beta_j &\sim N(0, \tau^2 \lambda_j^2 \sigma^2), \quad j = 1, \ldots, p \\ \lambda_j &\sim \text{Cauchy}^+(0, 1) \\ \tau &\sim \text{Cauchy}^+(0, 1) \end{aligned}$$

Here $\lambda_j$ are local shrinkage parameters (one per coefficient), $\tau$ is the global shrinkage parameter controlling overall sparsity.

• Deep Learning: Dropout, weight decay, batch normalization
• Compressed Sensing: Sparse signal recovery
• Genomics: Gene expression analysis
• Neuroimaging: Brain connectivity
• Finance: Portfolio optimization
• Recommendation Systems: Collaborative filtering

1. MLE fails in high dimensions ($p \geq 3$)
2. Shrinkage helps through bias-variance trade-off
3. Regularization = Bayesian priors (duality)
4. Different penalties for different sparsity patterns
5. Modern methods combine best of both worlds
6. Context matters for method selection

Expression

Double descent is a phenomenon of over-parameterized statistical models that challenges traditional bias-variance intuition. Over-parameterized models such as deep neural networks have an interesting re-descending property in their risk characteristics.

As model complexity increases:

1. Classical regime: U-shaped bias-variance trade-off
2. Interpolation threshold: Risk peaks when parameters equal observations
3. Over-parameterized regime: Risk re-descends beyond interpolation

This phenomenon was first observed in linear regression and later in deep learning contexts.

The curve shows two distinct phases:

• First descent follows classical bias-variance trade-off
• Second descent occurs in over-parameterized regime
• Peak occurs at interpolation threshold where $p = n$

Modern machine learning practice increasingly uses over-parameterized models with parameters far exceeding the number of observations. Examples include large language models, image processing networks, and generative AI methods.

Key insights:

• More parameters don’t necessarily mean worse generalization
• Interpolators achieving zero training error can still generalize well
• Implicit regularization from minimum-norm solutions helps
• Traditional model selection may be unnecessarily conservative

Practical implication: Instead of carefully tuning to find optimal complexity, over-parameterize and get good performance automatically!

We demonstrate double descent using polynomial regression with a sinusoidal target function $f(x) = \sin(5x)$.

Setup:

• Only $n = 20$ training observations from $[-1, 1]$
• Gaussian noise with $\sigma = 0.3$
• Fit polynomials of degree $d = 1, 2, \ldots, 50$
• Use Legendre polynomial basis for numerical stability
• Estimate coefficients using Moore-Penrose pseudoinverse

Small sample size is crucial for observing double descent, creating a regime where parameters can substantially exceed observations.

Degree 1 (Underparameterized): Linear model too simple to capture oscillatory sine function. High bias, poor fit to both training and test data.

Degree 5 (Classical Optimum): Sweet spot of traditional bias-variance tradeoff. Sufficient complexity to capture main features without severe overfitting.

Degree 20 (Interpolation Threshold): Exactly as many parameters as observations, enabling perfect interpolation. Wild oscillations between data points lead to poor generalization.

Degree 50 (Over-parameterized): Despite having far more parameters than observations, achieves better test performance than interpolating model. This is the double descent effect!

The test error (blue line) exhibits characteristic double descent shape with three phases:

Classical Regime ($d < 5$): Increasing model complexity reduces both bias and test error. Traditional understanding of bias-variance tradeoff applies.

Interpolation Crisis ($d \approx n = 20$): Test error peaks dramatically as model perfectly fits training data while generalizing poorly. This is where classical theory predicts catastrophic overfitting.

Over-parameterized Regime ($d > 30$): Test error decreases again! Extreme over-parameterization leads to improved generalization despite ability to memorize training data.

The double descent phenomenon arises from implicit regularization effects of minimum-norm solutions in over-parameterized settings.

When $d > n$, the pseudoinverse solution corresponds to minimum $\ell_2$-norm coefficients among all possible interpolating solutions. This implicit bias toward simpler functions leads to surprisingly good generalization.

Mathematical insight: Among infinitely many solutions that perfectly fit the data, we automatically select the one with smallest coefficients, providing implicit regularization without explicit penalties.

Connection to deep learning: Similar implicit regularization occurs in over-parameterized neural networks through gradient descent dynamics.

Double descent has a natural Bayesian interpretation and is not in conflict with Occam’s razor that Bayesian models possess.

Occam’s Razor: Bayesian methods naturally favor simpler models through the marginal likelihood, penalizing complexity via corrections like BIC.

Reconciliation: Even though Bayesian methods shift posterior toward lower-complexity models, highly parameterized Bayesian models can have good risk properties due to the conditional prior of parameters given the model.

The key is that over-parameterized models don’t necessarily have high effective complexity when properly regularized (implicitly or explicitly).

The double descent phenomenon has significant practical implications for modern machine learning:

Model selection strategy: Given that finding the “sweet spot” is time-consuming and computationally expensive, over-parameterization provides good performance automatically without careful tuning.

Generality: Observed across wide range of models including deep neural networks, random forests, kernel methods, and linear models.

Challenge to intuition: Classical statistical wisdom about model complexity doesn’t apply in contemporary settings with over-parameterized models.

Modern AI success: Helps explain why massive models (GPT, BERT, etc.) generalize well despite billions of parameters.

Setup: $$y_i \mid \theta_i \sim N(\theta_i, \sigma^2), \quad i=1,\ldots,p, \quad p > 2$$

Goal: Estimate $\theta = (\theta_1, \ldots, \theta_p)$ under squared loss $$\mathcal{L}(\theta, \hat{\theta}) = \sum_{i=1}^p (\theta_i - \hat{\theta}_i)^2$$

MLE: $\hat{\theta}_i^{MLE} = y_i$

Risk: $R(\theta,\hat{\theta}^{MLE}) = p\sigma^2$ (constant!)

For $p \geq 3$, the MLE is inadmissible under squared error loss!

There exist estimators with uniformly lower risk than MLE.

James-Stein Estimator: $$\hat{\theta}_i^{JS} = \left(1 - \frac{(p-2)\sigma^2}{\|y\|^2}\right) y_i$$

Risk: $R(\theta, \hat{\theta}^{JS}) < R(\theta, \hat{\theta}^{MLE})$ for all $\theta$

• Shrinkage: Pulls estimates toward zero (or overall mean)
• Borrowing strength: Uses information across components
• Global regularization: Single shrinkage parameter
• Bias-variance trade-off: Small bias for large variance reduction

Key insight: Multivariate estimation is easier than univariate when using appropriate shrinkage!

Hierarchical Model: $$\begin{aligned} y_i \mid \theta_i &\sim N(\theta_i, \sigma^2) \\ \theta_i \mid \mu, \tau^2 &\sim N(\mu, \tau^2) \end{aligned}$$

Empirical Bayes: - Estimate $\hat{\mu} = \bar{y}$ - Estimate $\hat{\tau}^2$ from data - Use in posterior mean calculation

Result: Closely related to James-Stein estimator

James-Stein Risk Calculation: $$R(\theta, \hat{\theta}^{JS}) = p\sigma^2 - (p-2)\sigma^2 \mathbb{E}\left[\frac{1}{\|\theta + \epsilon\|^2/\sigma^2}\right]$$

Horseshoe Posterior Mean: $$\mathbb{E}[\theta_i \mid y_i] = \left(1 - \kappa_i\right) \cdot 0 + \kappa_i \cdot y_i$$ where $\kappa_i$ depends on $\tau$ and $y_i$

MCMC for Horseshoe:

1. Classical Regime ($d < 5$): Traditional bias-variance tradeoff

2. Interpolation Crisis ($d \approx n = 20$): Test error peaks at perfect fit

3. Over-parameterized Regime ($d > 30$): Test error decreases with extreme over-parameterization

Key insight: Implicit regularization through minimum-norm solutions