Polson–Sokolov Lab · Least squares and diagnostics

Least Squares Lab

Drag, add or remove points and watch the least-squares line, SSE and R² update. Turn on the squared residuals to see what the line minimizes, try to beat it with your own line, then use the diagnostics to find the points that move the slope.

← All labs

Based on Chapter 12, Linear Regression, of Polson and Sokolov, Bayes, Artificial Intelligence, and Deep Learning (vsokolov.org/html/_book).

Data point Least-squares line Squared residual Cook's D > 0.5 Cook's D > 1
Diagnostics

Which points drive the fit?

Residuals vs. fitted

Should look like random scatter around zero. A curve means the line is the wrong shape.

Standardized residuals vs. leverage

Red dashed contours mark Cook's D = 0.5 and 1. Points outside them are influential.

Cook's distance by point

Combines residual size and leverage. Investigate D > 0.5; D > 1 is a strong warning.

The table lists the most influential points first. Click a row to highlight that point everywhere. "Drop" removes it and leaves the old line on the plot for comparison.

PointxyFitted ŷResidual eLeverage hStd. resid rCook's Dβ̂₁ without it
Fit, check, transform

Least squares picks the line with the smallest total square area

Leavitt's 25 pulsating stars made the point first: least squares minimizes the average squared error, and any other rule on the same data fits visibly worse. The demos below load classic diagnostic cases.

  1. The line minimizes the sum of squared residuals

    Each residual eᵢ = yᵢ − ŷᵢ is the vertical miss at one point. Turn on the squares: each square's side is a residual, so its area is eᵢ². Least squares moves the line until the total area SSE is as small as possible. Any line you draw has a larger SSE.

  2. The fitted line always passes through (x̄, ȳ)

    Because β̂₀ = ȳ − β̂₁x̄, the line pivots around the point of means. The slope is the correlation rescaled into the right units, β̂₁ = rxy · sy / sx.

  3. Leverage and residual size combine into influence

    A point far from x̄ has high leverage hᵢ. It can pull the line toward itself, so its own residual can look small. Cook's distance multiplies the squared standardized residual by hᵢ/(1 − hᵢ). In the cigarettes data, country K plays the role of the United States in the classic data: removing it changes the slope a lot, but the slope stays positive.

  4. When the residuals fan out, take logs

    On the raw scale a few giant mammals dominate and the residuals are huge and uneven. On the log–log scale the cloud is a straight band and the slope is an elasticity. The reference fit is log(Brain) = 2.18 + 0.74 log(Body): a 1% heavier body goes with a 0.74% heavier brain.

  5. Always look at the biggest residuals

    On the log scale the largest standardized residual belongs to the chinchilla, entered as 64 g instead of 6.4 g. Press "Fix the typo" and the only big outlier left is the human.

Least-squares formulas
β̂₁ = Σ(xᵢ − x̄)(yᵢ − ȳ) / Σ(xᵢ − x̄)² = rxy·sy/sx
β̂₀ = ȳ − β̂₁·x̄
Fit quality
SSE = Σ eᵢ², R² = 1 − SSE / Σ(yᵢ − ȳ)²
s = √(SSE / (n − 2))
Leverage and standardized residual
hᵢ = 1/n + (xᵢ − x̄)² / Σ(xⱼ − x̄)²
rᵢ = eᵢ / (s·√(1 − hᵢ)) rough 95% in (−2, 2)
Cook's distance with p = 2 coefficients
Dᵢ = (1/p) · rᵢ² · hᵢ / (1 − hᵢ)
Log–log model
Y = A·X^b ⇔ log Y = β₀ + β₁ log X

Takeaway for students. A regression without a look at its residuals is half a regression. Before you report a slope, check which points drive it, refit without them, and see whether the conclusion survives.

The SLR model

For each X, the Y values are normal, their means lie on the line, σ is the same at every X, and the observations are independent.

Y = β₀ + β₁X + ε, ε ~ N(0, σ²)

House prices

The Saratoga fit gives β̂₀ = 13.44 and β̂₁ = 0.11, in thousands of dollars per square foot. Each extra square foot adds about $110.

Ŷ at 2,200 sq ft = 13.44 + 0.11 × 2,200 ≈ 262

Five-step strategy

  1. Plot the data.
  2. Fit the model.
  3. Run diagnostics: 4-in-1, Cook's D.
  4. Interpret coefficients, t, R², s.
  5. Predict, and check out of sample.

Try this in class

  1. Load "Clean line" and turn on the squared residuals. Drag one point straight up and watch its square grow. Ask why one far point costs more than two near ones.
  2. Turn on "Try your own line" and press "Flat at ȳ". Its SSE equals the total sum of squares, so R² = 1 − SSELS/SSEflat. Then ask students to beat least squares by dragging the handles.
  3. Load "Cigarettes". Read Cook's D for country K, then press "Drop" in its row. The slope roughly doubles and the old line stays on the plot. Is the conclusion still "more smoking, more cancer"?
  4. Drag a point to the far right of the cigarette plot, first on the line, then far off it. Leverage stays the same and Cook's D jumps only in the second case.
  5. Load "Mammals" on the raw scale and look at the residuals vs. fitted plot. Switch on log–log, find the biggest standardized residual, then fix the typo and compare R² and s.