Polson–Sokolov Lab · Least squares and diagnostics

Least Squares Lab

Drag, add or remove points and watch the least-squares line, SSE and \(R^2\) update. Turn on the squared residuals to see what the line minimizes, try to beat it with your own line, then use the diagnostics to find the points that move the slope.

← All labs

Based on Chapter 12, Linear Regression, of Polson and Sokolov, Bayes, Artificial Intelligence, and Deep Learning (vsokolov.org/html/_book).

Data point Least-squares line Squared residual Cook's D > 0.5 Cook's D > 1
Diagnostics

Which points drive the fit?

Residuals vs. fitted

Should look like random scatter around zero. A curve means the line is the wrong shape.

Standardized residuals vs. leverage

Red dashed contours mark Cook's D = 0.5 and 1. Points outside them are influential.

Cook's distance by point

Combines residual size and leverage. Investigate D > 0.5; D > 1 is a strong warning.

The table lists the most influential points first. Click a row to highlight that point everywhere. "Drop" removes it and leaves the old line on the plot for comparison.

PointxyFitted ŷResidual eLeverage hStd. resid rCook's Dβ̂₁ without it
Fit, check, transform

Least squares picks the line with the smallest total square area

Leavitt's 25 pulsating stars made the point first: least squares minimizes the average squared error, and any other rule on the same data fits visibly worse. The demos below load classic diagnostic cases.

  1. The line minimizes the sum of squared residuals

    Each residual \(e_i = y_i - \hat y_i\) is the vertical miss at one point. Turn on the squares: each square's side is a residual, so its area is \(e_i^2\). Least squares moves the line until the total area SSE is as small as possible. Any line you draw has a larger SSE.

  2. The fitted line always passes through \((\bar x, \bar y)\)

    Because \(\hat\beta_0 = \bar y - \hat\beta_1 \bar x\), the line pivots around the point of means. The slope is the correlation rescaled into the right units, \(\hat\beta_1 = r_{xy} \cdot s_y / s_x\).

  3. Leverage and residual size combine into influence

    A point far from \(\bar x\) has high leverage \(h_i\). It can pull the line toward itself, so its own residual can look small. Cook's distance multiplies the squared standardized residual by \(h_i/(1 - h_i)\). In the cigarettes data, country K plays the role of the United States in the classic data: removing it changes the slope a lot, but the slope stays positive.

  4. When the residuals fan out, take logs

    On the raw scale a few giant mammals dominate and the residuals are huge and uneven. On the log–log scale the cloud is a straight band and the slope is an elasticity. The reference fit is \(\log(\text{Brain}) = 2.18 + 0.74 \log(\text{Body})\): a 1% heavier body goes with a 0.74% heavier brain.

  5. Always look at the biggest residuals

    On the log scale the largest standardized residual belongs to the chinchilla, entered as 64 g instead of 6.4 g. Press "Fix the typo" and the only big outlier left is the human.

Least-squares formulas
\(\displaystyle \hat\beta_1 = \sum (x_i - \bar x)(y_i - \bar y) / \sum (x_i - \bar x)^2 = r_{xy} \cdot s_y / s_x\)
\(\displaystyle \hat\beta_0 = \bar y - \hat\beta_1 \cdot \bar x\)
Fit quality
\(\displaystyle \text{SSE} = \sum e_i^2, \qquad R^2 = 1 - \text{SSE} / \sum (y_i - \bar y)^2\)
\(\displaystyle s = \sqrt{\text{SSE} / (n - 2)}\)
Leverage and standardized residual
\(\displaystyle h_i = 1/n + (x_i - \bar x)^2 / \sum (x_j - \bar x)^2\)
\(\displaystyle r_i = e_i / (s \cdot \sqrt{1 - h_i}) \qquad \text{rough 95}\%\text{ in } (-2, 2)\)
Cook's distance with p = 2 coefficients
\(\displaystyle D_i = (1/p) \cdot r_i^2 \cdot h_i / (1 - h_i)\)
Log–log model
\(\displaystyle Y = A \cdot X^b \iff \log Y = \beta_0 + \beta_1 \log X\)

Takeaway for students. A regression without a look at its residuals is half a regression. Before you report a slope, check which points drive it, refit without them, and see whether the conclusion survives.

The SLR model

For each X, the Y values are normal, their means lie on the line, \(\sigma\) is the same at every X, and the observations are independent.

\[ Y = \beta_0 + \beta_1 X + \varepsilon, \quad \varepsilon \sim N(0, \sigma^2) \]

House prices

The Saratoga fit gives \(\hat\beta_0 = 13.44\) and \(\hat\beta_1 = 0.11\), in thousands of dollars per square foot. Each extra square foot adds about $110.

\[ \hat Y \text{ at 2,200 sq ft} = 13.44 + 0.11 \times 2{,}200 \approx 262 \]

Five-step strategy

  1. Plot the data.
  2. Fit the model.
  3. Run diagnostics: 4-in-1, Cook's D.
  4. Interpret coefficients, \(t\), \(R^2\), \(s\).
  5. Predict, and check out of sample.

Try this in class

  1. Load "Clean line" and turn on the squared residuals. Drag one point straight up and watch its square grow. Ask why one far point costs more than two near ones.
  2. Turn on "Try your own line" and press "Flat at ȳ". Its SSE equals the total sum of squares, so \(R^2 = 1 - \text{SSE}_{\text{LS}}/\text{SSE}_{\text{flat}}\). Then ask students to beat least squares by dragging the handles.
  3. Load "Cigarettes". Read Cook's D for country K, then press "Drop" in its row. The slope roughly doubles and the old line stays on the plot. Is the conclusion still "more smoking, more cancer"?
  4. Drag a point to the far right of the cigarette plot, first on the line, then far off it. Leverage stays the same and Cook's D jumps only in the second case.
  5. Load "Mammals" on the raw scale and look at the residuals vs. fitted plot. Switch on log–log, find the biggest standardized residual, then fix the typo and compare \(R^2\) and \(s\).