Polson–Sokolov Lab · Simple linear regression

Regression Model Lab

Each point is drawn from its own normal distribution. The center of that normal sits on the line \(a\cdot x + b\), and its spread \(\sigma\) is the same at every \(x\). The red curves show those distributions at a few values of \(x\).

← All labs

Based on Chapter 12, Linear Regression, of Polson and Sokolov, Bayes, Artificial Intelligence, and Deep Learning (vsokolov.org/html/_book).

Data point (xᵢ, yᵢ) True mean line a·x + b N(a·x + b, σ²) at that x ±2σ band (95%) Least-squares fit

Model

Slope, a1.00

Intercept, b2.00

Noise SD, σ1.50

The noise draws stay fixed while you move a slider, so you can watch the same points slide and stretch.

Sample

Number of points, n150

Normal curves shown5

Show

Normal curves at chosen x Highlight points near each curve ±2σ band Residuals to the true line Least-squares fit

True model vs. estimate from this sample

	True	Estimate
Slope	–	–
Intercept	–	–
Noise SD σ	–	–
Points within ±1σ	68.3%	–
Points within ±2σ	95.4%	–

The estimated \(\sigma\) is the residual standard error, \(\sqrt{\text{SSE}/(n - 2)}\).

The line is a line of means

For each \(x\), the model says \(y\) is random, and its average is \(a\cdot x + b\). The line does not pass through the points. It passes through the centers of the red curves.

\[ E[y \mid x] = a\cdot x + b \]

The noise is normal with one \(\sigma\)

Every red curve has the same width. That is the constant-variance assumption (homoscedasticity). The errors are also independent of each other and of \(x\).

\[ \varepsilon_i = y_i - (a\cdot x_i + b) \sim N(0, \sigma^2), \quad \text{iid} \]

Three ways to write the same model

These all say the same thing. The last form is the one the picture shows: a normal distribution for each y, centered on the line.

\[ y_i = a\cdot x_i + b + \varepsilon_i \]

\[ y_i \mid x_i \sim N(a\cdot x_i + b, \sigma^2) \]

Try this in class

Drag \(\sigma\) down to 0.1. The points collapse onto the line and the red curves become spikes. Drag it up to 4 and the same points spread out, but the line of centers does not move.
Look at the highlighted points inside one vertical slice. Their spread matches the red curve next to them. Each slice is its own small normal sample.
Set \(n = 20\) and press "New sample" several times. The green fitted line wobbles around the blue true line. Set \(n = 1{,}000\) and it barely moves.
Count the points outside the shaded \(\pm 2\sigma\) band. About 1 in 20 should fall outside it. Check the bottom rows of the table.
Change the slope to a negative value. The curves move with the line, and their shape stays the same.