Linear Regression

Updated Started

Correlation tells us how strong a linear relationship is. Linear regression lets us use it: given a house's size, predict its price.

One independent variable

We hypothesise a linear model

yi=βxi+αy_i = \beta x_i + \alpha

where yiy_i is the price of house ii, xix_i its size, β\beta the slope and α\alpha the intercept. Least squares picks the α\alpha and β\beta that minimise the sum of squared errors (residuals), our cost function SS:

S=∑i=1n(yi−y^i)2=∑i=1n(yi−α−βxi)2S = \sum_{i=1}^n (y_i - \hat{y}_i)^2 = \sum_{i=1}^n (y_i - \alpha - \beta x_i)^2

At the minimum, the partial derivatives of SS with respect to α\alpha and β\beta are both zero. (The chain rule and partial derivatives notes cover the calculus.)

Finding α\alpha

Differentiating with the chain rule, with u=yi−α−βxiu = y_i - \alpha - \beta x_i: the outer u2u^2 becomes 2u2u, and the inner term differentiates to −1-1 with respect to α\alpha. Setting the result to zero:

0=∑i=1n−2(yi−α−βxi)0 = \sum_{i=1}^n -2(y_i - \alpha - \beta x_i)

Dividing by −2-2 and splitting the sum, with ∑i=1nα=nα\sum_{i=1}^n \alpha = n\alpha:

0=∑i=1nyi−nα−β∑i=1nxi⇒α=∑yin−β∑xin0 = \sum_{i=1}^n y_i - n\alpha - \beta \sum_{i=1}^n x_i \quad\Rightarrow\quad \alpha = \frac{\sum y_i}{n} - \beta \frac{\sum x_i}{n}

Those are just the means, so

α=yˉ−βxˉ(1)\alpha = \bar{y} - \beta \bar{x} \tag{1}

The fitted line always passes through (xˉ,yˉ)(\bar{x}, \bar{y}).

Finding β\beta

This time the inner term differentiates to −xi-x_i:

0=∑i=1n−2xi(yi−α−βxi)⇒0=∑i=1n(xiyi−αxi−βxi2)0 = \sum_{i=1}^n -2x_i(y_i - \alpha - \beta x_i) \quad\Rightarrow\quad 0 = \sum_{i=1}^n (x_i y_i - \alpha x_i - \beta x_i^2)

Substituting α\alpha from (1):

0=∑i=1n(xiyi−yˉxi+βxˉxi−βxi2)=∑i=1n(xiyi−yˉxi)−β∑i=1n(xi2−xˉxi)0 = \sum_{i=1}^n (x_i y_i - \bar{y} x_i + \beta \bar{x} x_i - \beta x_i^2) = \sum_{i=1}^n (x_i y_i - \bar{y} x_i) - \beta \sum_{i=1}^n (x_i^2 - \bar{x} x_i)

so

β=∑i=1n(xiyi−yˉxi)∑i=1n(xi2−xˉxi)(2)\beta = \frac{\sum_{i=1}^n (x_i y_i - \bar{y} x_i)}{\sum_{i=1}^n (x_i^2 - \bar{x} x_i)} \tag{2}

which is the same as cov(x,y)/var(x)\text{cov}(x, y) / \text{var}(x).

In short

  1. Calculate β\beta with (2).
  2. Substitute it into (1) to get α\alpha.
  3. Predict with y^=βx+α\hat{y} = \beta x + \alpha.
def get_beta(x: np.ndarray, y: np.ndarray) -> float:
    numerator = np.sum((x * y) - (y.mean() * x))
    denominator = np.sum((x**2) - (x.mean() * x))
    return numerator / denominator
 
 
def get_alpha(x: np.ndarray, y: np.ndarray) -> float:
    return y.mean() - get_beta(x, y) * x.mean()

On ten random points this gives α=1.18\alpha = 1.18, β=0.94\beta = 0.94, identical to scikit-learn's LinearRegression:

Least squares line through ten random points

And on price against horsepower from the UCI automobile dataset:

Least squares line of car price against horsepower

Gradient Descent fits the same line numerically instead of in closed form.

Multiple regression

With several independent variables, for example minutes a data scientist spends on a social network:

minutes=β1 friends+β2 work hours+β3 has PhD+α\text{minutes} = \beta_1 \,\text{friends} + \beta_2 \,\text{work hours} + \beta_3 \,\text{has PhD} + \alpha

the single number xix_i becomes a vector xi1,…,xikx_{i1}, \ldots, x_{ik}:

yi=β1xi1+⋯+βkxik+αy_i = \beta_1 x_{i1} + \cdots + \beta_k x_{ik} + \alpha

and the parameters become a vector β\beta. We assume the columns of xx are linearly independent: no column can be written as a weighted sum of the others. Otherwise the coefficients are not uniquely determined.

Derivation adapted from Linear Regression Intuition by Ridley Leisy; multiple regression from Joel Grus, Data Science from Scratch, chapter 15.

Original notebook

Linked from

Revision log

  1. Migrated from data-science-notes

Tags: Statistics