Where do these equations for linear regression come from?

Question

I found the following equations in my notes:

Linear Approximation by Least Squares

A first approximation is the regression line: the least squares approximation to the data. If written in the form $y = c_0 + c_1 (x - \bar{x})$, the normal equations are diagonal: $$ \begin{pmatrix} m + 1 & 0 \\ 0 & \sum_i (x_i - \bar{x})^2 \end{pmatrix} \begin{pmatrix} c_0 \\ c_1 \end{pmatrix} = \begin{pmatrix} \sum_i y_i \\ \sum_i (x_i - \bar{x}) y_i \end{pmatrix}. $$

The coefficients are: $$ c_0 = \bar{y}, \quad c_1 = \frac{\sum_{i=0}^m (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=0}^m (x_i - \bar{x})^2}. $$

I have seen the expression for $c_1$ being derived elsewhere (see Mathworld and Finding equation of best fit line in simple linear regression), but I know not where the equality $c_0 = \overline{y}$ nor the above matrix from which the equation is derived come from. In fact, it seems to me that the equation contradicts the result in formula (13) in the Mathworld article, namely

$$c_0 = \frac{\overline{y}\sum_i x_i^2-\overline{x}\sum_ix_iy_i}{\sum_ix_i^2-n\overline{x}^2}.$$

Answer 1

Let $\mathbb Y$ denote the $(m+1)\times 1$ matrix whose $i$-th entry is $y_i$, let $\mathbf 1$ denote the $(m+1)\times 1$ matrix with all entries equal to $1$, and let $\mathbf Z$ denote the $(m+1)\times 1$ matrix whose $i$-th entry is $x_i-\overline x$. Finally, let $\mathbf c = \begin{pmatrix}c_0\\c_1 \end{pmatrix}$.

Define the block matrix $\mathbb X = \begin{pmatrix}\mathbf 1 | \mathbf Z \end{pmatrix}$, so that $$\sum_{i=0}^m [y_i - c_0 -c_1(x_i-\overline x)]^2 = \|\mathbb Y - \mathbb X\mathbf c \|_2^2.$$ From least squares theory, the normal equation writes as $\mathbb X^\top\mathbb X \mathbf c = \mathbb X^\top \mathbb Y$.

By block matrix multiplication, $\mathbb X^\top\mathbb X$ is equal to $\begin{pmatrix}\mathbf 1^\top \mathbf 1 & \mathbf 1^\top \mathbf Z \\\mathbf Z^\top\mathbf 1 & \mathbf Z^\top\mathbf Z \end{pmatrix}$. The quantity $\mathbf 1^\top \mathbf 1$ is $\sum_{i=0}^m 1^2 = m+1$, while $\mathbf 1^\top \mathbf Z = \mathbf Z^\top\mathbf 1 = \sum_{i=0}^m \mathbf Z_i=0$. At last, $\mathbf Z^\top\mathbf Z = \sum_{i=0}^m (x_i-\overline x)^2$.

Similarly, $\mathbb X^\top \mathbb Y = \begin{pmatrix}\mathbf 1^\top \mathbb Y\\ \mathbf Z^\top \mathbb Y \end{pmatrix} = \begin{pmatrix}\sum_{i=0}^m y_i\\ \sum_{i=0}^m (x_i-\overline x)y_i \end{pmatrix}$. So the normal equation in your notes is correct.

Solving for $\mathbf c$ yields $\mathbf c = \begin{pmatrix} m+1 & 0 \\ 0 & \sum_{i=0}^m (x_i-\overline x)^2 \end{pmatrix}^{-1} \begin{pmatrix}\sum_{i=0}^m y_i\\ \sum_{i=0}^m (x_i-\overline x)y_i \end{pmatrix} = \begin{pmatrix} \frac 1{m+1}\sum_{i=0}^m y_i\\ [\sum_{i=0}^m (x_i-\overline x)y_i]/[\sum_{i=0}^m (x_i-\overline x)^2]\end{pmatrix}.$

To get the form of $c_1$ that appears in your notes, observe that $$\sum_{i=0}^m (x_i-\overline x)(y_i-\overline y) = \sum_{i=0}^m (x_i-\overline x)y_i-\overline y \sum_{i=0}^m (x_i-\overline x)=\sum_{i=0}^m (x_i-\overline x)y_i-0.$$

Answer 2

The solution $c_0=\bar y$ can be easily obtained from (28) of your linked Mathworld article: $$ c_0=\bar y-c_1(\bar x-\bar x)=\bar y. $$ To see this you just need to substitute each $x_i$ by $x_i-\bar x$ so that the regression problem $y=c_0+c_1(x-\bar x)$ now becomes $y=c_0+c_1x$. This is equivalent to shift the $x$ data.

You can also shift the data in another way: we have $y=c_0-c_1\bar x+c_1 x$ and we can substitute $c_0-c_1\bar x$ by $c_0$. Once you solve out this new $c_0$, you can get the original $c_0$ by adding $c_1\bar x$: $$ \begin{aligned} \frac{\overline{y}\sum_i x_i^2-\overline{x}\sum_ix_iy_i}{\sum_ix_i^2-n\overline{x}^2}+c_1\bar x&=\frac{\overline{y}\sum_i x_i^2-\overline{x}\sum_ix_iy_i+\bar x(\sum_i x_iy_i-n\bar x\bar y)}{\sum_ix_i^2-n\overline{x}^2}.\\ &=\frac{\overline{y}\sum_i x_i^2-\bar x\cdot n\bar x\bar y}{\sum_ix_i^2-n\overline{x}^2}\\ &=\bar y. \end{aligned} $$

The same result shows there is no contradiction.