Deriving the Line of Best Fit

We will define the set of points P as follows

P=\left\{(1,1),(2,2),(3,4),\ldots\right\}

And the line of best fit will be defined as

y=mx+b

Finally to express the error of a line just sum the error for each point. The error for a point is the squared difference between the point’s $y$ coordinate and the lines estimate for the point’s $x$ coordinate, $mx+b$ . The reason for squaring it is to make it always positive. Otherwise a positive error and negative error could cancel out. The reason for squaring instead of using an absolute value is for ease of the derivative. As an expression for each point it is $(y-mx-b)^2$ . The total error for the whole line can be written as:

E(m,b)=\sum_{(x,y)\epsilon P}\left(y-mx-b\right)^2

Going forward I will be leaving the bounds off of the summations, as they will all be the same and only muddle up the notation. The next step after finding the error function is to find the minimum. It would be really convenient if there was one minimum and it was the only critical point on the function. After expanding the function it is evident that its highest coefficient term of both $x$ and $y$ is parabolic.

\sum\left(y^2+m^2x^2+b^2-2ymx-2yb-2mxb\right)

That shows that it will have only one critical point. Next is checking if the critical point is a minimum or if it is a maximum or saddle point. As $b$ goes to positive or negative infinity the error function must go to positive infinity.

$A diagram showing how changing the y-intercept effects the error$

Less obviously but with similar logic the same is true for $m$ .

$A diagram showing how changing the slope effects the error$

Having shown that the critical point is the point we are looking for, the next step is to find it. We will start by taking the partial derivatives of the error function.

\frac{\partial E}{\partial m}=\sum\left(2mx^2-2yx+2xb\right)

\frac{\partial E}{\partial s}=\sum\left(2b-2y+2mx\right)

Now all that remains is to find where both of those equal zero. Starting with $\frac{\partial E}{\partial s}$ first set it to zero, then divide both sides by two. Then distribute the summations and you are left with:

0=\sum b-\sum y+\sum mx

The $m$ can be distributed out of the summation and the $\sum y$ can be moved to the other side.

\sum y=\sum b+m\sum x

For the next step remember what bounds on the summations are. The term $\sum b$ means the sum of $b$ added for each point. The term $\sum x$ means the total of all the $x$ coordinates. If we divide both sides by the number of points then $\sum b$ will change to $b$ and $\sum x$ will change to the average $x$ value. Now we have the equation

\bar{y}=m\bar{x}+b

showing that line of best fit goes through the average point. The other equation doesn’t turn out as neat, but still isn’t too bad to solve. Doing very similar steps as to the last equation you can find that

m\sum {x^2}+b\sum x=\sum {xy}

Changing all the summations to averages will not be as nice as last time, but still make it easier to work with.

m\bar{x^2}+b\bar{x}=\bar{xy}

You can then plug in the other formula tothat and solve for $m$ to get

m=\frac{\bar{xy}-\bar{x}\bar{y}}{\bar{x^2}\bar{x}^2}

The result from that can be used with the $\bar{y}=m\bar{x}+b$ equation to find the line of best fit.

Created on: December 31, 2023