Statistics: Derive Linear Regression

Derive Linear Regression Equations

Linear regression is a tool that allows to compute a linear law at minimum square error from an input and output set

I need to extend linear regression to multidimensional input and output.

First I make sure I can derive regular linear regression.

1Definitions

2Algorithm

1. I develop the error to extract dependence on the input and output set only
2. I develops the sum series into accumulators that can be computed once at the start of the process and reduce computation of the linear regression in a manipulation of only five variables
3. I compute the partial derivative of the error in respect of each linear regression parameter
4. The minimum of the function is a point of slope zero. I generate one equation per each partial derivative to search for the minimum
5. Solving the system for the linear regression parameters gives me the parameters to achieve minimum square error
6. By extending the computation to multiple input and output dimension, I can achieve my multidimensional linear regression

3Execution

3.1Develop the Error

Compute error metric in function of training set and linear regression parameters

Introduce accumulator FOMS

3.2Partial Derivatives

Objective: find the parameters Bias and Gain that results in the minimum value of the error metric.

The minimum is a point with a slope of zero, so the problem becomes finding the combination of Gain and Bias that results in the derivative of the error being zero.

To make my life easier, I see that the square root is a monotone function.

If I find the minimum of the argument, I also find the minimum of the square root.

First I compute the partial derivatives of the error argument in respect to Bias and Gain.

3.3Minimum Error

Make a system of equation with the partial derivatives to find the minimum of the square error

4Recap

Linear regression that achieves minimum square error

5Conclusions

From a training set with input and output vectors, it's possible to compute the linear regression that achieves minimum square error.

Furthermore, it's possible to compute such error from accumulator FOMS that allows to compute the linear regression parameters from five FOMS independently from the size of the original set.