Gauss-Markov theorem

In Simple or Multiple Linear Regression, the model parameters are most often calculated by the Least Squares method. The main advantage of this method is its mathematical simplicity, which allows an easy identification of the statistical properties of the calculated estimators, in particular their bias (which is 0) and their covariance matrix. But there is no a priori reason to believe that these estimators are particularly good (low Mean Square Error of the parameters and of model predictions ).

The Gauss-Markov theorem is here to somewhat soften our worries about the quality of the Least Squares estimator of the vector of the model parameters. This theorem states that among all possible estimators that are both :

    * Linear in the observations,

    * And unbiased,

the Least Squares estimator is the one with the smallest variance.

-----

More precisely, we'll show that if :

    * β^* denotes the vector of parameters estimated by the Least Squares method,

    * β⁺ denotes any vector linear in the observations and estimating the true vector β without bias,

then the covariance matrix of  β⁺ (that we denote Var(β⁺)) is obtained by adding a positive semi-definite matrix to the covariance matrix of β^*

and we'll calculate the matrix Z.

From this result we'll deduce that :

    1) The variances of the individual components of β^* are smaller than the variances of the corresponding components of β ⁺.

    2) The adjusted values of the model as well as its predictions have smaller Mean Square Errors (MSE) when using the vector of parameters β^* than when using the vector of parameters β ⁺.

For these reasons, the Least Squares estimator β^* is sometimes said to be a "BLUE", the acronym of Best Linear Unbiased Estimator.

 ________________________________________________________

In this Tutorial, we give two demonstrations of Gauss-Markov theorem as described above.

From this result, we'll deduce that the vector of parameters estimated by the Least Squares method is better (smaller variance of the parameters and smaller MSE of the model predictions) than any other unbiased estimator built as a linear function of the observations.

GAUSS-MARKOV THEOREM

______________________________________________________

Related readings :