Bias

Quite generally, the term "bias" refers to a systematic (i.e. not random) difference between a quantity and a predition of this quantity.

Bias of an estimator

Let q^* be an estimator of a parameter q of a probability distribution. q^* is said to be unbiased if its expectation is equal to the value of the parameter :

E[q^*] = q 

In other words, the value of an unbiased estimator is, on the average, equal to the value to be estimated.

If the expectation of the estimator is different from the value of the parameter, the estimator is said to be "biased", and its bias is then defined as the difference between the expectation of the estimator and the true value of the parameter :

Bias = E[q^*] - q 

The value of the bias depends, in general, on the value of the parameter.

Bias and Mean Square Error

The lack of bias is clearly an attractive feature of an estimator, but it is not a fundamental one. The quality of an estimator, for a given sample size, is rather measured by its Mean Square Error (MSE), the expected value of the squared difference between the value of the estimator and the value to be estimated :

MSE = E[(q^* - q)²]

It can be shown that :

MSE = Var(q^*) + Bias(q^*)²

and it is then quite possible that a certain amount of bias be more than compensated for by a low variance (see for instance Ridge Regression, and the animation about the bias-variance tradeoff).

Removing the bias of an estimator

Besides, it is sometimes possible to remove the bias of a biased estimator. The most classical example is that of the estimator of a variance :

    * The estimator obtained by the method of moments :

s² = 1/n .S_i (x_i - )²        with        = 1/n .S_ix_i  

is biased ;

    * Whereas the "corrected" estimator :

s' ² = 1/(n - 1) .S_i (x_i - )²

is unbiased (lower image of the above illustration).

Minimum Variance Unbiased Estimator (MVUE)

Unbiased estimators are easier to analyze than convergent but biased ones because the bias may depend on the value of the parameter in a complex manner. Unbiased estimators have therefore been studied quite extensively.

In particular, it is common for a parameter to have several, or even an infinity of unbiased estimators. For example :

    * For the normal distribution N(µ, s²) :

        - The sample mean,  and

        - The sample median,

are both unbiased estimators of the mean µ of the distribution.

    * For the uniform distribution in [0, q] :

        - Twice the sample median,  and

        - [(n + 1)/n].x_(n) , where x_(n) is the rightmost observation of the n-sample

are both unbiased estimators of q.

It is of course desirable to identify and use the (unbiased) estimator with the smallest variance. This estimator is called the "Minimum Variance Unbiased Estimator" (MVUE) of q, and we'll denote it by q^*_M.

-----

In the Tutorial below, we establish two important properties of a Minimum Variance Unbiased Estimator :

Cramér-Rao lower bound

What is the smallest variance that can be achieved by an unbiased estimator ?

In many cases, it is possible to identify a lower bound to the variance of an unbiased estimator. This lower bound is then given by the Cramér-Rao inequality,

that we describe is some detail here.

Rao-Blackwell theorem

If an unbiased estimator of q is not a MVUE, it may be improved (i.e. its variance may be reduced) provided that a sufficient statistic for q is available. This is the object of the Rao-Blackwell theorem.

The new, improved estimator is not garanteed, though, to be a MVUE.

Bias of a model

Local bias

At each and every point of the space of the input variables, the role of a model is to make a prediction of the value of a certain quantity (e.g. the value of the response variable in predictive models). As the fitted model depends on the sample, this prediction is a random variable, that is used as an estimator of the quantity whose value is to be predicted.

This estimator may be :

    * Unbiased (e.g. Simple Linear Regression under the standard conditions).

    * But it may also be biased (see Ridge Regression, and the first part of the animation on the bias-variance tradeoff).

Global bias

One may also consider the average bias over the range of the input variables, ponderated by the joint probability density of these variables. This quantity measures the global, or average bias of the model, that is, its ability to accommodate the shape of the deterministic part of the proces that generated the data (lower image of this illustration) :

The reader may refer to the second part of the animation on the bias-variance tradeoff.

________________________________________________________________

In this Tutorial, we establish two important properties of a Minimum Variance Unbiased Estimator :

    * It is unique : the variance of any other unbiased estimator of the parameter is strictly larger than that of the MVUE.

    * An unbiased estimator is a MVUE if and only if it is uncorrelated with any unbiased estimato of 0.

MINIMUM VARIANCE UNBIASED ESTIMATORS

 ________________________________________________

Related readings :