Estimators  (Combination of independent..)

If you are not familiar with the notion of estimator, we suggest that you first read this.

Suppose that one quantity Q (for example the mean m of a distribution), has to be estimated. For some reason, the estimation is conducted by two independent  teams, each team using its own pet estimator:

    * Team 1  uses estimator Q₁, that produces number Q^*₁ as an estimate of Q. Estimator Q₁ is known to be unbiased.

    * Team 2  uses estimator Q₂, that produces number Q^*₂ as another estimate of Q. Estimator Q₂ is also known to be unbiased.

Is there a way to merge these two independent estimates into a single, improved estimate Q^*_c of Q ? By "improved", we mean that :

    * The expectation of Q^*_c , considered as a random variable, must still be Q, that is, Q^*_c must be one realization of a new unbiased estimator of Q (just as Q₁ and Q₂ are) that we'll denote Q_c,

    * And the variance σ² of this estimator must be less than both the variance σ₁² of Q₁ and the variance σ₂² of Q₂. In fact, we'll obtain even more, as we are going to identify the one and only Q_c with minimal variance.

A simple idea is to use the average of Q^*₁ and Q^*₂  as a combined estimation of Q :

Q^*_c = (Q^*₁ + Q^*₂)/2

This means that we use :

Q_c  = 1/2.(Q₁ + Q₂)

as the new synhetic estimator of Q.
This idea, natural as it is, is not very good. Q_c is indeed unbiased, but that its variance, although certainly smaller than the larger of σ₁² and σ₂², may be larger than the smaller of σ₁² and σ₂², which means that Q_c  may be worse than the best of Q₁ and Q₂.
A better idea is to seek a more general linear combination of Q₁ and Q₂ : 

Q_c  = λ₁.Q₁ + λ₂.Q₂

such that :

    * Q_c is unbiased,

    * and with minimal variance.
 

We show here that the solution is :

and that the variance of Q_c is then :

which is clearly smaller than both σ₁² and σ₂².

So if the variances σ₁² and σ₂² of Q₁ and Q₂ are known, the best possible estimate of Q is :

Q^*_c = λ₁.Q^*₁ + λ₂ .Q^*₂

with the values of  λ₁ and λ₂ as above.

Warning : this does not mean that Q^*_c is closer to the true value of  Q than Q^*₁ and Q^*₂ (although it is certainly closer to Q than at least one of  Q^*₁ or Q^*₂). It only means that Q_c has a narrower distribution around Q than Q₁ and Q₂ (or any linear combination of Q₁ and Q₂). In other words, it is still quite possible that the error (Q^*_c - Q) be larger than either  (Q^*_c - Q₁) or (Q^*_c - Q₂) (but certainly not both).

________________________________

We show here that if :

    * Q (the quantity to be estimated) is the mean of a distribution,

    * Q₁ and Q₂ are the averages of two samples of different sizes,

then Q_c is not just the best linear combination of Q₁ and Q₂ : it is in fact the very best estimator possible.

________________________________

Combining independent estimators is somewhat related to Weighted Least Squares in Simple Linear Regression. The expression for Q_c may be re-written :

Q_c = (σ₁².σ₂² / (σ₁² + σ₂²)).[ Q₁/ σ₁² + Q₂ / σ₂²]

which shows that the best estimator for Q is a linear combination of Q₁ and Q₂ with coefficients inversely proportional to the estimators' variances.

In Weighted Linear Regression, the Weighted Least Squares line is the best estimator of y (the response variable), and it is obtained by minimizing the sum of the squares of the weighted residuals, each residual being ponderated by the inverse of the local variance of y.

 _____________________________________________

The following animation illustrates the concept of "best linear combination of independent estimators".

____________________________________________

Related readings :