OPTIMAL LINEAR COMBINATION

OF INDEPENDENT UNBIASED ESTIMATORS

We denote :

    * Q the quantity to be estimated,

    * Q₁ an unbiased estimator of Q, with known variance σ₁².

    * Q^*₁ a realization of Q₁.

    * Q₂ a second unbiased estimator of Q, with known variance σ₂².

    * Q^*₂  a realization of Q₂.

    
Q₁ and Q₂  are assumed to be independent (or rather, uncorrelated).

We seek values for  λ₁ and λ₂ such that :

Q_c  = λ₁.Q₁ + λ₂.Q₂

    * is unbiased,

    * and has the lowest possible variance.

1) Estimator Q_c must be unbiased.

    We want Q_c's expectation to be Q (the common mean of the unbiased estimators Q₁ and Q₂). So, if E denotes the expectation, we want :

E[λ₁.Q₁ + λ₂.Q_{2 ]} = Q

 But :

E[λ₁.Q₁ + λ₂.Q₂] = λ₁.E[Q₁] + λ₂.E[Q₂ ] = λ₁.Q + λ₂.Q = ( λ₁ + λ₂).Q

 So we must have :

( λ₁ + λ₂).Q = Q   or   λ₁ + λ₂ = 1

We will further denote the two coefficients λ₁ = λ and λ₂ = (1 - λ).

 2) Estimator Q_c must have minimal variance

    The variance of Q_c  is :

Var(Q_c) = Var(λ.Q₁ + (1 -λ).Q₂)    

Because Q₁ and Q₂ are uncorrelated, we have :

  Var(λ.Q₁ + (1 -λ).Q₂) = Var(λ.Q₁) + Var((1 -λ).Q₂)  

So :

Var(Q_c) = σ² = λ².Var(Q₁) + (1 -λ)².Var(Q₂) = λ².σ₁² + (1 -λ)².σ₂²

We want σ² to be minimal, so we take it's derivative with respect to λ, and make it equal to 0 :

 dσ² / dλ = 2.(λ.σ₁² - (1 -λ).σ₂²) = 0

or :

The second derivative of σ² is :

d²σ²/dλ² = 2.(σ₁² + ₂²) > 0

so the extremum of σ₂² is indeed a minimum.

The minimal value of σ² is therefore :

σ² = λ₁².σ₁²  + λ₂². σ₂² = σ₂⁴ .σ₁²/ (σ₁² + σ₂²)²  +  σ₂² .σ₁⁴ /(σ₁² + σ₂²)²

or :

This value is smaller than both σ₁² and σ₂². For example, the ratio of σ² to σ₁² is :

σ² / σ₁² = σ₂² / (σ₁² + σ₂²) < 1

__________________________________

Imagine that we are trying to estimate the mean of a distribution, for example the true position of a star on the celestial sphere.

Suppose that both Q^*₁ and Q^*₂ were obtained with the same telescope, whose measurement uncertainty is characterized by the single variance σ_i². But :

    * Q^*₁ was obtained as the average of n₁ measurements,

    * while Q^*₂ was obtained as the average of n₂ measurements the next night.

The variance of the distribution of Q₁ is σ₁² = σ_i² / n₁, while that of  Q₂ is σ₂² = σ_i² / n₂. So the minimal variance of Q_c is :

σ² = (σ_i² / n₁).(σ_i² / n₂) / (σ_i² / n₁ + σ_i² / n₂)

or

σ² = σ_i² /( n₁ + n₂)

Now imagine that the two samples are merged into one single large sample with n = (n₁ + n₂) observations. It is a known fact that the average of these n observations is the very best estimator of the true mean (i.e., no other unbiased estimator of the mean has a lower variance than the sample average). The variance of the distribution of the average of the large sample is :  

σ_i² /( n₁ + n₂)

which is just what we found for the minimal variance of Q_c. So, at least in this case, we know for sure that Q_c is the very best estimator possible.