Rao-Blackwell theorem

The Rao-Blackwell theorem provides a method for improving the performance of an unbiased estimator of a parameter (i.e. reduce its variance) provided that a sufficient statistic for this estimator is available.

The rationale behind the Rao-Blackwell theorem

Let q^* be this unbiased estimator. Following an informal but useful line of thought often encountered in estimation, we'll say that the reason why this estimator is not a Minimum Variance Unbiased Estimator (MVUE) may be that it does not carry with it all the information useful for estimating q, but which is nonetheless available in the sample.

We can also say that a statistic T that is sufficient for q carries with it all the information in the sample that is useful for estimating q. But a sufficient statistic has no particular reason for being unbiased.

We can therefore imagine "combining" q^* and T into a new estimator U that would retain the best of both worlds :

    * The unbiasedness of q^*,

    * And the information in T about q that is missing in q^* by transfering this extra information into this new estimator.

We would then obtain an unbiased estimator of q that would be hopefully better (lower variance), or at least not worse than q^*.

-----

The Rao-Blackwell theorem identifies this combination.

The Rao-Blackwell theorem

Let :

    * q^* be an unbiased estimator of q.

    * T be a sufficient statistic for q.

Then the random variable :

    1) Is a statistic, that is a function of the sample that does not depend on q.

    2) Its expectation is equal to q, and U is therefore an unbiased estimator of q.

    3) Its variance is no larger than the variance of q^*.

    4) If its variance happens to be equal to that of q^*, then q^* is a function of T.

Rao-Blackwell and MVUE

As a Corollary of Rao-Blackwell theorem, we'll show that :

    * If q admits a sufficient statistic T,

    * And if q has a Minimum Variance Unbiased Estimator (MVUE), then this MVUE is a function of T.

Blackwellization

Combining as above a sufficient statistic and an unbiased estimator for the purpose of reducing the variance of this estimator is sometimes dubbed "blackwelling" the estimator. It is called upon when it is believed that the estimator can possibly be improved, for example because its variance is larger than the Cramér-Rao lower bound.

Yet :

    * Blackwellization usually leads to cumbersone calculations, as we'll see in the Tutorial below.

    * The Rao-Blackwell theorem says nothing about the quality of the new estimator, and certainly not that is the best possible unbiased estimator (MVUE).

_________________________________________________ 

In this Tutorial, we demonstrate the Rao-Blackwell Theorem.

In a first section, we give an intuitive line of reasoning that leads to this particular combination of an unbiased estimator and a sufficient statistic. We then proceed with the demonstration proper.

Introducing a sufficient statistic was suggested on heuristic grounds for the purpose of bringing to the estimator some additional information about the parameter. But it will turn out that it has another fundamental role in overcoming an unexpected difficulty : the expectation of an estimator conditionally to another statistic is usally not a statistic (we'll give a counter example), unless the conditioning statistic is sufficient.

THE RAO-BLACKWELL THEOREM

____________________________________________________________

In this Tutorial, we put the Rao-Blackwell Theorem to work on an example.

The problem is as follows :

    * A r.v. variable is Poisson distributed with unknown parameter l. A sample {x₁, x₂, ..., x_n} is drawn from the distribution, and we want to use this        sample for estimating the probability for X to be equal to 0.

We will examine a first "natural" estimator, that we'll reject because of its bias and the intractable calculations associated to its MSE.

We'll then identify a second estimator that will turn out to be unbiased. We'll reduce (with some difficulty) the variance of this estimator by a "blackwellization" procedure.

Calculations are a bit cumbersome, but we'll be rewarded by the discovery of a good unbiased estimator that would probably have been impossible to identify by a more direct method.

A FIRST EXAMPLE OF BLACKWELLIZATION

 _________________________________________

We now address a second example of blackwellization of an unbiased estimator.

The problem is as follows :

    * A random variable follows an exponential distribution with unknown parameter l. A sample {x₁, x₂, ..., x_n} is drawn from this distribution, and we want        to estimate the probability for X to be larger than t :

P{X > t} = ?

So this is a lifetime problem : we want to estimate the probability for a component to live longer than t before breakdown.

-----

This Table of Contents shows a similarity in the approach to this problem and that used for the previous problem. We comment on this similarity below.

A SECOND EXAMPLE OF BLACKWELLIZATION

The problem is slightly more difficult than the preceding one. It leads to an improved estimator that lies out of reach of intuition alone. To our knowledge, the variance of this estimator is unknown at this time.

______________________________________

These two problems belong to a class of problems to which the Rao-Blackwell Theorem brings powerful and original solutions.

They can be expressed as follows :

    * One considers a probability distribution p(x, q).

    * A sample {x₁, x₂, ..., x_n} is drawn from this distribution.

    * One wants to estimate quantities like :

P{a  X  b}

This probability can be expressed as a function f(a, b, q). Usually, a good unbiased estimator of q is available, and it is therefore natural to try to estimate P by substituting the estimator of q for q into f(.).

Unfortunately :

    * The estimator thus obtained is biased.

    * Studying its properties (bias, variance, MSE) leads to intractable calculations.

-----

Another approach is to estimate P by the proportion of observations that lie in the segment [a b]. This "naive" estimator is always unbiased, but it does not take the nature of the distribution into account, and one can therefore suspect that it is rather weak (large variance), and can be improved. This is what the Rao-Blackwell Theorem does provided that a sufficient statistic for q is available, and that this statistic proves to be also sufficient for P.

Most classic distributions can be succesfully tackled this way. The improved estimators thus found are always completely beyond guessing, and more often than not, one cannot calculate their variances. But by the Rao-Blackwell Theorem, these variances are smaller than the variances of the naive estimators.

 ___________________________________________________________

Related readings :