UMVUE  (Uniformly Minimum Variance Unbiased Estimator)

Definition of a UMVUE

Let p(x; θ) be a probability distribution. The quality of an estimator of the parameter θ is measured by its Mean Square Error (MSE) which reduces to the estimator variance when the estimator is unbiased.

In the class of unbiased estimators of θ, is there an estimator "better" than all the other estimators in the class, that is :

    * Whose variance is smaller than the variance of any other unbiased estimator,

    * For all values of the parameter θ ?

It is not always the case, but when it happens to be true, this estimator is called a Uniformly Minimum Variance Unbiased Estimator (UMVUE) of θ, the term "Uniformly" meaning "for all values of θ".

An unbiased estimator may have the lowest variance among all unbiased etimators only for certain values of θ. It is then called a Locally Minimum Variance Unbiased Estimator.

The concept generalizes straightforwardly to that of the UMVUE of a function g(θ) of a parameter θ.

Properties of a UMVUE

Existence

A function g(θ) of the parameter θ does not necessarily have a UMVUE. For example, we'll show that the parameter θ of the uniform distribution

U[θ, θ + 1] has no UMVUE.

Uniqueness

We'll show that when it exists, a UMVUE is unique. Any "other" unbiased estimator with the same variance as a UMVUE is in fact identical to this estimator.

Is a UMVUE efficient ?

Under certain regularity conditions, the variance of an unbiased estimator cannot be smaller than a certain theoretical limit known as the Cramér-Rao lower bound. An unbiased estimator whose variance is equal to the Cramér-Rao lower bound is said to be efficient.

Some UMVUEs are efficient but most are not, as their variances are larger than the Cramér-Rao lower bound. We give here an example of a UMVUE that is not efficient.

-----

Recall that, given a probability distribution p(x; θ), there is at most a single function g(θ) that can be efficiently estimated, whereas we'll see that many functions of θ can have UMVUEs. This confirms the fact that the concept of UMVUE is both weaker and more general than the concept of efficient estimator.

This relative weakness may also be detected in the relationships that these two types of estimators entertain with the concept of sufficient statistic (see below).

UMVUE and MSE

A UMVUE, even if it is efficient, does not necessarily have the lowest Mean Squared Error (MSE) of all known estimators of the estimated quantity. We give here examples of estimators which, although biased, have a lower MSEs than the corresponding UMVUEs.

Characteristic properties

Let p(x; θ) be a probability distribution. An unbiased estimator of 0 is a statistic whose expectation is equal to 0 for all values of the parameter θ. We'll establish the following necessary and sufficient condition for an unbiased estimator to be a UMVUE :

-----

We'll show that this condition is equivalent to this other condition :

    * Let U be an unbiased estimator of 0,

    * And let θ^* be an unbiased estimator of g(θ).

θ^* is the UMVUE of g(θ) if and only if

In other words, an unbiased estimator θ^* of g(θ) is the UMVUE of g(θ) if and only if :

    * For any unbiased estimator of 0 U, 

    * (θ^*.U ) is also an unbiased estimator of 0.

______________

These two necessary and sufficient conditions are useful for :

    1) Proving that a given estimator is a UMVUE.

    2) Proving that a given estimator is not a UMVUE.

    3) Proving that a given g(θ) has no UMVUE, or even that no g(θ) has a UMVUE.

UMVUE and sufficiency

Let T be any sufficient statistic for θ. We'll show that a UMVUE of a function g(θ), when it exists, is necessarily a function of T.

This result is less restrictive than the similar result about efficient estimators, which states that an efficient estimator is not just a function of a sufficient statistic, but is a sufficient statistic.

Building a UMVUE

The above characteristic properties are not of much use for building UMVUEs. But there exist two powerful methods for building UMVUEs :

    1) The Lehmann-Scheffé theorem states that blackwellizing any unbiased estimator of g(θ) by a sufficient and complete statistic generates the (unique) UMVUE of g(θ).

The procedure for building a UMVUE then consists in :

    - First identifying an unbiased estimator of g(θ), without regards for its quality (variance).

    - Then identifying a complete statistic for g(θ).

    - Finally blackwellizing the unbiased estimator by the complete statistic. This is usually the difficult part as conditional expectations usually lead to rather cumbersome calculations. The result is the unique UMVUE of g(θ).

-----

    2) The Corollary of Lehmann-Scheffé's Theorem, which states that if an unbiased estimator of θ is a function of a statistic that is complete for θ (and therefore for g(θ)), then this estimator is the unique UMVUE of g(θ).

This result can be used two ways :

        * Most frequently, one identifies more or less easily an unbiased estimator of g(θ) which luckily turns out to be based on complete statistic. This estimator is then the UMVUE of g(θ).

        * But one may occasionally use the Corollary the other way around : given a complete statistic T, one tries to identify the conditions that a function h(T) of this complete statistic must verify in order to be an unbiased estimator of g(θ). If these conditions can be satisfied, then h(T) is the unique UMVUE of g(θ).

We'll use this approach when we calculate the UMVUEs of :

    * Any analytic function of the parameter λ of the Poisson(λ) distribution.

    * Any differentiable function of the parameter θ of the uniform distribution U[0, θ].

_________________________________________________________

In this Tutorial, we establish three important properties of Uniformly Minimum Variance Unbiased Estimators (UMVUE) :

    * Such an estimator, when it exists, is unique.

    * An unbiased estimator is a UMVUE if and only if it is uncorrelated with any unbiased estimator of 0. We then give another N&S condition for an unbiased estimator to be a UMVUE.

    * A UMVUE is a function of any sufficient statistic.

-----

We finally insist that although the demonstrations are given only for UMVUEs of a parameter θ, they are in fact valid for any function g(θ) of the parameter.

UNIFORMLY MINIMUM VARIANCE UNBIASED ESTIMATOR (UMVUE)

________________________________________________________________

In this Tutorial, we identify some UMVUEs pertaining to the normal distribution N(µ, σ²).

    * We first get out of the way the UMVUE's of µ (when the variance is known) and σ² (when the mean is known).

    * Encouraged by these first easy successes, we set out to calculate the UMVUEs of the first integer powers of the mean. The outcome is not up to our hopes : the UMVUEs are indeed identified up to the fourth power, but the calculation for each power is more intricate than for the previous one, and seems to be hopelessly complex for the fifth power. Worse, no regular pattern seems to be emerging from these calculations (i.e. general formula or recursion formula), so our efforts for calculating the UMVUEs of the powers of µ stop here.

    * Calculations about the variance are usually more complex than similar calculations about the mean. So we turn to the UMVUEs of the powers of the standard deviation σ with some diffidence. To our surprise, we'll be able to identify a general expression for the UMVUE of σ ^r, where r does not even have to be an integer, and can even be negative (down to a certain limit). In addition, the two cases "Mean known" and "Mean unknown"  lead to virtually identical calculations.

    * This result paves the way to the calculation of the UMVUEs of a large number of quantities of the form µ ^pσ ^r. As an example, we'll calculate the UMVUE of µ / σ.

All these UMVUEs are identified by calling on the Corollary of the Lehmann-Scheffé theorem.

__________

We then address the important question of the unbiased estimation the cumulative distribution function of the normal distribution. The Fundamental Theorem of Statistics (see here) states that the empirical distribution function is a convergent estimator of the distribution function F(x) = P{X ≤ x}, but says nothing about estimating distribution functions with finite samples, not even whether this estimator is biased or not. So the "empirical distribution function" approach will not be pursued any further.

In the special case of the normal distribution, though, it is possible to calculate the UMVUE of P{X ≤ x} from finite samples.

    * When the variance is known, the sample mean  is the UMVUE of the distribution mean. One could then imagine that

(where Φ(.) denotes the distribution function of the standard normal distribution) is the UMVUE of the distribution function of the normal distribution N(µ, σ²), but it is not so. We'll identify the genuine UMVUE.

    * Things get more complicated when the variance is unknown. We'll again identify the UMVUE of the distribution function of the normal distribution, but we'll have to accept several difficult results without proof.

-----

These are rare examples where UMVUEs are calculated by using the Lehmann-Scheffé theorem itself rather than its Corollary. Another example is found here, where we calculate the UMVUE of the function e^-λ of the parameter λ of the exponential distribution Exp(λ).

NORMAL DISTRIBUTION : SOME UMVUEs

__________________________________________________

In this Tutorial, we keep on identifying some classical UMVUEs.

    * We first identify directly the UMVUE of the square of the parameter λ of the Poisson(λ) distribution.

    * We then derive a much more general expression for the UMVUE of any analytic function g(λ) of the Poisson parameter λ. This expression is then used for calculating the UMVUEs of :

        - g(λ) = e^-λ, a result already obtained here as an application of the Rao-Blackwell theorem, and here as an application of the Corollary of the Lehmann-Scheffé theorem.

        - Any positive integer power λ^r of λ. As a special case, we obtain again the UMVUE of λ². As an even more special case, we'll see that the sample mean  is the UMVUE of λ.

-----

    * We calculate the UMVUE of the parameter θ of the uniform distribution U[0, θ], and then generalize this result to any differentiable function g(θ).

    * We finally come to studying the uniform distribution U[θ, θ + 1], and discover that there is no UMVUE for θ or any function of θ in this configuration.

We already knew that U[θ, θ + 1] has no complete statistic for θ, and that, consequently, neither the Lehmann-Scheffé theorem nor its Corollary can be called on. Yet, the Lehmann-Scheffé theorem is only a sufficient condition of existence of a UMVUE, and θ having no complete statistic does not imply that no function g(θ) has a UMVUE : so the lack of UMVUE for any function of the parameter θ of U[θ, θ + 1] has to be demonstrated, which we'll do.

POISSON DISTRIBUTION

UNIFORM DISTRIBUTION

__________________________________________

We now describe a family of distributions p(x; θ) known as "power series distributions" and whose general expression is

where x is a non negative integer, and θ > 0.

The identification of the UMVUEs of the integer powers θ ^r of the parameter θ is then particularly simple.

The UMVUEs of the powers of the parameter λ of the Poisson(λ) distribution that we calculated in the preceding Tutorial will appear as a special case.

POWER SERIES DISTRIBUTIONS

___________________________________________________________

We now calculate some UMVUEs pertaining to the Bernoulli, geometric and negative binomial distributions.

    * We know that the sample mean  is the UMVUE of the parameter p (the mean) of the Bernoulli distribution b(p). We now identify the UMVUE of the variance σ² = p(1 - p) of this distribution. This will require using the Lehmann-Scheffé theorem (not the Corollary).

Note that we identify here a biased estimator of σ² = p(1 - p) with a lower MSE than this UMVUE.

    * We then calculate the UMVUE of the parameter p of the geometric distribution. This may seem unnecessary, as we'll later calculate a more general UMVUE pertaining to the negative binomial distribution, of which the geometric distribution is a special case. But we'll see that this step cannot be by-passed.

    * We'll easily identify the UMVUE of the mean µ = k/p of the negative binomial distribution, and, as a special case, that of the mean µ = 1/p of the geometric distribution. So we feel confident that identifying the UMVUE of p will cause no problem. It will turn out that we'll rather painfully identify the UMVUE of p^r, with r a positive integer in a narrow range (which, fortunately, includes r = 1). We are not aware of the existence of UMVUEs valid for any integral power of p.

Although the geometric distribution is a special case of the negative binomial distribution, this result is not usable for identifying the UMVUE of the parameter p of the geometric distribution because of the restrictions on the values of the integral exponent r. So our previous effort (see above) was necessary after all.

    * The same method will the be used for identifying the UMVUE of the variance of the negative binomial distribution. The result applies to the geometric distribution as a special case.

MORE UMVUEs

_____________________________________________________

Related readings :