Minimal sufficient statistic

We noticed that the transform T ' = f(T)  of a sufficient statistic T by a function f(.) is usually not sufficient : only when f(.) is one-to-one is f(T) always still sufficient. But we also noticed that f(T) may occasionally be sufficient even if f(.) is not one-to-one. Within the paradigm of the "useful" and "useless" information carried by a statistic T, f(.) then sheds away some of the useless information, but retains all of the useful information.

T ' may be thought of as a "leaner" sufficient statistic than T.

A natural question is then whether there is a limit to the amount of useless information that can be removed from a sufficient statistic without also removing some of the useful information, thus spoiling the "sufficient" nature of the statistic.

The answer to this question comes in two steps :

    1) It can be shown that, under some mild conditions, there always exist sufficient statistics that contain as little useless information as possible : any transform of these statistics by a function that is not one-to-one then necessarily produces a new statistic that is not sufficient anymore. Such statistics are called minimal sufficient statistics.

    2) But there is no guarantee that a minimal sufficient statistic does not contain some residual useless information.

It may be, though, that the useless information can be completely eradicated, and the minimal sufficient statistic then contains nothing but useful information for the estimation of the parameter. These "super" minimal sufficient statistics are called complete statistics.

We insist on the fact that a minimal sufficient statistic is not necessarily complete, as we show here.

-----

We may therefore visualize the set of all sufficient statistics of a parameter θ as an "ocean" in which :

    * Transforming a sufficient statistic by a one-to-one function produces another sufficient statistic at the same "depth".

    * Transforming a sufficient statistic by a not one-to-one function and yet obtaining another sufficient statistic takes us to a greater depth.

    * A minimal sufficient statistics lies on the floor of the ocean : no function of this statistic that is not one-to-one can produce another sufficient statistic. A minimal sufficient statistic may be, but is not necessarily complete.

Formal definition of a minimal sufficient statistic

The above considerations could lead to a definition of a minimal sufficient statistic (MSS). Unfortunately, this definition would not be operational, contrary to the following universally accepted definition :

Uniqueness of a minimal sufficient statistic

It is not clear from the definition alone whether a minimal statistic is unique or not. It is not, and clearly any one-to-one function of a minimal sufficient statistic is also minimal. But it is easily shown that given two minimal sufficient statistics M₁ and M₂, it is always the case that there exists a one-to-one correspondence between M₁ and M₂.

So, within a one-to-one correspondence, a minimal sufficient statistic is unique.

Partitioning the sample space

The definition of a MSS can be given the following geometric interpretation.

In the n-dimensional space spanned by the n variables {x₁, x₂, ..., x_n}, a n-sample X is represented by a single point. Consider any statistic T(X) and a fixed number t. The set A_t of the representative points of the samples such that T(X) = t is usually a (n - 1)-dimensional subspace of the sample space :

The sample space is partitionned into a set {A_t} of "sheets", each sheet A_t being identified by a value t.

Let f(.) be a function that is not one-to-one. Under the action of f(.), {A_t} will be transformed into another set of sheets {B_{f (t)}}. For a given t :

    * All the points in A_t will be in the same B_{f (t)},

    * But because f(.) is not one-to-one, there may be some other number t' such that f(t' ) = f(t). Then A_t and A_t' have the same image B_{f (t)} = B_{f (t' )} under

f(.).

Let now T ' be another statistic defined by T ' = f(T). It partitions the sample space into {B_{f (t)}} and several f(A_t)s may be contained into the same B_{f (t)}.

Consequently :

The transform T ' = f(T) of a statistic T by a function f(.) that is not one-to-one
generates a coarser partition of the sample space than T does.

By definition, a minimal sufficient statistic then generates the coarsest partition among all partitions generated by all sufficient statistics.

Identifying a Minimal Sufficient Statistic

Just as the definition of a sufficient statistic proved inconvenient for identifying sufficient statistics and was advantageously replaced by the Factorization Theorem for this purpose, the sole definition of a minimal sufficient statistic is essentially useless for finding minimal sufficient statistics. Fortunately, we'll identify a condition for a statistic to be sufficient minimal which makes the identification of a MSS much easier.

-----

Let p(x, θ) be a distribution known up to the value of the parameter θ, and T(X) a statistic (that needs not be assumed sufficient). We denote f_θ (X) the sample distribution.

Consider the following two conditions :

    1) For any two samples X and Y such that T(X) = T(Y) (and that therefore belong to the same sheet as in the above illustration), the ratio

f_θ (X) / f_θ (Y)

considered as a function of θ turns out, in fact, not to depend on θ.

    2) Conversely, for any pair of samples X and Y such that f_θ (X) / f_θ (Y) does not depend on θ, it turns out that T(X) = T(Y).

We'll show that :

This result is sometimes called the "Lehmann-Scheffé condition", or even the "Lehmann-Scheffé theorem on minimal sufficient statistics" despite the risk of confusion with the Lehmann-Scheffé theorem (see here).

____________________________________________________________________________

In this Tutorial, we first show that a minimal sufficient statistic is unique within a one-to-one function.

We then etablish the classical condition for a statistic to be minimal sufficient.

We finally review the sufficient statistics previously identified and show that they are in fact not only sufficient, but also minimal sufficient.

The case of the U[θ, θ + 1] uniform distribution will show that the dimension of a minimal sufficient statistic does not necessarily match the dimension of the parameter to be estimated.

MINIMAL SUFFICIENT STATISTIC

 _____________________________________________________

Related readings :