Central Limit Theorem

One of the cornerstones of the theory of random variables.

General idea of the Central Limit Theorem

Let X be a continuous random variable described by a probability density with a mean µ and a variance σ². In a nutshell, the Central Limit Theorem (CLT) states that for large samples, the sample mean _n of this distribution is approximately normally distributed. Moreover, it adds that the distribution of the sample mean can be made as close to normal as wanted : all it takes is to consider larger and larger samples.

In addition, for any distribution (with a mean and a variance) and any sample size n, although the distribution of _n is usually unknown (with some notable exceptions like normal, Chi-square, binomial or Poisson distributions), it is always true that :

    * The mean of _n is µ, the mean of the distribution.

    * The variance of _n is σ²/n.

So one can build the standardized variable

whose mean is 0 and whose variance is 1.

The CLT then states that for large samples, the distribution of _n^' is approximately N(0, 1), and that the approximation gets better and better as n grows without limit.

-----

This is the most commonly perceived idea of what the CLT is all about. But we'll see that, although correct, this idea does not tell the whole story.

Animation

The following animation illustrates the CLT.

The Central Limit Theorem

Let's now formulate the Central Limit Theorem in terms that are both more general and more correct.

    1) Since N(0, 1) is a continuous distribution, it would seem that the CLT applies only to continuous distributions. In fact it applies to any distribution (continuous, discrete or other) with a mean and a variance.

    2) The CLT does not address the convergence of probability density functions (or of probability mass functions), but the convergence of  distribution functions.

It states that, for any distribution with a mean and a variance, the distribution function of the standardized sample mean _n^' converges to the distribution function Φ(.) of the standard normal distribution when the sample size n grows without limit. This fundamental result is expressed as :

We are in a situation where the distribution function of a r.v. (the standardized sample mean) converges to a limit function (the distribution function of the standard normal distribution) : this kind of convergence is called "convergence in distribution".

Central Limit Theorem and probability distributions

Discrete random variables

The distribution function of a discrete rv is a step function (and is therefore not continuous), but an infinite sequence {F_n(x)} of discontinuous functions may very well converge to a continuous limit F(x) (as shown in this illustration), and such is indeed the case when the CLT is applied to a discrete rv.

Consider for example the binomial distribution. Its probability mass function assigns probabilities to integer values of the variable. The sample mean also has a probability mass function made of "spikes". The distance between these spikes tends to 0 as the sample size grows without limit, but such a sequence of functions does not converge to anything in the sense of Calculus (see animation).

But the distribution function of the standardized sample mean, although is is not continuous (step function), does converge to the distribution function of the standard normal distribution.

Convergence of probability density functions

Recall that just because a sequence {F_n} of differentiable functions converges to a differentiable limit F does not mean that that the sequence {F_n^' } of derivatives converges to the derivative F^' of F as shown in this illustration

Here, although {F_n} converges to F(x) = 0 (whose derivative is 0), the sequence of derivatives clearly does not converge to 0 (or to anything for that matter).

Recall also that a probability density is the derivative of the corresponding distribution function.

Now if the random variable X has a probability density, then so does the standardized sample mean. But the above example tells us that just because the distribution function of the sample mean converges to that of a standard normal variable (CLT), we should not blindly believe that the probability density function of the standardized sample mean converges to the standard normal distribution.

-----

In fact, it is true that under rather weak conditions, if X has a probability density, then the probability density of the standardized sample mean converges to the standard normal distribution N(0, 1). But this result is not the CLT, and requires a (difficult) demonstration not given in this Glossary. It can be explained in loose terms by the fact that the convolution of a function by itself has the side effect of "smoothing out" the oscillations of the function. Iterate the convolution an infinite number of times, and you end up with a perfectly smooth function : the gaussian function.

Limitations of the Central Limit Theorem

Although the range of applicability of the CLT is immense, the CLT is not universal. In particular, it demands that the considered distribution has both a mean and a variance. If such is not the case, then the CLT breaks down.

The most obvious failure of the CLT is to be found with the Cauchy distribution, which does not have any moment. In this particular case, the sample mean always has the same distribution (Cauchy), irrespective of the sample size.

Two other classic distributions with no means are Fisher's F_{n, 1}et F_{n, 2}.

Generalizations of the Central Limit Theorem

There are many versions of the CLT that differ by the assumptions that are formulated about the considered probability distribution. For example :

    * A version weaker than the one we stated above assumes that the distribution has a moment generating function. This is the version we demonstrate in the Tutorial below.

    * Conversely, a stronger version does not refer to the sample mean of a distribution, but rather to the mean of a series of independent rvs {X_n}that are only assumed to have the same mean µ and the same variance σ².

There exist even stronger versions of the CLT, that is, that formulate even weaker assumptions about the random variables whose limit distribution of the standardized sum is seeked.

Importance of the Central Limit Theorem

Beyond its major theoretical significance, the Central Limit Theorem has an important practical consequence. It is quite common that a certain quantity can be considered as resulting from the addition of a large number of small and independent contributions with identical distributions. The CLT then explains why it is so frequently observed that these quantities are normally distributed, without having to worry about the nature of this common elementary probability distribution

______________________________________________________________________________

In this Tutorial, we demonstrate the Central Limit Theorem (CLT). More precisely, we demonstrate one version of the CLT, which assumes that the distribution under consideration has a moment generating function. This assumption is unnecessary, but it makes the demonstration much simpler. It is also reasonable as most of the distributions commonly encountered have a mgf.

We first establish a preliminary result about a certain class of undertermined limits of the form 0/0.

We then address the demonstration proper. Although it is not extremely complicated, it is probably a good idea to first give an outline that will help the reader to follow the various steps.

The final step of the demonstration calls on the convergence property of the moment generating function, that we mentioned but the demonstration of which is unfortunately beyond the bounds of this Glossary.

THE CENTRAL LIMIT THEOREM

_______________________________________________________

Related readings :