Central Limit Theorem

One of the cornerstones of the theory of random variables.

General idea of the Central Limit Theorem

Let X be any random variable described by a probability distribution with a mean µ and a variance s². In a nutshell, the Central Limit Theorem (CLT) states that for large samples, the sample mean of this distribution is approximately normally distributed. Moreover, it adds that the distribution of the sample mean can be made as close to normal as wanted : all it takes is to consider larger and larger samples.

The Central LImit Theorem

Let's formulate the CLT more precisely. The sample mean is a random variable, whose distribution is generally unknown (with some notable exceptions like normal, Chi-square, binomial or Poisson distributions), even if the "mother" distribution is known. Nevertheless, the Central Limit Theorem states that as the sample size grows without limit, the distribution function of the sample mean converges towards the distribution function of a normal distribution.

The distribution of the sample mean is unknown, but :

    * The mean of this distribution is known : it is µ, the mean of the mother distribution.

    * The variance of this distribution is s²/n, where s² is the variance of the mother distribution, and n the sample size.

We can therefore make the foregoing statement more detailed by stating that the distribution function of the standardized sample mean converges towards the distribution function of the standard normal distribution (convergence "in distribution").

This convergence means that for any given number x, the probability that  X x  tends towards F(x) as n tends to infinity, with F(x) being the distribution function of the standard normal distribution.

The Central Limit Theorem is therefore :

as n grows without limit.

Central Limit Theorem and probability density

We initially stated that the CLT asserts that the probability distribution of the (standardized) sample mean becomes closer and closer to a (standardized) normal distribution. But this formulation is too restrictive, for the CLT is valid for discrete r.v. as well (binomial, Poisson etc...), which do not have probability density functions. The reason is that the CLT bears on distribution functions, not on probability densities.

Consider for example the binomial distribution. It has a probability mass function, that assigns probabilities to integer values of the variable. The sample mean also has a probability mass function made of "spikes". The distance between these spikes tends to 0 as the sample size grows without limit, but such a series of functions does not converge towards anything in the sense of Calculus (see animation).

But the distribution function of the standardized sample mean, although is is not continuous ("staircase" function), does converge towards the distribution function of the standard normal distribution.

-----

If the r.v. has a probability density, then it is true that the probability density of the standardized sample mean converges towards the normal standard probability density, but this is not obvious, and requires a (difficult) demonstration.

Limitations of the Central Limit Theorem

Although the range of applicability of the CLT is immense, the CLT is not universal. In particular, it demands that the considered distribution has both a mean and a variance. If such is not the case, then the CLT breaks down.

The most obvious failure of the CLT is to be found with the Cauchy distribution, which has no moments at all. In this particular case, the sample mean always has the same distribution (Cauchy), irrespective of the sample size.

Two other classic distributions with no means are Fisher's F_{n, 1}et F_{n, 2}.

Generalizations of theCentral Limit Theorem

There are many versions of the CLT that differ by the assumptions that are formulated about the considered probability distribution. For example :

    * A version weaker than the one we stated above assumes that the distribution has a moment generating function. This is the version we demonstrate in the Tutorial below.

    * Conversely, a stronger version does not refer to the sample mean of a distribution, but rather to the mean of a series of independent r.v. {X_n}that are only assumed to have the same mean µ and the same variance s².

There exist even stronger versions of the CLT, that is, that formulate even weaker assumptions about the random variables whose limit distribution of the standardized sum is seeked.

Importance of the Central Limit Theorem

Beyond its major theoretical significance, the Central Limit Theorem has an important practical consequence. It is quite common that a certain quantity can be considered as resulting from the addition of a large number of small and independent contributions with identical distributions. The CLT then explains why it is so frequently observed that these quantities are normally distributed, without having to worry about the nature of this common elementary probability distribution

Animation

The following animation illustrates the CLT.

Upper frame

    a) The green rectangle is a uniform distribution. A sample drawn from this distribution is also displayed, and the tall vertical red line is the sample mean.

You may change the sample size by using the "Nb. Points" buttons. A new sample is drawn each time you click on one of these buttons.

    b) You can carve out any (limited support) density function you like by repetitively clicking inside the frame (even within the green area).

The red gaussian curve has the same mean and variance as the green density function.

Lower frame

The red gaussian curve is the standard normal distribution N(0, 1), which will be used as a reference. It remains unchanged as the green density in the upper frame, or the sample size, is changed. Its horizontal and vertical scales are arbitrary, and not related to those of the upper frame.

According to the CLT, the distribution of the "standardized sample mean":

 should get closer and closer to this gaussian as n grows without limit.

This is what the animation now illustrates.

Animation

Click on "Go", and watch the progressive build-up of the histogram of the sample mean of your distribution.

For some dnsities,  the histogram may be too tall for the frame. Reduce then the vertical scale "on the fly" with the "Vert. scale" button.

We suggest the following experiments (among many others) :

* Keep the initial uniform distribution, and observe the distributions of the sample mean for increasing values of n.

• For n = 2, observe that the distribution is triangular. This is demonstrated here.
• Even for low values of n, the distribution is very close to the reference gaussian curve.

* For any distribution you build, set the number of points to 1. The distribution of the sample "mean" will just be a duplicate of the original distribution.

* Build a distribution that is "as different as possible" from a normal distribution. For example, you may build a concave parabola-shaped distribution with 0 p.d.f. at the mean. No point will ever be created there. Yet, for large values of n, this is where the p.d.f. of the sample mean will reach its largest value!

* Keep the same original "concave" distribution, and set the number of points to 2. (To change the value of n while retaining the current distribution, first click on "Pause", then change n before clicking on "Go" again).

The distribution of the sample mean is now strongly modulated, with 3 "humps" and 2 "troughs". Can you interpret this structure :

• Either in terms of self-convolution of the current density,
• or with a more heuristic approach, asking "For a given abscissa x, what kind of likely sample will have x for the value of its sample mean ?".

* Now retain this same distribution, and launch again the animation for larger and larger values of n.

• The number of "humps" gets larger as n gets larger, but the "troughs" are more and more shallow. Can you explain why ?
• Observe that the histogram gets closer to the reference gaussian as n gets larger.

_________________________________________________________________

In this Tutorial, we demonstrate the Central Limit Theorem (CLT). More precisely, we demonstrate one version of the CLT, which assumes that the distribution under consideration has a moment generating function. This assumption is unnecessary, but it makes the demonstration much simpler. It is also reasonable as most of the distributions commonly encountered have a m.g.f.

We first establish a preliminary result in calculus about the limit of a certain class of undertermined forms.

We then address the demonstration proper. Although it is not extremely complicated, it is probably a good idea to first give an outline that will help the reader to follow the various steps.

The final step of the demonstration calls on the convergence property of the moment generating function, that we mentioned but the demonstration of which is unfortunately beyond the bounds of this Glossary.

THE CENTRAL LIMIT THEOREM

_______________________________________________________

Related readings