Normal  (distribution)

Also known as "gaussian distribution".

By far the most well known probability distribution.

Definition of the normal distribution

A random variable X is said to be normally distributed if its probability density function (pdf) is :

for a certain pair of values of the two parameters µ and s.

The coefficient before the exponential is there only to ensure that the integral of this function is equal to 1.

    * µ is clearly a location parameter,

    * and s may be interpreted as a shape, or more specifically a spread parameter.

Although the "true" parameter is s, applications are usually concerned with s², and consequently, the normal distribution with parameters µ and s will be denoted N(µ, s²). 

-----

The symmetric bell-shaped curve of the normal distribution is ubiquitous, and you'll see it many times throughout this site and just about everywhere else. We give here an illustrated example that displays both :

    * A normal distribution (upper green curve) together with a sample drawn from this distribution. The mean of this sample is marked by a red point, from which a vertical red line extends downwards.

    * Another normal distribution (lower red curve) which is the theoretical distribution of the sample mean of the upper curve (we demonstrate this result.in the Tutorial below).

Change the value of the standard deviation of the upper normal curve with the vertical cursor in the upper-right corner of the animation, and observe the changes of spreads of the two gaussian curves.

Change the sample size, and observe that the lower gaussian becomes narrower as the sample getsd larger.

The other controls are self-explanatory.

Moments

We are so used to thinking in terms of :

    * µ being the mean of the normal distribution

    * s being its standard deviation,

that we sometimes forget that this requires a demonstration, which is given below.

-----

So, with a little anticipation, it is to be noted that the normal distribution is thus entirely defined by its first two moments.

Standard normal distribution

We'll see in many instances that the special normal distribution N(0, 1) plays a central role is Statistics. It is called the standard normal distribution.

Origin of the normal distribution

The normal distribution was first introduced as the limit of the binomial distribution B(n, p) for very large values of n. The true binomial distribution is then almost impossible to compute exactly because of the factorials that are conducive to handling very large numbers, and the need was felt for an approximate formula that was easier to calculate. Some rather laborious calculations led de Moivre to be the first to identify the above analytical form. More precisely, de Moivre showed that if X is distributed as B(n, p),  the distribution of the r.v. Y defined by :

that is, the standardized version of X, converges towards N(0, 1) as n tends to infinity.

The Central Limit Theorem

But it later turned out that the normal distribution has a much deeper origin. What de Moivre observed with the binomial distribution is just a special case of a much more general setting.

If  {X_i} is an infinite series of independent r.v., we can construct another infinite series of r.v. defined by :

So Y_n is defined as the sum of the first n X_is.

Then it can be shown that, under some not very restrictive conditions, the distribution of :

converges towards the distribution N(0, 1) as n grows without limit.

This fundamental result is known as the Central Limit Theorem.

You'll easily convince yourself that the result obtained by de Moivre is a special case of the much more general Central Limit Theorem, with the X_i being independent Bernoulli variables, all with the same parameter p.

-----

The Central Limit Theorem "explains" why so many distributions encountered in applications resemble closely normal distributions. Many quantities can be interpereted as resulting from the averaged effect of a large number of independent random causes. Then, whatever the natures of the distributions of these causes, the effect will be closely normally distributed.

The normal distribution therefore appears as a universal distribution.

Distributions derived from the normal distribution

The normal distribution gives rise to several very important distributions :

The Chi-square distribution

The sample variance s² is such that :

(n - 1)s²/s²

follows a distribution that does not depend on s² and which is known as the " distribution with (n - 1) degrees of freedom". The properties of this very important distribution are detailed here.

It is because this distribution is known that it is possible to elaborate confidence intervals and tests pertaining to the variance of the normal distribution.

-----

Besides the sample variance of the normal distribution, several other quantities follow approximately distributions. This leads to several important tests that are known globally as "Chi-square tests".

Student's t distribution

The sample mean  is distributed as N(µ, s²/n) (see Tutorial below), and the standardized sample mean is therefore distributed as N(0, 1). This result leads to confidence intervals and the most basic tests (the so-called "t tests") bearing on the value of the mean of a normal distribution with known variance.

But more often than not, the variance s² is unknown, which makes standardizing the sample mean impossible. One can nevertheless replace s² by the sample variance, thus creating a quantity T that does not depend on s², and whose distribution is known : it is Student's t distribution with (n - 1) degrees of freedom, where n is the sample size.

It is then possible to construct confidence intervals and tests bearing on :

    * The value of the mean of a normal distribution whose variance is unknown,

    * The difference between the means of two normal distributions of equal but unknown variances.

-----

Other quantities also follow t distributions, in particular during the elaboration of confidence intervals on the values of the parameters of a Multiple Linear Regression.

Fisher's F distribution

The ratio of the variances of two samples drawn from two independent normal distributions with equal variances follows a distribution known as Fisher'F distribution. The properties of the F distribution are detailed here.

-----

Other quantities also follow the F distribution, which plays a central role :

    * In Analysis of Variance (ANOVA),

    * In Linear Regression (Simple of Multiple) for tests bearing on the overall validity of adjusted models.

Independence of the sample mean and the sample variance

We show here that the sample mean and the sample variance s² of a normal distribution are independent random variables. This important result is a characteristic property of the normal distribution : a distribution such that the sample mean and the sample variance are independent is necessarily normal (difficult).

Sufficient statistic

We show here that the sample mean  is a sufficient statistic for the distribution mean µ.

Exponential family

We show here that the normal distribution belongs to the exponential family. From this result, we'll deduce that the sample mean is an efficient estimator of the mean µ.

 ____________________________________________________________

In this first Tutorial, we establish the basic properties of the normal distribution.

    * We first calculate the mean and variance from their definitions. Calculating the mean is easy, calculating the variance is a bit more complicated. We won't be surprised to discover that µ is the mean of the normal distribution, and s² its variance.

    * We then calculate the moment generating function of the normal distribution, and then derive the mean and the variance from this mgf

-----

We insist on the importance of the standard normal distribution N(0, 1), which is the cornerstone of the simplest of the confidence intervals and t test.

BASIC PROPERTIES OF THE NORMAL DISTRIBUTION

______________________________________________

We now proceed with the estimation of the parameters of the normal distribution.

We first use the "natural" method of moments, then the method of Maximum Likelihood. We'll see that the ML solution is unique (which is not always the case). We'll also see that the method of moments provides a slightly better estimator of the distribution variance than the method of ML does, which is the exception rather than the rule.

We illustrate these result with an interactive animation.

ESTIMATION OF THE PARAMETERS

OF THE NORMAL DISTRIBUTION

 ________________________________________________________

We establish here two important properties of normally distributed random variables :

    * The linearly transform of a normal r.v. is also normal, and we calculate its parameters.

    * A linear combination of independent normal r.v.s is also normal, and we calculate its parameters. The condition "independent" is important, and we give here two examples of pairs of normally distributed r.v. {X, Y}, that are not independent and whose sums Z = X + Y are not normally distributed.

We then calculate the distribution of the sample mean of a normal distribution, a result widely used in applications.

BASIC PROPERTIES OF

NORMALLY DISTRIBUTED R.V.

 ________________________________________________________

This exercise and animation illustrate the concept of "random parameter".

We define the r.v. X by describing the procedure to obtain one realization of X.

    * Y is a r.v. with the normal distribution N(0, s²). Let m be a realization of Y.

    * Consider now the normal distribution N(m, s²). We draw an observation from this distribution, and consider this observation as a realization of X.

In other words, the distribution of X may be interpreted as :

    * A normal distribution N(m, s²),

    * Whose parameter µ is itself a normally distributed r.v. : µ~ N(0, s²).

-----

What is the distribution of X ?

We give two solutions in the Tutorial below.

    * The first solution is short, and relies on results obtained in the previous Tutorial.

    * The second solution is longer, but is a good example of calculation of a probability density knowing the density conditionally to an auxiliary variable, as well as the distribution of this conditioning variable.

    * The distribution of µ is the green gaussian curve in the upper part of the animation. Its standard deviation (s) can be changed with the vertical cursor at the right of this curve.

    * The distribution N(0, s²) is the blue gaussian curve in the middle section of the animation. Its standard deviation (s) can be changed with the vertical cursor at the right of this curve.

    * The theoretical distribution of X is the red curve in the lower section of the animation. Observe that it changes when either s or s is changed.

-----

    * Click repetitively on "Next". Every time you click, a green point is drawn from the N(0, s²) distribution. This point defines the position of the mean of the blue distribution, from which a blue point is then drawn.

The goal of the exercise is to calculate the distribution of the blue points.

-----

    * Click on "Go", and observe the progressive build-up of the histogram of the distribution of X.

NORMAL DISTRIBUTION WHOSE MEAN

IS ITSELF A NORMAL R.V.

 ___________________________________________________________

Related readings