Chi-square distribution

Definition of the Chi-square distribution

The distribution of the sample mean of a N(0,1) distribution is N(0,1/n), where n is the sample size. What is the distribution of the sample variance ?

The variance was defined so as to adequately describe the dispersion of the observations in a sample about the mean of the distribution. It turns out that the average of the squared distances of the observations in the sample to the distribution mean has good mathematical properties that justify a posteriori the definition of the variance.

So one is quite naturally lead, for the special case of the N(0,1) distribution, to study the distribution of the sum of the squared differences of the observations in the sample to the distribution mean 0, that is, in fact, to the sum of the squared values of these observations.

The reason why we focus on the sum of these squares rather than on their average value is explained in the Tutorial below, see "Additivity".

So, by definition, the Chi-square distribution is the distribution of the sum of the squared values of the observations drawn from the N(0,1) distribution. It is denoted by the symbol χ², which is pronounced "Ky square".

More precisely, and more formally :

• Let {X₁, X₂, ..., X_n} be n independent r.v., all ~ N(0,1).
• Then the χ²_n is defined as the distribution of the sum X₁² + X₂² +...+ X_n².

Degrees of freedom

So there is not one χ² distribution, but a family of distributions, indexed by n. This parameter is called the "number of degrees of freedom" of the distribution (this same expression is found in several other distribution families, like Student's t or Fisher's F).

The "Chi-square distribution with n degrees of freedom" is therefore the distribution of the sum of n independent squared r.v. all ~N(0,1).

The general normal distribution

Sum of the squared differences

Let X be any normally distribution r.v. :

 X ~ N(µ, ²)

Recall that the change of variable :

X ' = (X - µ)/

turns any normal r.v. X into a standard normal variable X ' ~ N(0,1).

So if X ~ N(µ, ²), the sum of the squared standardized observations of a n-sample is distributed as χ²_n.

Variance

In practice, one is more interested by the distribution of the average value of the squared observations rather than by their sum. Let :

s² = 1/n.Σ_i(X_i - µ)²

We then have :

ns²/² = Σ_i [(X_i - µ)/]²

Hence

ns²/² ~ χ²_n

as the sum of n square independent N(0, 1) variables.

Estimating the distribution mean

So far, we assumed that the distribution mean µ was known. In practice, this is rarely the case. So one is lead to replace the true mean µ by its estimated value

= 1/n.Σ_ix_i

in the above expression.

But whereas µ is a constant,  is the realization of the random variable

= 1/n.Σ_i X_i

and there is now no reason to believe that the modified ns²/² is χ² distributed anymore.

Distribution of the sample variance

We finally get to the question of interest to the analyst : "What is the distribution of the sample variance of the normal distribution ?".

Fundamental result

Let :

S² = 1/(n - 1).Σ_i (X_i - )²

which is the sample variance, an unbiased estimator of the distribution variance.

We will show that :

So it appears that replacing the distribution mean by the sample mean does not change the nature of the distribution of the sample variance (it remains χ²) : it simply reduces the number of degrees of freedom of the distribution by one unit.

This result is fundamental.

Note that replacing the distribution variance by the sample variance has a deeper effect on the distribution of the standardized sample mean : this distribution is then no longer normal, but is a t distribution instead (see here).

Losing a degree of freedom

The transition from "n" to "n - 1" is called "losing a degree of freedom". This phenomenon is quite general, and will be encountered in other circumstances involving χ² distributions, t distributions or F distributions. It is a consequence of replacing an unknown parameter by its estimated value.

Independence of the sample mean and the sample variance

In the course of demonstrating the above result, we'll incidentally demonstrate another important result :

This result may also be regarded as a simple consequence of Cochran's Theorem.

This is a characteristic property of the normal distribution : a distribution such that the sample mean and the sample variance are independent r.v. is necessarily normal (difficult, and not demonstrated in this site).

Animation

This interactive animation illustrates the Chi-square distribution.

Properties of the Chi-square distribution

We'll establish the following properties of the Chi-square distribution.

Probability density function

The probability density function of the Chi-square distribution with n degrees of freedom is :

where Γ is the Gamma function.

The Chi-square distribution thus appears to be a special case of the Gamma distribution Γ(α, β) with α = n/2 and β = 1/2.

Mean

The mean of the Chi-square distribution with n degrees of freedom is :

Variance

The variance of the Chi-square distribution with n degrees of freedom is :

Moments of all orders

The moment of order p of the Chi-square distribution is :

This is an immediate consequence of the corresponding result for the Gamma distribution.

Mode

            * For n > 2, the Chi-square distribution has a unique mode :

The mode is therefore trailing behind the mean by 2 whatever the number of degrees of freedom.

    * For n = 2, the Chi-square distribution is the exponential distribution with intercept  .5.

    * For n = 1, the vertical axis is an asymptote for the Chi-square distribution which then has no mode.

Moment generating function

The moment generating function of the Chi-square distribution with n degrees of freedom is :

We'll establish this result directly. It will be the link between the Gamma and the Chi-square distributions.

Chi-square distribution and tests

Testing the value of a variance

Now that we know the distribution of the estimated variance of a normal distribution, it is possible to elaborate tests about the variance of the population.

For example, suppose we want to test the null hypothesis H₀ : ² = ₀²  against the alternative hypothesis H₁ : ² ≠ ₀² at the significance level α for a n-sample drawn from a normal population. We'll compare the value of the quantity (n - 1)S²/₀² (the test statistic) to the quantiles α /2 and (1 - α/2) of the

χ²_{n - 1} distribution (upper and lower image of the following illustration).

Comparing two variances

Comparing two variances is hardly more difficult. We have two samples of respective sizes n₁ and n₂ drawn from two independent and normally distributed populations whose variances we want to compare.

The test statistic is then :

F = S₁² / S₂²

When the populations have identical variances, the distribution of this statistic is Fisher's F distribution.

This question is addressed in more detail here.

Quadratic forms in multivariate normal variables

Statistics is often facing quadratic forms is multivariate normal variables, in particular :

    * In Analysis of Variance (ANOVA),

    * In Multiple Linear Regression.

Under certain conditions that are detailed here, these quadratic forms follow (exactly) a Chi-square distribution.

The Chi-square distribution as an asymptotic distribution

The importance of the Chi-square distribution extends beyond the issue of the variance of a normal distribution. Two important test statistics follow distributions whose exact forms are unknown, but that converge to a χ² distribution for larger and larger samples (asymptotic distribution) :

    * Pearson's Chi-square, which is formally the common statistic of all Chi-square tests, and which is constructed from considerations about how to build a goodness-of-fit test for the multinomial distribution (see here).

    * Wilks' G², which serves exactly the same purpose as Pearson's Chi-square, but which is obtained by applying the standard Likelihood Ratio Test procedures to the goodness-of-fit problem of the multinomial distribution.

___________________________________________________________

In this Tutorial, we establish the basic properties of the Chi-square distribution. In fact, these properties can be deduced form those of the Gamma distribution of which the Chi-square distribution is just a special case.

It is nevertheless necessary to first prove this assertion, and this accounts for the largest part of the Tutorial. We'll do it by calculating the moment generating function (mgf) of the Chi-square distribution from first principles. We'll recognize this mgf as that of a particular Gamma distribution and, calling on the uniqueness property of the mgf, we'll deduce the probability density function of the Chi-square distribution as well as its basic properties.

-----

Calculating the pdf of the Chi-square distribution is useful :

    * For calculating the quantile tables needed for tests.

    * But in some occasions, it is also useful for itself. For example, it will come in handy for identifying a sufficient statistic for the variance of the normal distribution (see here).

BASIC PROPERTIES OF THE CHI-SQUARE DISTRIBUTION

________________________________________________________________________

We now demonstrate the fundamental result :

(n - 1)S ²/² ~ χ²_n-1

which expresses the fact that replacing the true distribution mean by the sample mean :

    * Preserves the "χ²" nature of the distribution of the sample variance,

    * But causes a loss of one degree of freedom of this distribution.

We first go over the 2-observation sample case as it can be represented graphically, as well as the demonstration and the final result.

We then move on to the demonstration for samples of any size. We'll use an elementary demonstration that does not call on Linear Algebra.

-----

This demonstration will incidentally establish another very important result :

The sample mean and the sample variance of the normal distribution are independent random variables.

This result is also a consequence :
   * Of Cochran's Theorem.
   * Of Basu's Theorem.

DISTRIBUTION OF THE SAMPLE VARIANCE

OF THE NORMAL DISTRIBUTION

________________________________________________________________

Related readings: