Cochran's Theorem

An important theorem from which is derived the Chi-square nature of the distributions of certain quantities in :

    * Linear Regression,

    * Analysis of Variance.

Additivity and reverse additivity of the Chi-square distribution

Additivity

We showed that if two independent r.v. X₁ and X₂ are distributed respectively as _m and _n , then X = X₁ + X₂  is disteributed as _{m + n} . This is the additivity property of the  distribution.

Reverse additivity

In many situations, the question is rather the other way around. One has three variables X, X₁ and X₂ such that :

    * X = X₁ + X₂

    * X~r   for a certain integer r,

    * X₁~q   for a certain integer p < r.

It is then quite tempting to believe that X₂ is a r.v. :

    * Independent of X₁,

    * and distributed as _q, with r = p + q.

This intuition is justified, and can easily be demonstrated by using the moment generating function of the Chi-square distribution.

Generalisation of the reverse additivity

Cochran's Theorem can be regarded as a generalization of the above result. The theorem exists under many different forms, and one of them is :

    * Let Q be a quadratic form in the multinormal vector x and distributed as _r.

    * Let Q₁, Q₂, ..., Q_k be k quadratic forms such that :

Q = Q₁ + Q₂, ...+ Q_k

The first k - 1 quadratic forms Q_i are assumed :

    * To be independent.

    * And Chi-square distributed.

Q_i ~ _ri 

with :

_i r_i < r               i = 1, 2, ..., (k - 1)

Then the last quadratic form, Q_k :

    1) Is independent of the Q_i, i = 1, 2, ..., k - 1.

    2) And is also Chi-square distributed :

Q_k ~ _rk 

    3) The number of degrees of freedom r_k is such that :

_i r_i = r        i = 1, 2, ..., k

This is clearly a generalization of the reverse additivity property of the Chi-square distribution.

Cochran's Theorem

Although the above result is quite useful as such, we believe appropriate to give a more general version, that we fully demonstrate in the Tutorial below.

The reader may be concerned to see matrices and Linear Algebra pop-up somewhat unexpectedly. Yet, it should be noted that :

    * The Chi-square distribution appears most often as the distribution of a quadratic form in a multinormal vector (this is true even for the basic definition of the distribution),

    * And the distribution and independence properties of quadratic forms derive from mathematical results that are not of statistical nature, but rather rely on properties (symmetry, idempotence, rank) of certain matrices, and therefore pertain to Linear Algebra.

In addition, a survey of the evolution of the demonstrations of Cochran's Theorem is a very convincing experience in favor of Linear Algebra, that leads to compact, elegant and quite comprehensible demonstrations.

Cochran's Theorem, Linear Algebra and Statistics

Notice that propositions (1) and (2) make no reference to Statistics but to Linear Algebra only. Statistics is to be found only in proposition (3). We'll see that the equivalence of (2) and (3) is very easy to establish because of the results already obtained about quadratic forms in multinormal vectors.

In other words, the largest part of the demontration of Cochran's Theorem is devoted to demonstrating the equivalence of (1) and (2).

Geometric interpretation of Cochran's Theorem

We suggest that you first report to the properties of projection matrices.

-----

It then appears that Cochran's Theorem can be interpreted as follows :

    * Let E be r-dimensional a vector space.

    * Let E₁, E₂, ..., E_k be pairwise orthogonal subspaces, with the sum of the dimensions of these subspaces equal to r.

    * Finally, let x~N(0, I_r) be a standard mutinormal vector.

Then :

    * For all i, the squared length of the orthogonal projection of x on E_i is a r.v. distributed as [dim(E_i)].

    * These r.v. are independent.

Conversely :

    * If the squared lengths of the projections of the vector x~N(0, I_r) on k subspaces  are independent  random variables,

    * And if the sum of the numbers of degrees of freedom of these distributions is equal to r, the dimension of E,

Then :

    * The subspaces E_i are orthogonal, and their direct sum is E (every vector x can be expressed uniquely as a sum of k vectors, one in each E_i).

-----

There is an interesting formal similarity between the geometric interpretation of Cochran's Theorem and the generalization of the Pythagorean Theorem to a r-dimensional space. This last theorem reads :

    * Let E be a r-dimensional vector space.

    * Let E₁, E₂, ..., E_k be pairwise orthogonal subspaces whose dimensions add-up to r.

    * Consider the vector x.

Then :

    * The sum of the squared lengths of the orthogonal projections of x on the E_i is equal to the squared length of x.

Conversely :

    * If the sum of the squared lengths of the orthogonal projections of x on the k subspaces E_i is equal to the squared length of x,

    * Then these subspaces are orthogonal, and their direct sum is E.

So, in this formal analogy :

    * "Orthogonal vectors" correspond to "independent  variables".

    * "Sum of the squared lengths" correspond to "sum of the degrees of freedom of the  distributions".

_____________________________________________________________

In this Tutorial, we demonstrate Cochran's Theorem.

    * We first establish a few elementary results in Linear Algebra that we'll need for the demonstration. This section represents the largest part of the Tutorial.

    * Demonstrating the "algebraic" part of Cochran's Theorem is then just a matter of applying these results to the theorem's assumptions.

    * The "statistics" part is very short, but is so only because we previously established important results about the distribution and independence of quadratic forms in multinormal vectors.

-----

We conclude this Tutorial with an example of utilization of Cochran's Theorem. We'll show that :

    * The sample mean,  and

    * The sample variance

of the normal distribution are independent random variables.

In addition, we'll rediscover that (n - 1)S²/² (with S² the sample variance) is distributed as _{n - 1}.

COCHRAN'S THEOREM

 ____________________________________________________

Related readings :