An important theorem from which is derived the Chi-square nature of the distributions of certain quantities in :
* Linear Regression,
* Analysis of Variance.
We showed that if two independent r.v. X1 and X2 are distributed respectively as m and n , then X = X1 + X2 is disteributed as m + n . This is the additivity property of the distribution.
In many situations, the question is rather the other way around. One has three variables X, X1 and X2 such that :
* X = X1 + X2
* X~r for a certain integer r,
* X1~q for a certain integer p < r.
It is then quite tempting to believe that X2 is a r.v. :
* Independent of X1,
* and distributed as q, with r = p + q.
This intuition is justified, and can easily be demonstrated by using the moment generating function of the Chi-square distribution.
Cochran's Theorem can be regarded as a generalization of the above result. The theorem exists under many different forms, and one of them is :
* Let Q be a quadratic form in the multinormal vector x and distributed as r.
* Let Q1, Q2, ..., Qk be k quadratic forms such that :
Q = Q1 + Q2, ...+ Qk
The first k - 1 quadratic forms Qi are assumed :
* To be independent.
* And Chi-square distributed.
Qi ~ ri
i ri < r i = 1, 2, ..., (k - 1)
Then the last quadratic form, Qk :
1) Is independent of the Qi, i = 1, 2, ..., k - 1.
2) And is also Chi-square distributed :
Qk ~ rk
3) The number of degrees of freedom rk is such that :
i ri = r i = 1, 2, ..., k
This is clearly a generalization of the reverse additivity property of the Chi-square distribution.
Although the above result is quite useful as such, we believe appropriate to give a more general version, that we fully demonstrate in the Tutorial below.
* Let x~N(0, In ) be a standard multinormal vector.
* Let Ai (i = 1, 2, ..., k) be k symmetric matrices, all of the same order, and denote :
A = A1 + A2 + ... + Ak
* Let Qi = x'Ai x be the quadratic form associated to Ai.
Then the following three propositions are equivalent :
(1) A² = A and r(A) = i r(Ai)
(2) Ai² = Ai for all i, and Ai Aj = 0 for i j
(3) The Qi are independent and Qi ~ ri where ri is the rank of Ai.
The reader may be concerned to see matrices and Linear Algebra pop-up somewhat unexpectedly. Yet, it should be noted that :
* The Chi-square distribution appears most often as the distribution of a quadratic form in a multinormal vector (this is true even for the basic definition of the distribution),
* And the distribution and independence properties of quadratic forms derive from mathematical results that are not of statistical nature, but rather rely on properties (symmetry, idempotence, rank) of certain matrices, and therefore pertain to Linear Algebra.
In addition, a survey of the evolution of the demonstrations of Cochran's Theorem is a very convincing experience in favor of Linear Algebra, that leads to compact, elegant and quite comprehensible demonstrations.
Notice that propositions (1) and (2) make no reference to Statistics but to Linear Algebra only. Statistics is to be found only in proposition (3). We'll see that the equivalence of (2) and (3) is very easy to establish because of the results already obtained about quadratic forms in multinormal vectors.
In other words, the largest part of the demontration of Cochran's Theorem is devoted to demonstrating the equivalence of (1) and (2).
We suggest that you first report to the properties of projection matrices.
It then appears that Cochran's Theorem can be interpreted as follows :
* Let E be r-dimensional a vector space.
* Let E1, E2, ..., Ek be pairwise orthogonal subspaces, with the sum of the dimensions of these subspaces equal to r.
* Finally, let x~N(0, Ir) be a standard mutinormal vector.
* For all i, the squared length of the orthogonal projection of x on Ei is a r.v. distributed as [dim(Ei)].
* These r.v. are independent.
* If the squared lengths of the projections of the vector x~N(0, Ir) on k subspaces are independent random variables,
* And if the sum of the numbers of degrees of freedom of these distributions is equal to r, the dimension of E,
* The subspaces Ei are orthogonal, and their direct sum is E (every vector x can be expressed uniquely as a sum of k vectors, one in each Ei).
There is an interesting formal similarity between the geometric interpretation of Cochran's Theorem and the generalization of the Pythagorean Theorem to a r-dimensional space. This last theorem reads :
* Let E be a r-dimensional vector space.
* Let E1, E2, ..., Ek be pairwise orthogonal subspaces whose dimensions add-up to r.
* Consider the vector x.
* The sum of the squared lengths of the orthogonal projections of x on the Ei is equal to the squared length of x.
* If the sum of the squared lengths of the orthogonal projections of x on the k subspaces Ei is equal to the squared length of x,
* Then these subspaces are orthogonal, and their direct sum is E.
So, in this formal analogy :
* "Orthogonal vectors" correspond to "independent variables".
* "Sum of the squared lengths" correspond to "sum of the degrees of freedom of the distributions".
In this Tutorial, we demonstrate Cochran's Theorem.
* We first establish a few elementary results in Linear Algebra that we'll need for the demonstration. This section represents the largest part of the Tutorial.
* Demonstrating the "algebraic" part of Cochran's Theorem is then just a matter of applying these results to the theorem's assumptions.
* The "statistics" part is very short, but is so only because we previously established important results about the distribution and independence of quadratic forms in multinormal vectors.
We conclude this Tutorial with an example of utilization of Cochran's Theorem. We'll show that :
* The sample mean, and
* The sample variance
of the normal distribution are independent random variables.
In addition, we'll rediscover that (n - 1)S²/² (with S² the sample variance) is distributed as n - 1.
Rank of the product of two matrices
Decomposition of a matrix into the product of two full rank matrices
Left inverse of a full rank matrix
Decomposition of an idempotent matrix
Trace of a product of matrices
Rank and trace of an idempotent matrix
Cochran's Theorem in Linear Algebra
Decomposition of the matrices and chaining of the components
(1) implies (2)
(2) implies (1)
Cochran's Theorem in Statistics
(3) implies (2)
(2) implies (3)
Example : sample mean and sample variance of the normal distribution
Related readings :
Want to contribute to this site ?