Quadratic forms

Definition of quadratic forms

Let {x₁, x₂ , ..., x_n} be n (non random) variables. A quadratic form Q is, by definition, an expression such as :

Q = _ij a_ij x_i x_j

where the a_ij (the coefficients of the form) are real numbers.

So a quadratic form is a second degree, homogenous (no constant term) polynom in the x_i.

Studying quadratic forms is made quite a bit easier by using matrix notation.

    * The ordered set of variables {x₁, x₂ , ..., x_n} is considered as a vector x with coordinates (x₁, x₂ , ..., x_n ). So we'll write :

x' = (x₁, x₂ , ..., x_n )

with " ' " denoting transposition, which simply means that the x_i are written as a row (as opposed to "as a column").

    * A denotes the matrix whose general term is [a_ij].

    * A quadratic form Q is then defined by the matrix equation :

A is called the matrix of the quadratic form.

It is easily shown that A may be assumed to be symmetric without loss of generality.

-----

Quadratic forms are an important chapter of Linear Algebra. We will only retain the aspects of quadratic forms that are useful to the Statistician.

Expectation of a quadratic form

Let :

    * x be a random vector with mean µ and covariance matrix .

    * Q = x'Ax. be a quadratic form.

We'll show that :

This result makes no assumption about the nature of the distribution of x, with the exception of the existence of the first two orders moments.

Quadratic forms in a multivariate normal vector and the Chi-square distribution

Important parts of Statistics like :

    * Multiple Linear Regression,

    * Variance Analysis,

extensively use the fact that if x is a multivariate normal vector :

x~N(µ, )

then certain quadratic forms is x are Chi-square distributed.

Chi-square distribution

Let {X₁, X₂ , ..., X_n} be n standard normal independent variables :

X_i ~ N(0, 1)     for all i

Recall that, by definition, the Chi-square distribution with n degrees of freedom is the distribution of the variable X² defined as :

X ² = _i X_i² 

and is usually denoted :

X ² ~ _n

So the variable X² is defined as a very special quadratic form in the X_i in which :

    * There is no cross-term (a_ij = 0 for i  j),

    * and all the a_ii are equal to 1.

The matrix of the form is then just the identity matrix of order n, denoted I_n.

-----

Note that these n variables may conveniently be represented by the single multivariate normal standard variable :

x~N(0, I_n)

Quadratic forms in a standard multivariate normal variable and the Chi-square distribution

It is then natural to ask whether there exist other quadratic forms in a spherical multinormal variable of unit variance that also have a Chi-square distribution. In other words, do matrices P exist such that :

x'Px ~ _??

for x~N(0, I_p) ?

Recall that a symmetric matrix is said to be "idempotent" if P ² = P.

We'll show that a necessary and sufficient condition for x'Px to have a Chi-square distribution is that P be an idempotent matrix. In addition, we'll show that the rank of P and the number of degrees of freedom of the  distribution are then equal.

In other words :

In fact, we'll show a bit more than that as we'll consider variables ~N(µ, I) that are not necessarily centered on the origin.

A symmetric idempotent matrix P or rank r is interpreted as the matrix of an orthogonal projection operator on a r-dimensional subspace. The projection of the vector x is Px.

If  P ² = P, then :

and x'Px is therefore the squared length of the projection of x on the subspace.

The above result is then interpreted as follows :

    * A quadratic form in a spherical, unit variance multinormal vector is Chi-square distributed with r degrees of freedom if and only if it is the squared length of the projection of x on some r-dimensional subspace.

In the above illustration :

    * The vector x has a spherical multinormal distribution with unit variance.

    * Px is the projection of x on the r-dimensional subspace .

    * The squared length of Px is distributed as _r.

Quadratic forms in a general multinormal variable and the Chi-square distribution

We then address the general issue of quadratic forms in a multivariate normal variable with an arbitrary covariance matrix  :

x~N(µ, )

We'll show that the quadratic form Q = x'Ax follows a (non central) Chi-square distribution with r degress of freedom if and only if both the following conditions are satisfied :

the value of the noncentrality parameter being  µ'Aµ.

-----

So, in the general case, the interpretation in terms of the distribution of the squared length of the projection of x on a subspace is lost, and A is not a projection matrix.

-----

This situation is encountered, for instance when studying the Mahalanobis distance.

Independence of two quadratic forms in a multivariate normal vector, Craig's Theorem

Independence of two random variables is a condition that makes life of a Statistician a lot easier. In particular, many results pertaining to the sum and the ratio of two random variables explicitely assume that these variables are independent. For example, the formal definition of Fisher's F distribution involves the ratio of two independent variables.

It is therefore natural to ask under what conditions two quadratic forms in a multivariate normal vector x are independent.

Independence of two linear forms

We'll first show that two linear forms A'x et B'x in a multinormal vector x with covariance matrix  are independent if and only if :

This result is important by itself but, in addition, it will be needed later when we address the issue of the independence of quadratic forms.

Independence of quadratic forms

We'll then show that :

In the special case where x is spherical, this condition becomes the simpler AB = 0.

-----

It is not assumed that the Chi-square distributions of the quadratic forms are central or have the same numbers of degrees of freedom.

Craig's Theorem

The above result explicitely assumes that the quadratic forms are Chi-square distributed. As it happens, this assumption is unnecessary and is introduced just as to make the demonstration easier.

More generally :

which is the same as before but with the assumption about the  distributions of the quadratic forms removed. The result is then known as Craig's Theorem, whose demonstration lies beyond the bounds of this Glossary.

Cochran's Theorem

The results about the nature of the distribution and the independence of quadratic forms in multinormal vectors as described here are a key element of a most important theorem in Statistics known as Cochran's Theorem, which is stated and demonstrated here.

__________________________________________________________________

In the first section of this Tutorial, we calculate the expectation of a quadratic form with no assumption about the distribution of the variable. This section is independent of the remainder of the Tutorial.

-----

We then establish a necessary and sufficient condition for a quadratic form Q :

Q = x'Px

in a multivariate normal variable to be Chi-square distributed.

    * We first address the special case of a spherical multinormal distribution with unit variance (identity covariance matrix).

        - The "sufficient" condition states that if P is a projection matrix with rank r, then Q is Chi-square distributed with r degrees of freedom.

        - The "necessary" part states that if Q is Chi-square distributed with s degrees of freedom, then P has to be a projection matrix with rank s. The demonstration is a bit more difficult, and the moment generating function (m.g.f.) of the quadratic form will be of central importance.

    * We then go over the general case (arbitrary covariance matrix) by identifying a transformation that will turn the problem into the already solved "special case" problem, a common strategy when studying the multivariate normal distribution. So the Table of Contents of the general case is quite short, as the brunt of the work has already been done when solving the special case. Once the transformation has been identified, only a small additional effort will be needed to express the result as a function of , the covariance matrix of x.

QUADRATIC FORMS IN MULTINORMAL VECTORS

AND THE CHI-SQUARE DISTRIBUTION

___________________________________________________________________________

We now address the issue of the independence of two quadratic forms is a multivariate normal vector.

We first solve the simpler problem of the independence of two linear forms. This result will anyway be needed in the remainder of the Tutorial.

-----

We then move on to quadratic forms and establish a necessary and sufficient condition for two such forms to be independent. The reader won't be surprised to see us first solve the case of quadratic forms in a spherically symmetric, unit variance multinormal vector. The general case (arbitrary covariance matrix) will then be solved by identifying a transformation that turns the general case into the already solved special case.

-----

Note that we always consider quadratic forms with Chi-square distributions. As it turns out, this assumption is unnecessary and is introduced only to make the demonstration easier. The condition remains valid without it and then bears the name of Craig's Theorem, which is very difficult to demonstrate.

INDEPENDENCE OF QUADRATIC FORMS

 ___________________________________________________

Related readings :