Projection matrix

Orthogonal projection

Let E be a vector space, and S a subspace of E.

Project a vector x orthogonally on S, and denote u the projection of x. How can we calculate u ?

The problem is not to be ignored by the statistician as it appears in several important occasions :

    * Principal Component Analysis (PCA) is all about orthogonal projections.

    * Multiple Linear Regression is fundamentally a problem in orthogonal projection.

    * The distributional and independence properties of quadratic forms in multivariate normal vectors are also fundamental in problems of variance decomposition (ANOVA and Multiple Linear Regression), and call on the concept of orthogonal projection.

Projection matrices

Orthogonal projection problems can be nicely represented and treated within the framework of Linear Algebra.

Projection of a vector

"Orthogonal projection on S" is a linear operator, and can therefore be conveniently represented by a matrix P_S. We'll show that if Z_S is a matrix whose columns form an orthonormal basis of the subspace S,  then the orthogonal projection u of any vector x is given by :

Uniqueness of the projection matrix

From the above result, it would seem that the projection matrix P_S depends on the particular orthonormal basis chosen for spanning S. In fact, we'll show that P_S does not depend on the choice of this basis. In other words, let W_S be another matrix whose columns also form an orthonormal basis of S.

We'll show that :

therefore leading to the same projection matrix.

Projection on the orthogonal complement

Denote S_z the set of vectors that are orthogonal to all vectors in S. It is a subspace of E called the orthogonal complement of S (see above illustration).

We'll show that  P_Sz, the matrix of the orthogonal projection operator on this subspace is given by :

where I_n is the identity matrix of order n, with n the dimension of the complete space E.

Non orthonormal basis

Suppose that the available basis of S, denote it X_S, is not orthonormal. How can we calculate the projection matrix P_S ?

    * One possibility is to first construct an orthonormal basis from X_S (for instance using the Gram-Schmidt orthonormalization procedure), and then use this basis for calculating P_S.

    * But one may also directly calculate P_S. We'll show that if X_S is a matrix whose columns form a (non orthonormal) basis of S, then :

The reader already familiar with Multiple Linear Regression will recognize this result as the "hat" matrix used for fitting the model to the data.

-----

If the columns of  X_S turn out to form an orthonormal basis of S after all, then it should be clear that the above expression reduces to the one we found for orthonormal basis (Why ?).

Characteristic properties of projection matrices

We'll first establish that a projection matrix is always symmetric.

Among symmetric matrices, projection matrices have two characteristic properties :

Idempotence

A projection matrix P_S is idempotent :

which simply means that if u is already the projection of a vector x, then the projection of u is just u : you can't usefully iterate a projection.

-----

But we'll also demonstrate the slightly more difficult converse statement : if P is a symmetric and idempotent matrix of rank r, then it is the projection matrix on some r-dimensional subspace S of E (that we'll identify).

Eigenvalues of a projection matrix

We'll show that each eigenvalue of a projection matrix is either equal to 0 or to 1, and we'll identify its eigenvectors.

But we'll also show the converse : if the eigenvalues of a symmetric matrix are either equal to 0 or to 1, then this matrix is a projection matrix, and we'll identify the projection subspace.

___________________________________________________________ 

In this Tutorial, we establish :

    * The form of the orthogonal projection matrix on a subspace, whether the available basis of this subspace is orthonormal or not. In addition, we show that this matrix is unique, and does not depend on the particular basis chosen for spanning the projection subspace.

    * We then establish two characteristic properties of projections matrices (that we first show to be always symmetric) :

        - Idempotence.

        - Eigenvalues equal to either 1 or 0.

    * Finally, we show how the action of an operator that is represented by a symmetric matrix can be interpreted in terms of orthogonal projections.

PROJECTION MATRICES

____________________________________________________

Related readings :