Symmetric matrix

Definition

A square matrix A = [a_ij] is said to be symmetric if for all i and all j :

a_ij = a_ji  

Two elements that are symmetrical with respect to the first diagonal are equal : a symmetric matrix A is therefore equal to its transpose A'.

The family of symmetric matrices is particularly rich in "good" properties. In particular, the eigenvalues of a (real) symmetric matrix are real, and so are its eigenvectors. In addition, these eigenvectors form an orthonormal base (see below).

Two sub-families of the family of symmetric matrices have properties and interpretations that are useful for the statistician :

    * Projection matrices,

    * And positive (semi-) definite matrices.

Symmetric matrices, Statistics and Modeling

Statistics and Data Modeling call on symmetric matrices on many occasions :

    * A covariance matrix is symmetric by definition (and in addition is positive semidefinite, see below).

    * A projection matrix (see below) projects the data space onto a linear subspace. Principal Components Analysis (PCA) and Linear Regression can both be interpreted in terms of projections onto subspaces, and therefore in terms of projection matrices.

    * Ridge Regression can be interpreted as a modified Regression on Principal Components that operates on the spectral decomposition (see below) of the data covariance matrix.

    * Establishing the properties of quadratic forms in normal random variables also relies on projection matrices.

Eigenvalues and eigenvectors of a symmetric matrix

Symmetric matrices enjoy many important properties. In the Tutorial below, we'll establish some of these properties that will be needed in other places of this site.

Eigenvalues and eigenvectors of a symmetric matrix are real

    * The eigenvalues of a symmetric matrix are real. Recall that just because the coefficients of a matrix are real certainly does not imply that the eigenvalues of this matrix are real.

    * A (real) eigenvalue may be associated to a complex eigenvector. But it is then always possible to find a real eigenvector associated to this eigenvalue. We'll consider only these real eigenvectors, and we'll therefore consider that the eigenvectors of a symmetric matrix are real.

The eigenvectors of a symmetric matrix are orthogonal

We'll show that two eigenvectors associated to two distinct eigenvalues are orthogonal. A consequence is that if all the eigenvalues of the matrix are distinct, the normalized eigenvectors form an orthonormal reference frame.

This result remains true even when there are multiple eigenvalues, but it is then quite a bit more difficult, and we'll have to state it without proof.

-----

So, quite generally :

Spectral decomposition of a symmetric matrix

Denote U the square matrix of order p whose columns are the eigenvectors of a symmetric matrix A. We'll show that :

where D = diag(₁, ₂ , ... , _n ) is the diagonal matrix of the eigenvalues of A.

This fundamental expression is called the spectral decomposition of the symmetric matrix A.

As the eigenvectors of A form an orthonormal base, the matrix U is an orthogonal matrix.

-----

Expanding the above expression leads to :

A = _iiu_iu_i'

where {u_i} is the set of the eigenvectors of A.

We'll see that it can be interpreted in terms of projectors on the eigenvectors of A.

-----

This result is central for many questions pertaining to covariance matrices, in particular :

    * In Principal Components Analysis,

    * In Ridge Regression.

Projection matrices

A symmetric matrix P is said to be a projection matrix if it is also idempotent, that is, if :

P ² = P

We'll justify this definition by showing that P is then the matrix of a linear operator that projects Uⁿ onto a subspace E of dimension r (where r is equal to the rank of P).

In addition, we'll show that (I_n - P) is then the projection operator on the subspace E_z of the vectors that are orthogonal to all vectors in E.

-----

These results are central :

    * For studying the multivariate normal distribution.

    * In Principal Components Analysis.

    * In Linear Regression (either Simple or Multiple), which can be interpreted as a special type of orthogonal projection.

Positive (semi-) definite matrices

A symmetric matrix A is said to be positive semi-definite if, for any non 0 vector x :

If this inequality is always strict, the matrix is said to be positive definite.

A covariance matrix is always at least positive semi-definite.

-----

We'll show, among other things that :

    * The inverse of a positive definite matrix exists and is positive definite.

    * Any square submatrix of a positive definite matrix is positive definite.

    * A positive definite matrix admits a unique symmetric "square root", which turns out to be also positive definite.

    * A symmetric matrix with eigenvalues (₁, ₂ , ... , _n ) is :

        - Positive semi-definite iff _i  0 for all i.

        - Positive definite iff _i > 0 for all i.

    * Any square submatrix of a pd matrix is pd. This result is of great importance in studying the multivariate normal distribution.

    * If A is a positive definite with distinct eigenvalues (₁, ₂ , ... , _n ), then the set of points x such that :

x'A^-1x = 1

is an ellipsoid. The principal axes of the ellipsoid :

        - Are borne by the eigenvectors u_i of A.

        - Their half lengths are (_i)^-1/2 .

-----

This last result has an unmistakable flavor of multivariate normal density, with A being the covariance matrix of the distribution. The ellipsoid is then a surface of constant density.

In Principal Components Analysis, _i is the variance of the projection of the cloud of observations on the Principal Component defined by u_i.

This ellipsoid is visualized in the two following interactive animations illustrating :

    * The bivariate normal distribution (see here),

    * The concept of covariance matrix (see here).

________________________________________________________________

In this Tutorial, we go over those properties of symmetric matrices that will be needed throughout this site for establishing important results about :

    * Principal Components Analysis,

    * Linear Regression (Simple and Multiple),

    * Ridge Regression,

    * Multivariate normal distribution,

    * Quadratic forms in normal variables.

----

The Tutorial is certainly not as extensive and rigorous as a genuine course in Linear Algebra. In addition, some classic or difficult results will be stated without proof. Yet it is written so as to as to make the fundamental properties of symmetric matrices accessible to statisticians in their daily practice.

SYMMETRIC MATRICES

 _____________________________________________________

Related readings :