Positive definite matrix

Definition of a positive definite matrix

A symmetric matrix A is said to be positive semi-definite if, for any non 0 vector x :

If this inequality is strict, the matrix is said to be positive definite.

-----

Positive semidefinite matrices are important in Statistics essentially because :

    * The covariance matrix  of a random vector is always at least positive semidefinite,

    * And conversely, any positive semidefinite matrix is the covariance matrix of some random vector (in fact, of infinitely many),

as we show here.

This is true in particular of a multinormal vector, a fact of central importance for establishing the properties of the multivariate normal distribution.

        - If is positive definite, the distribution has a probability density, which occupies the entire space.

        - If  is of order n but of rank r < n,  the distribution is degenerate and is entirely contained is a r-dimensional subspace.

The distribution then has no probability density function anymore, but an appropriate change of reference frame makes it appear as a non degenerate r-dimensional distribution.

    * A sample covariance matrix is always at least positive semidefinite. This remark is the basis of all data analysis techniques based on maximizing projected variances, the most prominent of which being Principal Components Analysis (PCA).

Properties of positive definite matrices

A positive (semi-)definite matrix enjoys all the good properties of symmetric matrices. In addition, its positiveness brings about new important properties, of which we list a few with no attempt at being systematic :

    * A positive definite matrix is nonsingular, and therefore has an inverse, which turns out to be also positive definite.

    * A symmetric matrix with eigenvalues (₁, ₂ , ... , _n ) is :

        - Positive semi-definite iff  _i  0 for all i.

        - Positive definite iff  _i > 0 for all i.

    * Any principal square submatrix of a positive definite matrix is positive definite. This result is of great importance in studying the multivariate normal distribution.

    * A positive definite matrix A has a unique symmetric "square root" F :

F² = A

which is also positive definite.

A psd matrix has many non symmetric square roots, among which the one obtained by the Cholesky factorization is of particular interest.

    * The Singular Value Decomposition of a positive definite matrix is identical to its Spectral Decomposition (see here). This alone is enough to consider pd matrices as the most "regular" of all matrices.

    * If A is positive definite with distinct eigenvalues (₁, ₂ , ... , _n ), then the set of points x such that :

x'A^-1x = 1

is an ellipsoid. The principal axes of the ellipsoid :

        - Are borne by the eigenvectors u_i of A.

        - Their half lengths are (_i)^-1/2 .

-----

This last result has an unmistakable flavor of multivariate normal density, with A being the covariance matrix of the distribution. The ellipsoid is then a surface of constant density.

In Principal Components Analysis, _i is the variance of the projection of the cloud of observations on the Principal Component defined by u_i.

This ellipsoid is visualized in the two following interactive animations illustrating :

    * The bivariate normal distribution (see here),

    * The concept of covariance matrix (see here).

 ______________________________________________________________

In this Tutorial, we demonstrate some of the important properties of positive semidefinite matrices. This is not just a pointless catalogue : every one of the properties in the list is actually used in this site.

POSITIVE DEFINITE MATRICES

____________________________________________________

Related readings :