Positive definite matrix
A symmetric matrix A is said to be positive semi-definite if, for any non 0 vector x :
If this inequality is strict, the matrix is said to be positive definite.
Positive semidefinite matrices are important in Statistics essentially because :
* The covariance matrix of a random vector is always at least positive semidefinite,
* And conversely, any positive semidefinite matrix is the covariance matrix of some random vector (in fact, of infinitely many),
as we show here.
This is true in particular of a multinormal vector, a fact of central importance for establishing the properties of the multivariate normal distribution.
- If is positive definite, the distribution has a probability density, which occupies the entire space.
- If is of order n but of rank r < n, the distribution is degenerate and is entirely contained is a r-dimensional subspace.
The distribution then has no probability density function anymore, but an appropriate change of reference frame makes it appear as a non degenerate r-dimensional distribution.
* A sample covariance matrix is always at least positive semidefinite. This remark is the basis of all data analysis techniques based on maximizing projected variances, the most prominent of which being Principal Components Analysis (PCA).
A positive (semi-)definite matrix enjoys all the good properties of symmetric matrices. In addition, its positiveness brings about new important properties, of which we list a few with no attempt at being systematic :
* A positive definite matrix is nonsingular, and therefore has an inverse, which turns out to be also positive definite.
* A symmetric matrix with eigenvalues (1, 2 , ... , n ) is :
- Positive semi-definite iff i 0 for all i.
- Positive definite iff i > 0 for all i.
* Any principal square submatrix of a positive definite matrix is positive definite. This result is of great importance in studying the multivariate normal distribution.
* A positive definite matrix A has a unique symmetric "square root" F :
F² = A
which is also positive definite.
A psd matrix has many non symmetric square roots, among which the one obtained by the Cholesky factorization is of particular interest.
* The Singular Value Decomposition of a positive definite matrix is identical to its Spectral Decomposition (see here). This alone is enough to consider pd matrices as the most "regular" of all matrices.
* If A is positive definite with distinct eigenvalues (1, 2 , ... , n ), then the set of points x such that :
x'A-1x = 1
is an ellipsoid. The principal axes of the ellipsoid :
- Are borne by the eigenvectors ui of A.
- Their half lengths are (i)-1/2 .
This last result has an unmistakable flavor of multivariate normal density, with A being the covariance matrix of the distribution. The ellipsoid is then a surface of constant density.
In Principal Components Analysis, i is the variance of the projection of the cloud of observations on the Principal Component defined by ui.
This ellipsoid is visualized in the two following interactive animations illustrating :
* The bivariate normal distribution (see here),
* The concept of covariance matrix (see here).
In this Tutorial, we demonstrate some of the important properties of positive semidefinite matrices. This is not just a pointless catalogue : every one of the properties in the list is actually used in this site.
POSITIVE DEFINITE MATRICES
Diagonal psd matrices
A positive definite matrix is non singular
Transpose of a p(s)d matrix is p(s)d
A matrix similar to a positive definite matrix is positive (semi-) definite
Eigenvalues of a p(s)d matrix are positive (non negative)
Product of any matrix with its transpose is p(s)d
Rank of any matrix is equal to the number of non-zero eigenvalues of its transpose product
Eigenvalues of a transpose product and the reverse transpose product are equal
Inverse of a positive definite matrix is positive definite
Any principal square submatrix of a positive definite matrix is positive definite
The diagonal elements of a positive semidefinite matrix are non negative
"Square roots" of a positive (semi-) definite matrix
Related readings :
Want to contribute to this site ?