Random vector

Let {X₁, X₂ ,  ..., X_n} be a set of n random variables.

It is often convenient to consider this set as a single object x = {X₁, X₂ ,  ..., X_n} that will be called a random vector. This is particularly true when this set is to be submitted to linear transformations, thus making Linear Algebra the natural mathematical tool.

Just as a random variable is described by its probability distribution, a random vector is is described by the joint probability distribution of the n variables that make up the vector.

The above illustration represents the probability distribution of a random vector of dimension 2.

-----

The moments of a random vector are defined just as for a random variable (see below). But because of the "internal structure" of a random vector that a simple random variable is lacking, the definitions are both a bit more complex and richer that those pertaining to random variables. In particular :

    * Multiplicative constants must be replaced by constant (not random) matrices, or constant vectors.

    * Variances must be replaced by covariance matrices.

Expectation of a random vector

Definition

By definition, the expectation of a random vector is the vector whose components are the expectations of the r.v. making up the vector :

µ = E[x] is the center of gravity of the joint probability distribution of {X₁, X₂ ,  ..., X_n}.

Linearity

It is immediately verified that the expectation is linear.

    * If x and y are two random vectors, and if A and B are two constant matrices :

E[Ax + By] = AE[x] + BE[y] 

    * Also :

E[AxB] = AE[x]B

which is for vectors what E[aX] = aE[X] is for random variables.

We demonstrate this result in the course of calculating the covariance matrix of a multivariate normal distribution.

    * If b is a constant vector :

E[Ax + b] = AE[x] + b

Variance of a random vector

Covariance matrix

The definition of the variance of a random vector is the same as that of a random variable. Denote µ the vector E[X]. Then, by definition :

Var(x) = E[(x - µ)(x - µ)']

Var(x) is a symmetric, positive (semi-) definite square matrix that is called the covariance matrix of x, or the covariance matrix of the set of variables {X₁, X₂ ,  ..., X_n}. It is usually denoted .

This expression may also be written :

Var(x) = E[xx'] - µµ'

which is for random vectors what  :

Var(X) = E[X ²] - µ²

is for random variables.

Variance of the transform of a vector

It is easily shown that if b is a constant vector :

which is for random vectors what  :

Var(aX + b) = a²Var(X)

is for random variables.

We demonstrate this result in the course of calculating the covariance matrix of a multivariate normal distribution.

Variance of an inner product

If the matrix A' is a unique row vector a', and if the vector b is reduced to a unique component b, then a'X + b is just a random variable, and the "Var" of the left hand term is just the ordinary variance of a r.v.., and not a covariance matrix.

If, in addition, b = 0, the above expression gives the variance of the inner product of a fixed vector and a random vector.

Variance of the projection of a vector

In particular, if a is a unit vector, Var(a'X) is the variance of the projection P of the random vector X on the straight line D defined by the fixed unit vector a.

We therefore have :

where  is the covariance matrix of X.

This result is very useful, in particular when studying the multivariate normal distribution and in Discriminant Analysis.

Covariance of two random vectors

Let :

    * x be a random vector mx1:

x = {X₁, X₂ ,  ..., X_m }

    * y be a random vector nx1:

y = {Y₁, Y₂ ,  ..., Y_n}

Definition

The covariance Cov(x, y) of these two random vectors is defined exactly as is defined the covariance of two random variables :

Cov(x, y) = E[(x₁ - µ_x )(x₂ - µ_y )']

where µ_x and µ_y are the mean vectors of x and of y.

Developping this expression shows that Cov(x, y) is a matrix with m rows and n columns whose generic term is :

[Cov(x, y)]_ij = Cov(X_i, Y_j )

where "Cov" in the right hand side term is the ordinary covariance between two random variables.

-----

The covariance of two random vectors will appear naturally when we partition the covariance matrix of a multivariate normal distribution (see here).

Properties of the covariance of two random vectors

It is easily verified that :

            - Transpose

Cov(y, x) = [Cov(x, y)]'

            - Variance of a sum or random vectors

If x and y have the same number of components :

 Var(x y) = Var(x) + Var(y)  2.Cov(x, y)

for Cov(x, y) is then a symmetric matrix.

            - Orthogonality

If x and y have the same number of components, they are said to be orthogonal if :

x'y = 0

It is then immediately verified that :

Cov(x, y) = 0

and that therefore :

 Var(x y) = Var(x) + Var(y)

 ____________________________________________________________

Related readings :