Search
STATLECT > Fundamentals of probability

Covariance matrix

Let X be a Kx1 random vector. The covariance matrix of X (or variance-covariance matrix of X), denoted by [eq1], is defined as follows:[eq2]provided the above expected value exists and is well-defined. It is a multivariate generalization of the definition of variance for a scalar random variable Y:[eq3]

Structure

Let X_1, ..., $X_{K}$ denote the K components of the vector X. From the definition of [eq4], it can easily be seen that [eq5] is a $K imes K$ matrix with the following structure:[eq6]

Therefore, the covariance matrix of X is a square $K imes K$ matrix whose generic $left( i,j ight) $-th entry is equal to the covariance between X_i and $X_{j}$.

Since the covariance between X_i and $X_{j}$ is equal to the variance of X_i when $i=j$ (i.e. [eq7]), the diagonal entries of the covariance matrix are equal to the variances of the individual components of X.

Example_ Suppose X is a $2 imes 1$ random vector with components X_1 and $X_{2} $. Let:[eq8]By the symmetry of covariance, it must also be:[eq9]Therefore, the covariance matrix of X is:[eq10]

Formula for computing the covariance matrix

The covariance matrix of a Kx1 random vector X can be computed as follows:[eq11]

nav_button Proof

The above formula can be derived as follows:[eq12]

This formula also makes clear that the covariance matrix exists and is well-defined only as long as the vector of expected values [eq13] and the matrix of second cross-moments [eq14] exist and are well-defined.

More details

The following subsections contain more details about the covariance matrix.

Addition to constant vectors

Let $ain U{211d} ^{K}$ be a constant Kx1 vector and let X be a Kx1 random vector. Then:[eq15]

nav_button Proof

This is a consequence of the fact that [eq16] (by linearity of the expected value):[eq17]

Multiplication by constant matrices

Let $b$ be a constant $M imes K$ matrix and let X be a Kx1 random vector. Then:[eq18]

nav_button Proof

This is easily proved using the fact that [eq19] (by linearity of the expected value):[eq20]

Linear transformations

Let $ain U{211d} ^{K}$ be a constant Kx1 vector, $b$ be a constant $M imes K$ matrix and X a Kx1 random vector. Then, combining the two properties above, one obtains:[eq21]

Symmetry

The covariance matrix [eq5] is a symmetric matrix, i.e. it is equal to its transpose:[eq23]

Semi-positive definiteness

The covariance matrix [eq5] is a positive-semidefinite matrix, i.e. for any $1 imes K$ vector $ain U{211d} ^{K}$:[eq25]This is easily proved using the Multiplication by constant matrices property above:[eq26]where the last inequality follows from the fact that variance is always positive.

Covariance between linear transformations

Let a and $b$ be two constant $1 imes K$ vectors and X a Kx1 random vector. Then, the covariance between the two linear transformations $aX$ and $bX$ can be expressed as a function of the covariance matrix:[eq27]

nav_button Proof

This can be proved as follows:[eq28]

Cross-covariance

The term covariance matrix is sometimes also used to refer to the matrix of covariances between the elements of two vectors. Let X be a Kx1 random vector and Y be a $L imes 1$ random vector. The covariance matrix between X and Y (or cross-covariance between X and Y), denoted by [eq29], is defined as follows:[eq30]provided the above expected value exists and is well-defined. It is a multivariate generalization of the definition of covariance between two scalar random variables. Let X_1, ..., $X_{K}$ denote the K components of the vector X and $Y_{1}$, ..., $Y_{L}$ denote the $L$ components of the vector Y . From the definition of [eq31], it can easily be seen that [eq32] is a $K imes L$ matrix with the following structure:[eq33]Note that [eq32] is not the same as [eq35]. In fact, [eq36] is a $L imes K$ matrix equal to the transpose of [eq32]:[eq38]

Solved exercises

Below you can find some exercises with explained solutions:

  1. Exercise set 1 (use of the covariance matrix)

by
About | Contacts | Privacy and terms of use | Sitemap