The covariance matrix of a random vector is a square matrix that contains all the covariances between the entries of the vector.

Table of contents

Here is a definition.

Definition Let be a random vector. The covariance matrix of , or variance-covariance matrix of , denoted by , is defined as follows:provided the above expected values exist and are well-defined.

It is a multivariate generalization of the definition of variance for a scalar random variable :

Let , ..., denote the components of the vector .

From the definition of , it can easily be seen that is a matrix with the following structure:

Therefore, the covariance matrix of is a square matrix whose generic -th entry is equal to the covariance between and .

Since when , the diagonal entries of the covariance matrix are equal to the variances of the individual entries of .

Here is an example.

Suppose that is a random vector with components and .

Let

By the symmetry of covariance, it must also be

Therefore, the covariance matrix of is

The covariance matrix of a random vector can be computed using the formula

Proof

The above formula can be derived as follows:

This formula also makes clear that the covariance matrix exists and is well-defined only as long as the vector of expected values and the matrix of second cross-moments exist and are well-defined.

The following subsections contain more details about the covariance matrix.

Let be a constant vector and let be a random vector. Then,

Proof

This is a consequence of the fact that (by linearity of the expected value):

Let be a constant matrix and let be a random vector. Then,

Proof

This is easily proved using the fact that (by linearity of the expected value):

Let be a constant vector, be a constant matrix and a random vector. Then, by combining the two properties above, we obtain

The covariance matrix is a symmetric matrix, that is, it is equal to its transpose:

Proof

The proof is as follows:

The covariance matrix is a positive-semidefinite matrix, that is, for any vector :

Proof

This is easily proved by using the Multiplication by constant matrices property above:where the last inequality follows from the fact that variance is always positive.

Let and be two constant vectors and a random vector. Then, the covariance between the two linear transformations and can be expressed as a function of the covariance matrix:

Proof

This can be proved as follows:

The term covariance matrix is sometimes also used to refer to the matrix of covariances between the elements of two vectors.

Let be a random vector and be a random vector.

The covariance matrix between and , or cross-covariance between and is denoted by .

It is defined as follows:provided the above expected values exist and are well-defined.

It is a multivariate generalization of the definition of covariance between two scalar random variables.

Let , ..., denote the components of the vector and , ..., denote the components of the vector .

From the definition of , it can easily be seen that is a matrix with the following structure:

Note that is not the same as . In fact, is a matrix equal to the transpose of :

Below you can find some exercises with explained solutions.

Let be a random vector and denote its components by and .

The covariance matrix of is

Compute the variance of the random variable defined as

Solution

By using a matrix notation, can be written aswhere we have definedTherefore, the variance of can be computed by using the formula for the covariance matrix of a linear transformation:

Let be a random vector and denote its components by , and .

The covariance matrix of is

Compute the following covariance:

Solution

Using the bilinearity of the covariance operator, we obtainThe same result can be obtained by using the formula for the covariance between two linear transformations. Definingwe have

Let be a random vector whose covariance matrix is equal to the identity matrix:

Define a new random vector as follows:where is a matrix of constants such that

Derive the covariance matrix of .

Solution

By the formula for the covariance matrix of a linear transformation, we have

Please cite as:

Taboga, Marco (2021). "Covariance matrix", Lectures on probability theory and mathematical statistics. Kindle Direct Publishing. Online appendix. https://www.statlect.com/fundamentals-of-probability/covariance-matrix.

The books

Most of the learning materials found on this website are now available in a traditional textbook format.

Featured pages

- Gamma distribution
- Normal distribution
- Hypothesis testing
- Statistical inference
- Uniform distribution
- Gamma function

Explore

Main sections

- Mathematical tools
- Fundamentals of probability
- Probability distributions
- Asymptotic theory
- Fundamentals of statistics
- Glossary

About

Glossary entries

- Continuous random variable
- Precision matrix
- Convolutions
- Posterior probability
- Null hypothesis
- Loss function

Share

- To enhance your privacy,
- we removed the social buttons,
- but
**don't forget to share**.