This lecture deals with the Wishart distribution, which is a multivariate generalization of the Gamma distribution.
In previous lectures we have explained that:
A Chi-square random variable with degrees of freedom can be seen as a sum of squares of independent normal random variables having mean 0 and variance 1.
A Gamma random variable with parameters and can be seen as a sum of squares of independent normal random variables having mean 0 and variance .
A Wishart random matrix with parameters and can be seen as a sum of outer products of independent multivariate normal random vectors having mean 0 and covariance matrix . In this sense, the Wishart distribution can be considered a generalization of the Gamma distribution (take point 2 above and substitute normal random variables with multivariate normal random vectors, squares with outer products and the variance with the covariance matrix ).
At the bottom of this page you can find a brief review of some basic concepts in matrix algebra that will be helpful in understanding the remainder of this lecture.
Wishart random matrices are characterized as follows:
Definition Let be a absolutely continuous random matrix. Let its support be the set of all symmetric and positive definite real matrices:Let be a symmetric and positive definite matrix and . We say that has a Wishart distribution with parameters and if its joint probability density function is:whereand is the Gamma function.
The parameter needs not be an integer, but, when is not an integer, can no longer be interpreted as a sum of outer products of multivariate normal random vectors.
The following proposition provides the link between the multivariate normal distribution and the Wishart distribution:
Proposition Let be independent random vectors all having a multivariate normal distribution with mean and covariance matrix . Let . Define:Then has a Wishart distribution with parameters and .
The proof of this proposition is quite lengthy and complicated. The interested reader might have a look at:
Ghosh, M. and Sinha, B. K. (2002) "A simple derivation of the Wishart distribution", The American Statistician, 56, 100-101.
The expected value of a Wishart random matrix is:
We do not provide a fully general proof, but we prove this result only for the special case in which is integer and can be written as:(see subsection above). In this case:where we have used the fact that the covariance matrix of can be written as:(see the lecture entitled Covariance matrix).
The concept of covariance matrix is well-defined only for random vectors. However, when dealing with a random matrix, one might want to compute the covariance matrix of its associated vectorization (if you are not familiar with the concept of vectorization, see the review of matrix algebra below for a definition). Therefore, in the case of a Wishart random matrix , we might want to compute the following covariance matrix:
Since , the vectorization of , is a random vector, is a matrix.
It is possible to prove that:where denotes the Kronecker product and is the transposition-permutation matrix associated to (see the review of matrix algebra below for a defintion).
The proof of this formula can be found in: Muirhead, R.J. (2005) Aspects of multivariate statistical theory, Wiley.
There is a simpler expression for the covariances between the diagonal entries of :
Again, we do not provide a fully general proof, but we prove this result only for the special case in which is integer and can be written as:(see above). To compute this covariance, we first need to compute the following fourth cross-moment:where denotes the -th component () of the random vector (). This cross-moment can be computed by taking the fourth cross-partial derivative of the joint moment generating function of and and evaluating it at zero (see the lecture entitled Joint moment generating function). While this is not complicated, the algebra is quite tedious. I recommend doing it with computer algebra, for example utilizing the Matlab Symbolic Toolbox and the following four commands:
syms t1 t2 s1 s2 s12;
The result of the computations is:Using this result, the covariance between and is derived as follows:
As the Wishart distribution involves outer products of multivariate normal random vectors, we briefly review here the concept of outer product.
If is a column vector, the outer product of with itself is the matrix obtained from the multiplication of with its transpose:
Example If is the random vectorthen its outer product is the random matrix
A matrix is symmetric if and only if:i.e. if and only if equals its transpose.
A matrix is said to be positive definite if and only if for any real vector such that .
All positive definite matrices are also invertible.
The proof is by contradiction. Suppose a positive definite matrix were not invertible. Then would not be full rank, i.e. there would be a vector such thatwhich, premultiplied by , would yield:But this is a contradiction.
Let be a matrix and denote by the -th entry of (i.e. the entry at the intersection of the -th row and the -th column). The trace of , denoted by , is the sum of all the diagonal entries of :
Given a matrix , its vectorization, denoted by , is the vector obtained by stacking the columns of on top of each other.
Example If is a matrix:the vectorization of is the random vector:
For a given matrix , the vectorization of will in general be different from the vectorization of its transpose . The transposition permutation matrix associated to is the matrix such that:
Given a matrix and a matrix , the Kronecker product of and , denoted by , is a matrix having the following structure:where is the -th entry of .
Most learning materials found on this website are now available in a traditional textbook format.