To understand conditional probability distributions, you need to be familiar with the concept of conditional probability, which has been introduced in the lecture entitled Conditional probability.
We discuss here how to update the probability distribution of a random variable after observing the realization of another random variable , i.e., after receiving the information that another random variable has taken a specific value . The updated probability distribution of will be called the conditional probability distribution of given .
The two random variables and , considered together, form a random vector . Depending on the characteristics of the random vector , different procedures need to be adopted in order to compute the conditional probability distribution of given . In the remainder of this lecture, these procedures are presented in the following order:
first, we tackle the case in which the random vector is a discrete random vector;
then, we tackle the case in which is a continuous random vector;
finally, we briefly discuss the case in which is neither discrete nor continuous.
Important: note that if we are able to update the probability distribution of when we observe the realization of (i.e., when we receive the information that ), then we are also able to update the probability distribution of when we receive the information that a generic event has happened: it suffices to set , where is the indicator function of the event , and update the distribution of based on the information .
In the case in which is a discrete random vector (as a consequence is a discrete random variable), the probability mass function of conditional on the information that is called conditional probability mass function.
Definition Let be a discrete random vector. We say that a function is the conditional probability mass function of given if, for any ,where is the conditional probability that given that .
How do we derive the conditional probability mass function from the joint probability mass function ? The following proposition provides an answer to this question.
Proposition Let be a discrete random vector. Let be its joint probability mass function, and the marginal probability mass function of . The conditional probability mass function of given isprovided .
This is just the usual formula for computing conditional probabilities (conditional probability equals joint probability divided by marginal probability):
Note that the above proposition assumes knowledge of the marginal probability mass function , which can be derived from the joint probability mass function by marginalization (go here if you do not remember how).
Example Let the support of be and its joint probability mass function beLet us compute the conditional probability mass function of given . The support of isThe marginal probability mass function of evaluated at isThe support of isThus, the conditional probability mass function of given is
In the case in which , there is, in general, no way to unambiguously derive the conditional probability mass function of , as we will show below with an example. The impossibility of deriving the conditional probability mass function unambiguously in this case (called by some authors the Borel-Kolmogorov paradox) is not particularly worrying, as this case is seldom relevant in applications. The following is an example of a case in which the conditional probability mass function cannot be derived unambiguously (the example is a bit involved; the reader might safely skip it on a first reading).
Example Suppose we are given the following sample space:i.e. the sample space is the set of all real numbers between and . It is possible to build a probability measure on , such that assigns to each sub-interval of a probability equal to its length, that is,This is the same sample space discussed in the lecture on zero-probability events. Define a random variable as follows:and another random variable as follows:Both and are discrete random variables and, considered together, they constitute a discrete random vector . Suppose we want to compute the conditional probability mass function of conditional on . It is easy to see that . As a consequence, we cannot use the formulabecause division by zero is not possible. It turns out that also the technique of implicitly deriving a conditional probability as a realization of a random variable satisfying the definition of a conditional probability with respect to a partition (see the lecture entitled Conditional probability as a random variable) does not allow to unambiguously derive . In this case, the partition of interest is , whereand can be viewed as the realization of the conditional probability when . The fundamental property of conditional probability is satisfied in this case if and only if, for a given , the following system of equations is satisfied:which impliesThe second equation does not help determining . So, from the first equation it is evident that is undetermined (any number, when multiplied by zero, gives zero). One can show that also the requirement that be a regular conditional probability does not help to pin down . What does it mean that is undetermined? It means that any choice of is legitimate, provided the requirement is satisfied. Is this really a paradox? No, because conditional probability with respect to a partition is defined up to almost sure equality, is a zero-probability event, so the value that takes on does not matter (roughly speaking, we do not really need to care about zero-probability events, provided there is only a countable number of them).
In the case in which is a continuous random vector (as a consequence is a continuous random variable), the probability density function of conditional on the information that is called conditional probability density function.
Definition Let be a continuous random vector. We say that a function is the conditional probability density function of given if, for any interval ,and is such that the above integral is well defined.
How do we derive the conditional probability mass function from the joint probability density function ?
Deriving the conditional distribution of given is far from obvious: whatever value of we choose, we are conditioning on a zero-probability event ( - see here for an explanation); therefore, the standard formula (conditional probability equals joint probability divided by marginal probability) cannot be used. However, it turns out that the definition of conditional probability with respect to a partition can be fruitfully applied in this case to derive the conditional probability density function of given :
Proposition Let be a continuous random vector. Let be its joint probability density function, and be the marginal probability density function of . The conditional probability density function of given isprovided .
To prove thatis a legitimate choice, we need to prove that conditional probabilities calculated by using the above conditional density function satisfy the fundamental property of conditional probability:for any and . Thanks to some basic results in measure theory, we can confine our attention to the events and that can be written as follows:For these events, it is immediate to verify that the fundamental property of conditional probability holds. First, by the very definition of a conditional probability density function, we have thatFurthermore, is also a function of . Therefore, the product is a function of , so we can use the transformation theorem to compute its expected value: The last equality proves the proposition.
Example Let the support of be and its joint probability density function beLet us compute the conditional probability density function of given . The support of isWhen , the marginal probability density function of is ; when , the marginal probability density function isThus, the marginal probability density function of isWhen evaluated at , it isThe support of isThus, the conditional probability density function of given is
In general, when is neither discrete nor continuous, we can characterize the distribution function of conditional on the information that . We define the conditional distribution function of given as follows.
Definition We say that a function is the conditional distribution function of given if and only ifwhere is the conditional probability that given that .
There is no immediate way of deriving the conditional distribution of given . However, we can characterize it using the concept of conditional probability with respect to a partition, as follows.
Define the events as follows:and a partition of events aswhere, as usual, is the support of .
Then, for any we havewhere is the probability that conditional on the partition . As we know, is guaranteed to exist and is unique up to almost sure equality. Of course, this does not mean that we are able to compute it. Nonetheless, this characterization is extremely useful, because it allows us to speak of the conditional distribution of given in general, without a need to specify whether and are discrete or continuous.
The following sections contain more details about conditional distributions.
We have discussed how to update the probability distribution of a random variable after observing the realization of another random variable , that is, after receiving the information that . What happens when and are random vectors rather than random variables? Basically, everything we said above still applies with straightforward modifications.
Thus, if and are discrete random vectors, then the conditional probability mass function of given isprovided .
If and are continuous random vectors then the conditional probability density function of given isprovided . In general, the conditional distribution function of given is
As we have explained above, the joint distribution of and can be used to derive the marginal distribution of and the conditional distribution of given . This process can also go in the reverse direction: if we know the marginal distribution of and the conditional distribution of given , then we can derive the joint distribution of and . For discrete random variables, we have thatFor continuous random variables, we have that
Below you can find some exercises with explained solutions.
Let be a discrete random vector with supportand joint probability mass functionCompute the conditional probability mass function of given .
The marginal probability mass function of evaluated at isThe support of isThus, the conditional probability mass function of given is
Let be a continuous random vector with supportand its joint probability density function beCompute the conditional probability density function of given .
The support of isWhen , the marginal probability density function of is ; when , the marginal probability density function of isThus, the marginal probability density function of isWhen evaluated at the point , it becomesThe support of isThus, the conditional probability density function of given is
Let be a continuous random variable with supportand probability density functionLet be another continuous random variable with supportand conditional probability density functionFind the marginal probability density function function of .
The support of the vector isand the joint probability function of and isThe marginal probability density function of is obtained by marginalization, integrating out of the joint probability density function
Thus, for we trivially have (because ), while for we haveThus, the marginal probability density function of is
Most of the learning materials found on this website are now available in a traditional textbook format.