Search
STATLECT > Fundamentals of probability

Probability

Probability is used to quantify the likelihood of things that can happen, when it is not yet known whether they will happen. Sometimes probability is also used to quantify the likelihood of things that could have happened in the past, when it is not yet known whether they actually happened.

Since we usually speak of the "probability of an event" the next section introduces a formal definition of the concept of event. We then discuss the properties that probability needs to satisfy. Finally, we discuss some possible interpretations of the concept of probability.

Sample space, sample points and events

Let Omega be a set of things that can happen. We say that Omega is a sample space, or space of all possible outcomes, if it satisfies the following two properties:

  1. Mutually exclusive outcomes. Only one of the things in Omega will happen. That is, when we learn that [eq1] has happened, then we also know that none of the things in the set [eq2] has happened.

  2. Exhaustive outcomes. At least one of the things in Omega will happen.

An element omega in Omega is called a sample point, or a possible outcome. When (and if) we learn that [eq3] has happened, $overline{omega }$ is called the realized outcome.

A subset $Esubseteq Omega $ is called an event (in the section below you will see that not every subset of Omega is, strictly speaking, an event; however, on a first reading you can be happy with this definition). Note that Omega itself is an event, because every set is a subset of itself, and the empty set $emptyset $ is also an event, because it can be considered a subset of Omega.

Example_ Suppose that we toss a die. Six numbers, from 1 to $6$, can appear face up, but we do not yet know which one of them will appear. The sample space is:[eq4]Each of the six numbers is a sample point. The outcomes are mutually exclusive, because only one number at a time can appear face up. The outcomes are also exhaustive, because at least one of the six numbers in Omega will appear face up, after we toss the die. Define:[eq5]E is an event (a subset of Omega). In words, the event E can be described as 'an odd number appears face up'. Now define:[eq6]Also F is an event (a subset of Omega). In words, the event F can be described as 'the number $6$ appears face up'.

Probability and its properties

The probability of an event is a real number, attached to the event, that tells us how likely that event is. Suppose E is an event. We denote the probability of E by [eq7].

Probability needs to satisfy the following properties:

  1. Range. For any event E, [eq8].

  2. Sure thing. [eq9].

  3. Sigma-additivity (or countable additivity). Let [eq10] be a sequence of events. Let all the events in the sequence be mutually exclusive, i.e. [eq11] if $i eq j$. Then:[eq12]

The first property is self-explanatory. It just means that the probability of an event is a real number between 0 and 1.

The second property says that at least one of all the things that can possibly happen will happen with probability 1.

The third property is a bit more cumbersome. It can be proved (see below) that if sigma-additivity holds, then also the following holds:[eq13]

This property, called finite additivity, while very similar to sigma-additivity, is easier to interpret. It says that if two events are disjoint, then the probability that either one or the other happens is equal to the sum of their individual probabilities.

Example_ Suppose that we flip a coin. The possible outcomes are either tail ($T$) or head (H), i.e.:[eq14]There are a total of four subsets of Omega (events): Omega itself, the empty set $emptyset $, the event $left{ T ight} $ and the event $left{ H ight} $. The following assignment of probabilities satisfies the properties enumerated above:[eq15]All these probabilities are between 0 and 1, so the range property is satisfied. [eq9], so the sure thing property is satisfied. Also sigma-additivity is satisfied, because:[eq17]and the four couples $left( T,H ight) $, [eq18], [eq19], [eq20] are the only four possible couples of disjont sets.

Before ending this section, two remarks are in order. First, we have not discussed the interpretations of probability, but below you can find a brief discussion of the interpretations of probability. Second, we have been somewhat sloppy in defining events and probability, but you can find a more rigorous definition of probability below.

Other properties of probability

The following subsections discuss other properties enjoyed by probability.

The probability of the empty set is 0

Here we prove that [eq21]. Define a sequence of event as follows: $E_{1}=Omega $, $E_{2}=emptyset $, ..., $E_{n}=emptyset $, ... . The sequence is a sequence of disjoint events, because the empty set is disjoint from any other set. Then: [eq22]that is:[eq23]Since [eq9],[eq25]which implies [eq21].

A sigma-additive function is additive

It is easy to prove that a sigma-additive function is also additive. In fact, let E and F be two events and $Ecap F=emptyset $. Define a sequence of events as follows: $E_{1}=E$, $E_{2}=F$, $E_{3}=emptyset $, ..., $E_{n}=emptyset $, ... . The sequence is a sequence of disjoint events, because the empty set is disjoint from any other set. Then:[eq27]since [eq21].

Probability of the complement

Let E be an event and $E^{c}$ its complement (i.e. the set of all elements of Omega that do not belong to E). Note that:[eq29]and that E and $E^{c}$ are disjoint sets. Then, using the sure thing property and finite additivity, we obtain:[eq30]Thus:[eq31]In words, the probability that an event does not occur ([eq32]) is equal to one minus the probability that it occurs.

Probability of a union

Let E and F be two events. We have already seen how to compute [eq33] in the special case in which E and F are disjoint. In the more general case (E and F are not necessarily disjoint), the formula is:[eq34]This is proved as follows. First note that:[eq35]so that:[eq36]Furthermore the event $Ecup F$ can be written as follows:[eq37]and the three events on the right hand side are disjoint. Thus:[eq38]

Monotonicity of probability

If two events E and F are such that $Esubseteq F$, then:[eq39]i.e., if E occurs less often than F (because F contemplates more occurrences), then the probability of E must be less than the probability of F. This is easily proved using additivity:[eq40]Since, by the range property, [eq41], it follows that:[eq42]

Interpretations of probability

This subsection briefly discusses some common interpretations of probability. Although none of these interpretations is sufficient per se to clarify the meaning of probability, they all touch upon important aspects of probability.

Classical interpretation of probability

According to the classical definition of probability, when all the possible outcomes of an experiment are equally likely, the probability of an event is the ratio between the number of outcomes that are favorable to the event and the total number of possible outcomes. While intuitive, this definition has two main drawbacks:

  1. it is circular, because it uses the concept of probability to define probability: it is based on the assumption of 'equally likely' outcomes, where equally likely means 'having the same probability';

  2. it is limited in scope, because it does not allow to define probability when the possible outcomes are not all equally likely.

Frequentist interpretation of probability

According to the frequentist definition of probability, the probability of an event is the relative frequency of the event itself, observed over a large number of repetitions of the same experiment. In other words, it is the limit to which the ratio:[eq43]converges when the number of repetitions of the experiment tends to infinity. Despite its intuitive appeal, also this definition of probability has some important drawbacks:

  1. it assumes that all probabilistic experiments can be repeated many times, which is false;

  2. it is also somewhat circular, because it implicitly relies on a Law of Large Numbers, which can be derived only after having defined probability.

Subjectivist interpretation of probability

According to the subjectivist definition of probability, the probability of an event is related to the willingness of an individual to accept bets on that event. Suppose a lottery ticket pays off 1 dollar in case the event occurs and 0 in case the event does not occur. An individual is asked to set a price for this lottery ticket, at which she must be indifferent between being a buyer or a seller of the ticket. The subjective probability of the event is defined to be equal to the price thus set by the individual. Also this definition of probability has some drawbacks:

  1. different individuals can set different prices, therefore preventing an objective assessment of probabilities;

  2. the price an individual is willing to pay to participate in a lottery can be influenced by other factors that have nothing to do with probability; for example, an individual's betting behavior can be influenced by her preferences.

Probability and events - A more rigorous definition

A more rigorous definition of event

The definition of event given above is not entirely rigorous. Often, statisticians work with probability models where some subsets of Omega are not considered events. This happens mainly for the following two reasons:

  1. sometimes, Omega is a really complicated set; to make things simpler, attention is restricted to only some subsets of Omega;

  2. sometimes, it is possible to assign probabilities only to some subsets of Omega; in these cases, only the subsets to which probabilities can be assigned are considered events.

Denote by $ ciFourier $ the set of subsets of Omega which are considered events. $ ciFourier $ is called the space of events. In rigorous probability theory, $ ciFourier $ is required to be a sigma-algebra on Omega. $ ciFourier $ is a sigma-algebra on Omega if it is a set of subsets of Omega satisfying the following three properties:

  1. Whole set. [eq44].

  2. Closure under complementation. If $Ein ciFourier $ then also [eq45] ($E^{c}$, the complement of E with respect to Omega, is the set of all elements of Omega that do not belong to E).

  3. Closure under countable unions. If $E_{1}$, $E_{2}$, ..., $E_{n}$,... are a sequence of subsets of Omega belonging to $ ciFourier $ then:[eq46]

Why is a space of events required to satisfy these properties? Besides a number of mathematical reasons, it seems pretty intuitive that they must be satisfied. Property a) means that the space of events must include the event 'something will happen', quite a trivial requirement! Property b) means that if 'one of the things in the set E will happen' is considered an event, then also 'none of the things in the set E will happen' is considered an event. This is quite natural: if you are considering the possibility that an event will happen, then, by necessity, you must also be simultaneously considering the possibility that the same event will not happen. Property c) is a bit more complex. However, the following property, implied by c), is probably easier to interpret:[eq47]It means that if 'one of the things in E will happen' and 'one of the things in F will happen' are considered two events, then also 'either one of the things in E or one of the things in F will happen' must be considered an event. This simply means that if you are able to separately assess the possibility of two events E and F happening, then, of course, you must be able to assess the possibility of either one or the other happening. Property c) simply extends this intuitive propery to countable collection of events: the extension is needed for mathematical reasons, to derive certain continuity properties of probability measures.

A more rigorous definition of probability

The definition of probability given above was not entirely rigorous. Now that we have defined sigma-algebras and spaces of events, we can make it completely rigorous. Let Omega be a sample space. Let $ ciFourier $ be a sigma-algebra on Omega (a space of events). A function [eq48] is a probability measure if and only if it satisfies the following two properties:

  1. Sure thing. [eq9].

  2. Sigma-additivity. Let [eq50] be any sequence of elements of $ ciFourier $ such that $i eq j$ implies [eq51]. Then:[eq52]

Nothing new has been added to the definition given above. This definition just clarifies that a probability measure is a function defined on a sigma-algebra of events. Hence, it is not possible to properly speak of probability for subsets of Omega that do not belong to the sigma-algebra.

A triple [eq53] is called a probability space.

Solved exercises

Below you can find some exercises with explained solutions:

  1. Exercise set 1 (sample spaces, probability and events).

by
About | Contacts | Privacy and terms of use | Sitemap