Probability Axioms and Basic Rules, introduction to probability lesson

A machine reports exactly one of the four states ready, inspection, adjustment and stopped, and a model gives them the probabilities $0.1$ , $0.2$ , $0.3$ and $0.4$ . Is that a valid probability model, and may we add $0.1$ and $0.2$ to get the probability that the machine is ready or in inspection? The numbers look reasonable, but looking reasonable is not a test. This lesson states the three axioms that every probability assignment satisfies and derives from them how disjoint events add, why the empty event has probability zero and why an event that contains another never has a smaller probability.

Key terms

Probability axioms

Definition A probability model has three parts. The first is a sample space $\Omega$ , the set of all outcomes. The second is a collection of subsets of $\Omega$ called events. It contains $\Omega$ , the complement of each of its members and every countable union of its members. In a finite or countable sample space every subset can be taken as an event. The third part is a function $P$ that assigns a real number $P(E)$ to every event $E$ . The function $P$ is a probability when it satisfies three axioms.

Nonnegativity. Every event $E$ has $P(E) \ge 0$ .
Normalization. The whole sample space has $P(\Omega) = 1$ .
Countable additivity. If $E_1, E_2, E_3, \dots$ is a sequence of pairwise disjoint events, then
$P\Bigl(\bigcup_{k=1}^{\infty} E_k\Bigr) = \sum_{k=1}^{\infty} P(E_k).$

In plain words The axioms are consistency rules for a probability assignment. No event gets a negative probability, the whole sample space carries all of the probability, and an event made of nonoverlapping pieces that can be listed one after another gets the total of their probabilities. The axioms do not say which numbers fit a real machine. Choosing the model takes information about the situation, and the axioms then decide whether the chosen numbers are coherent.

Example For the machine take $\Omega = \{R, I, A, S\}$ , the outcomes ready, inspection, adjustment and stopped, with the outcome masses

\begin{gathered} p_R = 0.1, \quad p_I = 0.2, \\[0.4em] p_A = 0.3, \quad p_S = 0.4. \end{gathered}

Define $P(E)$ as the total mass of the outcomes in $E$ , for each of the $16$ subsets of $\Omega$ . Every such total is nonnegative, the total for $\Omega$ is $0.1 + 0.2 + 0.3 + 0.4 = 1$ , and the total mass of a union of disjoint events is the sum of their separate totals. So all three axioms hold. The event “ready or inspection” is $\{R, I\}$ , with probability $0.1 + 0.2 = 0.3$ .

Nonnegativity and normalization

Definition Nonnegativity requires $P(E) \ge 0$ for every event $E$ , and the value $0$ is allowed. Normalization requires $P(\Omega) = 1$ , where $\Omega$ is the certain event, the event that contains every outcome. Suppose $\Omega$ is finite or countable and every subset is an event, and write $p_\omega = P(\{\omega\})$ for the mass of outcome $\omega$ . A probability on such a space has

\begin{gathered} p_\omega \ge 0 \ \text{ for every } \omega, \\[0.4em] \sum_{\omega \in \Omega} p_\omega = 1, \end{gathered}

and every event has $P(E) = \sum_{\omega \in E} p_\omega$ . Conversely, any masses with these two properties define a probability through that sum.

In plain words Zero counts as nonnegative but not as positive, so an outcome can have probability $0$ and still be an outcome listed in the model. Together with additivity, normalization says that the probabilities of a complete finite or countable list of mutually exclusive outcomes add to exactly one, the whole allocation.

Example A model that lists three mutually exclusive outcomes covering every case cannot give them the probabilities $0.5$ , $0.4$ and $0.3$ , because the total is $1.2$ . Declaring one outcome rare does not fix the total, and dividing every value by $1.2$ is justified only when a modeling argument calls for it. Either some of the numbers are wrong, or the listed outcomes are not really mutually exclusive and count some cases twice, and finding the error comes before any repair.

Pairwise disjoint events

Definition A finite or infinite list of events $E_1, E_2, \dots$ is pairwise disjoint, or mutually exclusive, if $E_i \cap E_j = \varnothing$ whenever $i \ne j$ . Pairwise disjoint nonempty events whose union is $\Omega$ form a partition of $\Omega$ .

In plain words No outcome belongs to two different events of the list. It is not enough that no outcome belongs to all of them at once, because every pair has to be checked.

Example In the machine model the events $\{R, I\}$ , $\{A\}$ and $\{S\}$ are pairwise disjoint, and since their union is $\Omega$ they also form a partition. The events $\{R, I\}$ , $\{I, A\}$ and $\{A, R\}$ have no outcome common to all three, yet each pair shares an outcome, so they are not pairwise disjoint. Their probabilities add to $0.3 + 0.5 + 0.4 = 1.2$ , which counts every outcome of their union $\{R, I, A\}$ twice. The union itself has probability $0.6$ .

Countable additivity

Definition For every sequence $E_1, E_2, E_3, \dots$ of pairwise disjoint events,

P\Bigl(\bigcup_{k=1}^{\infty} E_k\Bigr) = \sum_{k=1}^{\infty} P(E_k),

where the infinite sum means the limit of the partial sums $\sum_{k=1}^{n} P(E_k)$ as $n \to \infty$ . The axiom also covers finite lists. If $E_1, \dots, E_n$ are pairwise disjoint, extend the list with $E_k = \varnothing$ for every $k > n$ . The longer list is still pairwise disjoint and has the same union, and each added term is $0$ because $P(\varnothing) = 0$ , as shown below. This gives finite additivity,

P\Bigl(\bigcup_{k=1}^{n} E_k\Bigr) = \sum_{k=1}^{n} P(E_k).

For two disjoint events $E$ and $F$ it reads $P(E \cup F) = P(E) + P(F)$ . The axiom makes no claim about uncountable collections of events.

In plain words When an event splits into finitely many nonoverlapping pieces, or into infinitely many that can be listed as first, second, third and so on, its probability is the total of the pieces. The rule does not reach collections too large to list in a sequence. A model that picks a point uniformly from the interval $[0, 1]$ , built in later lessons, gives every single point probability $0$ and the whole interval probability $1$ , although the interval is the union of its points.

Example Let $\Omega = \{1, 2, 3, \dots\}$ with mass $2^{-k}$ on outcome $k$ . The partial sums are

\sum_{k=1}^{n} 2^{-k} = 1 - 2^{-n},

so the mass still missing halves at every step. No finite partial sum reaches $1$ , but their limit is $1$ , so normalization holds. The event that the outcome is even is the disjoint union of the one-outcome events $\{2\}$ , $\{4\}$ , $\{6\}$ and so on. Countable additivity and the sum of a geometric series with first term and ratio $\tfrac14$ give

\begin{aligned} P(\text{even}) &= \sum_{j=1}^{\infty} 2^{-2j} \\[0.4em] &= \frac{1/4}{1 - 1/4} = \frac13. \end{aligned}

Probability of the empty event

Definition Every probability model has $P(\varnothing) = 0$ . To prove it, apply countable additivity to the sequence $\varnothing, \varnothing, \varnothing, \dots$ , which is pairwise disjoint with union $\varnothing$ . This gives

P(\varnothing) = \sum_{k=1}^{\infty} P(\varnothing).

The same nonnegative number added infinitely often has a finite total only when that number is $0$ , so $P(\varnothing) = 0$ . The converse is false. An event can have probability $0$ without being empty, and an event can have probability $1$ without being all of $\Omega$ . An event with probability $1$ is said to occur almost surely.

In plain words The event that contains no outcome gets no probability. Probability zero is a statement about mass, not about membership, so a nonempty event can carry zero mass. In the continuous models with a density in later lessons, each single exact value has probability $0$ even though it is an outcome of the model.

Example Change the machine model to $p_R = 0$ , $p_I = 0.2$ , $p_A = 0.3$ and $p_S = 0.5$ . The masses are nonnegative and total $1$ , so the model is valid. The event $\{R\}$ is not empty, since it contains the outcome ready, yet $P(\{R\}) = 0$ . The event $\{I, A, S\}$ has probability $0.2 + 0.3 + 0.5 = 1$ but is not the whole sample space. Whether a zero mass describes an excluded operating mode or an idealization is a question about the machine, not about the axioms.

Monotonicity

Definition If $E$ and $F$ are events with $E \subseteq F$ , then $P(E) \le P(F)$ . For the proof, split $F$ into the disjoint events $E$ and $F \setminus E$ . Additivity and nonnegativity give

\begin{aligned} P(F) &= P(E) + P(F \setminus E) \\[0.4em] &\ge P(E). \end{aligned}

Taking $F = \Omega$ gives $P(E) \le 1$ , so every event satisfies $0 \le P(E) \le 1$ .

In plain words If every outcome of $E$ also lies in $F$ , then $F$ has at least the probability of $E$ , because $F$ is $E$ plus an extra piece whose probability is not negative. The rule compares nested events. Two events where neither contains the other can have their probabilities in either order. A strict containment does not force a strict inequality, since the extra piece may have probability $0$ .

Example In the original machine model $\{R, I\} \subseteq \{R, I, A\}$ , and indeed $0.3 \le 0.6$ . The difference $0.3$ is the probability of the extra piece $\{A\}$ . In the changed model with $p_R = 0$ , the event $\{I, A, S\}$ is strictly smaller than $\Omega$ and still has the same probability $1$ .

Check yourself

A model has three mutually exclusive outcomes $L$ , $M$ and $H$ for low, medium and high that cover every case, with masses $p_L = 0.2$ and $p_M = 0.5$ . Find $p_H$ and the probability of low or high. Then explain why a claim that medium has probability $1.1$ cannot hold in any model.

Show the answer

Normalization requires $p_L + p_M + p_H = 1$ , so $p_H = 1 - 0.2 - 0.5 = 0.3$ . This value is nonnegative, so the completed model passes both checks. The events $\{L\}$ and $\{H\}$ are disjoint, so additivity gives $P(\{L, H\}) = 0.2 + 0.3 = 0.5$ . A probability of $1.1$ is impossible because $\{M\} \subseteq \Omega$ , and monotonicity gives $P(\{M\}) \le P(\Omega) = 1$ .

Each list gives probabilities to three mutually exclusive outcomes that cover every case. List (a) is $0.5$ , $0.5$ , $0$ , list (b) is $0.7$ , $0.4$ , $-0.1$ and list (c) is $0.35$ , $0.35$ , $0.35$ . Which lists define a valid probability model?

Show the answer

List (a) is valid. Its values are nonnegative and add to $1$ . The third outcome has probability $0$ , which nonnegativity allows, and it remains an outcome of the model.

List (b) is not valid. Its values add to $1$ , but $-0.1$ breaks nonnegativity, so a correct total is not enough.

List (c) is not valid. Its values are nonnegative but add to $1.05$ , which breaks normalization. Dividing by $1.05$ would produce numbers that pass both checks, but only a reason from the situation can justify that particular repair.

In the model on $\{1, 2, 3, \dots\}$ with mass $2^{-k}$ on outcome $k$ , find the probability that the outcome is a multiple of $3$ and the probability that the outcome is at least $4$ .

Show the answer

Let $M$ be the event that the outcome is a multiple of $3$ . It is the union of the disjoint one-outcome events $\{3\}$ , $\{6\}$ , $\{9\}$ and so on, so countable additivity gives a geometric series with first term and ratio $\tfrac18$ ,

\begin{aligned} P(M) &= \sum_{j=1}^{\infty} 2^{-3j} \\[0.4em] &= \frac{1/8}{1 - 1/8} = \frac17. \end{aligned}

For the second event, the one-outcome events $\{1\}$ , $\{2\}$ , $\{3\}$ and the event $\{4, 5, 6, \dots\}$ are pairwise disjoint and their union is $\Omega$ . Write $q$ for the probability of $\{4, 5, 6, \dots\}$ . Additivity and normalization give

\frac12 + \frac14 + \frac18 + q = 1,

so the outcome is at least $4$ with probability $q = \tfrac18$ . Summing the masses directly agrees, since $\sum_{k=4}^{\infty} 2^{-k} = \tfrac{1/16}{1 - 1/2} = \tfrac18$ .

Probability Axioms and Basic Rules

Key terms

Probability axioms

Nonnegativity and normalization

Pairwise disjoint events

Countable additivity

Probability of the empty event

Monotonicity

Check yourself

How this lesson connects

Builds on

Used later in