Introduction to Probability reference sheet

This sheet follows the thirteen units of the course in order. Each unit starts with the big idea in a few sentences, then gives the definitions, formulas and rules you will look up most often, with the conditions they need. When you want the full explanation, a proof or more examples, follow the lesson links under each topic.

Events are sets of outcomes in a sample space $\Omega$, with complement $A^{c}$, empty event $\varnothing$, intersection $A \cap B$ and union $A \cup B$. A conditional probability $P(A \mid B)$ is defined only when $P(B) > 0$. Random variables are capital letters such as $X$, and lowercase letters such as $x$ stand for their values. A discrete variable has the PMF $p_X(x) = P(X = x)$, a continuous one has a density $f$, and every random variable has the CDF $F(t) = P(X \le t)$. Expectations are written $E[X]$, together with $\operatorname{Var}(X)$, $\operatorname{SD}(X)$ and $\operatorname{Cov}(X, Y)$, and a formula that contains an expectation assumes it is finite unless it says otherwise. For observations $X_1, \dots, X_n$ the partial sum is $S_n = X_1 + \cdots + X_n$ and the sample mean is $\bar{X}_n = S_n/n$, also written $\bar{X}$.

Distributions are written $\operatorname{Bin}(n, p)$, $\operatorname{Pois}(\lambda)$, $\operatorname{Uniform}(a, b)$, $\operatorname{Exp}(\lambda)$ and $N(\mu, \sigma^2)$. The exponential distribution is written with its rate $\lambda$, so its mean is $1/\lambda$, and the second entry of $N(\mu, \sigma^2)$ is the variance. The standard normal CDF is $\Phi$. Geometric and negative binomial waiting times count the trials up to and including the success they wait for. Markov chains use a row-stochastic transition matrix $P$ and row vectors of state probabilities. Logarithms are natural, and decimals from $\Phi$, $e^{x}$ or long products are rounded.

Probability Foundations

A probability is a number attached to an event under a stated model. The model lists the possible outcomes and assigns probabilities that obey three axioms, and every other rule, such as the complement rule and the addition rule, follows from those axioms.

Models, outcomes and events

An experiment is a procedure together with a rule for what is recorded, and a trial is one run of it. An outcome is one complete recorded result, the sample space $\Omega$ is the set of all outcomes, and an event is a set of outcomes. An event occurs on a trial when the observed outcome belongs to it. In a finite model it is enough to give each outcome $\omega$ a probability $p_\omega \ge 0$ so that these numbers add to $1$, and then every event $A$ has

Three kinds of probability are easy to mix up.

For independent trials in which $A$ has the same probability $p$, for every fixed positive margin the chance that the relative frequency misses $p$ by more than that margin tends to $0$ as $n$ grows. This is the law of large numbers. At a fixed $n$ the relative frequency can still differ from $p$.

Lessons Probability Models and Uncertainty

Describing events

De Morgan’s laws hold for any events,

and for any number of events in the same way. Events are disjoint, or mutually exclusive, when $A \cap B = \varnothing$. Nonempty, pairwise disjoint events whose union is $\Omega$ form a partition, so every outcome lies in exactly one of them. For a count $N$ with integer values and an integer $k$, at least $k$ means $N \ge k$ and includes $k$, and its complement is $N \le k - 1$. One outcome that lies in $A$ but not in $B$ is a counterexample that disproves the claim $A \subseteq B$.

Lessons Sample Spaces and Events, Translating Event Language

The axioms and basic rules

A probability $P$ gives a number to every event and satisfies three axioms.

1. Nonnegativity. Every event has $P(E) \ge 0$.
2. Normalization. The certain event has $P(\Omega) = 1$.
3. Countable additivity. For pairwise disjoint events $E_1, E_2, \dots$ the probability of the union is the sum $P(E_1) + P(E_2) + \cdots$.

The other basic rules follow from these.

The complement rule is the usual way to handle at least one. The complement of at least one of $A_1, \dots, A_n$ is none of them, so

This step needs no independence. Independence enters only when $P(\text{none})$ is written as the product of the factors $1 - P(A_i)$, a step that mutual independence of the events justifies.

Lessons Probability Axioms and Basic Rules, The Complement Rule

The addition rule

For any two events,

The overlap is subtracted once because $P(A) + P(B)$ counts it twice. Two events split $\Omega$ into the four disjoint regions $A \cap B$, $A \cap B^{c}$, $A^{c} \cap B$ and $A^{c} \cap B^{c}$, and every event built from $A$ and $B$ is a union of some of them. Exactly one of the two events occurs with probability $P(A) + P(B) - 2P(A \cap B)$, and neither occurs with probability $1 - P(A \cup B)$. When the overlap is unknown, the union bound still gives

and for two events $\max\bigl(P(A), P(B)\bigr) \le P(A \cup B) \le P(A) + P(B)$, with equality on the right exactly when $P(A \cap B) = 0$. For example, if $P(A) = 0.5$, $P(B) = 0.4$ and $P(A \cap B) = 0.2$, then $P(A \cup B) = 0.7$, exactly one occurs with probability $0.5$ and neither occurs with probability $0.3$.

Lessons The Addition Rule

Equally likely outcomes

If $\Omega$ is finite and nonempty and all of its outcomes are equally likely, then every event $A$ has

where $\lvert A \rvert$ is the number of outcomes in $A$. Equal likelihood is an assumption about the experiment, and it can hold at one level of detail and fail at a coarser one. Count the event and the sample space at the same level, one where the outcomes are equally likely, which is often the level of ordered sequences. Two fair dice rolled independently have $36$ equally likely ordered outcomes, so a total of $7$ has probability $6/36 = 1/6$, while the eleven possible totals are not equally likely. Three tosses of a fair coin have $8$ equally likely sequences, and exactly two heads has probability $3/8$.

Lessons Equally Likely Outcomes

Simulation

A simulation generates outcomes from a model, repeats the experiment $n$ times and reports the relative frequency

where $S_n$ is the number of simulated trials in which the event occurred. A pseudorandom generator is deterministic, so the same algorithm, seed and sequence of calls reproduce the same run, and a different seed usually gives a different estimate. For independent simulated trials the estimate varies around the model probability, and for every fixed positive margin the chance of a miss larger than that margin tends to $0$ as $n$ grows. A simulator built on the wrong model estimates the probability of the event in that wrong model, and running it longer does not remove the difference from the intended probability. How large the sampling error is likely to be is worked out in the unit on repeated sampling.

Lessons Probability Simulation

Counting and Equally Likely Models

When the outcomes are equally likely, a probability is a ratio of two counts. Every counting method starts from what one outcome records, then asks whether order matters, whether items can repeat and which objects can be told apart.

Sum and product rules

• Product rule. Suppose each outcome is built by $k$ successive choices, the choice at stage $i$ has $n_i$ options whatever the earlier choices were, and different sequences of choices give different outcomes. Then there are $n_1 n_2 \cdots n_k$ outcomes. The options themselves may depend on the earlier choices, as long as their number does not.
• Sum rule. If a set splits into pairwise disjoint cases, its size is the sum of the sizes of the cases. For two overlapping sets, $\lvert S_1 \cup S_2 \rvert = \lvert S_1 \rvert + \lvert S_2 \rvert - \lvert S_1 \cap S_2 \rvert$.
• Cases and trees. When the number of options differs from path to path, split into cases at the stage where it differs, count each case and add. In a decision tree each outcome is one leaf.
• Restrictions. Give each stage only the options that satisfy every restriction, and it often helps to fill the most restricted stage first.

A code of $4$ digits has $10^4 = 10000$ possibilities, and $10 \cdot 9 \cdot 8 \cdot 7 = 5040$ of them repeat no digit.

Lessons Sum and Product Rules for Counting

Permutations and combinations

Choose $k$ objects from $n$ distinct objects.

Here $n! = n(n - 1) \cdots 2 \cdot 1$ for $n \ge 1$, and $0! = 1$. There are $n!$ arrangements of $n$ distinct objects in a row, and for $n \ge 1$ there are $(n - 1)!$ arrangements around a circle when rotations count as the same. Without replacement no selection exists when $k > n$. Counting multisets is not the same as finding their probabilities, because unordered records of independent draws are usually not equally likely, as the last topic of this unit shows.

For $0 \le k \le n$ the binomial coefficient $\binom{n}{k}$ counts the subsets of size $k$ of a set of $n$ elements. It satisfies $\binom{n}{k} = \binom{n}{n - k}$, the rows of Pascal’s triangle add up to $\binom{n}{0} + \cdots + \binom{n}{n} = 2^n$, and Pascal’s identity

holds for $1 \le k \le n - 1$. For every integer $n \ge 0$ the binomial theorem states

with $0^0$ read as $1$. From $10$ people, a chair and a secretary can be chosen in $10 \cdot 9 = 90$ ways and a committee of three in $\binom{10}{3} = 120$ ways.

Lessons Permutations, Combinations and Binomial Coefficients

Repeated objects

Suppose $n$ objects come in $k$ types, with $n_i$ objects of type $i$ that cannot be told apart and $n_1 + \cdots + n_k = n$. The number of distinct arrangements in a row is the multinomial coefficient

The same number counts the ways to split $n$ distinct objects into named groups of sizes $n_1, \dots, n_k$, and it is the coefficient of $x_1^{n_1} \cdots x_k^{n_k}$ in $(x_1 + \cdots + x_k)^n$, so these coefficients add up to $k^n$. Choosing the positions of each type in turn gives the same count as the product $\binom{n}{n_1}\binom{n - n_1}{n_2} \cdots$. Whether two objects can be told apart depends on what the outcome records. The letters of LEVEL have $5!/(2!\,2!\,1!) = 30$ arrangements, and those of MISSISSIPPI have $11!/(1!\,4!\,4!\,2!) = 34650$.

Lessons Repeated Objects and Multinomial Counting

Inclusion-exclusion

For three events $A$, $B$ and $C$, write $S_1 = P(A) + P(B) + P(C)$ for the sum of the single probabilities, $S_2$ for the sum of the three pairwise intersection probabilities and $S_3 = P(A \cap B \cap C)$. These three sums give the probabilities in the table.

For $n$ events, with $S_k$ the sum of the probabilities of all intersections of $k$ of them,

The same identities hold for the sizes of finite sets, and none of them needs independence or equally likely outcomes. Among the integers $1$ to $100$, the number divisible by $2$, $3$ or $5$ is $50 + 33 + 20 - 16 - 10 - 6 + 3 = 74$.

Lessons Inclusion-Exclusion

Birthday and collision problems

Suppose each of $n$ individuals receives one of $m$ categories, independently and with probability $1/m$ for each category. A collision means that at least two individuals share a category, and its complement has probability

• With $m = 365$ equally likely birthdays, a shared birthday has probability about $0.476$ for $n = 22$ people and about $0.507$ for $n = 23$.
• A pair named in advance matches with probability $1/m$ whatever $n$ is. A collision is the union of $\binom{n}{2}$ such pair events, which can occur together, so the sum gives only the bound $P(\text{collision}) \le \binom{n}{2}/m$.
• If $n > m$, a collision is certain by the pigeonhole principle, whatever the category probabilities.
• With unequal category probabilities $p_1, \dots, p_m$, two independent individuals match with probability $p_1^2 + \cdots + p_m^2 \ge 1/m$. For any $n$, independent individuals who share these probabilities collide at least as often as under equal probabilities.

Lessons Birthday and Collision Problems

Choosing a counting method

1. Write down what one outcome records, and check that these outcomes are equally likely under the way the selection is actually made.
2. Decide whether order matters, whether repetition is allowed and which objects can be told apart, then use the table of selections above or the multinomial coefficient.
3. Count the event and the sample space at the same level of detail.
4. When an event such as at least one splits into many cases, count its complement instead.

Erasing order can destroy equal likelihood. For two fair dice the unordered record of a $1$ and a $2$ comes from the two ordered outcomes $(1, 2)$ and $(2, 1)$ and has probability $2/36$, while a double $1$ comes only from $(1, 1)$ and has probability $1/36$, so the $21$ unordered records are not equally likely. A record produced by $m$ of the equally likely outcomes of $\Omega$ has probability $m/\lvert \Omega \rvert$.

Lessons Choosing a Counting Method

Conditioning, Independence and Evidence

Conditioning on an event $B$ with $P(B) > 0$ keeps only the outcomes in $B$ and rescales their probabilities to total $1$. Independence is a product rule, not a statement about overlap, and Bayes’ theorem reverses a conditional probability by combining the evidence with the base rate.

Conditional probability

For events $A$ and $B$ with $P(B) > 0$,

If $P(B) = 0$, the ratio is $0/0$ and the conditional probability is undefined, which is different from $0$. With equally likely outcomes, $P(A \mid B) = \lvert A \cap B \rvert / \lvert B \rvert$. For a fixed $B$ the rule $A \mapsto P(A \mid B)$ is itself a probability, so for example $P(A^{c} \mid B) = 1 - P(A \mid B)$. Reversing the condition changes the denominator. When $P(A) > 0$ and $P(B) > 0$, the probabilities $P(A \mid B)$ and $P(B \mid A)$ are equal exactly when $P(A) = P(B)$ or $P(A \cap B) = 0$.

In a two-way table of counts with one member drawn uniformly at random, a cell divided by the grand total is a joint probability and a row or column total divided by the grand total is a marginal probability. A cell divided by its row total is a conditional probability given the row, and a cell divided by its column total is a conditional probability given the column. In the constructed table below, $P(T \cap D) = 18/200 = 0.09$, $P(D \mid T) = 18/30 = 0.6$ and $P(T \mid D) = 18/24 = 0.75$.

Lessons Conditional Probability, Conditional Probability Tables

Independence

Events $A$ and $B$ are independent when

• When $P(B) > 0$ this is equivalent to $P(A \mid B) = P(A)$, and when $P(A) > 0$ to $P(B \mid A) = P(B)$.
• If $A$ and $B$ are independent, so are $A$ and $B^{c}$, $A^{c}$ and $B$, and $A^{c}$ and $B^{c}$.
• An event of probability $0$ or $1$ is independent of every event.
• Events $A_1, \dots, A_n$ are mutually independent when the product rule holds for every subcollection of two or more of them. Pairwise independence, the rule for every two, does not imply mutual independence.
• For mutually independent events, none of them occurs with probability $\prod_i \bigl(1 - P(A_i)\bigr)$, and at least one occurs with probability $1 - \prod_i \bigl(1 - P(A_i)\bigr)$.

Disjoint and independent are different properties.

So disjoint events with positive probabilities are always dependent, since $P(A \cap B) = 0 < P(A)\,P(B)$. Overlapping events can go either way. For one fair die the events $\{2, 4, 6\}$ and $\{1, 2\}$ are independent, because their intersection $\{2\}$ has probability $1/6 = \tfrac12 \cdot \tfrac13$. The union of independent events still needs the overlap term,

Lessons Independent Events, Disjoint Events and Independence

Trees and the chain rule

The multiplication rule $P(A \cap B) = P(A)\,P(B \mid A)$ holds whenever $P(A) > 0$, with or without independence. Repeating it gives the chain rule. If $P(A_1 \cap \cdots \cap A_{n-1}) > 0$, then

In a probability tree the branches leaving a node are the disjoint and exhaustive outcomes of the next stage. Each carries its probability given the history at that node, so the branches from one node add to $1$. The probability of a path is the product of its branch probabilities, and the probability of an event is the sum over the paths that make it up. Two cards drawn without replacement from a standard deck are both aces with probability $\tfrac{4}{52} \cdot \tfrac{3}{51} = \tfrac{1}{221}$.

Lessons Probability Trees and the Chain Rule

Total probability and Bayes’ theorem

Let $B_1, \dots, B_k$ be a partition of $\Omega$ with every $P(B_i) > 0$. For every event $E$,

a weighted average of the rates in the groups, with the group probabilities as weights. If a partition also has groups of probability $0$, they contribute nothing to $P(E)$, so they are left out and the sum runs over the groups of positive probability. Bayes’ theorem reverses the condition. If $P(H) > 0$ and $P(E) > 0$, then

and when also $P(H) < 1$, the partition $H$, $H^{c}$ gives the denominator

For a test, let $F$ be the event that the condition is present and $A$ the event of a positive result.

Sensitivity and specificity condition on the true state, so neither of them is the posterior probability, and judging the posterior without the base rate is base rate neglect. With $P(F) = 0.05$, sensitivity $0.9$ and false positive rate $0.1$, a positive result has probability $0.045 + 0.095 = 0.14$, and

Lessons The Law of Total Probability, Bayes’ Theorem and Base Rates

Conditional independence and odds

For an event $C$ with $P(C) > 0$, events $A$ and $B$ are conditionally independent given $C$ when $P(A \cap B \mid C) = P(A \mid C)\,P(B \mid C)$. Conditional independence and ordinary independence do not imply each other. Two reports that both respond to the same hidden state can be conditionally independent given each value of the state and still be dependent overall. To process evidence one piece at a time, use the current posterior as the next prior and condition each new likelihood on everything already observed. Under conditional independence given each hypothesis, the single-report likelihoods can be reused, while a report that only copies another one adds no evidence.

For $0 \le p < 1$, the odds in favor of an event with probability $p$ are $o = p/(1 - p)$, and conversely $p = o/(1 + o)$. Odds of $a : b$ in favor, with $a, b \ge 0$ and $a + b > 0$, mean $p = a/(a + b)$. For a hypothesis $H$ with $0 < P(H) < 1$ and evidence $E$ with $P(E \mid H^{c}) > 0$, write $O(H) = P(H)/P(H^{c})$ for the prior odds and $O(H \mid E)$ for the posterior odds. The odds form of Bayes’ theorem is

In the test example the prior odds $1/19$ times the likelihood ratio $9$ give posterior odds $9/19$, which is the probability $9/28$ again. For pieces of evidence that are conditionally independent given $H$ and given $H^{c}$, the likelihood ratios multiply.

Lessons Conditional Independence and Sequential Evidence, Probability and Odds

Discrete Random Variables and Expected Value

A random variable assigns a number to each outcome, and its distribution says how probability is spread over those numbers. Expected value is the probability-weighted average of the values, a single summary that need not be a possible value.

Random variables and PMFs

A random variable is a function $X$ from $\Omega$ to the real numbers, and $\{X \in B\}$ is the event of the outcomes at which its value lies in $B$. It is discrete when $P(X \in S) = 1$ for some finite or countably infinite set $S$. Its support is the set of values with positive probability, and its probability mass function is

for every real $x$. A function $p$ is a PMF exactly when $p(x) \ge 0$ for every $x$, finitely or countably many values have $p(x) > 0$, and those masses add to $1$. For a discrete $X$ and every set $B$ of real numbers

where only the support values in $B$ contribute. If the masses are known only up to a factor, $p(x) = c\,g(x)$ on a set $S$ with $g \ge 0$ and a finite positive total, then $c = 1/\sum_{x \in S} g(x)$. For $p(x) = cx$ on $\{1, 2, 3, 4\}$ this gives $c = 1/10$.

Lessons Discrete Random Variables, Probability Mass Functions

Cumulative distribution functions

The CDF of any random variable is $F(t) = P(X \le t)$ for every real $t$. It is nondecreasing and right-continuous, it tends to $0$ as $t \to -\infty$ and to $1$ as $t \to \infty$, and its left limit is $F(t^-) = P(X < t)$. The jump of $F$ at $a$ is $F(a) - F(a^-) = P(X = a)$, so the CDF of a discrete variable jumps exactly at its support values, and with finite support it is a step function. For $a < b$,

For a discrete variable, including or excluding an endpoint matters exactly when the endpoint has positive probability. When $F$ is continuous it never matters.

Lessons Cumulative Distribution Functions

Functions of a random variable

For $Y = g(X)$ with $X$ discrete, each value of $X$ passes its mass to its output, and masses with the same output add,

A map that is one-to-one on the support moves the masses without merging them. When two support values have the same output their masses merge, and when $Y$ takes that output its value does not determine the value of $X$. If $X$ is uniform on $\{-2, -1, 0, 1, 2\}$, then $Y = X^2$ has $p_Y(0) = 1/5$ and $p_Y(1) = p_Y(4) = 2/5$. The positive part $X^{+} = \max(X, 0)$ moves the mass of every negative value to $0$.

Lessons Functions of Discrete Random Variables

Expected value

For a discrete $X$, the expected value, or mean, is

a finite number when $\sum_x \lvert x \rvert\,p_X(x) < \infty$, which always holds for finite support. Each value is weighted by its own probability, and for finite support the mean lies between the smallest and the largest value. A fair die has $E[X] = 3.5$, which is not a possible roll. For a function $g$ with $\sum_x \lvert g(x) \rvert\,p_X(x) < \infty$,

and the value $E[g(X)]$ can differ from $g(E[X])$, as it does for $g(x) = x^2$ and a fair die.

• Infinite mean. A variable with no negative values has either a finite mean or $E[X] = +\infty$. If $P(X = 2^k) = 2^{-k}$ for $k = 1, 2, \dots$, every term of the series equals $1$ and the mean is infinite. This needs infinitely many values.
• Undefined mean. If the positive and the negative values both contribute infinite sums, $E[X]$ is undefined.
• Conditional mean. For an event $A$ with $P(A) > 0$, the mean computed with the conditional masses $P(X = x \mid A)$ is $E[X \mid A]$.
• Mode. A mode is a value with the largest mass. There can be several, and a mode can differ from the mean.

Lessons Expected Value

Expected value and decisions

If an action pays a random amount $G$ and costs a fixed amount $c$, the net payoff $N = G - c$ has $E[N] = E[G] - c$. The fair price, the fee that makes the expected net payoff $0$, is $c^{*} = E[G]$. A ticket that costs $c \ge 0$ and pays $a > 0$ with probability $p$ and nothing otherwise has expected net payoff $ap - c$, which is $0$ at the break-even probability $p^{*} = c/a$, a probability only when $c \le a$. A $2 ticket that pays $10 with probability $0.15$ has expected net payoff $10(0.15) - 2 = -0.5$ dollars, and it would break even at $p^{*} = 0.2$.

Choosing the action with the largest expected payoff, or the smallest expected cost, is one decision criterion among several. Others protect the worst case or limit the probability of exceeding a budget, and different criteria can rank the same actions differently. An expected value describes the model it was computed from, so a sensitivity check under other plausible inputs is one guard against model risk.

Lessons Expected Value and Decisions

Expectation, Spread, Sums and Averages

Expectation is linear whether or not the variables are independent, which turns expected counts into sums of probabilities. Variance measures spread around the mean, the variances of independent variables add, and together these facts give the mean and the standard error of a sample mean.

Linearity and indicators

If $X_1, \dots, X_n$ have finite expectations, $a_1, \dots, a_n$ and $c$ are constants and $n$ is fixed, then

with no independence needed. The product rule $E[XY] = E[X]\,E[Y]$ is a different statement. It holds for independent $X$ and $Y$ with finite expectations, and it can also hold for dependent variables, so it does not show independence.

The indicator $I_A$ of an event $A$ equals $1$ on $A$ and $0$ elsewhere, with $E[I_A] = P(A)$, $I_{A^{c}} = 1 - I_A$ and $I_{A \cap B} = I_A I_B$. If $N$ counts how many of the events $A_1, \dots, A_n$ occur, then $N = I_{A_1} + \cdots + I_{A_n}$ and

even when the events are dependent. If $n$ hats are handed back to their $n$ owners in a uniformly random order, each owner gets their own hat with probability $1/n$, so the expected number of matches is $1$ for every $n$. Six rolls of a fair die show on average $6\bigl(1 - (5/6)^6\bigr) \approx 3.99$ different faces.

Lessons Linearity of Expectation, Indicator Random Variables

Variance and standard deviation

For $X$ with finite mean $\mu$,

The variance is finite exactly when $E[X^2] < \infty$, and then $\operatorname{Var}(X) = E[X^2] - (E[X])^2$. It equals $0$ exactly when $P(X = \mu) = 1$. Variance is in squared units, and the standard deviation is in the units of $X$. For a fair die $E[X^2] = 91/6$, so $\operatorname{Var}(X) = 91/6 - 49/4 = 35/12$ and $\operatorname{SD}(X) \approx 1.71$. For constants $a$ and $b$ and finite variance,

A shift leaves the spread unchanged, and a change of units with factor $a$ multiplies the standard deviation by $\lvert a \rvert$. The mean absolute deviation $E\bigl[\lvert X - \mu \rvert\bigr]$ is another measure of spread, and it never exceeds $\operatorname{SD}(X)$.

Lessons Variance, Standard Deviation and Changing Scale

Sums of random variables

For discrete $X$ and $Y$ on the same sample space, with joint masses $p_{X,Y}(x, y) = P(X = x, Y = y)$, the sum $S = X + Y$ has

In general the separate distributions of $X$ and $Y$ do not determine the distribution of the sum. For independent $X$ and $Y$ the joint masses factor, and the formula becomes the discrete convolution

The mean always adds, $E[X + Y] = E[X] + E[Y]$. With finite variances,

where the covariance, defined in the unit on joint laws, is $0$ for independent variables. So the variances of independent variables add, while their standard deviations do not. The total of two fair dice rolled independently has $P(S = 7) = 6/36$ and $\operatorname{Var}(S) = 35/6$.

Lessons Sums of Discrete Random Variables

Sample means

A random sample of size $n$ is a list $X_1, \dots, X_n$ of independent and identically distributed, or IID, random variables. A statistic is computed from the sample by a fixed rule, and its sampling distribution is its distribution under the model. If the $X_i$ are IID with mean $\mu$ and finite variance $\sigma^2$, then

The mean needs only a common mean. The variance needs uncorrelated observations with a common variance, which IID observations have. The standard deviation of a statistic is called its standard error. A histogram of simulated values of a statistic only approximates its sampling distribution.

Lessons Sample Means and Sampling Distributions

Discrete Probability Distributions

A few families of discrete distributions describe the most common random mechanisms, and each family belongs to a particular experiment. Its formulas apply only when that experiment’s conditions hold, so check the conditions before using a formula.

Families at a glance

In the binomial, geometric and negative binomial rows, the trials are independent and every trial is a success with the same probability $p$.

A discrete uniform variable on a set of $N$ values gives each of them the mass $1/N$, so an event containing $m$ of the values has probability $m/N$. The mean and variance in the table hold for consecutive integers, and for evenly spaced values with gap $d$ the variance is $d^2(N^2 - 1)/12$. A fair die has mean $3.5$ and variance $35/12$. No distribution gives the same positive mass to infinitely many values, because the masses would add to more than $1$.

Lessons The Discrete Uniform Distribution

Bernoulli and binomial distributions

A variable $B \sim \operatorname{Bernoulli}(p)$ with $0 \le p \le 1$ has $P(B = 1) = p$ and $P(B = 0) = 1 - p$. Every indicator $I_A$ is $\operatorname{Bernoulli}\bigl(P(A)\bigr)$. The variance $p(1 - p)$ is at most $1/4$, with equality only at $p = 1/2$, and it is $0$ when $p$ is $0$ or $1$.