Stationary Distributions

Definition Let $X_0, X_1, X_2, \dots$ be a time-homogeneous Markov chain on a finite state space $S$ with transition matrix $P$, so that $P(X_{n+1} = j \mid X_n = i) = P_{ij}$ whenever $P(X_n = i) > 0$. The entries of $P$ are nonnegative and its rows each sum to $1$. Write the distribution of $X_n$ as a row vector $\pi_n$ with entries $\pi_n(j) = P(X_n = j)$. By the law of total probability $\pi_{n+1} = \pi_n P$, so $\pi_n = \pi_0 P^n$. A row vector $\pi$ is a stationary distribution of the chain if

If $\pi_0 = \pi$, then $\pi_1 = \pi P = \pi$, and by induction $\pi_n = \pi$ for every $n \ge 0$.

Definition Written out entry by entry, the equation $\pi P = \pi$ is the system of balance equations, one for each state $j$,

The right side is the probability of being in $j$ one step after a start from $\pi$. The equations are linear and homogeneous, so the zero vector and every multiple of a solution solve them as well. A stationary distribution is a solution that also has nonnegative entries and satisfies the normalization $\sum_j \pi(j) = 1$. Because every row of $P$ sums to $1$, the sum of all the balance equations reduces to $\sum_j \pi(j) = \sum_j \pi(j)$. So any one of the equations follows from the others, and the normalization takes its place.

Definition When the chain has distribution $\pi_n$ at time $n$, the probability flow from state $i$ to a different state $j$ is $\pi_n(i) P_{ij}$. It equals $P(X_n = i, X_{n+1} = j)$, the probability that the step from time $n$ to time $n + 1$ goes from $i$ to $j$. Subtracting $\pi(j) P_{jj}$ from both sides of the balance equation for $j$ and using $\sum_k P_{jk} = 1$ gives the equivalent form

So a probability vector is stationary exactly when, at every state, the total flow out to other states equals the total flow in from other states. This is a balance of totals. The stronger pairwise condition $\pi(i) P_{ij} = \pi(j) P_{ji}$ for every pair of states, called detailed balance, also makes $\pi$ stationary, but a stationary distribution need not satisfy it.

Definition The state distribution converges to a distribution $\pi$ if $\pi_n(j) \to \pi(j)$ as $n \to \infty$ for every state $j$. Whether this happens, and what the limit is, can depend on the initial distribution $\pi_0$, so a convergence statement has to say which starting distributions it covers. On a finite state space every such limit is stationary, because letting $n \to \infty$ in $\pi_{n+1} = \pi_n P$ gives $\pi = \pi P$. A stationary distribution, on the other hand, need not be the limit from other starts. For the two-state matrix above and every initial distribution, the probability $a_n = P(X_n = A)$ satisfies

so $a_n \to 0.6$ and $\pi_n \to (0.6, 0.4)$ from every start.

Definition Suppose the chain can return to state $i$, meaning $(P^n)_{ii} > 0$ for some $n \ge 1$. The period of $i$ is the greatest common divisor of all such $n$. The state is periodic if its period is at least $2$ and aperiodic if its period is $1$. A chain whose states all have the same period $d \ge 2$ is called periodic, and in it a return to any state can happen only after a multiple of $d$ steps. Started from a single state, such a chain can cycle through its states in a fixed rhythm, so its state distribution need not converge, even when the state space is finite and a stationary distribution exists.

Definition A chain has a nonunique stationary distribution when the balance equations together with nonnegativity and normalization have more than one solution. Every chain with finitely many states has at least one stationary distribution. If $\pi$ and $\pi'$ are two different ones, then every mixture $t\pi + (1 - t)\pi'$ with $0 \le t \le 1$ is stationary as well, so there are infinitely many. For a two-state chain the stationary distribution is unique unless $P$ is the identity matrix, as the general two-state solution above shows.