Unit 12 of 13

Random Processes and Introductory Markov Chains

  • 9 lessons
  • Videos 74 to 82

This unit follows random quantities through time in Bernoulli processes, random walks, Poisson processes and Markov chains, including stationary distributions and hitting times.

A Bernoulli process gives binomial counts and geometric waiting times, and a simple random walk adds independent steps, so its position is tied to a binomial count. A Poisson process has independent and stationary increments, Poisson counts and exponential gaps between arrivals. A time-homogeneous Markov chain moves between states by one transition matrix, whose powers give multi-step probabilities. Stationary distributions, convergence in a two-state example, absorption probabilities from first-step analysis and checks of the Markov assumption complete the unit.

Every lesson will be one video. The videos are coming soon, and each lesson page already has the definitions and an example for every key term.

Bernoulli Processes and Random Walks

  1. Bernoulli Processes

    How a Bernoulli process orders independent success or failure trials by a time index, why its cumulative count is binomial and its first success time geometric, and how count and waiting time events match.

    Video 74

  2. Simple Random Walks

    How a simple random walk adds independent unit steps, why its position is tied to a binomial count with a parity restriction, how drift sets its mean while its variance grows, and how path, endpoint and visit events differ.

    Video 75

Poisson Processes

  1. Poisson Process Basics

    How a homogeneous Poisson process counts arrivals in continuous time, why rate times interval length is the expected count, how stationary and independent increments differ, and why overlapping counts are dependent.

    Video 76

  2. Arrival Counts and Waiting Times

    How the first arrival of a Poisson process gives an exponential waiting time, how interarrival times add up to the kth arrival time, how waiting events become count events, and why the wait is memoryless.

    Video 77

Markov Chains

  1. Markov Chains and Transition Matrices

    What states and the Markov property are, what makes a chain time-homogeneous, how a row-stochastic transition matrix and an initial distribution describe a Markov chain, and how the one-step update advances its distribution.

    Video 78

  2. Multi-Step Markov Probabilities

    How n-step transition probabilities of a Markov chain add path products, why they are entries of a matrix power, how the Chapman-Kolmogorov relation splits a path, and how ending in a state differs from visiting it.

    Video 79

  3. Stationary Distributions

    What a stationary distribution of a Markov chain is, how the balance equations and probability flow find one, and why convergence and uniqueness need separate checks, shown by a periodic chain and a nonunique case.

    Video 80

Hitting Times and Model Checking

  1. Hitting Times and Absorption

    How hitting times and hitting probabilities of a Markov chain are defined, how first-step analysis with boundary conditions gives absorption probabilities and expected times, and why a hitting time can be infinite.

    Video 81

  2. Choosing and Checking Markov Models

    How to choose the states and time step of a Markov model, refine a state that hides history, estimate an empirical transition matrix from counts, check the Markov and time homogeneity assumptions, and state model limitations.

    Video 82