Unit 13 of 15

Reinforcement Learning and Sequential Decisions

  • 3 lessons
  • Videos 81 to 83

In reinforcement learning the data depend on the actions taken. This unit separates exploration in bandit problems from sequential decisions with states and computes value updates by hand.

A multi-armed bandit chooses again and again among actions with unknown average rewards. Epsilon-greedy and upper confidence bound rules balance exploration against exploitation, and regret measures what the choices cost. A Markov decision process adds states that actions can change, and the Bellman equations relate the value of a state to the rewards and values that follow. Temporal-difference learning and Q-learning update value estimates in small tables from single transitions. Deep reinforcement learning and control theory are left for further study.

Every lesson will be one video. The videos are coming soon, and each lesson page already has the definitions and an example for every key term.

Exploration in Bandit Problems

  1. Multi-Armed Bandits and Exploration

    How a bandit learner estimates action values from the rewards it observes, how greedy, epsilon-greedy and upper confidence bound rules trade exploration against exploitation, and how regret measures the cost.

    Video 81

Markov Decision Processes and Value Learning

  1. Markov Decision Processes and Bellman Equations

    How a Markov decision process specifies states, actions, transition probabilities and rewards, how a policy and the discounted return define a value function, and what the Bellman expectation and optimality equations say.

    Video 82

  2. Temporal-Difference Learning and Q-Learning

    How temporal-difference learning updates a value table from single transitions by bootstrapping on the TD error, how Q-learning and on-policy learning differ in their targets, and how terminal states end a return.

    Video 83