Reinforcement learning (RL) is the third major machine learning paradigm — alongside supervised and unsupervised learning. Instead of learning from labelled examples, an RL agent learns by interacting with an environment, receiving a reward signal, and adjusting its behaviour to maximise cumulative reward over time.

The learning signal is delayed and sparse. A chess program only receives +1 (win) or −1 (loss) at the very end of the game. It must figure out which of the 40–80 moves it played actually caused that outcome — the credit assignment problem, one of the central challenges in RL.

Three key differences from supervised learning: (1) there is no teacher providing correct action labels; (2) actions affect future states, so decisions are not independent; and (3) training data is generated by the agent's own behaviour — a non-stationary, self-influencing distribution.

At every discrete time step t, the agent-environment interaction follows a four-step cycle: (1) agent observes state s_t; (2) selects action a_t via policy π; (3) environment transitions to s_t+1; (4) environment emits reward r_t+1. This repeats until a terminal state (episodic) or runs indefinitely (continuing).

The most fundamental tension in RL is exploration vs exploitation. Exploitation means choosing the action believed best right now. Exploration means trying something new to discover potentially better strategies. Every RL algorithm must balance these two competing imperatives.

Every RL problem is formalised as a Markov Decision Process — defined by the 5-tuple (S, A, P, R, γ): the state space S, action space A, transition probability P(s'|s,a), reward function R(s,a), and discount factor γ ∈ [0,1].

The cornerstone assumption is the Markov Property: the future depends only on the current state, not on the full history. Formally: P(s_t+1 | s_t, a_t, …, s0, a0) = P(s_t+1 | s_t, a_t). The current state encodes all information needed to act optimally.

When the agent cannot observe the full state, the problem becomes a Partially Observable MDP (POMDP) — the agent must maintain a belief state, a probability distribution over possible states, greatly increasing difficulty.

The agent's goal is not to maximise the next reward, but the cumulative discounted reward over the entire episode. The return G_t from time step t is the sum of all future rewards, each discounted by γᵏ where k is how many steps away it lies.

The discount factor γ ∈ [0, 1] controls how much the agent values future rewards. With γ = 0 the agent is purely myopic. With γ = 1 all future rewards are equally valued (only valid for finite-horizon tasks). Typical values are γ = 0.95–0.99. The recursive form G_t = r_t+1 + γ·G_t+1 is the key identity that makes Bellman equations tractable.

A policy π maps states to actions. A deterministic policy π(s) = a always picks the same action; a stochastic policy π(a|s) outputs a probability distribution over actions. The goal: find the optimal policy π* that maximises expected return.

The state value function V^π(s) answers: "How good is state s under policy π?" The action-value function Q^π(s,a) is more granular: "How good is taking action a in state s, then following π?" Q-values are the direct targets of Q-learning and DQN. The advantage A^π(s,a) = Q − V measures how much better action a is compared to the average action from s — used in actor-critic methods to reduce variance.

Richard Bellman's 1957 insight is the mathematical spine of all RL. The Bellman Expectation Equation expresses V^π(s) recursively: the value of state s equals the immediate reward plus the discounted value of the next state, averaged over all possible next states. This converts an infinite-horizon problem into a tractable recursive computation.

The Bellman Optimality Equation does the same for the optimal value function V*: the optimal value is obtained by always choosing the best possible action. Every major RL algorithm — Q-learning, DQN, PPO, SAC — is explicitly or implicitly solving a form of this equation.

RL algorithms split along two major axes: (1) whether they use an explicit model of transition dynamics, and (2) whether they optimise a value function, directly a policy, or both via an actor-critic hybrid.

Dynamic Programming (DP) is a family of algorithms that solve MDPs exactly — given perfect knowledge of the environment model: every transition probability P(s'|s,a) and every reward R(s,a). Because real environments rarely hand us this model, DP is not deployed in production RL — but it is the indispensable theoretical bedrock every modern algorithm is built on.

Two main algorithms arise from DP: Policy Iteration (evaluate then improve a fixed policy in alternating steps) and Value Iteration (combine both steps into a single max-Bellman update). Both converge; they differ in computational trade-offs.

Given a fixed policy π, Policy Evaluation computes V^π(s) for every state by repeatedly applying the Bellman expectation equation until the values stop changing (convergence threshold θ).

Once we have V^π for a fixed policy π, the Policy Improvement Theorem guarantees we can construct a strictly better (or equal) policy by acting greedily with respect to V^π.

Policy Iteration alternates between Policy Evaluation and Policy Improvement until the policy stabilises. Despite each evaluation requiring many inner iterations, the outer loop converges in very few steps — typically 2–10 even for large MDPs.

// Policy Iteration
Initialise π(s) = arbitrary policy for all s ∈ S
Initialise V(s) = 0 for all s ∈ S

LOOP (outer — policy improvement):
  // ── Step 1: Policy Evaluation (inner loop) ──
  LOOP:
    Δ = 0
    FOR each state s ∈ S:
      v = V(s)
      V(s) ← Σ_a π(a|s) Σ_s' P(s'|s,a) [R(s,a,s') + γ·V(s')]
      Δ = max(Δ, |v - V(s)|)
    IF Δ < θ: BREAK   // inner convergence

  // ── Step 2: Policy Improvement ──
  policy_stable = TRUE
  FOR each state s ∈ S:
    old_action = π(s)
    π(s) ← argmax_a Σ_s' P(s'|s,a) [R(s,a,s') + γ·V(s')]
    IF old_action ≠ π(s): policy_stable = FALSE

  IF policy_stable: RETURN π, V   // converged to π*, V*

Value Iteration eliminates the inner evaluation loop by fusing policy evaluation and improvement into a single Bellman optimality update. Instead of summing over π(a|s), it takes the max over actions — implicitly acting greedily at every step.

// Value Iteration
Initialise V(s) = 0 for all s ∈ S
Set θ = 1e-6 (convergence threshold), γ ∈ [0,1)

LOOP:
  Δ = 0
  FOR each state s ∈ S:
    v = V(s)
    // Bellman optimality operator — single sweep, no inner loop
    V(s) ← max_a [ Σ_s' P(s'|s,a) * (R(s,a,s') + γ * V(s')) ]
    Δ = max(Δ, |v - V(s)|)
  IF Δ < θ: BREAK   // values converged

// Extract greedy policy from V*
FOR each state s ∈ S:
  π*(s) ← argmax_a [ Σ_s' P(s'|s,a) * (R(s,a,s') + γ * V(s')) ]

RETURN π*, V*

DP is mathematically elegant but practically limited. Three hard walls prevent it from scaling to real-world problems — and each wall motivated a different branch of modern RL.

Despite its impracticality, DP is the theoretical target all RL algorithms approximate. Monte Carlo methods replace full sweeps with sampled episodes. TD learning (Chapter 7.3) bootstraps from partial trajectories. Deep RL (Chapter 7.5+) replaces the value table with a neural network — enabling Atari, Go, and robotic control at scale.

Monte Carlo (MC) methods learn directly from complete episodes of experience — no model of P(s'|s,a) needed. The idea is disarmingly simple: visit state s many times, record the actual return G_t each time, and average them. Like estimating the probability of heads by flipping a coin many times and averaging the results.

Two variants: First-visit MC averages returns from only the first visit to s per episode; Every-visit MC averages returns from every visit. Both converge to V^π with enough data.

The prediction problem: estimate V^π(s) for a given policy π. Algorithm: run many episodes following π, compute actual returns at each visit to s, average them. Classic example — Blackjack: state is (player sum, dealer card, usable ace?); policy is hit if sum < 20 else stick. After 10,000 episodes a clear value landscape emerges.

Control means finding the optimal policy, not just evaluating a given one. MC Control applies the GPI (Generalised Policy Iteration) loop using MC evaluation — but requires special care around exploration.

TD learning is the central idea of modern RL — it combines the best of Monte Carlo and Dynamic Programming. Like MC: learns from raw experience, no model needed. Like DP: bootstraps — updates V(s) using the current estimate of V(s'), not the full actual return. The payoff: update after every single step, not after the whole episode.

TD(0) is the simplest TD algorithm — 1-step bootstrapping. It updates V(s_t) immediately after observing the next reward and state, using only one step of lookahead.

// TD(0) Policy Evaluation
Initialise V(s) = 0 for all s ∈ S
Set α = 0.1, γ = 0.9

FOR each episode:
  Initialise state s
  LOOP (each step t until terminal):
    a  ← π(a|s)                    // follow policy
    r, s' ← env.step(a)            // observe reward and next state

    δ = r + γ * V(s') - V(s)       // TD error
    V(s) ← V(s) + α * δ            // online update — no need to wait!

    s ← s'
  END LOOP
END FOR

// Worked trace (chain: A → B → C → GOAL, γ=0.9, α=0.1):
// t=0: s=A, r=0, s'=B  → δ=0+0.9×0−0=0    → V(A)=0
// t=1: s=B, r=0, s'=C  → δ=0+0.9×0−0=0    → V(B)=0
// t=2: s=C, r=+1, s'=∅ → δ=1+0.9×0−0=+1   → V(C)=0.1
// (next episode, s=C again):
//   δ = 1+0−0.1 = +0.9  → V(C)=0.1+0.1×0.9=0.19
// After many episodes:  V(C)→1.0, V(B)→0.9, V(A)→0.81

TD(0) only updates the immediately preceding state; MC updates all states in an episode. TD(λ) interpolates between them via a parameter λ ∈ [0,1]: at λ=0 it is pure TD(0); at λ=1 it is equivalent to MC. The mechanism is the eligibility trace eₜ(s) — a running score of how recently and frequently each state was visited.

Watkins (1989) introduced Q-Learning — still the most widely taught RL control algorithm. It learns the optimal action-value function Q*(s,a) directly, regardless of what policy is being followed during training. This makes it off-policy: the agent can explore freely, and the update still converges to the optimal values.

// Q-Learning Algorithm
Initialise Q(s,a) = 0 for all s ∈ S, a ∈ A
Set α=0.1, γ=0.99, ε=1.0

FOR each episode:
  s = env.reset()
  LOOP until terminal:
    // ε-greedy action selection
    IF random() < ε:  a = random action       // explore
    ELSE:              a = argmax_a Q(s,a)     // exploit

    r, s' = env.step(a)

    // Off-policy update: always uses max over next actions
    best_next = max over a' of Q(s', a')
    Q(s,a) += α * (r + γ * best_next - Q(s,a))

    s = s'

  ε = max(ε_min, ε * ε_decay)    // decay exploration rate

In tabular RL the Q-function is stored as a 2-D table: rows are states, columns are actions, cells hold Q(s,a) estimates. Each transition updates one cell. Below: a step-by-step trace through a 4-state chain MDP, and a full Gymnasium implementation.

import gymnasium as gym
import numpy as np

env = gym.make('FrozenLake-v1', is_slippery=False)
n_states  = env.observation_space.n   # 16 states (4×4 grid)
n_actions = env.action_space.n        # 4 actions

Q = np.zeros((n_states, n_actions))

alpha, gamma  = 0.1, 0.99
epsilon       = 1.0
eps_min       = 0.01
eps_decay     = 0.995
n_episodes    = 5000

for ep in range(n_episodes):
    state, _ = env.reset()

    for _ in range(200):
        # ε-greedy action
        if np.random.random() < epsilon:
            action = env.action_space.sample()
        else:
            action = np.argmax(Q[state])

        next_state, reward, done, truncated, _ = env.step(action)

        # Q-Learning update
        td_target        = reward + gamma * np.max(Q[next_state]) * (not done)
        Q[state, action] += alpha * (td_target - Q[state, action])

        state = next_state
        if done or truncated: break

    epsilon = max(eps_min, epsilon * eps_decay)

print("Learned Q-table:\n", Q.reshape(4, 4, 4).round(2))

The exploration-exploitation dilemma is central to all RL: to learn good values you must explore, but to perform well you must exploit. ε-greedy is the simplest solution — take a random action with probability ε, else take the best known action. ε is annealed from 1.0 (pure explore) down to a small floor (mostly exploit).

SARSA (Rummery & Niranjan, 1994) is Q-learning's on-policy cousin. The name comes from the 5-tuple it uses per update: (S_t, A_t, R_t+1, S_t+1, A_t+1). The key difference: the update uses the actual next action taken — not the greedy max.

// SARSA Algorithm
FOR each episode:
  s = env.reset()
  a = ε_greedy(Q, s)        // select FIRST action

  LOOP until terminal:
    r, s' = env.step(a)
    a' = ε_greedy(Q, s')    // select NEXT action BEFORE updating

    // SARSA: uses actual (s, a, r, s', a') — not max
    Q(s,a) += α * (r + γ * Q(s',a') - Q(s,a))

    s = s'
    a = a'                  // carry forward the chosen action

The on/off-policy distinction is subtle but has a concrete consequence: near danger. Q-learning sees the world through the lens of optimal behaviour and ignores its own exploration noise. SARSA accounts for the fact that it sometimes takes random actions — and learns a more cautious policy as a result.

The Cliff Walking benchmark (Sutton & Barto) makes the on/off-policy difference vivid. A 4×12 grid: start bottom-left, goal bottom-right. The entire bottom row between them is cliff — R=−100 and agent resets. Normal steps cost R=−1.

Tabular Q-learning stores one number per (state, action) pair. That works fine for small, discrete environments — but consider Atari Pong: 160×210 pixels, 128 colours per pixel → ~10^568,000 possible states. A Q-table is not just impractical, it is cosmologically impossible. Continuous control (robot joints, autonomous driving) has an infinite state space.

The solution: approximate Q(s,a) ≈ Q(s,a;θ) using a parameterised function (neural network with weights θ). The network generalises — similar inputs produce similar outputs — so states never seen before can still receive reasonable Q-value estimates. A Deep Q-Network (DQN) is a CNN mapping raw pixels directly to Q-values.

The DQN input is 4 stacked 84×84 grayscale frames — the stack gives the network a sense of motion (velocity of the ball, direction of movement) that a single frame cannot convey. Three convolutional layers extract spatial features; two FC layers decode those features into one Q-value per action.

import torch
import torch.nn as nn

class DQN(nn.Module):
    """DQN as in the original Atari paper (Mnih et al. 2015)"""
    def __init__(self, n_actions: int):
        super().__init__()
        self.conv = nn.Sequential(
            nn.Conv2d(4, 32, kernel_size=8, stride=4), nn.ReLU(),  # 4 stacked frames
            nn.Conv2d(32, 64, kernel_size=4, stride=2), nn.ReLU(),
            nn.Conv2d(64, 64, kernel_size=3, stride=1), nn.ReLU(),
        )
        self.fc = nn.Sequential(
            nn.Flatten(),
            nn.Linear(64 * 7 * 7, 512), nn.ReLU(),
            nn.Linear(512, n_actions)   # one output per action
        )

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # x: (batch, 4, 84, 84) — normalise pixels to [0, 1]
        return self.fc(self.conv(x / 255.0))

net = DQN(n_actions=6)
x = torch.randint(0, 255, (32, 4, 84, 84), dtype=torch.float32)
q_values     = net(x)                      # (32, 6) — Q-value per action
best_actions = q_values.argmax(dim=1)      # (32,)  — greedy action
print(f"Output shape: {q_values.shape}")   # torch.Size([32, 6])

Naively applying gradient descent to TD errors with a neural network diverges. DQN introduces two stabilisation innovations that together tame the instability and make deep RL practical.

The full DQN training loop combines all the pieces into a clean 7-step cycle that repeats millions of times until convergence.

Van Hasselt et al. (2016) identified that standard DQN systematically overestimates Q-values. The culprit: maxa' Q(s',a';θ⁻) always selects the highest value — if any Q-value is overestimated (inevitable early in training), it picks that inflated value, and the bias propagates. Fix: decouple action selection from action evaluation.

Wang et al. (2016) observed that in many states the choice of action barely matters — an empty screen in Pong, a static field in Breakout. Only near the ball does the action have a significant effect. The Dueling DQN architecture exploits this by splitting Q(s,a) into two components: V(s) (how good is this state, regardless of action) and A(s,a) (how much better is this specific action than average).

Hessel et al. (DeepMind, 2018) asked: what if we combine all the improvements to DQN into one agent? The result — Rainbow — obliterated all prior benchmarks on Atari while using far fewer environment interactions.

Q-learning and DQN are powerful — but they require enumerating a Q-value for every possible action. That works for discrete actions (6 Atari buttons) but breaks immediately for continuous action spaces: robot joint torques, steering angles, portfolio weights. How do you take argmax over an infinite set? You can't.

Policy gradient methods sidestep this by directly parameterising the policy π(a|s;θ) as a neural network and performing gradient ascent on expected return J(θ). They also naturally produce stochastic policies — essential in adversarial games where determinism is exploitable (rock-paper-scissors, poker).

Williams (1992) proved that the gradient of expected return with respect to policy parameters has a remarkably clean form — computable purely from sampled trajectories, without knowledge of the environment model.

REINFORCE is the simplest instantiation of the policy gradient theorem — Monte Carlo style: collect a full episode, compute discounted returns G_t at every step, then update the policy. Unbiased but high-variance.

import torch, torch.nn as nn
import gymnasium as gym

class PolicyNetwork(nn.Module):
    def __init__(self, obs_dim, action_dim):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(obs_dim, 128), nn.ReLU(),
            nn.Linear(128, 64),      nn.ReLU(),
            nn.Linear(64, action_dim)   # raw logits
        )
    def forward(self, x): return self.net(x)

env      = gym.make('CartPole-v1')
policy   = PolicyNetwork(4, 2)          # 4 obs dims, 2 actions
optim    = torch.optim.Adam(policy.parameters(), lr=3e-4)
gamma    = 0.99

for episode in range(1000):
    state, _   = env.reset()
    log_probs, rewards = [], []

    while True:                                         # collect full episode
        state_t = torch.tensor(state, dtype=torch.float32)
        logits  = policy(state_t)
        dist    = torch.distributions.Categorical(logits=logits)
        action  = dist.sample()                         # stochastic action
        log_probs.append(dist.log_prob(action))         # log π(a|s;θ)
        state, reward, done, trunc, _ = env.step(action.item())
        rewards.append(reward)
        if done or trunc: break

    # Compute discounted returns G (backwards accumulation)
    G, returns = 0, []
    for r in reversed(rewards):
        G = r + gamma * G
        returns.insert(0, G)
    returns = torch.tensor(returns)
    returns = (returns - returns.mean()) / (returns.std() + 1e-8)  # normalise

    # Policy gradient loss — negative for gradient ASCENT
    loss = -sum(lp * g for lp, g in zip(log_probs, returns))
    optim.zero_grad(); loss.backward(); optim.step()

REINFORCE works in theory but is slow in practice because G_t estimates from single trajectories have extremely high variance. The fix: subtract a baseline b(s_t) from the return. The baseline doesn't change the expected gradient — it just centres it, dramatically reducing variance.

Using V(s) as a baseline requires learning V(s). The natural solution: train a separate Critic network alongside the policy (Actor). The Critic computes the TD error δ_t — a single-step advantage estimate — and feeds it back to guide the Actor's gradient updates. This enables online, per-step learning without waiting for an episode to end.

Mnih et al. (DeepMind, 2016) scaled Actor-Critic to deep networks with a key insight: parallelism solves the correlation problem. Instead of a replay buffer, run many independent workers simultaneously — each in its own environment copy. Their diverse experiences are naturally uncorrelated.

A policy trained purely to maximise return tends to collapse to deterministic — it finds one good action per state and stops exploring everything else. This is catastrophic in environments where exploration is critical or where the optimal policy is genuinely stochastic.

The fix: add an entropy bonus H(π) to the objective. Entropy measures how spread-out a probability distribution is — maximising it encourages the policy to remain uncertain and keep exploring. The coefficient β controls the exploration-exploitation trade-off. Entropy regularisation is used in A3C, PPO, and is a cornerstone of SAC (Soft Actor-Critic), where maximising entropy is part of the fundamental objective.

Schulman et al. (OpenAI/Berkeley, 2015) identified the root instability in policy gradient methods: step size. Too large a step and the new policy is so different that the old value estimates are invalid — training collapses catastrophically and cannot recover. Too small and learning is painfully slow. This is fundamentally different from supervised learning where labels are fixed — in RL, a bad update corrupts the very data distribution used for future learning.

TRPO's solution: constrain each update so the new policy stays within a trust region — KL(π_old ‖ π_new) ≤ δ. The surrogate objective is a valid lower bound on true performance inside this region, guaranteeing monotonic improvement. The catch: enforcing this hard constraint requires second-order optimisation (Fisher information matrix, conjugate gradients) at O(|θ|²) cost — infeasible for large networks.

Schulman et al. (OpenAI, 2017) asked: can we get TRPO's stability with standard first-order optimisation? The answer is PPO — the most widely used deep RL algorithm in 2024. It powers ChatGPT/Claude/Gemini alignment (RLHF), OpenAI Five, AlphaStar, Boston Dynamics locomotion, and continuous control across robotics.

PPO replaces TRPO's hard KL constraint with a clipped surrogate objective. Define the probability ratio r_t(θ) = π_new(a_t|s_t) / π_old(a_t|s_t). Instead of constraining this ratio via Lagrange multipliers, simply clip it to [1−ε, 1+ε] and take the minimum of clipped and unclipped — creating a flat region where the gradient is zero, which naturally prevents too-large updates.

// PPO Algorithm (pseudocode)
Initialise policy π_θ, value function V_φ
Set ε=0.2, γ=0.99, λ=0.95, K=10, T=2048, B=64, c1=0.5, c2=0.01

LOOP:
  // Phase 1: Collect rollout with current policy
  FOR t = 0 to T-1 (across N envs):
    a_t  ~ π_θ(·|s_t)
    s_t1, r_t1 = env.step(a_t)
    Store (s_t, a_t, r_t1, s_t1, log π_θ(a_t|s_t), V_φ(s_t))

  // Phase 2: Compute GAE advantages
  FOR each step t (backwards):
    delta = r_t1 + γ·V_φ(s_t1) - V_φ(s_t)
    A_hat[t] = delta + (γ·λ)·A_hat[t+1]      // GAE
  A_hat = (A_hat - mean(A_hat)) / std(A_hat)  // normalise

  // Phase 3: K epochs of mini-batch updates
  θ_old ← θ                                    // freeze reference policy
  FOR epoch = 1 to K:
    FOR each mini-batch of B samples:
      r_t  = π_θ(a_t|s_t) / π_old(a_t|s_t)   // probability ratio
      L_CLIP = mean(min(r_t*A_hat, clip(r_t,1-ε,1+ε)*A_hat))
      L_VF   = mean((V_φ(s_t) - V_target)²)
      L_H    = mean(H(π_θ(·|s_t)))
      loss   = -(L_CLIP - c1*L_VF + c2*L_H)
      backprop(loss); clip_grad_norm_(0.5); optimizer.step()

from stable_baselines3 import PPO
from stable_baselines3.common.env_util import make_vec_env
from stable_baselines3.common.callbacks import EvalCallback
import gymnasium as gym

vec_env = make_vec_env("LunarLander-v2", n_envs=8)

model = PPO(
    policy="MlpPolicy", env=vec_env,
    clip_range=0.2,          # ε — trust region boundary
    n_steps=2048,            # T steps per env per rollout
    batch_size=64,           # mini-batch size B
    n_epochs=10,             # K update epochs per rollout
    gamma=0.99,              # discount factor
    gae_lambda=0.95,         # GAE λ
    ent_coef=0.01,           # entropy coefficient c2
    vf_coef=0.5,             # value loss coefficient c1
    max_grad_norm=0.5,       # gradient clipping
    learning_rate=3e-4,
    verbose=1, tensorboard_log="./ppo_tb/"
)

eval_cb = EvalCallback(gym.make("LunarLander-v2"),
    best_model_save_path="./ppo_best/", eval_freq=10_000, n_eval_episodes=10)
model.learn(total_timesteps=1_000_000, callback=eval_cb)

Schulman et al. (2016) formalised a key trade-off: the advantage estimate A_t used in policy gradient updates can be computed at different "depths" — ranging from a pure TD(0) estimate (low variance, high bias) to a full Monte Carlo return (zero bias, high variance). GAE smoothly interpolates between these extremes with parameter λ, and λ=0.95 has proven the sweet spot across nearly every task.

Haarnoja et al. (Berkeley + Google, 2018) introduced SAC — the dominant algorithm for continuous control. While PPO is on-policy (needs fresh data every update), SAC is off-policy — it reuses all past experience from a replay buffer. Its fundamental innovation is maximum entropy RL: the agent is rewarded not just for getting high return, but for maintaining a stochastic (uncertain) policy.

from stable_baselines3 import SAC
import gymnasium as gym

env = gym.make("HalfCheetah-v4")

model = SAC(
    policy="MlpPolicy", env=env,
    learning_rate=3e-4,
    buffer_size=1_000_000,     # replay buffer capacity
    batch_size=256,            # mini-batch size
    tau=0.005,                 # soft target network update τ
    gamma=0.99,
    ent_coef="auto",           # auto-tune α
    target_entropy="auto",     # H̄ = -|A|
    learning_starts=10_000,    # fill buffer before training
    train_freq=1,              # update every step
    gradient_steps=1,
    verbose=1
)
model.learn(total_timesteps=1_000_000)
model.save("sac_halfcheetah")

Fujimoto et al. (2018) dissected three specific failure modes of DDPG (Deep Deterministic Policy Gradient) and engineered a targeted fix for each. TD3 is a lean, deterministic off-policy algorithm that is an important baseline for continuous control benchmarks.

Model-free RL needs millions of real environment interactions. Atari DQN requires 50M frames — equivalent to 38 days of continuous play. For real robots or industrial systems, that is prohibitively expensive and dangerous. Model-based RL addresses this by learning a world model M_ψ(s,a) → (s', r) — a differentiable simulator — and generating cheap synthetic experience to train the policy.

Sutton (1991) introduced the Dyna architecture — deceptively simple yet powerful. Every real environment step generates one real transition and K=100 model-generated transitions. The policy receives 101× more gradient updates per real step at negligible extra compute cost (the model is a neural network, not the real environment).

Ha & Schmidhuber (2018) showed that compressing observations into a compact latent space and predicting the future in that latent space enables policies trained purely in imagination. DreamerV3 (Hafner et al., 2023) is the apex of this line: a single model with one set of hyperparameters that masters Atari, DMControl, Crafter, BSuite, and Minecraft diamond collection (4/7 tasks — the first RL agent to achieve this).

The core is the RSSM (Recurrent State Space Model) — a latent dynamics model with deterministic memory (GRU recurrent path) and stochastic uncertainty (discrete latent variables). The actor-critic trains entirely inside the RSSM by rolling out K=15 imagined steps from the current latent state — never touching the real environment during policy updates.

Schrittwieser et al. (DeepMind, 2020) built on AlphaZero (which required the full rules of Go/Chess to run MCTS) and asked: what if we learn the rules from scratch? MuZero learns three functions: a Representation network (observation → latent state h), a Dynamics network ((h, a) → next latent h' + predicted reward r), and a Prediction network (h → policy π + value v). MCTS runs entirely in the learned latent space — never invoking the real environment during planning.

The result: state of the art simultaneously on Atari (57 games), Go, Chess, and Shogi — all with one algorithm, no domain knowledge, no hand-coded rules. The dynamics model need not predict realistic pixels or observations — it only needs to produce accurate value and reward estimates for planning purposes.

MuZero Reanalyse (2021) further improves sample efficiency by replaying stored positions with the latest network, re-running MCTS to generate improved training targets. The successor EfficientZero (2021) achieves human-level Atari performance in just 2 hours of real game time — a 20× improvement over MuZero's original sample efficiency.

Chess has ~10⁴⁷ positions — hard but solvable for minimax search engines like Stockfish. Go has ~2×10¹⁷⁰ — minimax is computationally infeasible, and classic evaluation functions can't reliably assess Go positions. Go requires intuition more than calculation. That intuition was beyond computers — until March 2016.

AlphaZero required explicit game rules for MCTS expansion — it couldn't play Atari because the rules are embedded in emulator code. MuZero (DeepMind, 2020) solved this: it learns its own transition model from experience and runs MCTS entirely in the learned latent space, never calling the real environment during planning.

Ouyang et al. (OpenAI, 2022) scaled RLHF to GPT-3 (175B) and produced InstructGPT — the model that became ChatGPT. Three stages, each with distinct datasets, objectives, and failure modes. Understanding each stage explains why alignment is hard.

Reward Hacking is the central failure mode of RLHF. PPO maximises the reward model score — but the RM is an imperfect proxy for human preferences. The model discovers RM weaknesses: verbosity (RM rewards long responses → pad with filler), sycophancy (RM rewards agreement → tell users what they want to hear), formatting exploitation (RM likes bullet points → overuse bullets everywhere). Goodhart's Law: when a measure becomes a target, it ceases to be a good measure.

RLHF is powerful but complex: three stages, a separate reward model, PPO instability, and massive compute cost. Two newer approaches simplify or improve it significantly.

Robotics exposes every limitation of RL simultaneously: expensive data, dangerous exploration, partial observability, sim-to-real transfer. Yet recent systems have overcome these barriers at scale — from dexterous hands to general-purpose manipulation.

Standard RL maximises expected cumulative reward — no constraint on how. In simulation, failure is cheap (just reset). In the real world, failure can be catastrophic: a self-driving car exploring randomly could kill someone; a medical AI exploring randomly could harm a patient. Safe RL adds explicit constraints to the optimisation.