On-Policy vs Off-Policy Learning: The Most Misunderstood Distinction in Reinforcement Learning

Almost everyone learns this distinction the same way: "On-policy learns from its own data. Off-policy learns from someone else's."

That sentence isn't wrong. But it's a slogan, not a definition — and slogans collapse the moment you ask a real question. Is DQN off-policy because it has a replay buffer? No. Is PPO genuinely on-policy? Not quite. Is DPO reinforcement learning, off-policy, or offline? Three separate questions, three different answers. And here's the one keeping LLM infrastructure teams awake since 2025: your training run can be secretly off-policy even when your algorithm, your config file, and your intentions all say on-policy.

We're going to build this properly. And before comparing the algorithms, we'll understand the two engines underneath all of them — temporal-difference learning and policy gradients — because once you see where the TD error comes from and why the policy gradient works, on-policy vs off-policy stops being a fact you memorize and becomes something you can derive.

Prerequisites: what a Markov Decision Process is, roughly what a policy and a value function are, basic probability. Python/NumPy for the code.

How this article handles the mathematics

I've read enough tutorials that open with a wall of Greek letters to know how that goes. So here's the deal.

This article explains the machinery in words. Where a formula matters, I'll write it as a sentence with the parts labeled — because the parts are what you need, and the notation is just shorthand for them.

There are exactly four symbolic equations in the whole piece. Four. Each one earns its place by carrying an argument no sentence can, and I'll flag each as it arrives so you know it's coming.

That's a deliberate choice, and it costs something: you won't find the derivations here. If you want to see the algebra worked out line by line, the "Going Deeper" section at the end names the exact chapters where each one lives. My aim is that you arrive at those pages already knowing what the equations are for — which, in my experience, is the difference between reading a derivation and following one.

Part 1: There Are Always Two Policies

Imagine a driving instructor in the passenger seat. The student is steering — hesitant, over-braking, drifting toward the curb. That's the driving that actually happens. Meanwhile the instructor is building a model of how driving should go, and correcting toward that ideal.

Two different things, same car, same moment. In RL they get names:

Behavior policy (written b) — the policy that actually generates the data. Whatever your agent really does. The source of experience. The student.

Target policy (written π) — the policy you're trying to learn and improve. What you want to end up with. The instructor's ideal.

One piece of notation, because it appears everywhere: π(a|s) reads as "the probability that policy π picks action a, given we're in state s." The vertical bar means "given." The important part isn't the symbols — it's that a policy is a distribution, not a lookup table. It might put 70% of its weight on left and 30% on right. That matters because distributions can be compared by dividing one by the other, and that division is going to turn out to be the whole story.

Now the definition:

On-policy: b = π_. The policy you're learning is exactly the one that produced the data._

Off-policy: b ≠ π_. You learn one policy from data generated by a different one._

That's it. No replay buffers, no sample efficiency, no "old data" — those are all consequences. To classify any algorithm, ask one question:

At the moment I compute this gradient, is the distribution that generated my data the same as the distribution I'm optimizing?

Why we'd ever want two policies

Here's the problem off-policy learning exists to solve.

You want the best policy. The best policy always picks the highest-value action — it's greedy. But you can't learn by following a greedy policy: how would you know the value of an action you've never tried? And you'd never try it, because your current estimate says something else is better. The estimate justifies the ignorance that produced it.

This is the exploration–exploitation deadlock, and there are exactly two ways out.

Way one — compromise (on-policy). Accept you'll never be greedy. Learn a policy that keeps some randomness forever — ε-greedy: take the best-known action most of the time, take a random one with probability ε. What you learn isn't optimal, but it's the best policy among those that keep exploring. Honest: you learn the thing you actually do.

Way two — separate the roles (off-policy). Let one policy explore while you learn about a different, greedy one. Now you can learn the optimal policy without paying the price of following it.

Route two unlocks something bigger. If the identity of the data-generating policy doesn't matter, your data can come from anywhere: human demonstrations, a checkpoint from three weeks ago, a random policy, a log file. Half of modern RL is built on that.

Part 2: How TD Learning Actually Works

This is the engine behind SARSA, Q-learning, DQN, and the critic inside PPO and GRPO. Worth doing properly.

The thing we're estimating

An agent wants to know "how good is it to be here?" — which means the total reward it expects from this point onward, with distant rewards counted a little less than immediate ones. That discounting isn't arbitrary; it does two jobs at once. It encodes real impatience (a reward now beats the same reward in a thousand steps), and it keeps the mathematics finite (without it, in a task that never ends, every state is worth infinity and nothing is comparable).

Call that expected total the value of a state. The whole problem becomes: how do you estimate an average when you can only observe one sample at a time?

The obvious method, and why it hurts

Play out the entire episode, compute what you actually got, average over many episodes. That's Monte Carlo, and it's obviously correct.

It also has two problems.

You have to wait for the end. In chess you learn nothing about move 3 until the game is over. In a task with no end, you wait forever.

The variance is enormous. The total return is a sum of hundreds of random rewards from hundreds of random transitions. One episode might be wildly unrepresentative.

A metaphor I find clarifying: Monte Carlo is a student who only ever learns from final exam grades. It's an unbiased signal — the exam does reflect their understanding — but it's rare, delayed, and noisy (they had a headache; the questions favored one chapter). They'd learn far faster from weekly homework, even though each homework is a less complete measure.

That weekly-homework alternative is TD learning.

The recursive insight

Here's the move that makes everything possible, and it's an observation about arithmetic rather than a theorem.

Take the total future reward and pull out the first term. What's left is the same kind of quantity, one step later. Which means:

The return from here equals the reward you get right now, plus the discounted return from wherever you land next.

Read that twice. We started with something defined over an infinite future, requiring us to look to the end of time. We've replaced it with a local relationship between two adjacent states. The infinite horizon is gone; we only ever need to look one step ahead.

Bootstrapping: using a guess in place of the future

The problem: that relationship needs the value of the next state — which is exactly what we're trying to learn.

So we do something that feels illegal the first time. We use our current estimate of the next state anyway. The learning target becomes:

target  =  the reward we actually observed  +  our own current guess about where we landed
              ↑                                    ↑
        real information from the world        made up (for now)

target  =  the reward we actually observed  +  our own current guess about where we landed
              ↑                                    ↑
        real information from the world        made up (for now)

This is bootstrapping — updating a guess using another guess. It's genuinely unsettling. If all our estimates are wrong, doesn't updating one wrong number with another produce garbage?

Here's why it doesn't, and it's the most common conceptual gap in TD learning:

The target isn't purely a guess. It contains one piece of real information — the observed reward — that came from the actual world._ Every update injects one grain of truth. That truth then propagates backward: the state next to the reward learns first, then the state before it learns from_ that_, and so on. Truth diffuses backward through the chain of guesses, one link per pass._

Remember the rumor-chain problem: a hundred people, each distorting the message by 10%, and the output is noise. But now put one person who can see the actual event at the end of the line, whispering the truth backward every day. Eventually the whole line converges — not because the whisperers improved, but because a real signal keeps entering and errors have nowhere to hide.

The update rule, in words

Now we have two estimates of the same quantity, made one step apart: what we believed before, and what we believe now that we've seen a real reward. The second is better — it has more real data in it. The gap between them is the TD error, usually written δ:

surprise (δ)  =  ( reward received  +  value of where I landed )  −  ( value I predicted )
                  └──────────── what I believe now ───────────┘      └── what I believed ──┘

surprise (δ)  =  ( reward received  +  value of where I landed )  −  ( value I predicted )
                  └──────────── what I believe now ───────────┘      └── what I believed ──┘

And the learning rule is just:

new estimate  =  old estimate  +  α × surprise

new estimate  =  old estimate  +  α × surprise

where α (the learning rate, between 0 and 1) says how much you trust this one new piece of evidence.

Three things worth noticing about this shape, because they're all load-bearing:

It's a running average in disguise. If you've ever computed an average incrementally — new average = old average plus a fraction of the gap to the new sample — this is the identical structure. RL just uses a fixed fraction instead of a shrinking one, and that's deliberate: the thing you're averaging is non-stationary, because your policy keeps improving. A fixed α means old information decays away instead of accumulating. You're choosing to forget, because the past describes a policy you no longer have.

It's a thermostat. The correction shrinks automatically as your guess approaches the target — when surprise goes to zero, the update goes to zero. Nothing has to tell it to stop.

It's learning from surprise, not from reward. Worth pausing on. There's a well-known line of neuroscience research (Schultz, Dayan & Montague, 1997) showing that dopamine neurons in primates fire in a pattern remarkably close to this: not for reward itself, but for reward that was unexpected — and they go quiet when an expected reward fails to arrive. Whether or not brains literally compute δ, the correspondence has shaped both fields. Learning from surprise may be a deep principle rather than an algorithmic convenience.

Watching truth propagate backward

Let's make the diffusion claim concrete, because it's the part people accept on faith and shouldn't have to.

Five states in a line: A → B → C → D → E → done. Every reward is zero except the last step, which gives +1. So the true value of every state is exactly 1.0 — you always eventually reach the reward. Start all estimates at zero.

TD learning — information seeps one step backward per episode
  episode 1:  A=0.000  B=0.000  C=0.000  D=0.000  E=0.500
  episode 2:  A=0.000  B=0.000  C=0.000  D=0.250  E=0.750
  episode 3:  A=0.000  B=0.000  C=0.125  D=0.500  E=0.875
  episode 4:  A=0.000  B=0.062  C=0.312  D=0.688  E=0.938
  episode 5:  A=0.031  B=0.188  C=0.500  D=0.812  E=0.969

TD learning — information seeps one step backward per episode
  episode 1:  A=0.000  B=0.000  C=0.000  D=0.000  E=0.500
  episode 2:  A=0.000  B=0.000  C=0.000  D=0.250  E=0.750
  episode 3:  A=0.000  B=0.000  C=0.125  D=0.500  E=0.875
  episode 4:  A=0.000  B=0.062  C=0.312  D=0.688  E=0.938
  episode 5:  A=0.031  B=0.188  C=0.500  D=0.812  E=0.969

Look at the diagonal. That's the wave. After episode 1 only E knows anything — it's the only state that touched real reward. After episode 2, D has learned from E. After episode 3, C learns from D. Knowledge marches backward at exactly one state per episode.

Nobody told A anything about the reward. A found out because it kept asking B, and B eventually had something worth saying.

Monte Carlo on the same problem behaves completely differently — it waits for the episode to end, then tells every state the full truth at once (all five sit at 0.500 after episode one, 0.750 after two, and so on). In this deterministic toy example MC actually converges faster, which is the right moment to state the trade-off honestly:

Monte Carlo_ is unbiased but high-variance, and can't start until the episode ends. TD is biased early (your estimates are wrong at first) but low-variance, and learns from every single step._

_The example above is deterministic, which erases MC's variance problem and flatters it. Add noise to the rewards and MC's estimates thrash while TD stays steady. _Almost all real environments are noisy, which is why TD dominates in practice.

Same trade-off as the exams: unbiased-but-rare versus biased-but-frequent. In a noisy world, frequent and slightly wrong beats rare and technically correct.

One last step: from states to actions

Knowing how good a state is doesn't tell you what to do. So we track the value of state-action pairs instead — written Q(s,a), read as "how much total reward do I expect if I take action a here." Everything above carries over unchanged.

Which leaves exactly one open question, and it's the question this entire article is about:

What do we put in the learning target?

Because that single choice decides whether an algorithm is on-policy or off-policy.

Part 3: SARSA vs Q-Learning — A One-Term Difference

Here are the two answers, and this is the first of the article's four symbolic equations — it earns its place because the difference is a single term and you need to see them side by side to believe it.

SARSA        Q ← Q + α [ R + γ·Q(next state, the action I will ACTUALLY take) − Q ]
Q-learning   Q ← Q + α [ R + γ·max Q(next state, the BEST available action)  − Q ]
                                 └──────────── the only difference ────────────┘

SARSA        Q ← Q + α [ R + γ·Q(next state, the action I will ACTUALLY take) − Q ]
Q-learning   Q ← Q + α [ R + γ·max Q(next state, the BEST available action)  − Q ]
                                 └──────────── the only difference ────────────┘

SARSA uses the action the agent is actually going to take. Your ε-greedy agent will sometimes do something foolish; SARSA counts that in. Asked "what happens after this state?", it answers: "whatever you're really going to do." The name spells it out — State, Action, Reward, State, Action — and that fifth element is the real next move.

Q-learning takes the max. It doesn't care what the agent does next. It asks: "what if you played optimally from here?" So while ε-greedy generates the data, Q-learning learns about the greedy policy. b ≠ π. Off-policy learning, smuggled in by a single max operator.

Two navigation apps. The first knows you're a nervous driver who occasionally misses turns, and routes you accordingly — wide roads, forgiving junctions. The second assumes you drive flawlessly and sends you down the fastest route, which includes a hairpin with no guardrail. The second app computed the objectively optimal route. It's also going to get you killed.

Cliff Walking

Sutton and Barto's textbook has the perfect demonstration.

A 4×12 grid. Start bottom-left, goal bottom-right. The bottom row between them is a cliff: step in, take −100, get teleported back to start. Every other step costs −1, so dawdling is punished.

The optimal path is obvious: hug the cliff edge, shortest route. But for an ε-greedy agent it's terrifying, because 10% of the time it moves randomly — and one of those random moves is down.

Real output:

--- What SARSA learns ---           --- What Q-learning learns ---
.***********                        ............
**.........*                        ............
*..........*                        ************
SXXXXXXXXXXG                        SXXXXXXXXXXG
Return of the greedy policy:  −17   Return of the greedy policy:  −13
Average return while training: −23  Average return while training: −61

--- What SARSA learns ---           --- What Q-learning learns ---
.***********                        ............
**.........*                        ............
*..........*                        ************
SXXXXXXXXXXG                        SXXXXXXXXXXG
Return of the greedy policy:  −17   Return of the greedy policy:  −13
Average return while training: −23  Average return while training: −61

Q-learning hugs the cliff and finds the genuinely optimal policy: −13, the best achievable here. SARSA takes the scenic route and settles for −17.

Now the second line. During training, behaving ε-greedily, Q-learning averages −61. SARSA averages −23. Q-learning spends its entire training life falling off a cliff.

The key insight, and it's what most treatments skip:_ SARSA is not weaker and not timid. SARSA gives the_ correct answer to a different question. Q-learning answers "what is the optimal policy?" SARSA answers "given that I will keep exploring, how should I behave?" If your agent really will act ε-greedily, SARSA's answer is the more useful one.

Off-policy teaches you the ideal. On-policy teaches you the best version of what you're actually going to do. When exploration is cheap — a simulator you reset a million times — the ideal is what you want. When it's expensive — a robot that breaks, a user who churns, an account that loses real money — the second question pays the rent.

The crossover, measured

Sweeping ε across both algorithms, 8 seeds each:

eps | SARSA train    QL train | SARSA greedy   QL greedy
   0.5 |      -135.8      -712.8 |       -131.4       -13.0
   0.3 |       -59.5      -219.9 |       -108.5       -13.0
   0.2 |       -44.5      -108.8 |       -131.4       -13.0
   0.1 |       -29.3       -51.6 |        -39.9       -13.0
  0.05 |       -20.8       -29.7 |        -17.0       -13.0
  0.01 |       -17.7       -14.9 |        -17.0       -13.0

eps | SARSA train    QL train | SARSA greedy   QL greedy
   0.5 |      -135.8      -712.8 |       -131.4       -13.0
   0.3 |       -59.5      -219.9 |       -108.5       -13.0
   0.2 |       -44.5      -108.8 |       -131.4       -13.0
   0.1 |       -29.3       -51.6 |        -39.9       -13.0
  0.05 |       -20.8       -29.7 |        -17.0       -13.0
  0.01 |       -17.7       -14.9 |        -17.0       -13.0

Three things, each teaching something:

The rightmost column never moves. Q-learning recovers the optimal policy at every exploration level, from wildly random to nearly greedy. That's the off-policy superpower in one column: the behavior policy can be almost anything and the target still converges to optimal.

The crossover is real, around ε ≈ 0.02. Above it SARSA earns more during training; below it Q-learning wins. That's b converging toward π — as ε → 0, ε-greedy becomes greedy, the two policies merge, and the gap disappears. The distinction dissolves exactly where theory says it should.

SARSA's greedy column is erratic — −131 at ε = 0.5, −17 at ε = 0.05. Not a bug. SARSA isn't estimating the optimal value function; it's estimating the value function of the ε-greedy policy. Those are answers to a different question, so extracting a greedy policy from them isn't guaranteed to give anything sensible when ε is large. Off-policy methods hand you a clean optimal value function. On-policy methods hand you one that's only meaningful in the context of the behavior it describes.

The code

python

import numpy as np
ROWS, COLS = 4, 12
START, GOAL = (3, 0), (3, 11)
ACTIONS = [(-1,0), (1,0), (0,-1), (0,1)]   # up, down, left, right
def step(s, a):
    r, c = s
    dr, dc = ACTIONS[a]
    nr, nc = min(max(r+dr, 0), ROWS-1), min(max(c+dc, 0), COLS-1)
    if nr == 3 and 1 <= nc <= 10:            # fell off the cliff
        return START, -100.0, False
    return (nr, nc), -1.0, (nr, nc) == GOAL
def eps_greedy(Q, s, eps, rng):
    if rng.random() < eps:
        return rng.integers(4)
    q = Q[s[0], s[1]]
    # Break ties randomly. If you don't, argmax always picks the same
    # direction and exploration gets systematically crippled — a bug
    # that costs people days.
    return int(rng.choice(np.flatnonzero(q == q.max())))
def train(algo, episodes=2000, alpha=0.5, eps=0.1, gamma=1.0, seed=0):
    rng = np.random.default_rng(seed)
    Q = np.zeros((ROWS, COLS, 4))
    returns = []
    for _ in range(episodes):
        s = START
        a = eps_greedy(Q, s, eps, rng)        # behavior policy b
        G = 0.0
        for _ in range(500):
            s2, r, done = step(s, a)
            G += r
            a2 = eps_greedy(Q, s2, eps, rng)  # the action we will REALLY take
            if algo == 'sarsa':
                # Target policy == behavior policy.
                target = r + gamma * Q[s2[0], s2[1], a2] * (not done)
            else:
                # Target policy == greedy policy. We ignore a2 entirely.
                # This one line is what makes the algorithm off-policy.
                target = r + gamma * Q[s2[0], s2[1]].max() * (not done)
            Q[s[0], s[1], a] += alpha * (target - Q[s[0], s[1], a])
            s, a = s2, a2
            if done:
                break
        returns.append(G)
    return Q, np.array(returns)

import numpy as np
ROWS, COLS = 4, 12
START, GOAL = (3, 0), (3, 11)
ACTIONS = [(-1,0), (1,0), (0,-1), (0,1)]   # up, down, left, right
def step(s, a):
    r, c = s
    dr, dc = ACTIONS[a]
    nr, nc = min(max(r+dr, 0), ROWS-1), min(max(c+dc, 0), COLS-1)
    if nr == 3 and 1 <= nc <= 10:            # fell off the cliff
        return START, -100.0, False
    return (nr, nc), -1.0, (nr, nc) == GOAL
def eps_greedy(Q, s, eps, rng):
    if rng.random() < eps:
        return rng.integers(4)
    q = Q[s[0], s[1]]
    # Break ties randomly. If you don't, argmax always picks the same
    # direction and exploration gets systematically crippled — a bug
    # that costs people days.
    return int(rng.choice(np.flatnonzero(q == q.max())))
def train(algo, episodes=2000, alpha=0.5, eps=0.1, gamma=1.0, seed=0):
    rng = np.random.default_rng(seed)
    Q = np.zeros((ROWS, COLS, 4))
    returns = []
    for _ in range(episodes):
        s = START
        a = eps_greedy(Q, s, eps, rng)        # behavior policy b
        G = 0.0
        for _ in range(500):
            s2, r, done = step(s, a)
            G += r
            a2 = eps_greedy(Q, s2, eps, rng)  # the action we will REALLY take
            if algo == 'sarsa':
                # Target policy == behavior policy.
                target = r + gamma * Q[s2[0], s2[1], a2] * (not done)
            else:
                # Target policy == greedy policy. We ignore a2 entirely.
                # This one line is what makes the algorithm off-policy.
                target = r + gamma * Q[s2[0], s2[1]].max() * (not done)
            Q[s[0], s[1], a] += alpha * (target - Q[s[0], s[1], a])
            s, a = s2, a2
            if done:
                break
        returns.append(G)
    return Q, np.array(returns)

Two lines of difference. One conceptual universe apart.

Part 4: How Policy Gradients Actually Work

TD learning is one engine. Here's the other — the one powering every modern LLM post-training method.

Why not just use values?

We just built a machine for estimating how good each action is. Why not stop there and always pick the best one?

Three reasons, each fatal in its own domain.

Continuous actions. "Take the best action" requires searching over every action. If your action is "torque on seven joints," that's a seven-dimensional optimization inside every single decision. Not feasible.

Sometimes randomness is optimal. In rock-paper-scissors the best policy is uniformly random. A greedy value-based policy is deterministic by construction and will get destroyed. Whenever your state is partially observed, this generalizes.

Indirection. Value methods optimize a proxy — accurate values — and hope good behavior falls out. But a small error in the values can flip which action looks best and change the policy completely. Policy gradients optimize the thing you actually care about, directly.

So: make the policy itself a network with parameters, and improve those parameters by gradient ascent on expected reward.

The hard part

Here's why this is structurally harder than supervised learning, and it's worth being precise.

In supervised learning, your dataset is fixed. The parameters only affect the function, not which data you see. Sample a batch, compute gradients, done.

In RL, the parameters appear inside the distribution itself. Change the policy and you change which trajectories get sampled. The "dataset" rewrites itself every time you take a step.

Concretely: the quantity you need to differentiate is a sum over every possible trajectory — an astronomically large set — and it's a sum of gradients weighted by probabilities, not a weighted average of anything you can draw samples of. There's nothing to sample.

We need to turn "the gradient of an average" into "an average of gradients." The first is uncomputable; the second is just a mini-batch.

The trick, and why it's the same trick twice

The fix is one line of calculus, and I'm flagging it now because we're going to use the exact same move again in Part 5, where it becomes importance sampling.

The move: multiply by one, then regroup.

You reintroduce a factor of the probability you'd divided out, which changes nothing mathematically, but restructures the expression into a weighted average — and weighted averages are things you can estimate by sampling. Run the policy, collect trajectories, average.

Two consequences fall out immediately, and both are lovely.

The environment disappears. A trajectory's probability includes the world's transition dynamics, which we don't know. But those don't depend on our parameters, so they differentiate to zero and vanish. We never needed a model of the world — not by clever design, but because the unknown terms happened to be constants. This is why policy gradients are model-free.

What's left is almost embarrassingly simple. The resulting algorithm — REINFORCE (Williams, 1992) — says:

Do stuff. See what score you got. If the score was good, make everything you did more likely. If it was bad, make it less likely.

Trial and error, with calculus attached.

What the gradient actually points at

Let's make that concrete, because "the gradient of the log-probability" stays abstract until you look at one.

Say you have three possible actions, currently equally likely, and you take action 2. The gradient is a vector that says:

raise the score of action 2,  lower the score of actions 1 and 3

raise the score of action 2,  lower the score of actions 1 and 3

That's it. It's the direction in parameter space that makes the action you took more likely and every other action less likely. Probability is a fixed budget of 1.0 — to give more to one action you must take it from the others, and the gradient does that bookkeeping automatically.

Then you multiply by how good the outcome was:

Good outcome → step along that direction. Action 2 becomes more likely. Reinforced.
Bad outcome → step against it. Action 2 becomes less likely, and — note this — actions 1 and 3 become more likely. Punishing one action automatically promotes the alternatives, without anyone specifying which.

One vector, one scalar, one multiplication. That's the whole learning rule.

The hidden correction (this is the important one)

There's a division buried inside that gradient — you divide by the probability of the action you took. It's the reason the logarithm appears in the textbook formula, and it is not there for numerical convenience. It's there to fix a bias.

Think about why. You're sampling actions from your own policy. An action taken 90% of the time appears in 90% of your samples. Without the division, that action would get counted twice over — once for being frequent in your data, once for having high probability in the formula. Common actions would dominate the gradient purely by being common.

You'd have built a machine that reinforces habits rather than rewards.

The division by the action's probability exactly cancels the frequency with which you sampled it.

Now notice what that division is: a correction that divides by the probability of the policy that generated the sample. That's an off-policy correction with your own policy in the denominator. It's already there, in every policy gradient — it's just that on-policy, the numerator is the same policy, the ratio equals 1, and it hides.

Hold onto this._ In Part 5 we're going to pull the numerator and denominator apart, and PPO's famous ratio term will appear. It was there all along. Nobody added it._

Two problems, and the fixes

Problem one: blaming the wrong actions. Naive REINFORCE multiplies every action's gradient by the score of the whole episode — including reward that arrived before the action was taken. An action at step 50 cannot have influenced a reward at step 10. So we use only the reward from that point onward.

The subtle part: including earlier rewards doesn't make the estimator wrong — their contribution averages to exactly zero, since your action can't correlate with a reward that preceded it. It just adds noise. And that reveals a principle that recurs constantly in RL:

If a term contributes nothing in expectation, delete it. It is pure variance._ Half the progress in policy gradient methods is finding zero-mean terms and removing them._

Problem two: the variance is still crushing. The fix is one of the most elegant things in the field. Subtract a baseline — any quantity that depends on the state but not on the action you chose.

The claim: subtracting it doesn't change the gradient at all on average. It only reduces variance. A free lunch.

Free lunches deserve suspicion, so here's the reason in one sentence. Probabilities always sum to exactly 1, no matter what your parameters are. So the gradient of that sum is zero — moving the parameters can't change a constant. Anything you multiply by that zero vanishes.

The constraint that makes something a probability distribution is what makes baselines free._ A structural fact about probability hands you an unlimited license to reduce variance._

What the baseline is actually worth: measured

The theory undersells this dramatically, so let's measure it.

Three-armed bandit. True rewards are 1000, 1002, 1001 — arm 2 is best, but all three are large and nearly identical. Not contrived: this is exactly what LLM reward models look like, where every response scores in a narrow band.

Measuring the gradient estimator over 20,000 samples, with and without a baseline:

baseline=none  | mean gradient = [-8.223  3.298  4.925] | std = [468.8 473.2 473.6]
baseline=mean  | mean gradient = [-0.336  0.330  0.006] | std = [  0.5   0.5   0.5]

baseline=none  | mean gradient = [-8.223  3.298  4.925] | std = [468.8 473.2 473.6]
baseline=mean  | mean gradient = [-0.336  0.330  0.006] | std = [  0.5   0.5   0.5]

Standard deviations: 470 versus 0.5. A thousandfold noise reduction from subtracting a constant.

Now read the means, which is the part I find most illuminating. With a baseline: [−0.336, +0.330, +0.006]. The true advantages — rewards minus their average — are [−1, +1, 0]. The baselined gradient reads out the structure almost exactly: arm 1 worse, arm 2 better, arm 3 neutral.

Without a baseline the mean points at arm 3 as the winner. It's wrong — and with that standard deviation, twenty thousand samples still can't tell the arms apart.

Does it matter for learning? Probability assigned to the best arm during training:

after 600  1200  1800  2400  3000 steps
  baseline=none   |      0.35   0.35   0.35   0.35   0.35
  baseline=mean   |      0.94   0.97   0.98   0.99   0.99

after 600  1200  1800  2400  3000 steps
  baseline=none   |      0.35   0.35   0.35   0.35   0.35
  baseline=mean   |      0.94   0.97   0.98   0.99   0.99

Without a baseline the agent never learns. It sits at chance forever, buried in noise.

Why:_ with no baseline,_ every action gets a score around +1000, so every action gets pushed up hard. The actual learning signal — the tiny differences between arms — is a 0.1% ripple on top of an enormous common signal saying "yes, more of everything." The baseline subtracts the common part and leaves only the differences, which is the only information that was ever there.

A teacher who gives every student 98/100 has technically graded them and communicated nothing. Grading on a curve is exactly a baseline — and it's what turns scores into information.

The best baseline turns out to be the average value of the state you were in. Subtract it and what remains is the advantage: how much better was this action than what I'd typically do here? It isolates the part of the outcome that was due to the choice — the only part the policy can be held responsible for. And it has a lucky property: the advantage of your average action is zero, so the update automatically shuts off where there's nothing left to learn and concentrates where a real improvement is available.

Where the two engines meet

We need the state's average value for the baseline. Where do we get it? We learn it — with TD learning, from Part 2.

That's actor-critic: the actor is the policy, updated by the policy gradient; the critic is the value function, updated by the TD error.

And now the fact I think is the most satisfying in this entire subject. If you estimate the advantage using a single step, it becomes:

advantage  ≈  ( reward + value of next state )  −  ( value of this state )

advantage  ≈  ( reward + value of next state )  −  ( value of this state )

That's the TD error. The exact object from Part 2, the one we called surprise.

The two engines of reinforcement learning are the same object seen from two sides.

The TD error is what the critic uses to correct its predictions — "I was wrong by this much, adjust." And it's also what the actor multiplies its gradient by — "that went better than expected, do more of it."

The critic's error is the actor's reward._ One number, two jobs. Surprise is simultaneously the signal for improving your predictions and the signal for improving your behavior_

This isn't a coding shortcut. It's why actor-critic dominates: you compute one number and it drives both halves. And it means everything from Part 2 — bias-variance, bootstrapping, and later the deadly triad — applies directly inside PPO and GRPO, because a TD learner is living inside every policy gradient method.

GAE (Schulman et al., 2015) is the natural finishing touch: a dial that slides smoothly between the one-step TD estimate (low variance, biased) and the full Monte Carlo return (unbiased, high variance). The exact trade-off from Part 2, now with a knob on it.

Back to our subject

We now have the full policy gradient, and here's the second symbolic equation of the article — because one symbol in it is the rest of this article:

∇J(θ)  =  𝔼_{τ ~ π_θ} [ Σₜ ∇log π_θ(aₜ|sₜ) · Âₜ ]
              ↑
      this subscript is the whole story

∇J(θ)  =  𝔼_{τ ~ π_θ} [ Σₜ ∇log π_θ(aₜ|sₜ) · Âₜ ]
              ↑
      this subscript is the whole story

Everything else in it you built: the log from the frequency correction, the sum over time from the trajectory, the Â from the critic's TD error. The subscript τ ~ π_θ reads "averaged over trajectories that this policy produces."

Which raises the obvious question: what if they weren't?

Part 5: Distribution Shift, and Why Off-Policy Is Hard

A story about the wrong distribution

In 1936, Literary Digest ran the largest election poll in American history. Ten million ballots mailed, 2.4 million returned — a sample size that still dwarfs modern polling. They predicted Alf Landon would beat Franklin Roosevelt in a landslide.

Roosevelt won 46 of 48 states. The magazine folded within two years.

The sample size was never the problem. The sampling distribution was. The mailing list came from telephone directories, automobile registrations, and club memberships — which in 1936 meant people doing fine during the Great Depression. They surveyed one population and drew conclusions about another. George Gallup got it right with 50,000 responses, because he sampled from the population he was actually asking about.

More data cannot fix the wrong distribution. In our field this failure has a name, and it is the only real problem in off-policy learning: distribution shift.

That subscript in the policy gradient is exactly the Literary Digest mailing list. If your data came from a different policy, your sample average estimates the wrong quantity — a correct answer to a question nobody asked.

And recall from Part 4 that this isn't a new problem bolted on. The hidden division inside the gradient was always correcting for the sampling distribution. It just silently assumed you sampled from your own policy. Break that assumption and the correction is now wrong by exactly the ratio between the two policies.

Which tells you precisely what the fix must be.

The fix

Same move as before: multiply by one, regroup. Only now you're converting an average over the distribution you want into an average over the distribution you have, and the price is a correction factor attached to every sample.

It's what pollsters do whOn-Policy vs Off-Policy Learning: The Most Misunderstood Distinction in Reinforcement Learning

From TD errors to GRPO — explained in words, with all the algebra kept in one place at the end

Almost everyone learns this distinction the same way: "On-policy learns from its own data. Off-policy learns from someone else's."

Prerequisites: what a Markov Decision Process is, roughly what a policy and a value function are, basic probability. Python/NumPy for the code.

How this article handles the mathematics

I've read enough tutorials that open with a wall of Greek letters to know how that goes. So here's the deal.

There are exactly four symbolic equations in the whole piece. Four. Each one earns its place by carrying an argument no sentence can, and I'll flag each as it arrives so you know it's coming.

Part 1: There Are Always Two Policies

Two different things, same car, same moment. In RL they get names:

Behavior policy (written b) — the policy that actually generates the data. Whatever your agent really does. The source of experience. The student.

Target policy (written π) — the policy you're trying to learn and improve. What you want to end up with. The instructor's ideal.

Now the definition:

On-policy: b = π_. The policy you're learning is exactly the one that produced the data._

Off-policy: b ≠ π_. You learn one policy from data generated by a different one._

That's it. No replay buffers, no sample efficiency, no "old data" — those are all consequences. To classify any algorithm, ask one question:

At the moment I compute this gradient, is the distribution that generated my data the same as the distribution I'm optimizing?

Why we'd ever want two policies

Here's the problem off-policy learning exists to solve.

This is the exploration–exploitation deadlock, and there are exactly two ways out.

Way two — separate the roles (off-policy). Let one policy explore while you learn about a different, greedy one. Now you can learn the optimal policy without paying the price of following it.

Part 2: How TD Learning Actually Works

This is the engine behind SARSA, Q-learning, DQN, and the critic inside PPO and GRPO. Worth doing properly.

The thing we're estimating

Call that expected total the value of a state. The whole problem becomes: how do you estimate an average when you can only observe one sample at a time?

The obvious method, and why it hurts

Play out the entire episode, compute what you actually got, average over many episodes. That's Monte Carlo, and it's obviously correct.

It also has two problems.

You have to wait for the end. In chess you learn nothing about move 3 until the game is over. In a task with no end, you wait forever.

The variance is enormous. The total return is a sum of hundreds of random rewards from hundreds of random transitions. One episode might be wildly unrepresentative.

That weekly-homework alternative is TD learning.

The recursive insight

Here's the move that makes everything possible, and it's an observation about arithmetic rather than a theorem.

Take the total future reward and pull out the first term. What's left is the same kind of quantity, one step later. Which means:

The return from here equals the reward you get right now, plus the discounted return from wherever you land next.

Bootstrapping: using a guess in place of the future

The problem: that relationship needs the value of the next state — which is exactly what we're trying to learn.

So we do something that feels illegal the first time. We use our current estimate of the next state anyway. The learning target becomes:

target  =  the reward we actually observed  +  our own current guess about where we landed
              ↑                                    ↑
        real information from the world        made up (for now)

target  =  the reward we actually observed  +  our own current guess about where we landed
              ↑                                    ↑
        real information from the world        made up (for now)

This is bootstrapping — updating a guess using another guess. It's genuinely unsettling. If all our estimates are wrong, doesn't updating one wrong number with another produce garbage?

Here's why it doesn't, and it's the most common conceptual gap in TD learning:

The target isn't purely a guess. It contains one piece of real information — the observed reward — that came from the actual world._ Every update injects one grain of truth. That truth then propagates backward: the state next to the reward learns first, then the state before it learns from_ that_, and so on. Truth diffuses backward through the chain of guesses, one link per pass._

The update rule, in words

surprise (δ)  =  ( reward received  +  value of where I landed )  −  ( value I predicted )
                  └──────────── what I believe now ───────────┘      └── what I believed ──┘

surprise (δ)  =  ( reward received  +  value of where I landed )  −  ( value I predicted )
                  └──────────── what I believe now ───────────┘      └── what I believed ──┘

And the learning rule is just:

new estimate  =  old estimate  +  α × surprise

new estimate  =  old estimate  +  α × surprise

where α (the learning rate, between 0 and 1) says how much you trust this one new piece of evidence.

Three things worth noticing about this shape, because they're all load-bearing:

It's a thermostat. The correction shrinks automatically as your guess approaches the target — when surprise goes to zero, the update goes to zero. Nothing has to tell it to stop.

Watching truth propagate backward

Let's make the diffusion claim concrete, because it's the part people accept on faith and shouldn't have to.

TD learning — information seeps one step backward per episode
  episode 1:  A=0.000  B=0.000  C=0.000  D=0.000  E=0.500
  episode 2:  A=0.000  B=0.000  C=0.000  D=0.250  E=0.750
  episode 3:  A=0.000  B=0.000  C=0.125  D=0.500  E=0.875
  episode 4:  A=0.000  B=0.062  C=0.312  D=0.688  E=0.938
  episode 5:  A=0.031  B=0.188  C=0.500  D=0.812  E=0.969

TD learning — information seeps one step backward per episode
  episode 1:  A=0.000  B=0.000  C=0.000  D=0.000  E=0.500
  episode 2:  A=0.000  B=0.000  C=0.000  D=0.250  E=0.750
  episode 3:  A=0.000  B=0.000  C=0.125  D=0.500  E=0.875
  episode 4:  A=0.000  B=0.062  C=0.312  D=0.688  E=0.938
  episode 5:  A=0.031  B=0.188  C=0.500  D=0.812  E=0.969

Nobody told A anything about the reward. A found out because it kept asking B, and B eventually had something worth saying.

Monte Carlo_ is unbiased but high-variance, and can't start until the episode ends. TD is biased early (your estimates are wrong at first) but low-variance, and learns from every single step._

_The example above is deterministic, which erases MC's variance problem and flatters it. Add noise to the rewards and MC's estimates thrash while TD stays steady. _Almost all real environments are noisy, which is why TD dominates in practice.

Same trade-off as the exams: unbiased-but-rare versus biased-but-frequent. In a noisy world, frequent and slightly wrong beats rare and technically correct.

One last step: from states to actions

Which leaves exactly one open question, and it's the question this entire article is about:

What do we put in the learning target?

Because that single choice decides whether an algorithm is on-policy or off-policy.

Part 3: SARSA vs Q-Learning — A One-Term Difference

SARSA        Q ← Q + α [ R + γ·Q(next state, the action I will ACTUALLY take) − Q ]
Q-learning   Q ← Q + α [ R + γ·max Q(next state, the BEST available action)  − Q ]
                                 └──────────── the only difference ────────────┘

SARSA        Q ← Q + α [ R + γ·Q(next state, the action I will ACTUALLY take) − Q ]
Q-learning   Q ← Q + α [ R + γ·max Q(next state, the BEST available action)  − Q ]
                                 └──────────── the only difference ────────────┘

Cliff Walking

Sutton and Barto's textbook has the perfect demonstration.

The optimal path is obvious: hug the cliff edge, shortest route. But for an ε-greedy agent it's terrifying, because 10% of the time it moves randomly — and one of those random moves is down.

Real output:

--- What SARSA learns ---           --- What Q-learning learns ---
.***********                        ............
**.........*                        ............
*..........*                        ************
SXXXXXXXXXXG                        SXXXXXXXXXXG
Return of the greedy policy:  −17   Return of the greedy policy:  −13
Average return while training: −23  Average return while training: −61

--- What SARSA learns ---           --- What Q-learning learns ---
.***********                        ............
**.........*                        ............
*..........*                        ************
SXXXXXXXXXXG                        SXXXXXXXXXXG
Return of the greedy policy:  −17   Return of the greedy policy:  −13
Average return while training: −23  Average return while training: −61

Q-learning hugs the cliff and finds the genuinely optimal policy: −13, the best achievable here. SARSA takes the scenic route and settles for −17.

Now the second line. During training, behaving ε-greedily, Q-learning averages −61. SARSA averages −23. Q-learning spends its entire training life falling off a cliff.

The key insight, and it's what most treatments skip:_ SARSA is not weaker and not timid. SARSA gives the_ correct answer to a different question. Q-learning answers "what is the optimal policy?" SARSA answers "given that I will keep exploring, how should I behave?" If your agent really will act ε-greedily, SARSA's answer is the more useful one.

The crossover, measured

Sweeping ε across both algorithms, 8 seeds each:

eps | SARSA train    QL train | SARSA greedy   QL greedy
   0.5 |      -135.8      -712.8 |       -131.4       -13.0
   0.3 |       -59.5      -219.9 |       -108.5       -13.0
   0.2 |       -44.5      -108.8 |       -131.4       -13.0
   0.1 |       -29.3       -51.6 |        -39.9       -13.0
  0.05 |       -20.8       -29.7 |        -17.0       -13.0
  0.01 |       -17.7       -14.9 |        -17.0       -13.0

eps | SARSA train    QL train | SARSA greedy   QL greedy
   0.5 |      -135.8      -712.8 |       -131.4       -13.0
   0.3 |       -59.5      -219.9 |       -108.5       -13.0
   0.2 |       -44.5      -108.8 |       -131.4       -13.0
   0.1 |       -29.3       -51.6 |        -39.9       -13.0
  0.05 |       -20.8       -29.7 |        -17.0       -13.0
  0.01 |       -17.7       -14.9 |        -17.0       -13.0

Three things, each teaching something:

The code

import numpy as np

ROWS, COLS = 4, 12
START, GOAL = (3, 0), (3, 11)
ACTIONS = [(-1,0), (1,0), (0,-1), (0,1)]   # up, down, left, right

def step(s, a):
    r, c = s
    dr, dc = ACTIONS[a]
    nr, nc = min(max(r+dr, 0), ROWS-1), min(max(c+dc, 0), COLS-1)
    if nr == 3 and 1 <= nc <= 10:            # fell off the cliff
        return START, -100.0, False
    return (nr, nc), -1.0, (nr, nc) == GOAL

def eps_greedy(Q, s, eps, rng):
    if rng.random() < eps:
        return rng.integers(4)
    q = Q[s[0], s[1]]
    # Break ties randomly. If you don't, argmax always picks the same
    # direction and exploration gets systematically crippled — a bug
    # that costs people days.
    return int(rng.choice(np.flatnonzero(q == q.max())))

def train(algo, episodes=2000, alpha=0.5, eps=0.1, gamma=1.0, seed=0):
    rng = np.random.default_rng(seed)
    Q = np.zeros((ROWS, COLS, 4))
    returns = []
    for _ in range(episodes):
        s = START
        a = eps_greedy(Q, s, eps, rng)        # behavior policy b
        G = 0.0
        for _ in range(500):
            s2, r, done = step(s, a)
            G += r
            a2 = eps_greedy(Q, s2, eps, rng)  # the action we will REALLY take

            if algo == 'sarsa':
                # Target policy == behavior policy.
                target = r + gamma * Q[s2[0], s2[1], a2] * (not done)
            else:
                # Target policy == greedy policy. We ignore a2 entirely.
                # This one line is what makes the algorithm off-policy.
                target = r + gamma * Q[s2[0], s2[1]].max() * (not done)

            Q[s[0], s[1], a] += alpha * (target - Q[s[0], s[1], a])
            s, a = s2, a2
            if done:
                break
        returns.append(G)
    return Q, np.array(returns)

import numpy as np

ROWS, COLS = 4, 12
START, GOAL = (3, 0), (3, 11)
ACTIONS = [(-1,0), (1,0), (0,-1), (0,1)]   # up, down, left, right

def step(s, a):
    r, c = s
    dr, dc = ACTIONS[a]
    nr, nc = min(max(r+dr, 0), ROWS-1), min(max(c+dc, 0), COLS-1)
    if nr == 3 and 1 <= nc <= 10:            # fell off the cliff
        return START, -100.0, False
    return (nr, nc), -1.0, (nr, nc) == GOAL

def eps_greedy(Q, s, eps, rng):
    if rng.random() < eps:
        return rng.integers(4)
    q = Q[s[0], s[1]]
    # Break ties randomly. If you don't, argmax always picks the same
    # direction and exploration gets systematically crippled — a bug
    # that costs people days.
    return int(rng.choice(np.flatnonzero(q == q.max())))

def train(algo, episodes=2000, alpha=0.5, eps=0.1, gamma=1.0, seed=0):
    rng = np.random.default_rng(seed)
    Q = np.zeros((ROWS, COLS, 4))
    returns = []
    for _ in range(episodes):
        s = START
        a = eps_greedy(Q, s, eps, rng)        # behavior policy b
        G = 0.0
        for _ in range(500):
            s2, r, done = step(s, a)
            G += r
            a2 = eps_greedy(Q, s2, eps, rng)  # the action we will REALLY take

            if algo == 'sarsa':
                # Target policy == behavior policy.
                target = r + gamma * Q[s2[0], s2[1], a2] * (not done)
            else:
                # Target policy == greedy policy. We ignore a2 entirely.
                # This one line is what makes the algorithm off-policy.
                target = r + gamma * Q[s2[0], s2[1]].max() * (not done)

            Q[s[0], s[1], a] += alpha * (target - Q[s[0], s[1], a])
            s, a = s2, a2
            if done:
                break
        returns.append(G)
    return Q, np.array(returns)

Two lines of difference. One conceptual universe apart.

Part 4: How Policy Gradients Actually Work

TD learning is one engine. Here's the other — the one powering every modern LLM post-training method.

Why not just use values?

We just built a machine for estimating how good each action is. Why not stop there and always pick the best one?

Three reasons, each fatal in its own domain.

So: make the policy itself a network with parameters, and improve those parameters by gradient ascent on expected reward.

The hard part

Here's why this is structurally harder than supervised learning, and it's worth being precise.

In supervised learning, your dataset is fixed. The parameters only affect the function, not which data you see. Sample a batch, compute gradients, done.

In RL, the parameters appear inside the distribution itself. Change the policy and you change which trajectories get sampled. The "dataset" rewrites itself every time you take a step.

We need to turn "the gradient of an average" into "an average of gradients." The first is uncomputable; the second is just a mini-batch.

The trick, and why it's the same trick twice

The fix is one line of calculus, and I'm flagging it now because we're going to use the exact same move again in Part 5, where it becomes importance sampling.

The move: multiply by one, then regroup.

Two consequences fall out immediately, and both are lovely.

What's left is almost embarrassingly simple. The resulting algorithm — REINFORCE (Williams, 1992) — says:

Do stuff. See what score you got. If the score was good, make everything you did more likely. If it was bad, make it less likely.

Trial and error, with calculus attached.

What the gradient actually points at

Let's make that concrete, because "the gradient of the log-probability" stays abstract until you look at one.

Say you have three possible actions, currently equally likely, and you take action 2. The gradient is a vector that says:

raise the score of action 2,  lower the score of actions 1 and 3

raise the score of action 2,  lower the score of actions 1 and 3

Then you multiply by how good the outcome was:

Good outcome → step along that direction. Action 2 becomes more likely. Reinforced.
Bad outcome → step against it. Action 2 becomes less likely, and — note this — actions 1 and 3 become more likely. Punishing one action automatically promotes the alternatives, without anyone specifying which.

One vector, one scalar, one multiplication. That's the whole learning rule.

The hidden correction (this is the important one)

You'd have built a machine that reinforces habits rather than rewards.

The division by the action's probability exactly cancels the frequency with which you sampled it.

Hold onto this._ In Part 5 we're going to pull the numerator and denominator apart, and PPO's famous ratio term will appear. It was there all along. Nobody added it._

Two problems, and the fixes

If a term contributes nothing in expectation, delete it. It is pure variance._ Half the progress in policy gradient methods is finding zero-mean terms and removing them._

The claim: subtracting it doesn't change the gradient at all on average. It only reduces variance. A free lunch.

The constraint that makes something a probability distribution is what makes baselines free._ A structural fact about probability hands you an unlimited license to reduce variance._

What the baseline is actually worth: measured

The theory undersells this dramatically, so let's measure it.

Measuring the gradient estimator over 20,000 samples, with and without a baseline:

baseline=none  | mean gradient = [-8.223  3.298  4.925] | std = [468.8 473.2 473.6]
baseline=mean  | mean gradient = [-0.336  0.330  0.006] | std = [  0.5   0.5   0.5]

baseline=none  | mean gradient = [-8.223  3.298  4.925] | std = [468.8 473.2 473.6]
baseline=mean  | mean gradient = [-0.336  0.330  0.006] | std = [  0.5   0.5   0.5]

Standard deviations: 470 versus 0.5. A thousandfold noise reduction from subtracting a constant.

Without a baseline the mean points at arm 3 as the winner. It's wrong — and with that standard deviation, twenty thousand samples still can't tell the arms apart.

Does it matter for learning? Probability assigned to the best arm during training:

after 600  1200  1800  2400  3000 steps
  baseline=none   |      0.35   0.35   0.35   0.35   0.35
  baseline=mean   |      0.94   0.97   0.98   0.99   0.99

after 600  1200  1800  2400  3000 steps
  baseline=none   |      0.35   0.35   0.35   0.35   0.35
  baseline=mean   |      0.94   0.97   0.98   0.99   0.99

Without a baseline the agent never learns. It sits at chance forever, buried in noise.

Why:_ with no baseline,_ every action gets a score around +1000, so every action gets pushed up hard. The actual learning signal — the tiny differences between arms — is a 0.1% ripple on top of an enormous common signal saying "yes, more of everything." The baseline subtracts the common part and leaves only the differences, which is the only information that was ever there.

A teacher who gives every student 98/100 has technically graded them and communicated nothing. Grading on a curve is exactly a baseline — and it's what turns scores into information.

Where the two engines meet

We need the state's average value for the baseline. Where do we get it? We learn it — with TD learning, from Part 2.

That's actor-critic: the actor is the policy, updated by the policy gradient; the critic is the value function, updated by the TD error.

And now the fact I think is the most satisfying in this entire subject. If you estimate the advantage using a single step, it becomes:

advantage  ≈  ( reward + value of next state )  −  ( value of this state )

advantage  ≈  ( reward + value of next state )  −  ( value of this state )

That's the TD error. The exact object from Part 2, the one we called surprise.

The two engines of reinforcement learning are the same object seen from two sides.

The TD error is what the critic uses to correct its predictions — "I was wrong by this much, adjust." And it's also what the actor multiplies its gradient by — "that went better than expected, do more of it."

The critic's error is the actor's reward._ One number, two jobs. Surprise is simultaneously the signal for improving your predictions and the signal for improving your behavior._

Back to our subject

We now have the full policy gradient, and here's the second symbolic equation of the article — because one symbol in it is the rest of this article:

∇J(θ)  =  𝔼_{τ ~ π_θ} [ Σₜ ∇log π_θ(aₜ|sₜ) · Âₜ ]
              ↑
      this subscript is the whole story

∇J(θ)  =  𝔼_{τ ~ π_θ} [ Σₜ ∇log π_θ(aₜ|sₜ) · Âₜ ]
              ↑
      this subscript is the whole story

Which raises the obvious question: what if they weren't?

Part 5: Distribution Shift, and Why Off-Policy Is Hard

A story about the wrong distribution

Roosevelt won 46 of 48 states. The magazine folded within two years.

More data cannot fix the wrong distribution. In our field this failure has a name, and it is the only real problem in off-policy learning: distribution shift.

Which tells you precisely what the fix must be.

The fix

It's what pollsters do when they reweight. Sample over-represents homeowners? Weight those responses below one. Under-represents renters? Weight them up. Same arithmetic, different century.

That factor is the importance sampling ratio — the third symbolic equation, and it's small:

ρ  =  π(a|s) / b(a|s)          "how much more likely was my target policy
                                to take this action than the one that did?"

ρ  =  π(a|s) / b(a|s)          "how much more likely was my target policy
                                to take this action than the one that did?"

The intuition is entirely in reading its values:en they reweight. Sample over-represents homeowners? Weight those responses below one. Under-represents renters? Weight them up. Same arithmetic, different century.

That factor is the importance sampling ratio — the third symbolic equation, and it's small:

ρ  =  π(a|s) / b(a|s)          "how much more likely was my target policy
                                to take this action than the one that did?"

ρ  =  π(a|s) / b(a|s)          "how much more likely was my target policy
                                to take this action than the one that did?"

The intuition is entirely in reading its values:

One sentence: listen harder to the samples that look more like what you'd actually do.

And here's where it falls apart

One step of correction is fine. But a trajectory has many steps, and correcting the whole thing means multiplying the ratios together.

Products of random numbers behave viciously. If each ratio is around 1.1 — a mild, seemingly harmless disagreement — then over 100 steps you get 1.¹¹⁰⁰ ≈ 13,780. If each is 0.9 instead, you get 0.000027.

Picture that rumor chain again: a hundred people, each distorting by only 10%. Nobody does anything unreasonable. What comes out has no relationship to what went in.

In practice you get an estimator where tens of thousands of samples contribute essentially nothing while three samples with enormous weights determine the entire update. It's mathematically unbiased — correct if you averaged infinitely many runs — and completely unusable with any finite sample.

Internalize this, because it explains half of modern RL:_ off-policy learning forces a permanent bias–variance bargain. Correct fully and variance explodes. Truncate the correction and variance drops but your estimate becomes biased. PPO's clipping, V-trace's truncated ratios, the TIS/MIS techniques in today's LLM stacks — all just different points on that same bargain. Once you see it, they stop being separate tricks._

So why does Q-learning escape this?

Elegant, and a little sneaky.

Q-learning takes the max, so it never needs the distribution over the next action at all. The target policy is deterministic and expressed directly by that operator — there's nothing to form a ratio with. And being one-step, it never accumulates a product along a trajectory. No product, no explosion.

But careful: distribution shift didn't disappear. You fixed the action distribution and did nothing about the state distribution. Q-learning still only learns in the states the behavior policy visits, while making claims about the states the target policy would visit.

With a lookup table that's harmless — visit every state enough and convergence is guaranteed. Swap the table for a neural network and the guarantee evaporates. Sutton and Barto call the failure mode the deadly triad:

Function approximation — an update in one state silently changes estimates for states you've never seen, because they share weights.
Bootstrapping — your target contains your own estimate, so errors feed themselves. (Note this is exactly the mechanism that made TD work in Part 2. The same self-reference that lets truth propagate backward also lets error propagate backward.)
Off-policy training — you update on one distribution while optimizing for another.

Any two together are safe. All three can diverge — not converge slowly, diverge, values running to infinity.

Three medications, each fine alone, fine in any pair, dangerous only in combination. Deep off-policy RL sits at that three-way intersection by construction. DQN's target network, TD3's twin networks, SAC's entropy regularization, CQL's conservatism penalty — each sedates one leg of the triad.

Part 6: What This Buys You in Practice

In an on-policy algorithm, the moment you update, all your data is stale — you changed the policy, so behavior and target no longer match. Pure on-policy methods must collect fresh data for every gradient step. Each experience is used once and discarded. An enormous luxury.

Off-policy algorithms have no such constraint. Store experience in a replay buffer and sample repeatedly. That's one of the two tricks that made DQN work on Atari (Mnih et al., 2015); the other is the target network. The buffer also breaks correlation between consecutive samples, which matters because SGD assumes roughly independent samples and consecutive game frames are anything but.

Now the correction, because this is the single most common confusion in the topic:

A replay buffer does not make an algorithm off-policy. An algorithm can have a replay buffer** because **it is already off-policy.

Causality runs that direction only. The max operator is what makes the data's origin irrelevant; the buffer exploits that freedom rather than creating it. Reverse the two and you'll misclassify half the algorithms you meet.

Look hard at the sample efficiency row — there's a hidden assumption. Off-policy methods are efficient in environment interactions, not necessarily in compute; chewing the same data twenty times costs GPU-hours.

So the real question is which resource is scarce. Interaction expensive (physical robot, real user, real money) → off-policy wins decisively. Interaction cheap and parallelizable (10,000 simulator copies) → on-policy stability is often the better trade. PPO's long reign was never about theoretical superiority; it was this practical balance plus the fact that it works without heroic tuning.

Part 7: The Gray Zone — Is PPO Actually On-Policy?

Here's where the binary you just learned turns out to be a spectrum.

PPO's objective is the fourth and last symbolic equation, and by now you've already built every piece of it:

L(θ) = 𝔼 [ min( r·Â ,  clip(r, 1−ε, 1+ε)·Â ) ]        where  r = π_new(a|s) / π_old(a|s)

L(θ) = 𝔼 [ min( r·Â ,  clip(r, 1−ε, 1+ε)·Â ) ]        where  r = π_new(a|s) / π_old(a|s)

That r is the importance sampling ratio from Part 5. Same object, different letter. And it's the same hidden division from Part 4, with numerator and denominator finally pulled apart.

The clip squashes the ratio into a narrow band — with the usual ε = 0.2, anything above 1.2 becomes 1.2 and anything below 0.8 becomes 0.8.

The min is the part that confuses people, so here it is in words. It makes the objective pessimistic, and it behaves differently depending on whether the action was good or bad:

Good action (positive advantage): the min caps how much credit you get for raising its probability. Push past the band and the objective stops improving — no gradient, no incentive. You get rewarded for moving toward good actions, but only so far.
Bad action (negative advantage): the penalty keeps applying until the ratio drops back into the band. Bad actions get pushed down, but not infinitely far in one update.

The asymmetry is deliberate: PPO can never be rewarded for a large policy change, but it can still be penalized for one. A one-way ratchet.

Now the real question: why does PPO need an importance ratio at all, if it's on-policy?

Because it isn't, quite. PPO collects a batch, then makes several passes over it — typically 4 to 10. The first gradient step is genuinely on-policy. After that first step, the new policy is no longer the old policy. From the second minibatch onward, PPO trains off-policy on data from a policy that no longer exists.

Which reframes clipping entirely. Textbooks say it "prevents excessively large updates" — true but shallow. The deeper reading: clipping constrains the region in which the importance sampling correction remains trustworthy. Once the ratio leaves the band, you've entered the high-variance regime from Part 5, and PPO simply refuses to act on it. TRPO's trust region states the same idea more explicitly: don't extrapolate far from where you actually measured.

The correct mental model:_ on-policy and off-policy aren't a binary flag. They're a continuum, and the axis is staleness — how old the data is relative to the policy being updated._

The real difference between algorithms is how much staleness they tolerate and what mechanism they use to pay for it. Hold this picture and new algorithms stop being new — you locate them on the line and ask which correction they chose.

Part 8: A Second Axis — Online vs Offline

Time to clear up the confusion that causes more errors than any other here. These are different axes, and people use them interchangeably constantly.

Online vs offline asks: is the agent collecting new interactions during training?
On-policy vs off-policy asks: is the data-generating policy the same as the one being learned?

The empty cell isn't an oversight. If your dataset is fixed and your policy is changing, the data came from some other policy the moment you took your first gradient step. Offline training is off-policy by construction.

Offline RL is the far end of the spectrum, with a signature failure mode: extrapolation error.

Here's how it kills you. Your network must assign values to state-action pairs appearing nowhere in the dataset. For some it will guess optimistically — neural networks extrapolate, and they do it without shame. Then the max operator does exactly what it's designed to do: select the highest-valued action, which means it specifically seeks out the optimistic errors. The policy moves toward them. And because you're offline, no new data can ever contradict the fantasy. The error compounds with nothing to check it.

Picture a medieval map with "here be dragons" across the unexplored regions. Offline RL's failure is a cartographer who, instead of writing the warning, confidently draws a shipping lane through the blank space — and routes the fleet through it.

Every fix is a version of the same instruction: be conservative about what your data didn't tell you. CQL (Kumar et al., 2020) pushes down values for out-of-distribution actions. IQL (Kostrikov et al., 2021) avoids ever querying one. BCQ (Fujimoto et al., 2019) constrains the policy to stay near the data. Different mathematics, one message.

Part 9: The Same Distinction in the LLM Era

Everything so far has been known since roughly the 1990s. Now the interesting part: this distinction reappeared, almost line for line, in LLM post-training.

The standard RLHF recipe is on-policy

The classical pipeline — SFT, then a reward model, then PPO (Stiennon et al., 2020; Ouyang et al., 2022) — is on-policy at heart. The model generates its own outputs, the reward model scores them, PPO updates on those fresh samples. A KL penalty toward a frozen reference model is added for stability, which is the trust region idea from Part 7 in different clothes.

The cost is heavy: four models in memory — policy, reference, reward, and value. GRPO (Shao et al., 2024) removes one with an idea you can now predict.

You know from Part 4 that the gradient needs a baseline, and that the ideal baseline is the state's average value. PPO trains an entire value network to estimate it. GRPO's observation: if you already sampled eight answers to the same prompt, the other seven are a baseline. Free, and no extra network. Subtract the group's mean; divide by its spread to normalize across prompts of wildly different difficulty.

And you can now predict what happens without it, because you've seen the experiment. Reward models score everything in a narrow high band — exactly the 1000/1002/1001 bandit from Part 4. Without a baseline, every response gets a large positive gradient and the model learns nothing about which was better. The 0.35-forever line.

(One honest caveat: subtracting the mean is provably free, but dividing by the spread is not — that proof covers additive baselines only, and dividing by a data-dependent quantity introduces bias. Several 2025 papers propose dropping it. Worth knowing the free lunch has a precise scope.)

GRPO remains fundamentally on-policy, and since DeepSeek-R1 it's the default for RLVR — RL with verifiable rewards, where correctness is checked by a program rather than a learned model.

DPO: the fully off-policy end

DPO (Rafailov et al., 2023) sits at the opposite extreme. It never samples during training; it optimizes directly over a fixed preference dataset. Offline and off-policy. Simplicity made it enormously popular.

Its weakness is exactly what Part 8 predicts. Preference data usually comes from other models, not yours. As training proceeds your model drifts from that distribution while the loss keeps optimizing against it — textbook distribution shift. Tajwar et al. (2024) tackled this directly, arguing preference fine-tuning should leverage on-policy data even when that data is lower quality. The field's answer has been iterative and online DPO: periodically regenerate the dataset from the current model. Which is to say — rescue the off-policy method by injecting on-policy-ness back into it.

The 2026 twist: your on-policy training is secretly off-policy

The part I'd most want you to remember.

Modern LLM RL stacks split generation from training. Rollouts run on high-throughput inference engines (vLLM, SGLang); gradients are computed in training backends (FSDP, Megatron). Different systems, different kernels, different numerical precision.

The problem: the same weights, given the same input, produce slightly different token probabilities in the two engines.

Which is exactly the condition from Part 1. The distribution that generated the data is not the distribution where the gradient is computed. Your nominally on-policy training became off-policy at the infrastructure level, through no algorithmic choice of yours.

It's the game of telephone, except both players are convinced they're the same person. And PPO's clipping can't catch it, because clipping was designed for algorithmic staleness, not numerical disagreement. In Mixture-of-Experts models it's worse: tiny numerical differences flip routing decisions, so a rounding error becomes a different expert becomes a materially different distribution.

The response should now feel entirely predictable — it's the correction from Part 5. Truncated and masked importance sampling reweight the gradient by the ratio between the two engines' probabilities, clipping or masking the extremes. There's a pleasing symmetry: mathematics built around 2000 for off-policy evaluation is now patching GPU kernel mismatches a quarter-century later. Good abstractions age well.

This is the concrete proof of the article's main claim._ If you treat on-policy vs off-policy as_ "which algorithm did I pick from the menu," you'll never notice your infrastructure violating the property on your behalf. If you understand it as a relationship between two distributions, you can run the same diagnosis on SARSA and on vLLM.

There's also a countercurrent. Pure on-policy RLVR is wasteful on hard tasks: if the model almost never produces a correct solution, reward is too sparse to learn from. So 2025–2026 has seen rapid adoption of mix-policy approaches — replay buffers of past successes, expert traces, asynchronous rollouts tolerating stale data. Deep RL learned this in 2015; LLM post-training is relearning it now. Same bargain, new hardware.

A final subtlety: there are actually three policies

To be precise about modern systems you need three, not two:

Behavior policy — what actually generated the rollouts (the inference engine; under asynchronous training, often a mixture over workers at different weight versions).
Old policy — what sits in the denominator of PPO/GRPO's ratio.
Target policy — what you're optimizing.

Standard theory assumes the first two are identical. Asynchronous or multi-engine systems break that, adding a second independent source of drift on top of the algorithmic one. Naming which of the three is misaligned is the prerequisite for diagnosing modern training instabilities — and the reason "just add clipping" often doesn't help.

Part 10: Common Misconceptions

"Off-policy means using a replay buffer." Causality reversed. The buffer is a consequence. There are off-policy algorithms with no buffer (tabular Q-learning), and methods that reuse data buffer-style while still being called on-policy (PPO's multiple passes).

"On-policy = online, off-policy = offline." The grid in Part 8 disproves it. DQN is simultaneously online and off-policy. Asynchronous RLHF is the same — live generation, stale training.

"SARSA is just a weaker Q-learning." Different target policies. The cliff experiment settles it: if your agent really will explore, SARSA's answer is more useful. Not weaker — a different question.

"Off-policy is always more sample efficient." In environment samples, usually. Factor in compute, stability, and tuning effort and the story changes. And as LLM RLVR shows, data that's too stale can degrade the signal rather than enrich it.

"PPO is an on-policy algorithm." Approximately true, precisely false. Multiple passes over one batch make you mildly off-policy — which is exactly why the ratio and clip exist. "Near-on-policy" is honest.

"Importance sampling solves distribution shift." It makes the estimator unbiased. It does not make it usable. In practice there's always a clip, a mask, a trust region, or a conservatism penalty — which means you've always accepted some bias. There is no free correction.

"Bootstrapping is just an approximation we tolerate." Understating it. Bootstrapping is what lets value information propagate backward through a chain of states — the wave in Part 2. It's also leg two of the deadly triad. The same mechanism is the source of both the power and the instability, which is why you can't simply remove it to fix divergence.

"The baseline is a variance-reduction hack." It's a mathematical identity, and it's not optional: the bandit experiment shows learning fails completely without it when rewards share a large common component — the normal case in LLM reward modeling.

"SFT is a kind of off-policy RL." Careful. SFT predicts the next token on fixed external data — the same family as behavior cloning. It inherits offline RL's distribution shift problems, and that part is fair. But there's no reward signal and no value estimation, so calling it off-policy RL muddies the terminology.

Part 11: A Decision Framework

1. How expensive is environment interaction? Real robot, real users, real money → lean off-policy; you need to reuse every sample. Fast, parallelizable simulation → you can afford on-policy stability.

2. Do I have pre-collected data, and can I collect more? Data but no new collection → offline RL with conservative methods and serious respect for extrapolation error. Both → hybrid: pretrain offline, fine-tune online.

3. Discrete or continuous actions? Discrete and moderate → the DQN family. Continuous → SAC (more sample efficient) or PPO (needs less tuning to work at all).

4. Will my agent keep exploring after training? If yes, remember SARSA: the optimal behavior for a policy that keeps exploring is not the optimal behavior for one that doesn't.

5. (For LLMs) How sparse is my reward? Dense, reasonable success rate → on-policy GRPO/PPO works cleanly. Very sparse → bring in off-policy support: expert traces, replay of past successes.

6. Is my advantage properly baselined? If rewards share a large common component — and reward models almost always produce this — an unbaselined gradient teaches nothing while looking like it's training.

7. Do generation and training follow the same numerical path? If no — and usually it's no — you have hidden off-policy drift inside a nominally on-policy run. Monitor the log-probability gap between engines.

Closing

We started with a slogan and ended with a relationship.

But look at what the road contained. TD learning came from a single observation about how sums factor, and we watched information crawl backward through a chain of states one episode at a time. The policy gradient came from multiplying by one and regrouping — and the same move, used again, gave us importance sampling. We found that the logarithm exists to cancel the bias of sampling from your own policy. We found that baselines are free because probabilities sum to one, and measured a thousandfold variance reduction from that fact. And we found that the critic's error and the actor's reward are the same number.

Then the distinction itself: on-policy and off-policy are not algorithm labels. They name the relationship between two distributions — the one that generated your data and the one you're optimizing. When it breaks you get distribution shift. The only honest fix is importance sampling, and its price is variance. Most of the diversity in RL algorithms is different answers to one question: how much bias will you accept to buy that variance down?

Once you hold that frame you know where to look in any new paper. Where is the ratio term? How is it constrained? How much staleness was this designed to survive? Those three questions work on Watkins' 1989 Q-learning thesis and on a 2026 asynchronous RLVR pipeline equally well.

In a field that reinvents itself every eighteen months, the fact that the same three questions have kept working for thirty-five years is the best argument I know for learning fundamentals instead of chasing releases.

Compact Glossary

Going Deeper

If you want the algebra behind what I described in words, here is exactly where each derivation lives. Sutton & Barto's Reinforcement Learning: An Introduction (2nd ed.) is freely available online and covers most of it:

Also worth reading: Precup, Sutton & Singh (2000) for the foundations of off-policy evaluation; Schultz, Dayan & Montague (1997) if the dopamine connection interested you.

Deep RL: Mnih et al. (2015) DQN; Schulman et al. (2015) GAE; Schulman et al. (2015, 2017) TRPO and PPO — read the trust region and clip as constraints on the importance sampling correction and both papers get much clearer; Haarnoja et al. (2018) SAC; Espeholt et al. (2018) IMPALA/V-trace, the direct ancestor of today's asynchronous LLM RL.

Offline RL: Levine et al. (2020) survey; Fujimoto et al. (2019) BCQ; Kumar et al. (2020) CQL; Kostrikov et al. (2021) IQL.

LLM post-training: Ouyang et al. (2022) InstructGPT; Rafailov et al. (2023) DPO; Shao et al. (2024) GRPO; Tajwar et al. (2024) on why on-policy data matters. Work from 2025–2026 on training–inference mismatch moves quickly enough that you should check recent arXiv listings rather than trusting any fixed citation, including this one.

Three Exercises, in Increasing Order of Value

1. Watch the wave. Reproduce the five-state chain from Part 2. Print the value table after every episode and watch information crawl backward. Then add noise to the rewards and watch TD stay steady while Monte Carlo thrashes.

2. Kill the baseline. Reproduce the bandit from Part 4. Set the true rewards to [0, 2, 1] instead of [1000, 1002, 1001] and watch the no-baseline version suddenly work fine. Then walk the common component back up — 10, 100, 1000 — and find where learning dies. That threshold is exactly what GRPO's group baseline exists to defeat.

3. Find the crossover. Take the Cliff Walking code, sweep ε from 0.5 to 0.01, and plot four curves: training return and greedy return for both algorithms. My numbers are in Part 3, but generating them yourself is different from reading them. When you see where the training curves cross — and why the Q-learning greedy line is flat all the way across — you'll have this entire article compressed into a single figure. That figure is worth more than the article.

Contents

How this article handles the mathematics

Part 1: There Are Always Two Policies

Why we'd ever want two policies

Part 2: How TD Learning Actually Works

The thing we're estimating

The obvious method, and why it hurts

The recursive insight

Bootstrapping: using a guess in place of the future

The update rule, in words

Watching truth propagate backward

One last step: from states to actions

Part 3: SARSA vs Q-Learning — A One-Term Difference

Cliff Walking

The crossover, measured

The code

Part 4: How Policy Gradients Actually Work

Why not just use values?

The hard part

The trick, and why it's the same trick twice

What the gradient actually points at

The hidden correction (this is the important one)

Two problems, and the fixes

What the baseline is actually worth: measured

Where the two engines meet

Back to our subject

Part 5: Distribution Shift, and Why Off-Policy Is Hard

A story about the wrong distribution

The fix

From TD errors to GRPO — explained in words, with all the algebra kept in one place at the end

How this article handles the mathematics

Part 1: There Are Always Two Policies

Why we'd ever want two policies

Part 2: How TD Learning Actually Works

The thing we're estimating

The obvious method, and why it hurts

The recursive insight

Bootstrapping: using a guess in place of the future

The update rule, in words

Watching truth propagate backward

One last step: from states to actions

Part 3: SARSA vs Q-Learning — A One-Term Difference

Cliff Walking

The crossover, measured

The code

Part 4: How Policy Gradients Actually Work

Why not just use values?

The hard part

The trick, and why it's the same trick twice

What the gradient actually points at

The hidden correction (this is the important one)

Two problems, and the fixes

What the baseline is actually worth: measured

Where the two engines meet

Back to our subject

Part 5: Distribution Shift, and Why Off-Policy Is Hard

A story about the wrong distribution

The fix

And here's where it falls apart

So why does Q-learning escape this?

Part 6: What This Buys You in Practice

Part 7: The Gray Zone — Is PPO Actually On-Policy?

Part 8: A Second Axis — Online vs Offline

Part 9: The Same Distinction in the LLM Era

The standard RLHF recipe is on-policy

DPO: the fully off-policy end

The 2026 twist: your on-policy training is secretly off-policy

A final subtlety: there are actually three policies

Part 10: Common Misconceptions

Part 11: A Decision Framework

Closing

Compact Glossary

Going Deeper

Three Exercises, in Increasing Order of Value