import plotly.io as pio; pio.renderers.default = "notebook_connected"
Lecture 22: From REINFORCE to GRPO (CS 189, Fall 2026)
This notebook builds the policy gradient step by step and trains a small language model with reinforcement learning against a checker.
| Notebook section | Slide |
|---|---|
| The Policy Gradient on One Prompt | The Log-Derivative Trick, Baselines |
| A Toy Verifiable Task | The Toy Task |
| REINFORCE and a Baseline (Lecture Demo 1) | Baselines in Practice |
| GRPO (Lecture Demo 2) | The GRPO Objective |
| Reward Hacking | Reward Hacking |
Everything runs on a laptop CPU in about a minute: the "language model" is a small network that writes the answer to an addition problem one token at a time.
import copy
import time
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import plotly.graph_objects as go
from plotly.subplots import make_subplots
torch.manual_seed(0)
NAVY, GOLD, GRAY, RED = '#002676', '#FDB515', '#808080', '#C0362C'
The Policy Gradient on One Prompt
Start with one prompt and $K=4$ possible responses. The policy is a softmax over logits $\boldsymbol{\theta}$, $\pi_{\boldsymbol\theta}(y)=\exp(\theta_y)/\sum_k\exp(\theta_k)$, and response $y$ earns reward $r(y)$. The objective is the expected reward
$$J(\boldsymbol\theta)=\sum_y \pi_{\boldsymbol\theta}(y)\,r(y)=\mathbb{E}_{y\sim\pi_{\boldsymbol\theta}}[r(y)].$$
Here we can compute $J$ and its gradient exactly. For a language model the sum runs over every possible response, so we cannot; we can only sample responses and score them.
r = torch.tensor([0.0, 1.0, 0.0, 0.0]) # only response 1 is correct
theta = torch.tensor([1.0, 0.0, 0.5, -0.5], requires_grad=True)
pi = torch.softmax(theta, 0)
J = (pi * r).sum()
J.backward()
exact_grad = theta.grad.clone()
print('pi =', pi.detach().numpy().round(3), ' J =', round(J.item(), 3))
print('exact gradient (autograd):', exact_grad.numpy().round(3))
pi = [0.455 0.167 0.276 0.102] J = 0.167 exact gradient (autograd): [-0.076 0.139 -0.046 -0.017]
The log-derivative trick. Since $\nabla\pi = \pi\,\nabla\ln\pi$,
$$\nabla J=\sum_y r(y)\,\nabla\pi_{\boldsymbol\theta}(y)=\sum_y \pi_{\boldsymbol\theta}(y)\,r(y)\,\nabla\ln\pi_{\boldsymbol\theta}(y)=\mathbb{E}_{y\sim\pi_{\boldsymbol\theta}}\big[r(y)\,\nabla\ln\pi_{\boldsymbol\theta}(y)\big].$$
An expectation can be estimated by sampling: draw $y\sim\pi_{\boldsymbol\theta}$ and use $r(y)\nabla\ln\pi_{\boldsymbol\theta}(y)$. For the softmax, $\nabla_{\boldsymbol\theta}\ln\pi_{\boldsymbol\theta}(y)=\mathbf{e}_y-\boldsymbol\pi$ (one-hot minus probabilities). This is REINFORCE (Williams, 1992).
def reinforce_estimates(theta, r, n, baseline=0.0, seed=0):
"""n single-sample REINFORCE estimates (r(y) - b) * grad ln pi(y), as an n x K tensor."""
g = torch.Generator().manual_seed(seed)
pi = torch.softmax(theta.detach(), 0)
y = torch.multinomial(pi, n, replacement=True, generator=g)
grad_log_pi = F.one_hot(y, len(pi)).float() - pi # e_y - pi
return (r[y] - baseline)[:, None] * grad_log_pi
est = reinforce_estimates(theta, r, 100_000)
print('mean of 100,000 estimates:', est.mean(0).numpy().round(3))
print('exact gradient: ', exact_grad.numpy().round(3))
mean of 100,000 estimates: [-0.076 0.139 -0.046 -0.017] exact gradient: [-0.076 0.139 -0.046 -0.017]
The average matches the exact gradient: the estimator is unbiased. But each single estimate is noisy. Now add 5 to every reward. The best response is unchanged and so is the exact gradient (adding a constant to $r$ adds a constant to $J$). The estimates, however, become much noisier: every sampled response is now "rewarded", so every estimate pushes up whatever was sampled.
A baseline. Subtracting any constant $b$ keeps the estimator unbiased, because $\mathbb{E}_{y\sim\pi}[\nabla\ln\pi(y)]=\sum_y\nabla\pi(y)=\nabla\sum_y\pi(y)=\nabla 1=0$. A good choice is $b\approx J$, the average reward. Then $r(y)-b$, the advantage, says whether a response was better or worse than usual.
J_val = (torch.softmax(theta.detach(), 0) * r).sum().item()
cases = {'r': (r, 0.0), 'r + 5': (r + 5, 0.0), 'r + 5, baseline b = J + 5': (r + 5, J_val + 5)}
fig = make_subplots(1, 3, subplot_titles=list(cases), shared_yaxes=True)
for k, (name, (rr, b)) in enumerate(cases.items()):
e = reinforce_estimates(theta, rr, 400 * 8, baseline=b, seed=1).view(400, 8, 4).mean(1) # averages of 8 samples
full = reinforce_estimates(theta, rr, 100_000, baseline=b, seed=2)
var = full.var(0).sum().item()
print(f'{name:28s} mean {full.mean(0).numpy().round(3)} total variance {var:6.3f}')
fig.add_trace(go.Scatter(x=e[:, 0], y=e[:, 1], mode='markers', marker=dict(color=NAVY, size=6, opacity=0.4),
showlegend=False), 1, k + 1)
fig.add_trace(go.Scatter(x=[exact_grad[0]], y=[exact_grad[1]], mode='markers', marker=dict(color=GOLD, size=14, symbol='star'),
name='exact gradient', showlegend=k == 0), 1, k + 1)
fig.update_xaxes(title='gradient for θ₀', range=[-2.2, 1])
fig.update_yaxes(title='gradient for θ₁', range=[-1, 2.2], col=1)
fig.update_layout(width=950, height=380, title='400 REINFORCE estimates, each the average of 8 samples (first two coordinates)')
fig
r mean [-0.076 0.139 -0.046 -0.017] total variance 0.137 r + 5 mean [-0.058 0.136 -0.049 -0.029] total variance 18.691 r + 5, baseline b = J + 5 mean [-0.077 0.139 -0.046 -0.017] total variance 0.101
A single-sample estimate takes only $K=4$ values (one per response), so each dot averages 8 samples. Same mean in all three panels; the spread is what changes. With the baseline, the variance returns to (here, even below) that of the original rewards. Lower variance means fewer samples for the same progress.
A Toy Verifiable Task
Task. The prompt is two numbers $a,b\in\{0,\dots,19\}$; the response is their sum written as digit tokens followed by an end token, for example 2 3 <eos>. The vocabulary has 11 tokens (the digits and <eos>), and a response has at most $T=3$ tokens.
Policy. A small network predicts each token from the prompt, the position, and the previous token, so
$$\ln\pi_{\boldsymbol\theta}(\mathbf{y}\mid\mathbf{x})=\sum_{t}\ln\pi_{\boldsymbol\theta}(y_t\mid\mathbf{x},y_{t-1}),$$
the same factorization as a language model (with a one-token memory instead of attention).
Checker. The reward is 1 if the answer is exactly right and 0 otherwise: a verifiable reward, as for math problems or unit tests.
N, EOS, V, T = 20, 10, 11, 3 # numbers 0..19, token 10 is <eos>, 11 tokens, at most 3 response tokens
prompts = torch.tensor([(a, b) for a in range(N) for b in range(N)]) # all 400 prompts
def answer_tokens(s):
digits = [int(c) for c in str(s)]
return digits + [EOS] * (T - len(digits))
answers = torch.tensor([answer_tokens(a + b) for a, b in prompts.tolist()])
def detok(y):
return ''.join(str(t) for t in y if t != EOS)
print('prompt (7, 16) ->', answers[7 * N + 16].tolist(), '=', detok(answers[7 * N + 16].tolist()))
prompt (7, 16) -> [2, 3, 10] = 23
class Policy(nn.Module):
"""pi(y_t | a, b, t, y_{t-1}): embeddings of the prompt, the position and the previous token, then an MLP."""
def __init__(self, d=64, hidden=128):
super().__init__()
self.emb_a, self.emb_b = nn.Embedding(N, d), nn.Embedding(N, d)
self.emb_pos, self.emb_prev = nn.Embedding(T, d), nn.Embedding(V + 1, d) # V = "start" token
self.mlp = nn.Sequential(nn.Linear(d, hidden), nn.ReLU(), nn.Linear(hidden, hidden), nn.ReLU(), nn.Linear(hidden, V))
def logits(self, x, y):
"""Logits for every response position (teacher forcing): x is B x 2, y is B x T, output B x T x V."""
prev = torch.cat([torch.full((len(x), 1), V), y[:, :-1]], 1)
h = self.emb_a(x[:, 0])[:, None] + self.emb_b(x[:, 1])[:, None] + self.emb_pos(torch.arange(T)) + self.emb_prev(prev)
return self.mlp(h)
def token_logprobs(policy, x, y):
"""ln pi(y_t | x, y_<t) for each token (B x T), and a mask that is 0 for the padding after the first <eos>."""
lp = F.log_softmax(policy.logits(x, y), -1).gather(-1, y[..., None])[..., 0]
is_eos = (y == EOS).long()
mask = ((torch.cumsum(is_eos, 1) - is_eos) == 0).float()
return lp, mask
@torch.no_grad()
def sample(policy, x, gen=None):
"""Sample one response per prompt, one token at a time."""
y = torch.full((len(x), T), EOS)
for t in range(T):
p = torch.softmax(policy.logits(x, y)[:, t], -1)
y[:, t] = torch.multinomial(p, 1, generator=gen)[:, 0]
after_eos = (torch.cumsum((y == EOS).long(), 1) - (y == EOS).long()) > 0
y[after_eos] = EOS
return y
def check(x, y):
"""The checker: reward 1 for exactly the right answer, else 0."""
return (y == answers[x[:, 0] * N + x[:, 1]]).all(1).float()
@torch.no_grad()
def accuracy(policy):
"""Probability that a sampled answer is right, averaged over all 400 prompts (computed exactly)."""
lp, mask = token_logprobs(policy, prompts, answers)
return (lp * mask).sum(1).exp().mean().item()
Start from an imperfect SFT model
RL usually starts from a supervised fine-tuned model (Lecture 19). Our demonstrator is sloppy: it writes the right sum only 40% of the time, and otherwise an answer off by 1 or by 10. Fine-tuning on its demonstrations gives a model that is right about as often as the demonstrator.
def noisy_demonstrations(copies=8, p_correct=0.4, seed=0):
g = torch.Generator().manual_seed(seed)
x = prompts.repeat(copies, 1)
s = x.sum(1)
error = torch.tensor([-10, -1, 1, 10])[torch.randint(4, (len(x),), generator=g)]
s = torch.where(torch.rand(len(x), generator=g) < p_correct, s, (s + error).clamp(0, 99))
return x, torch.tensor([answer_tokens(v) for v in s.tolist()])
def sft(x, y, epochs=400, lr=1e-3, seed=0):
torch.manual_seed(seed)
policy = Policy()
opt = torch.optim.Adam(policy.parameters(), lr)
for _ in range(epochs):
lp, mask = token_logprobs(policy, x, y)
loss = -(lp * mask).sum() / mask.sum() # next-token loss on the response tokens
opt.zero_grad(); loss.backward(); opt.step()
return policy
t0 = time.time()
sft_model = sft(*noisy_demonstrations())
print(f'SFT model: right {accuracy(sft_model):.0%} of the time ({time.time() - t0:.0f} s)')
g = torch.Generator().manual_seed(3)
x = torch.tensor([[7, 16]] * 8)
print('8 samples for 7 + 16:', [detok(y) for y in sample(sft_model, x, g).tolist()])
SFT model: right 43% of the time (4 s) 8 samples for 7 + 16: ['33', '23', '33', '33', '23', '23', '33', '23']
RL does not need the right answers, only the prompts and the checker. The checker never shows the model a correct answer; the model has to sample one and get rewarded for it.
(Lecture Demo 1) REINFORCE and a Baseline
Lecture: run from the "Baselines in Practice" slide.
Each step samples $G=8$ responses for each of 32 prompts, scores them, and takes one gradient step on
$$\text{loss}=-\frac{1}{BG}\sum_{i} A_i\,\ln\pi_{\boldsymbol\theta}(\mathbf{y}_i\mid\mathbf{x}_i),$$
with $A_i=r_i$ (REINFORCE) or $A_i=r_i-\bar r$, where $\bar r$ is the mean reward of the 8 responses to the same prompt (a per-prompt baseline). Minimizing this loss is gradient ascent on $J$. It is the SFT loss on the model's own samples, weighted by the advantage.
def advantages(r, G, method):
"""r: rewards for B prompts x G samples, flattened prompt by prompt."""
rg = r.view(-1, G)
if method == 'reinforce':
return r
if method == 'baseline':
return (rg - rg.mean(1, keepdim=True)).view(-1)
if method == 'group': # GRPO: standardize within the group
return ((rg - rg.mean(1, keepdim=True)) / (rg.std(1, keepdim=True) + 1e-4)).view(-1)
raise ValueError(method)
def train_pg(method, reward_fn=check, steps=300, B=32, G=8, lr=1e-3, seed=0):
"""Policy gradient with one gradient step per batch of samples. Returns the policy and a history of
(samples so far, mean reward, accuracy)."""
torch.manual_seed(seed)
gen = torch.Generator().manual_seed(seed)
policy = copy.deepcopy(sft_model)
opt = torch.optim.Adam(policy.parameters(), lr)
history = []
for step in range(steps):
x = prompts[torch.randint(len(prompts), (B,), generator=gen)].repeat_interleave(G, 0)
y = sample(policy, x, gen)
r = reward_fn(x, y)
A = advantages(r, G, method)
lp, mask = token_logprobs(policy, x, y)
loss = -(A * (lp * mask).sum(1)).mean()
opt.zero_grad(); loss.backward()
nn.utils.clip_grad_norm_(policy.parameters(), 1.0)
opt.step()
history.append(((step + 1) * B * G, r.mean().item(), accuracy(policy) if step % 10 == 9 else None))
return policy, history
def curve(history):
pts = [(n, a) for n, _, a in history if a is not None]
return [0] + [n for n, _ in pts], [accuracy(sft_model)] + [a for _, a in pts]
format_bonus = lambda x, y: 1.0 + check(x, y) # reward 2 for right, 1 for any answer: r + 1
runs = {
'REINFORCE, r ∈ {0, 1}': ('reinforce', check),
'REINFORCE, r + 1 ∈ {1, 2}': ('reinforce', format_bonus),
'baseline, r + 1 ∈ {1, 2}': ('baseline', format_bonus),
}
t0 = time.time()
results = {name: train_pg(method, reward_fn)[1] for name, (method, reward_fn) in runs.items()}
print(f'{time.time() - t0:.0f} s')
fig = go.Figure()
for (name, h), color in zip(results.items(), [GRAY, RED, NAVY]):
fig.add_trace(go.Scatter(x=curve(h)[0], y=curve(h)[1], name=name, line=dict(color=color)))
fig.update_layout(xaxis_title='responses sampled', yaxis_title='accuracy (probability of a right answer)', yaxis_range=[0, 1],
width=800, height=400, title='Policy gradient on the addition task')
fig
2 s
- RL raises the accuracy well beyond the 40% demonstrator: the checker can recognize answers it never demonstrated.
- With rewards in $\{0,1\}$, plain REINFORCE works: wrong answers get $A=0$ and are simply ignored.
- Adding 1 to every reward slows plain REINFORCE down: wrong answers are now pushed up too, just less than right ones, and the estimates are noisier. The baseline removes the shift.
(Lecture Demo 2) GRPO
Lecture: run from "The GRPO Objective" slide.
GRPO (Group Relative Policy Optimization; Shao et al., 2024) makes three changes:
- Group advantage. Standardize the rewards of the $G$ responses to the same prompt: $A_i=(r_i-\text{mean})/\text{std}$. No learned value network (critic) is needed.
- Reuse each batch for several steps. Sampling is the expensive part, so take $\mu$ gradient steps per batch. After the first step the samples come from an old policy $\pi_{\text{old}}$, so each token's term is weighted by the ratio $\rho_t=\pi_{\boldsymbol\theta}(y_t\mid\cdot)/\pi_{\text{old}}(y_t\mid\cdot)$ (importance sampling).
- Clip the ratio to $[1-\epsilon,1+\epsilon]$, so that one batch cannot move the policy far; and add a penalty $\beta\,\mathrm{KL}(\pi_{\boldsymbol\theta}\,\|\,\pi_{\text{ref}})$ that keeps the policy near the SFT model.
Per token: $\min\big(\rho_t A_i,\ \text{clip}(\rho_t,1-\epsilon,1+\epsilon)A_i\big)-\beta\,\mathrm{KL}_t$, averaged over the tokens of each response and then over responses.
def grpo_loss(lp, old_lp, ref_lp, mask, A, eps=0.2, beta=0.04):
"""GRPO loss (to minimize). lp, old_lp, ref_lp, mask: B x T token log-probs and mask; A: advantages (B)."""
ratio = torch.exp(lp - old_lp)
clipped = torch.clamp(ratio, 1 - eps, 1 + eps)
objective = torch.minimum(ratio * A[:, None], clipped * A[:, None])
kl = torch.exp(ref_lp - lp) - (ref_lp - lp) - 1 # unbiased, always >= 0 estimate of KL(pi || ref)
per_token = objective - beta * kl
return -((per_token * mask).sum(1) / mask.sum(1)).mean()
def train_grpo(reward_fn=check, steps=300, B=32, G=8, mu=4, lr=1e-3, eps=0.2, beta=0.04, seed=0):
torch.manual_seed(seed)
gen = torch.Generator().manual_seed(seed)
policy, ref = copy.deepcopy(sft_model), copy.deepcopy(sft_model)
opt = torch.optim.Adam(policy.parameters(), lr)
history = []
for step in range(steps):
x = prompts[torch.randint(len(prompts), (B,), generator=gen)].repeat_interleave(G, 0)
y = sample(policy, x, gen) # sampled from pi_old = the current policy
r = reward_fn(x, y)
A = advantages(r, G, 'group')
with torch.no_grad():
old_lp, mask = token_logprobs(policy, x, y)
ref_lp, _ = token_logprobs(ref, x, y)
for _ in range(mu): # mu gradient steps on the same samples
lp, _ = token_logprobs(policy, x, y)
loss = grpo_loss(lp, old_lp, ref_lp, mask, A, eps, beta)
opt.zero_grad(); loss.backward()
nn.utils.clip_grad_norm_(policy.parameters(), 1.0)
opt.step()
history.append(((step + 1) * B * G, r.mean().item(), accuracy(policy) if step % 10 == 9 else None))
return policy, history
t0 = time.time()
grpo_policy, results['GRPO (μ = 4)'] = train_grpo()
print(f'{time.time() - t0:.0f} s')
fig = go.Figure()
for name, color in [('REINFORCE, r ∈ {0, 1}', GRAY), ('baseline, r + 1 ∈ {1, 2}', NAVY), ('GRPO (μ = 4)', GOLD)]:
fig.add_trace(go.Scatter(x=curve(results[name])[0], y=curve(results[name])[1], name=name, line=dict(color=color, width=3)))
fig.update_layout(xaxis_title='responses sampled', yaxis_title='accuracy', yaxis_range=[0, 1], width=800, height=400,
title='GRPO takes four gradient steps per batch of samples')
fig
2 s
GRPO climbs fastest at first: it gets four gradient steps out of every batch of samples. The KL penalty keeps it close to the SFT model, so here it levels off slightly below plain REINFORCE. On this tiny problem, plain REINFORCE with 0/1 rewards is already strong; GRPO's advantages matter most for large models, where every sample is expensive and rewards vary in scale from prompt to prompt (standardizing within the group removes that scale).
x = torch.tensor([[7, 16]] * 8)
print('7 + 16, SFT model: ', [detok(y) for y in sample(sft_model, x, torch.Generator().manual_seed(3)).tolist()])
print('7 + 16, after GRPO:', [detok(y) for y in sample(grpo_policy, x, torch.Generator().manual_seed(3)).tolist()])
print(f'accuracy: SFT {accuracy(sft_model):.0%}, GRPO {accuracy(grpo_policy):.0%}')
7 + 16, SFT model: ['33', '23', '33', '33', '23', '23', '33', '23'] 7 + 16, after GRPO: ['23', '23', '23', '23', '23', '23', '23', '23'] accuracy: SFT 43%, GRPO 94%
What the clip does. For a token with $A>0$, once $\rho_t>1+\epsilon$ the clipped term is constant, so its gradient is zero: the token has already been pushed up enough for this batch. For $A<0$ the same happens below $1-\epsilon$. The next cell checks this directly.
lp = torch.log(torch.tensor([[1.5, 1.1, 0.7, 0.9]])).requires_grad_() # ratios 1.5, 1.1, 0.7, 0.9 (old_lp = 0)
for A in (1.0, -1.0):
lp.grad = None
grpo_loss(lp, torch.zeros(1, 4), lp.detach(), torch.ones(1, 4), torch.tensor([A]), beta=0.0).backward()
print(f'A = {A:+.0f}: ratio [1.5, 1.1, 0.7, 0.9] -> gradient on each token', lp.grad.numpy().round(3))
A = +1: ratio [1.5, 1.1, 0.7, 0.9] -> gradient on each token [[ 0. -0.275 -0.175 -0.225]] A = -1: ratio [1.5, 1.1, 0.7, 0.9] -> gradient on each token [[0.375 0.275 0. 0.225]]
With $A=+1$ the token at ratio 1.5 gets no gradient (already above $1+\epsilon=1.2$); with $A=-1$ the token at ratio 0.7 gets none (already below $0.8$).
Groups where every answer is right (or wrong) have standard deviation 0 and advantage 0: they teach nothing. As the policy improves, more groups are all right, so practical systems drop them and sample harder prompts (for example DAPO, Yu et al., 2025).
gen = torch.Generator().manual_seed(0)
for name, pol in [('SFT model', sft_model), ('after GRPO', grpo_policy)]:
x = prompts[torch.randint(len(prompts), (256,), generator=gen)].repeat_interleave(8, 0)
r = check(x, sample(pol, x, gen)).view(256, 8)
useless = ((r.sum(1) == 0) | (r.sum(1) == 8)).float().mean().item()
print(f'{name:11s}: {useless:.0%} of groups have all 8 answers right or all wrong (zero advantage)')
SFT model : 6% of groups have all 8 answers right or all wrong (zero advantage) after GRPO : 77% of groups have all 8 answers right or all wrong (zero advantage)
Reward Hacking
RL optimizes the reward you wrote, not the one you meant. Suppose the checker has a bug: it only checks the last digit of the answer. The policy learns to satisfy that checker.
def buggy_check(x, y):
"""Reward 1 if the last digit of the answer matches the last digit of a + b (the tens digit is never checked)."""
last = []
for row in y.tolist():
digits = row[:row.index(EOS)] if EOS in row else row
last.append(digits[-1] if digits else -1)
return (torch.tensor(last) == x.sum(1) % 10).float()
hacked_policy, hacked = train_grpo(reward_fn=buggy_check)
n, proxy, acc = zip(*[(n, np.mean([h[1] for h in hacked[max(0, i - 9):i + 1]]), a) for i, (n, _, a) in enumerate(hacked) if a is not None])
fig = go.Figure()
fig.add_trace(go.Scatter(x=n, y=proxy, name='reward from the buggy checker', line=dict(color=GOLD, width=3)))
fig.add_trace(go.Scatter(x=n, y=acc, name='true accuracy', line=dict(color=NAVY, width=3)))
fig.add_trace(go.Scatter(x=curve(results['GRPO (μ = 4)'])[0], y=curve(results['GRPO (μ = 4)'])[1],
name='true accuracy with the correct checker', line=dict(color=GRAY, dash='dash')))
fig.update_layout(xaxis_title='responses sampled', yaxis_range=[0, 1], width=800, height=400, title='Optimizing a flawed reward')
fig
x = torch.tensor([[7, 16]] * 8)
print('7 + 16 with the buggy checker:', [detok(y) for y in sample(hacked_policy, x, torch.Generator().manual_seed(3)).tolist()])
7 + 16 with the buggy checker: ['33', '23', '33', '33', '33', '33', '33', '23']
The reward climbs to nearly 1 while the true accuracy stays far lower: the policy has fixed the last digit, which the checker sees, but not the tens digit, which it never checks. Real examples include models that edit unit tests so that they pass, or that write longer answers because a reward model likes length. Checkers need to be as careful as the tests in a homework autograder.