In [1]:
import plotly.io as pio; pio.renderers.default = "notebook_connected"

Lecture 23: Generating Text (CS 189, Fall 2026)

This notebook samples text from a small open language model, compares decoding strategies, measures the effect of the KV cache, and implements speculative decoding.

Notebook section Slide
The Next-Token Distribution (Lecture Demo 1) Sampling demo
Temperature Temperature
Truncation: top-k, top-p, min-p Truncation Sampling
The KV Cache The KV Cache
Speculative Decoding (Lecture Demo 2) Speculative decoding demo

Models. Qwen3-1.7B is the target model and Qwen3-0.6B the draft model (they share a tokenizer). The first run downloads about 4.6 GB from Hugging Face. On a laptop the notebook takes a few minutes; set DEVICE below if you have a GPU.

In [2]:
import math
import time

import numpy as np
import torch
import plotly.express as px
import plotly.graph_objects as go
from plotly.subplots import make_subplots
from transformers import AutoModelForCausalLM, AutoTokenizer

DEVICE = 'cuda' if torch.cuda.is_available() else ('mps' if torch.backends.mps.is_available() else 'cpu')
DTYPE = torch.float32 if DEVICE == 'cpu' else torch.bfloat16
print('device:', DEVICE)
device: mps

Plotting Code

The function below draws the bar charts used in this notebook. It is not part of the course material.

In [3]:
def plot_next_token(probs, tokenizer, kept=None, top=15, title=''):
    """Bar chart of the `top` most likely next tokens; tokens removed by a sampler are gray."""
    p, idx = torch.topk(probs.float().cpu(), top)
    labels = [repr(tokenizer.decode([i])) for i in idx.tolist()]
    if kept is None:
        colors = ['#002676'] * top
    else:
        colors = ['#002676' if kept[i] else '#BFBFBF' for i in idx.tolist()]
    fig = go.Figure(go.Bar(x=labels, y=p.numpy(), marker_color=colors))
    fig.update_layout(title=title, yaxis_title='probability', width=800, height=350,
                      margin=dict(l=40, r=20, t=50, b=40))
    return fig

Loading the Models

In [4]:
tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen3-0.6B')
draft = AutoModelForCausalLM.from_pretrained('Qwen/Qwen3-0.6B', dtype=DTYPE).to(DEVICE).eval()
target = AutoModelForCausalLM.from_pretrained('Qwen/Qwen3-1.7B', dtype=DTYPE).to(DEVICE).eval()

def encode(text):
    return tokenizer(text, return_tensors='pt').input_ids.to(DEVICE)

@torch.no_grad()
def next_token_logits(model, ids):
    """Logits a for the token after `ids` (a 1 x n tensor of token ids)."""
    return model(ids).logits[0, -1].float()
W1117 15:56:24.745000 53156 site-packages/torch/distributed/elastic/multiprocessing/redirects.py:29] NOTE: Redirects are currently not supported in Windows or MacOs.

(Lecture Demo 1) The Next-Token Distribution

Lecture: run from the sampling demo slide.

The model maps a context to logits $\mathbf{a}\in\mathbb{R}^{V}$ over the vocabulary of $V$ tokens. The softmax turns them into the next-token distribution $p_i=\exp(a_i)/\sum_j\exp(a_j)$.

In [5]:
prompt = 'My favorite ice cream flavor is'
a = next_token_logits(target, encode(prompt))
p = torch.softmax(a, dim=-1)
print('vocabulary size V =', p.numel())
plot_next_token(p, tokenizer, title=f'Next token after "{prompt}"')
vocabulary size V = 151936

Temperature

Dividing the logits by a temperature $T>0$ before the softmax (Bishop, Section 12.3.2):

$$p_i(T)=\frac{\exp(a_i/T)}{\sum_j\exp(a_j/T)}\qquad\text{(Bishop, Eq. 12.35)}$$

  • $T\to0$: all the probability goes to the largest logit (greedy decoding).
  • $T=1$: the model's distribution.
  • $T\to\infty$: the uniform distribution.
In [6]:
def apply_temperature(a, T):
    """Next-token probabilities at temperature T (T = 0 means greedy)."""
    if T == 0:
        return torch.nn.functional.one_hot(a.argmax(), a.numel()).float()
    return torch.softmax(a / T, dim=-1)

def entropy_bits(p):
    p = p[p > 0]
    return -(p * torch.log2(p)).sum().item()

for T in (0.3, 1.0, 2.0):
    pT = apply_temperature(a, T)
    plot_next_token(pT, tokenizer, title=f'T = {T}: entropy {entropy_bits(pT):.2f} bits').show()

The same model and context produce very different text depending on $T$. Below, each temperature samples three continuations with a fixed seed.

In [7]:
@torch.no_grad()
def generate(model, prompt, n_new, sampler, seed=0, use_cache=True):
    """Autoregressive decoding. sampler(a) returns the probabilities to sample the next token from.

    With use_cache=True the keys and values of earlier tokens are kept (the KV cache), so each step
    processes only the newest token. With use_cache=False every step reprocesses the whole context.
    """
    g = torch.Generator().manual_seed(seed)
    ids = encode(prompt)
    out = model(ids, use_cache=use_cache)
    for _ in range(n_new):
        probs = sampler(out.logits[0, -1].float()).cpu()
        nxt = torch.multinomial(probs, 1, generator=g).to(DEVICE)
        ids = torch.cat([ids, nxt[None]], dim=1)
        if nxt.item() == tokenizer.eos_token_id:
            break
        if use_cache:
            out = model(nxt[None], past_key_values=out.past_key_values, use_cache=True)
        else:
            out = model(ids, use_cache=False)
    return tokenizer.decode(ids[0])

story = 'Once upon a time, a robot'
for T in (0.0, 0.7, 1.5):
    print(f'--- T = {T}')
    for seed in range(3 if T else 1):
        print(generate(target, story, 30, lambda a: apply_temperature(a, T), seed=seed))
--- T = 0.0
Once upon a time, a robot was born in the year 2023. It was programmed to do a certain task. The robot is now 10 years old.
--- T = 0.7
Once upon a time, a robot in a sterile, white room, with a red, green and blue light. The robot is named "Iris". It has a white, paper
Once upon a time, a robot was born. It was born with a very simple life, but the fact it was born in 2010 is all that matters. This
Once upon a time, a robot was a simple machine, able to perform one task. But now, the world is changing rapidly, and the robot is needed to perform more complex functions
--- T = 1.5
Once upon a time, a robot ZAAA started leather planes in deform « ??? » //Asนิยม

Okay, so I need to figure out what the answer is to this riddle
Once upon a time, a robot earns the South Beyond奇状的ห walnut fi ld empty پ their grasp itDMETHOD皋 design score efficiently prioritize㉡ activity cap essay customer� lime壁
Once upon a time, a robot has a pirate recorder reacts bc bean L...
Ben中国经济研讨会paper forever
bossﺍ.call doğ의 waarde
delta batch import method for sobject

Truncation: top-k, top-p, min-p

Sampling at $T=1$ occasionally picks a token from the long tail of unlikely tokens, and one bad token can derail the rest of the text. Truncation samplers remove the tail and renormalize what remains (Bishop, Section 12.3.2):

  • top-$k$: keep the $k$ most likely tokens.
  • top-$p$ (nucleus): keep the smallest set of most likely tokens whose probabilities sum to at least $p$.
  • min-$p$: keep tokens with $p_i\ge p_{\min}\cdot\max_jp_j$, a threshold relative to the most likely token.
In [8]:
def renormalize(p, keep):
    q = torch.where(keep, p, torch.zeros_like(p))
    return q / q.sum()

def top_k_keep(p, k):
    keep = torch.zeros_like(p, dtype=torch.bool)
    keep[torch.topk(p, k).indices] = True
    return keep

def top_p_keep(p, top_p):
    sorted_p, order = torch.sort(p, descending=True)
    before = torch.cumsum(sorted_p, 0) - sorted_p      # mass of the tokens ranked above each token
    keep_sorted = before < top_p                       # keep until the mass reaches top_p
    keep = torch.zeros_like(p, dtype=torch.bool)
    keep[order[keep_sorted]] = True
    return keep

def min_p_keep(p, min_p):
    return p >= min_p * p.max()

p = torch.softmax(a, dim=-1)
for name, keep in [('top-k, k = 5', top_k_keep(p, 5)), ('top-p, p = 0.8', top_p_keep(p, 0.8)),
                   ('min-p, p_min = 0.1', min_p_keep(p, 0.1))]:
    print(f'{name}: keeps {keep.sum().item()} tokens, {p[keep].sum().item():.2f} of the probability')
    plot_next_token(p, tokenizer, kept=keep.cpu(), title=name).show()
top-k, k = 5: keeps 5 tokens, 0.72 of the probability
top-p, p = 0.8: keeps 12 tokens, 0.81 of the probability
min-p, p_min = 0.1: keeps 4 tokens, 0.69 of the probability

The three samplers adapt differently: top-$k$ keeps a fixed number of tokens, while top-$p$ and min-$p$ keep few tokens when the model is confident and many when it is not. A common default for chat is $T\approx0.7$ with top-$p=0.9$.

In [9]:
nucleus = lambda a: renormalize(apply_temperature(a, 0.7), top_p_keep(apply_temperature(a, 0.7), 0.9))
for seed in range(3):
    print(generate(target, story, 30, nucleus, seed=seed))
Once upon a time, a robot in a factory produces a certain number of parts. The robot has a production rate of 10 parts per minute. The robot's efficiency is affected
Once upon a time, a robot was born. It was born in a very special year, 2015. The robot, named J, is very curious about the properties
Once upon a time, a robot was a simple machine, but with the advent of the modern era, the robot has become a very sophisticated machine. The history of the robot is one

The KV Cache

Generating token $n+1$ needs the keys and values of all $n$ earlier tokens in every layer. Without a cache they are recomputed at every step; with the cache each step computes the keys and values of one new token and appends them. The output is the same; the cost of a step drops from processing $n$ tokens to processing one.

In [10]:
def timed(f):
    if DEVICE == 'cuda':
        torch.cuda.synchronize()
    elif DEVICE == 'mps':
        torch.mps.synchronize()
    t0 = time.perf_counter()
    result = f()
    if DEVICE == 'cuda':
        torch.cuda.synchronize()
    elif DEVICE == 'mps':
        torch.mps.synchronize()
    return result, time.perf_counter() - t0

greedy = lambda a: apply_temperature(a, 0)
with_cache, t_cache = timed(lambda: generate(draft, story, 80, greedy, use_cache=True))
no_cache, t_none = timed(lambda: generate(draft, story, 80, greedy, use_cache=False))
print(f'with KV cache: {t_cache:.1f} s   without: {t_none:.1f} s   same text: {with_cache == no_cache}')
with KV cache: 4.6 s   without: 19.8 s   same text: True

The cache trades memory for time. Per token it stores a key and a value vector for every layer and every key-value head: $2\times\text{layers}\times\text{heads}_{kv}\times d_{\text{head}}$ numbers.

In [11]:
cfg = target.config
per_token = 2 * cfg.num_hidden_layers * cfg.num_key_value_heads * cfg.head_dim * 2  # bytes in bfloat16
print(f'Qwen3-1.7B: {per_token / 1024:.0f} KiB per token, {per_token * 32768 / 2**30:.1f} GiB for 32,768 tokens')
Qwen3-1.7B: 112 KiB per token, 3.5 GiB for 32,768 tokens

(Lecture Demo 2) Speculative Decoding

Lecture: run from the speculative decoding demo slide.

Decoding one token at a time reads every weight of the large model per token. Speculative decoding (Leviathan et al., 2023; Chen et al., 2023) lets a small draft model $q$ propose $k$ tokens, then checks all of them with one forward pass of the target model $p$:

  1. Draft $k$ tokens $x_1,\dots,x_k$ from $q$.
  2. Run the target once on the context plus the draft, which gives $p(\cdot)$ at all $k+1$ positions.
  3. For $i=1,\dots,k$: accept $x_i$ with probability $\min\left(1,\ p(x_i)/q(x_i)\right)$. At the first rejection, sample a replacement from the residual $\dfrac{\max(0,\ p-q)}{\sum_x\max(0,\ p(x)-q(x))}$ and stop.
  4. If all $k$ were accepted, sample one more token from the target's last distribution.

Every target call produces at least one token, and the accepted tokens are distributed exactly as if sampled from $p$.

In [12]:
def accept_or_resample(p, q, x, gen=None):
    """One step of the acceptance rule. Returns (accepted, token); token is x or the replacement."""
    if torch.rand((), generator=gen) < torch.clamp(p[x] / q[x], max=1.0):
        return True, x
    residual = torch.clamp(p - q, min=0)
    return False, torch.multinomial(residual / residual.sum(), 1, generator=gen).item()

@torch.no_grad()
def speculative_generate(prompt, n_new, k=4, T=1.0, seed=0):
    """Speculative decoding with `draft` proposing k tokens per target call.

    Written for clarity: each call reprocesses the whole context (no KV cache).
    Returns the text, the number of new tokens, the number of target calls, and the acceptance rate.
    """
    gen = torch.Generator().manual_seed(seed)
    ids = encode(prompt)
    n0, calls, accepted, proposed = ids.shape[1], 0, 0, 0
    while ids.shape[1] - n0 < n_new:
        proposal, qs = ids, []
        for _ in range(k):                                         # 1. draft k tokens
            q = apply_temperature(next_token_logits(draft, proposal), T).cpu()
            x = torch.multinomial(q, 1, generator=gen).item()
            qs.append(q)
            proposal = torch.cat([proposal, torch.tensor([[x]], device=DEVICE)], dim=1)
        logits = target(proposal).logits[0, -k - 1:].float()       # 2. one target call
        P = torch.stack([apply_temperature(l, T) for l in logits]).cpu()
        calls += 1
        n = ids.shape[1]
        for i in range(k):                                         # 3. accept or resample
            x = proposal[0, n + i].item()
            ok, tok = accept_or_resample(P[i], qs[i], x, gen)
            proposed += 1
            accepted += ok
            if not ok:
                ids = torch.cat([proposal[:, :n + i], torch.tensor([[tok]], device=DEVICE)], dim=1)
                break
        else:                                                      # 4. all accepted: one bonus token
            bonus = torch.multinomial(P[k], 1, generator=gen).item()
            ids = torch.cat([proposal, torch.tensor([[bonus]], device=DEVICE)], dim=1)
    return tokenizer.decode(ids[0]), ids.shape[1] - n0, calls, accepted / proposed

text, n_tokens, calls, alpha = speculative_generate(story, 60, k=4)
print(text)
print(f'\n{n_tokens} new tokens with {calls} target calls (instead of {n_tokens}); acceptance rate {alpha:.2f}')
Once upon a time, a robot exhibits the following behaviors: moving, speaking, and displaying emotion.研究人员 in the 1950s already thought about the question of artificial creativity. In the 1980s, computer scientists introduced the concept of ( fill in the blank ) of AI, which has since been widely used in the

63 new tokens with 30 target calls (instead of 63); acceptance rate 0.54

The output distribution is exactly p

The probability that the procedure outputs token $x$ at a position is

$$q(x)\min\left(1,\frac{p(x)}{q(x)}\right)+\Pr(\text{reject})\cdot\frac{\max(0,\,p(x)-q(x))}{\Pr(\text{reject})}=\min(p(x),q(x))+\max(0,\,p(x)-q(x))=p(x),$$

since $\Pr(\text{reject})=1-\sum_x\min(p(x),q(x))=\sum_x\max(0,\,p(x)-q(x))$. The check below draws 100,000 tokens with the rule, using the two models' actual next-token distributions for one context.

In [13]:
ids = encode(story)
p_target = apply_temperature(next_token_logits(target, ids), 1.0).cpu()
q_draft = apply_temperature(next_token_logits(draft, ids), 1.0).cpu()

gen = torch.Generator().manual_seed(0)
counts = torch.zeros_like(p_target)
for _ in range(100_000):
    x = torch.multinomial(q_draft, 1, generator=gen).item()
    _, tok = accept_or_resample(p_target, q_draft, x, gen)
    counts[tok] += 1
empirical = counts / counts.sum()
direct = torch.bincount(torch.multinomial(p_target, 100_000, replacement=True, generator=gen), minlength=p_target.numel()).float() / 100_000
tv = lambda a, b: 0.5 * (a - b).abs().sum().item()
print(f'total variation distance to p:  speculative samples {tv(empirical, p_target):.3f},  '
      f'samples drawn from p itself {tv(direct, p_target):.3f},  draft q {tv(q_draft, p_target):.3f}')
print(f'acceptance probability sum_x min(p, q) = {torch.minimum(p_target, q_draft).sum().item():.2f}')
total variation distance to p:  speculative samples 0.042,  samples drawn from p itself 0.042,  draft q 0.341
acceptance probability sum_x min(p, q) = 0.66

The speculative samples are as close to $p$ as 100,000 samples drawn from $p$ itself (the remaining distance is sampling noise spread over a vocabulary of 151,936 tokens), even though the draft $q$ is far from $p$.

How many tokens per target call?

If each draft token is accepted independently with probability $\alpha$, a target call produces $1+\alpha+\alpha^2+\dots+\alpha^k=\dfrac{1-\alpha^{k+1}}{1-\alpha}$ tokens on average. A larger $k$ helps only while $\alpha^k$ is not negligible, and each drafted token costs a draft call.

In [14]:
rows = []
for k in (1, 2, 4, 8):
    _, n_tokens, calls, alpha = speculative_generate(story, 60, k=k, seed=1)
    predicted = (1 - alpha ** (k + 1)) / (1 - alpha)
    rows.append((k, alpha, n_tokens / calls, predicted))
    print(f'k = {k}: acceptance {alpha:.2f}, tokens per target call {n_tokens / calls:.2f} (formula {predicted:.2f})')
k = 1: acceptance 0.67, tokens per target call 1.67 (formula 1.67)
k = 2: acceptance 0.76, tokens per target call 2.31 (formula 2.33)
k = 4: acceptance 0.68, tokens per target call 2.67 (formula 2.66)
k = 8: acceptance 0.61, tokens per target call 2.54 (formula 2.51)

Wall-clock time. This implementation counts target calls, the quantity that matters when the target is large and decoding is memory bound (a 70B model on a GPU). With a 1.7B target on a laptop, a forward pass is not memory bound and the draft calls are not much cheaper, so speculative decoding saves little or no time here. Production implementations (for example target.generate(..., assistant_model=draft) in Hugging Face, vLLM, SGLang) also use KV caches for both models.