In [1]:
import plotly.io as pio; pio.renderers.default = "notebook_connected"

Lecture 24: Test-Time Compute and Agents (CS 189, Fall 2026)

This notebook builds a small agent: a language model (Qwen3-1.7B) that can call two tools, a calculator and a search over short course notes. The model only ever writes text. A tool call is text in a fixed format; our code parses it, runs the tool, and appends the result to the conversation. The loop that does this is the agent.

In [2]:
import ast
import json
import operator
import re

import torch
from transformers import AutoModel, AutoModelForCausalLM, AutoTokenizer

DEVICE = 'cuda' if torch.cuda.is_available() else ('mps' if torch.backends.mps.is_available() else 'cpu')
DTYPE = torch.float32 if DEVICE == 'cpu' else torch.bfloat16
print('device:', DEVICE)
device: mps

Loading the Model

Qwen3-1.7B was post-trained to call tools in the format of its chat template. We turn off its "thinking" mode so each turn is short, and decode greedily so the runs are repeatable.

In [3]:
tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen3-1.7B')
model = AutoModelForCausalLM.from_pretrained('Qwen/Qwen3-1.7B', dtype=DTYPE).to(DEVICE).eval()
W1119 13:41:20.239000 30734 site-packages/torch/distributed/elastic/multiprocessing/redirects.py:29] NOTE: Redirects are currently not supported in Windows or MacOs.

Tool 1: A Calculator

Language models are unreliable at multiplying long numbers token by token. A tool returns the exact answer. The docstring matters: transformers turns the signature and docstring into the JSON schema the model sees. We evaluate only arithmetic (no eval), since the model writes the expression.

In [4]:
def calculator(expression: str) -> str:
    """Evaluate an arithmetic expression such as "48729 * 9317".

    Args:
        expression: The arithmetic expression to evaluate.
    """
    ops = {ast.Add: operator.add, ast.Sub: operator.sub, ast.Mult: operator.mul, ast.Div: operator.truediv,
           ast.Pow: operator.pow, ast.USub: operator.neg}

    def ev(node):
        if isinstance(node, ast.Constant):
            return node.value
        if isinstance(node, ast.BinOp):
            return ops[type(node.op)](ev(node.left), ev(node.right))
        if isinstance(node, ast.UnaryOp):
            return ops[type(node.op)](ev(node.operand))
        raise ValueError('unsupported expression')

    return str(ev(ast.parse(expression.replace(',', ''), mode='eval').body))


calculator("48729 * 9317")
Out[4]:
'454008093'

Tool 2: Searching the Course Notes

Retrieval as nearest-neighbor search. An embedding model $e$ (a small sentence encoder trained with a contrastive loss) maps the query $q$ and every passage $d$ to unit vectors, and we return the passages with the highest score

$$\text{score}(q, d) = e(q)^{\mathsf{T}} e(d).$$

The embedding of a text is the mean of the encoder's output vectors over its tokens, normalized to length one.

In [5]:
NOTES = [
    "Lecture 21: REINFORCE estimates the policy gradient as the average of R(tau) times the gradient of log pi_theta over sampled trajectories.",
    "Lecture 22: GRPO samples G answers per prompt and uses the group mean reward as the baseline.",
    "Lecture 22: Reward hacking happens when the policy finds a way to get high reward without solving the task.",
    "Lecture 23: The KV cache stores keys and values of past tokens; for Llama 3 70B it takes about 0.3 MB per token.",
    "Lecture 23: Speculative decoding lets a small draft model propose k tokens that the large model verifies in one pass.",
    "Lecture 23: Temperature T divides the logits before the softmax; T near 0 is greedy decoding.",
    "Lecture 19: LoRA adds a low-rank update AB to a frozen weight matrix W0.",
    "Lecture 19: CLIP trains image and text encoders with a contrastive loss over a batch of N pairs.",
]

emb_tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/paraphrase-MiniLM-L6-v2')
emb_model = AutoModel.from_pretrained('sentence-transformers/paraphrase-MiniLM-L6-v2').eval()


@torch.no_grad()
def embed(texts):
    """Unit-length embeddings e(text): mean of the encoder outputs over the tokens."""
    batch = emb_tokenizer(texts, padding=True, truncation=True, return_tensors='pt')
    h = emb_model(**batch).last_hidden_state
    mask = batch['attention_mask'].unsqueeze(-1)
    e = (h * mask).sum(1) / mask.sum(1)
    return torch.nn.functional.normalize(e, dim=-1)


NOTE_EMB = embed(NOTES)


def search_notes(query: str) -> str:
    """Search the CS 189 lecture notes and return the 2 most relevant passages.

    Args:
        query: What to look for.
    """
    scores = embed([query]) @ NOTE_EMB.T
    top = scores[0].topk(2).indices.tolist()
    return '\n'.join(NOTES[i] for i in top)


print(search_notes("GRPO baseline"))
Lecture 22: GRPO samples G answers per prompt and uses the group mean reward as the baseline.
Lecture 19: CLIP trains image and text encoders with a contrastive loss over a batch of N pairs.

What a Tool Call Looks Like

The chat template puts the tool schemas into the system prompt. The model answers with a JSON object between <tool_call> tags, and a tool result goes back as a <tool_response>. Below is the text the model sees for one finished exchange (the slide "How Tool Calling Works" shows a shortened version).

In [6]:
example = [
    {"role": "user", "content": "What is 48,729 times 9,317?"},
    {"role": "assistant", "content": '<tool_call>\n{"name": "calculator", "arguments": {"expression": "48729 * 9317"}}\n</tool_call>'},
    {"role": "tool", "content": "454008093"},
]
print(tokenizer.apply_chat_template(example, tools=[calculator, search_notes], tokenize=False,
                                    add_generation_prompt=True, enable_thinking=False))
<|im_start|>system
# Tools

You may call one or more functions to assist with the user query.

You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name": "calculator", "description": "Evaluate an arithmetic expression such as \"48729 * 9317\".", "parameters": {"type": "object", "properties": {"expression": {"type": "string", "description": "The arithmetic expression to evaluate."}}, "required": ["expression"]}, "return": {"type": "string"}}}
{"type": "function", "function": {"name": "search_notes", "description": "Search the CS 189 lecture notes and return the 2 most relevant passages.", "parameters": {"type": "object", "properties": {"query": {"type": "string", "description": "What to look for."}}, "required": ["query"]}, "return": {"type": "string"}}}
</tools>

For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call><|im_end|>
<|im_start|>user
What is 48,729 times 9,317?<|im_end|>
<|im_start|>assistant
<tool_call>
{"name": "calculator", "arguments": {"expression": "48729 * 9317"}}
</tool_call><|im_end|>
<|im_start|>user
<tool_response>
454008093
</tool_response><|im_end|>
<|im_start|>assistant
<think>

</think>


(Lecture Demo) The Agent Loop

generate asks the model for one turn. parse_tool_calls finds the JSON between <tool_call> tags. run_agent repeats: generate, run any tool calls, append the results, until the model answers without a tool call or a step limit is reached. Everything the agent knows is in messages.

In [7]:
@torch.no_grad()
def generate(messages, tools, max_new_tokens=300):
    """One assistant turn, decoded greedily."""
    ids = tokenizer.apply_chat_template(messages, tools=list(tools.values()), add_generation_prompt=True,
                                        enable_thinking=False, return_tensors='pt', return_dict=True)['input_ids'].to(DEVICE)
    out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False)
    text = tokenizer.decode(out[0, ids.shape[1]:], skip_special_tokens=True).strip()
    return {"role": "assistant", "content": text}


def parse_tool_calls(text):
    return [json.loads(m) for m in re.findall(r'<tool_call>\s*(.*?)\s*</tool_call>', text, re.S)]


def run_agent(task, tools, max_steps=8):
    messages = [{"role": "user", "content": task}]
    for step in range(max_steps):
        reply = generate(messages, tools)      # the LLM writes one turn
        messages.append(reply)
        calls = parse_tool_calls(reply["content"])
        if not calls:                          # no tool call: final answer
            return reply["content"], messages
        for call in calls:
            result = tools[call["name"]](**call["arguments"])
            messages.append({"role": "tool", "content": result})
    return "step limit reached", messages

First the calculator. The model writes the call, our code computes the product, and the model reads the result.

In [8]:
TOOLS = {f.__name__: f for f in (calculator, search_notes)}
answer, messages = run_agent("What is 48,729 times 9,317?", TOOLS)
for m in messages:
    print(f"[{m['role']}] {m['content']}")
[user] What is 48,729 times 9,317?
[assistant] <tool_call>
{"name": "calculator", "arguments": {"expression": "48729 * 9317"}}
</tool_call>
[tool] 454008093
[assistant] The result of 48,729 times 9,317 is 454,008,093.

Then retrieval: the model chooses the search tool and answers from the passage it gets back (the demo on the slides).

In [9]:
TOOLS = {f.__name__: f for f in (calculator, search_notes)}
answer, messages = run_agent("What baseline does GRPO use? "
                             "Check the course notes.", TOOLS)
for m in messages:
    print(f"[{m['role']}] {m['content']}")
[user] What baseline does GRPO use? Check the course notes.
[assistant] <tool_call>
{"name": "search_notes", "arguments": {"query": "GRPO baseline"}}
</tool_call>
[tool] Lecture 22: GRPO samples G answers per prompt and uses the group mean reward as the baseline.
Lecture 19: CLIP trains image and text encoders with a contrastive loss over a batch of N pairs.
[assistant] GRPO uses a baseline of samples G answers per prompt and employs the group mean reward as the baseline.

A Failure

Two steps are needed here: look up the per-token size of the KV cache, then multiply by the number of tokens. Watch what this small model does. (The right answer is about 0.3 MB times 128,000, roughly 40 GB.)

In [10]:
answer, messages = run_agent("According to the course notes, how much memory does the KV cache of Llama 3 70B "
                             "take for 128,000 tokens? Use the calculator.", TOOLS)
for m in messages:
    print(f"[{m['role']}] {m['content']}")
[user] According to the course notes, how much memory does the KV cache of Llama 3 70B take for 128,000 tokens? Use the calculator.
[assistant] <tool_call>
{"name": "calculator", "arguments": {"expression": "128000 * 70 * 1000000"}}
</tool_call>
[tool] 8960000000000
[assistant] The KV cache for Llama 3 70B with 128,000 tokens takes approximately 896 terabytes of memory.

With greedy decoding on our machine, the model skipped the search and called the calculator with numbers it invented (the "70" from "70B" times a million), then reported the result with confidence. Nothing in the loop checks that the inputs to a tool came from somewhere. Larger models, models trained with RL on multi-step tool use, and prompts that ask the model to plan first all make this less common. Try them:

  1. Rephrase the task so it asks for the lookup first, then the computation.
  2. Turn on thinking (enable_thinking=True in generate) and compare the transcripts.
  3. Add a third tool, and check whether the model chooses the right one.