(Notation: $V$ is the vocabulary — the finite set of tokens the model can emit — and $|V|$ its size. $\mathbf z \in \mathbb{R}^{|V|}$ are the logits, the network's unnormalised scores, one per token; $z_i$ is the entry for token $i$, and $\mathbf 1 = (1,\dots,1)$ is the all-ones vector of the same length.)
(a) Show $\operatorname{softmax}(\mathbf z)$ is a valid probability distribution: every entry is positive and the entries sum to 1.
(b) Prove shift-invariance: $\operatorname{softmax}(\mathbf z + c\mathbf 1) = \operatorname{softmax}(\mathbf z)$ for any constant $c$. Why does this justify subtracting $\max_j z_j$ before exponentiating — and why does that matter numerically?
(c) (numerical stability in practice) Take logits $\mathbf z = (1000,\ 1001,\ 1002)$. (i) What goes wrong if you compute $e^{z_i}$ directly in 64-bit floating point? (ii) Apply the shift from (b) with $c=-\max_j z_j$ and compute the softmax.
Temperature $T>0$ divides the logits before the softmax; write the resulting distribution
$$ P_T(W_t{=}i) \;=\; \frac{e^{z_i/T}}{\sum_j e^{z_j/T}}, $$
where $W_t$ is the random variable for the token emitted at step $t$ (capitals are random variables, lowercase their values), $z_i$ is the logit of token $i$, and the sum runs over the whole vocabulary; $T=1$ gives the model's own distribution. Below, $P_T(i)$ abbreviates $P_T(W_t{=}i)$.
Show $P_T$ concentrates on $\arg\max_i z_i$ as $T\to 0^+$ and tends to uniform as $T\to\infty$. (Hint: two logits $z_1>z_2$, analyse $P_T(1)/P_T(2)$.)
Give a distribution where top-$k$ ($k=2$) and top-$p$ ($p=0.9$) select different sets. What does this show?
Temperature, top-$k$ and top-$p$ change the next-token distribution by probability; none of them makes an output impossible. Constrained decoding (also guided generation, structured output) changes it by validity (Willard & Louf, 2023): a parser for the target language — a regular expression, a JSON Schema, a grammar — runs beside the generation, and in parser state $s_t$ only the tokens of the legal set $A(s_t) \subseteq V$ can be drawn:
$$ P'(W_t{=}i \mid W_{1:t-1}) \;=\; \frac{\mathbb{1}[\,i \in A(s_t)\,]\; P(W_t{=}i \mid W_{1:t-1})} {\sum_{j\in A(s_t)} P(W_t{=}j \mid W_{1:t-1})} $$
$W_{1:t-1}$ is the sequence generated so far, and the indicator $\mathbb{1}[\cdot]$ is $1$ where its condition holds and $0$ otherwise. After each draw the parser advances to $s_{t+1} = \delta(s_t, W_t)$; $A(s)$ is the set of tokens $a$ for which the transition $\delta(s,a)$ is defined. An implementation sets the logit of every illegal token to $-\infty$, which the softmax maps to probability $0$.
(a) Advice vs. enforcement. The prompt "reply with valid JSON only" and a JSON mask both raise the share of well-formed outputs. Argue with $P$ and $P'$ why only the mask guarantees a well-formed output. What does the probability that the model assigns to an illegal token indicate?
(b) Position in the sampler. A sampler applies temperature, then truncation (top-$k$ or top-$p$), then draws. Show that masking commutes with temperature scaling, and construct a distribution and a legal set $A(s_t)$ for which top-$p$ before masking leaves no legal token.
(c) Cost. (optional) A naive check tests every token of $V$ ($|V| \approx 10^5$) against the parser at every step. Willard & Louf (2023) write the target language as a finite-state machine with state set $Q$ and compute $A(s)$ for every $s \in Q$ once, before generation. Compare the cost per step of the two, state the offline cost in time and memory, and name a schema for which that cost is acceptable and one for which it is not.
(d) Syntax, not semantics. A mask enforces the schema {"tool_name": ..., "arguments": {...}}.
Give an output that is schema-valid and wrong, name the component of the agent that has to detect
it, and state the class of error that the mask removes.
(e) Unconstrained thought. A runtime constrains only the action of a ReAct step, not the thought. Give two reasons: one from the content of $A(s_t)$, one from the lecture's argument that intermediate tokens are serial computation. What is the expected observation when the whole reply is constrained? (Lab Exercise 2 measures it.)
For each task, state whether plain Chain-of-Thought suffices or ReAct (tools) is needed, and why:
"agent".report.md?(e) Plausible or faithful? The agent answers "42 lines" to (d) and its chain reads
"I opened report.md, counted the lines, and found 42.", but the trace contains no tool call.
An explanation is plausible to the degree that it convinces a human reader, and faithful to
the degree that it reflects the true reasoning process of the model
(Jacovi & Goldberg, 2020, §2). Which of the two is this
explanation? State the rule that the distinction gives for debugging an agent, and name the artifact
to trust in place of the chain.
ReAct is defined in one line (Yao et al., 2023, §2). At loop step $k$ the agent holds a context $c_k$ and selects $\hat a_k$ from the augmented action space
$$ \hat{\mathcal{A}} \;=\; \mathcal{A} \cup \mathcal{L}, $$
where $\mathcal{A}$ is the external action space (tool calls) and $\mathcal{L}$ the language space (free-form text). A thought $t_k \in \mathcal{L}$ returns no observation; its only effect is to append text to the context, and it does not advance the index $k$, which counts observations. An external action $a_k \in \mathcal{A}$ returns an observation $o_{k+1} \in \mathcal{O}$. A step with a thought gives $c_{k+1} = (c_k,\, t_k,\, a_k,\, o_{k+1})$, a step without a thought $c_{k+1} = (c_k,\, a_k,\, o_{k+1})$.
A run of the function run_agent of the lab tutorial The ReAct Loop with Tool Calls:
1 assistant: I need the two numbers added; the add(a, b) tool does that.
2 assistant: tool_calls=[ add(a=19, b=23) ]
3 tool: 42
4 assistant: The sum of 19 and 23 is 42.
(a) Classification. For each of lines 1–4, state whether it is a thought, an external action or an observation, and write the context update. Exactly one line is not an element of $\hat{\mathcal{A}}$. Which, and why?
(b) One space, not two phases. ReAct could have been defined as "think, then act". What does the union $\mathcal{A}\cup\mathcal{L}$ permit that a fixed think-then-act schedule does not? Which of the two definitions covers line 4?
(c) No observation. A thought only appends tokens to $c_k$. State one consequence for cost ($c_k$ is re-sent in full at every later step) and one for truth: was the content of a thought ever compared with the environment?
(d) Dense or sparse thoughts. Yao et al. (2023) find that thoughts should be dense for multi-hop reasoning and sparse for tasks with many environment actions. Take a trajectory of 50 steps and thoughts of 40 tokens; a thought emitted at step $k$ is re-sent in the prompt of each later step $k+1,\dots,50$. Compute the additional tokens of a dense policy (a thought at every step) over a sparse one (thoughts at steps 10, 20, 30, 40, 50) — first counting the thoughts alone, then counting every re-send. Which of the two numbers decides the design?
(e) The think tool. The lab tutorial Termination of a Run advertises think(thought) as a tool whose
implementation returns the constant "OK.". In the vocabulary of this exercise: into which set does
that tool move an element, and why is the constant return value the correct implementation?
Give one task that is better served by a general code tool (run_python(code); CodeAct,
Wang et al., 2024) and one that is better served by a
specific tool; justify. Which additional safeguard does a code tool require, and why?
The model keeps no state, so the loop assembles and sends the complete context on every call:
context sent to the LLM = history # system prompt + all turns + tool calls/results
+ tool schemas # the advertised tools
+ retrieved documents # (→ L-RAG)
+ long-term memory # (→ L-Memory)
(a) Cost and benefit. State one cost and one benefit of sending the complete context on every call.
(b) Growth. Which of the four components grow with the number of loop steps? Which are assembled anew by the runtime on each call instead of accumulated?
(c) Calculation. The system prompt has $F_{\text{sys}} = 300$ tokens, the tool schemas have $F_{\text{tools}} = 450$ tokens, and each loop step appends $d = 250$ tokens (the tool call and its result). Give the context size $|c_n|$ after $n$ steps and the cumulative number of tokens sent over all $n$ calls, and evaluate both at $n = 20$. Which of the two is billed, and what is its order of growth in $n$?
(d) Context limit. With a context window of 8192 tokens and the numbers of (c), at which step does the context no longer fit? Name the three standard measures that postpone that step — one for the history, one for retrieved material, one for procedural instructions — and the term of the formula that each reduces. Does one of them remove the re-sending?
(a) Whose field is it? Take the schema advertised in the lab's setup cell:
{ "type": "function",
"function": { "name": "add", "description": "Add two numbers.",
"parameters": { "type": "object",
"properties": { "a": {"type": "number"}, "b": {"type": "number"} },
"required": ["a", "b"] } } }
Sort every key into three groups and say who determines it:
Then: the model was never shown this JSON — the endpoint renders it into the model's own
template. What does the model actually condition on when it decides whether to call add?
(b) Portability. (optional) The OpenAI Chat-Completions format above is what OpenAI, Ollama and other
OpenAI-compatible endpoints accept. Providers' native APIs differ — Anthropic nests the schema
under input_schema, Gemini under function_declarations. Which of your three groups from (a) has
to change when you port a tool to another provider, and which does not? What does that tell you
about where the work of tool design actually sits?
The Agent–Computer Interface (ACI) is the interface that a tool presents to a model (Yang, Jimenez et al., 2024): the tool's name, its description, its argument names and types, and the shape of the result. The first three are the part of the tool schema that its author writes (the third group of Exercise 9); the shape of the result is stated in the description. Schluntz & Zhang (2024) report that on SWE-bench, a benchmark of GitHub issues to be resolved by a code change, they spent more time on optimising the tools than on the prompt.
A first version of a tool:
{"type": "function", "function": {
"name": "f",
"description": "file tool",
"parameters": {"type": "object",
"properties": {"p": {"type": "string"},
"m": {"type": "string"},
"n": {"type": "integer"}},
"required": ["p", "m", "n"]}}}
The tool returns n lines of the file at path p, from the start or from the end according to m
("head" or "tail"); p may be relative to the agent's working directory.
(a) Defects. List the defects of this schema as an interface for a model, and name for each
the design rule it violates: natural formats, a description written from the model's
perspective, poka-yoke arguments (mistake-proofing: the argument is shaped so that misuse is
hard, not merely forbidden), room to think, and testing on real inputs (Schluntz & Zhang,
2024). For p: what must the model have tracked over the whole trajectory for a relative path to be
correct, and how does an absolute path change that requirement?
(b) Corrected schema. Write the corrected schema: descriptive names, a description that a
colleague who cannot ask a question could act on, an enum where the set of values is fixed, an
explicit unit for a bare number, and a stated result shape. Which of the three groups of
Exercise 9(a) changed?
(c) Room to think. (optional) A tool whose first required field is a strict enum forces the model to
commit before it has reasoned. Name the two standard remedies and state, in terms of forward
passes, what they provide (compare Exercise 4(e)).
(d) Agent or workflow. (optional) Schluntz & Zhang (2024) reserve an agent — the model directs its own steps — for open-ended tasks with verifiable outcomes and recommend a workflow (LLM calls on predefined code paths) otherwise. State the property of a task that decides between the two, and classify the word-count round-trip of the lab tutorial Function-Calling Round-Trip.
Peer-reviewed papers
Preprints & other non-peer-reviewed papers
Blogs, documentation & other web sources