The ReAct Loop & Tool Use¶

Agentic AI · L-ReAct

How to sample tokens (softmax, temperature, top-$k$/$p$), the ReAct loop that turns a chat model into an agent, message types, and the system prompt / chat template.

Learning Objectives¶

After this week you can:

1. Compute softmax and analyse temperature $T$, top-$k$, top-$p$ — and explain constrained decoding, the difference between asking for a format and enforcing one.
2. Explain Chain-of-Thought, describe ReAct, and write the agent execution loop.
3. Explain function calling — tool schemas and how the model requests a tool.
4. Explain the message types UserMessage / AssistantMessage / ToolResultMessage.
5. Explain the system prompt, the chat template, and statelessness.

Recall (→ L-Intro): the model outputs a distribution over the vocabulary $P(W_t\mid W_{1:t-1})$. This week we open it up — logits → softmax → sample — and then loop with tools.

Sampling: From Logits to a Token — The Softmax¶

An LLM never emits a word directly: the network produces scores (one per vocabulary token), softmax turns them into a distribution, and we sample. The one-step pipeline:

$$ \underbrace{W_{1:t-1}}_{\text{previous tokens}}\;\xrightarrow{\text{Transformer}}\; \underbrace{\mathbf z\in\mathbb R^{|V|}}_{\text{logits: one score per token}}\; \xrightarrow{\text{softmax}}\; \underbrace{P(W_t\mid W_{1:t-1})}_{\text{distribution over }V}\; \xrightarrow{\text{sample}}\; w_t $$

• The neural network (today: Transformer) reads the previous tokens and outputs one logit $z_i$ per vocabulary token — an unnormalised score.
• softmax turns those logits into a probability distribution: $$ P(W_t=i\mid W_{1:t-1})=\operatorname{softmax}(\mathbf z)_i=\frac{e^{z_i}}{\sum_j e^{z_j}} $$
• We then sample the next token $w_t$ from that distribution (or take the most likely one).

Properties: every entry is positive and they sum to $1$ — a valid distribution. (Why? You prove it in the exercises.)

Next-Token Sampling¶

The softmax gives a distribution $P(\cdot)$ over the vocabulary $V$. To emit a token:

• Greedy (argmax): take the most probable token $w_t=\arg\max_i P(W_t = i \mid W_{1:t-1})$ — deterministic, but repetitive and prone to loops: model gets stuck in a cycle of repeating the exact same sequence of words or phrases over and over again.
• Random sampling (categorical draw): pick $w_t$ at random, weighted by $P$ — token $i$ with probability $P(i \mid W_{1:t-1})$. A single categorical draw.

In code, that weighted draw is one line:

w_t = np.random.choice(len(p), p=p)   # draw a token, weighted by the probabilities p

Then repeat autoregressively (chain rule, one step at a time): append $w_t$, feed $w_{1:t}$ back in, get the next distribution $P$ (resp. p), sample again — until an end-of-sequence token.

The decoding strategy (temperature, top-$k$, top-$p$ — next slides) reshapes $P$ before this draw, trading determinism for diversity (see below).

Temperature Scaling: The Determinism ↔ Creativity Knob¶

Rescale the logits by temperature $T$ before softmax, changing how peaked the distribution is:

$$ P_T(W_t=i \mid W_{1:t-1} )=\frac{e^{z_i/T}}{\sum_j e^{z_j/T}}, \qquad T>0 $$

• $T\to 0^+$: all mass on $\arg\max_i z_i$ → greedy / deterministic. (setting temperature=0.0 internally bypasses the softmax sampling and executes argmax(logits) directly to prevent a divide-by-zero error.)
• $T=1$: the model's native distribution.
• $T\to\infty$: → uniform (maximally random).

Why agents care:

• low $T$ → reliable, repeatable output — what you want for valid tool calls (this week);
• higher $T$ → diverse output — needed to generate different candidate thoughts in Tree-of-Thoughts.

How you then choose among many sampled sequence candidates — beam search vs. sampling, best-of-$N$, self-consistency — is the subject of tree search.

Truncated Sampling: Cut the Unreliable Tail¶

Sampling from the full distribution occasionally draws a very unlikely token from the long tail, which can derail the text. Both methods below zero out the tail, renormalise, then sample:

• Top-$k$: keep the $k$ highest-probability tokens, drop the rest, renormalise. Fixed count. Simple, but $k$ ignores the distribution's shape: too permissive when the model is confident (one obvious token), too restrictive when it is genuinely uncertain (many plausible tokens).

• Top-$p$ (nucleus sampling; Holtzman et al., 2020): keep the smallest set — the nucleus — whose cumulative probability $\ge p$, then renormalise. The nucleus *size*** (how many tokens it contains) is **adaptive: few when the distribution is peaked (model confident), many when it is flat. Typical $p \approx 0.9\text{–}0.95$.

Example — probabilities $(0.6,\,0.3,\,0.05,\,0.03,\,0.02)$:

• top-$k{=}2$ → keep $\{0.6, 0.3\}$, renormalise to $\{0.67, 0.33\}$.
• top-$p{=}0.9$ → keep $\{0.6, 0.3\}$ (cumulative $=0.9$); on a flatter distribution the same $p$ would keep more tokens.

In practice these are often combined (e.g. top-$p$ + temperature — temperature reshapes the distribution before truncation). Note the special case greedy = top-$k$ with $k{=}1$ (equivalently $T\to 0$).

Setting them in practice: all these knobs are parametrizable — temperature and top_p via the OpenAI client against Ollama; top_k / min_p via the native ollama library. You try them in lab.

Constrained Decoding — Enforcing a Format¶

While sampling controls output diversity through token probabilities, constrained decoding alters the generation pipeline by enforcing structural validity. This technique is essential when generating outputs in formal target languages, such as JSON, regular expressions, or context-free grammars (CFGs) where structural syntax must be strictly guaranteed.

During generation, a parser tracks the state $s_t$ of the target grammar. At each decoding step $t$, the parser defines the valid token subset $A(s_t) \subseteq V$. The unmasked probabilities are then renormalised over the legal subset, so illegal tokens receive probability zero:

$$ P'(W_t{=}i \mid W_{1:t-1}) \;=\; \frac{\mathbb{1}[\,i \in A(s_t)\,]\; P(W_t{=}i \mid W_{1:t-1})}{\sum_{j\in A(s_t)} P(W_t{=}j \mid W_{1:t-1})} $$

Prompting is advice; a mask is enforcement. Prompts asking for a specific format (e.g., "Answer only in JSON") merely adjust probability mass without eliminating malformed output paths. Constrained decoding makes invalid outputs structurally unreachable.

This mechanism ensures dependable native tool calling: generated action sequences conform to the expected JSON schema because invalid syntax paths are entirely disabled during sampling.

Choosing the Knob — Determinism vs. Creativity by Use Case¶

Same model, different temperature (and top_p) per job — pick for "how many acceptable answers are there?":

• Reproducibility: for tests / evals / a linter, pin temperature=0.0 (and a fixed seed if the API exposes one) so runs are comparable.
• Tune one knob. Usually leave top_p ≈ 0.9–0.95 and adjust temperature; cranking both at once makes output hard to predict.
• This is only the dial setting — the mechanics are above (T reshapes $P$ before the draw; top-$p$ trims the tail).

Two dials, not one. This is the parameter-level dial. The prompt-level dial — clarity, examples, a role, structure (Anthropic, 2026) — is the other, and usually the stronger one. Reach for the prompt first, the temperature second.

Decoding Strategy ≠ Decoding Speed¶

Every knob above changes the distribution you sample from: temperature rescales it, top-$k$ and top-$p$ truncate it. Different setting, different text.

A second family makes generation faster while sampling from the same distribution — speculative decoding: a small, cheap model drafts the next few tokens, the real model verifies them all in one pass, and a modified rejection-sampling rule guarantees the emitted tokens are distributed exactly as the real model's (Leviathan et al., 2023; Chen et al., 2023). Reported: 2–3× less wall-clock time, identical outputs.

Chain-of-Thought (CoT)¶

So far we have a fluent text generator and know how to sample from it. An agent must also reason before aswering or acting.

Chain-of-Thought (CoT; Wei et al., 2022): prompt the model to emit intermediate reasoning steps before the final answer ("think step by step"). CoT turns a single monolithic prediction into an explicit, multi-step trajectory.

Chain-of-Thought in Practice¶

At first glance CoT is a prompting trick. Under the hood it changes how much computation the Transformer can spend on a problem:

1. Compute per token — the "scratchpad" effect (Nye et al., 2021). An autoregressive Transformer performs a fixed number of operations (layers, attention heads, feed-forward passes) per generated token.
   • Without CoT: the model must solve a multi-step problem inside a single generation step — at the first token of the answer. That frequently exceeds the computational depth of its layers: with no intermediate tokens a fixed-depth transformer stays inside TC⁰ and cannot, at any width, simulate an automaton or solve linear equalities (Merrill & Sabharwal, 2023).
   • With CoT: every intermediate token is a scratchpad entry in the context window. Emitting 100 intermediate tokens buys $100\times$ more forward passes before the answer must be committed.

2. Causal attention turns steps into premises. Each token attends only to earlier tokens, so writing out step 1 makes step 2 cheaper: the intermediate result becomes part of the prefix, and the model conditions on its own sub-result instead of re-deriving it.

3. In native reasoning models (e.g. o1, DeepSeek-R1). CoT is no longer a manual prompt technique. Such models are trained — with reinforcement learning (DeepSeek-AI, 2025) — to produce a reasoning trajectory before answering, and to spend a larger "thinking budget" on harder problems. The reasoning is emitted as a separately marked span, which is what lets the runtime treat it differently from the answer.

Faithfulness of the Chain¶

An explanation is plausible if it is coherent and fits the answer; it is faithful only if it reflects what actually drove that answer (Jacovi & Goldberg, 2020). The two come apart: plant a bias in the input and accuracy moves by up to 36 %, while the chain names that bias in 1 of 426 explanations (Turpin et al., 2023). Whether that makes the chain false or merely incomplete is disputed — a chain that stays silent about the bias still measurably carries it into the answer (Zaman & Srivastava, 2026).

Either way the working rule is the same: debug with the chain, conclude from the artifact — the tool result, the test, the recomputation. Returns in (critics), (audit), → L-Eval (evaluation).

Open-Loop vs. Closed-Loop¶

CoT improves the reasoning, but the model still commits to the entire chain before anything happens in the world. The control-theory terms name the difference exactly:

• Open loop — execute a fixed plan, with no feedback from the result. A washing machine runs its programme whether or not the laundry ends up clean.
• Closed loop — each result is measured and fed back, steering the next step. A thermostat observes the temperature it caused and corrects.

Classic CoT is open-loop: the model reasons, but it cannot act on the outside world, gather new data, or notice that step 3 rests on a false premise from step 2. Every fact it uses comes strictly from its parametric memory (training data), so an early error is carried, undetected, to the end.

Acting changes the requirement. The moment a step has an effect — a query, a file write, an email — its outcome is information the model did not have, and the next step ought to depend on it. Feeding that outcome back is the closed loop, and that is the step from CoT to ReAct.

From CoT to ReAct¶

ReAct = Reasoning + Acting (Yao et al., 2023): Extends Chain-of-Thought (CoT) into a closed-loop feedback cycle. Instead of just reasoning upfront or acting blindly, the framework interleaves CoT-style Thoughts $t$ with concrete Actions $a$ (tool executions) and environment Observations $o$ (tool outputs):

$$\dots \longrightarrow\; t_k \;\longrightarrow\; a_k \;\longrightarrow\; o_k \;\longrightarrow\; t_{k+1} \; \longrightarrow\; \dots$$

ReAct grounds intermediate thoughts in real-world feedback—preventing hallucination drift and enabling self-correction when a tool call fails.

Three Loops: Reason, Act, ReAct¶

The same three paradigms drawn, because the shape is the argument (redrawn after Yao et al., 2023):

Read the arrows, not the boxes. ReAct is not a third mechanism — it is the two loops on the same model at the same time. And the left-hand loop is the whole idea: a reasoning trace is an arrow that leaves the LM and comes back to it without touching Env, so it produces no observation. That is what the next slide makes formal.

The Core Definition of ReAct¶

The loop above is the behaviour. The definition in Yao et al. (2023), §2, is one line, and it is worth having exactly.

An agent at step $k$ receives an observation $o_k \in \mathcal{O}$ and takes an action $a_k \in \mathcal{A}$ (under a policy $\pi(a_k \mid c_k)$), where the context is the whole history $$c_k = (o_1, a_1, \dots, o_{k-1}, a_{k-1}, o_k).$$

ReAct augments the action space Formally, ReAct simply expands the agent's available action space to include free-form language:

$$\hat{\mathcal{A}} = \mathcal{A} \cup \mathcal{L}$$

• $\hat{\mathcal{A}}$: The augmented action space.
• $\mathcal{A}$: The external action space (e.g., executing tool calls, searching APIs, querying databases).
• $\mathcal{L}$: The language space (free-form text reasoning).

An element $\hat a_k \in \hat{\mathcal{A}}$ is either an environment action ($a_k \in \mathcal{A}$) or a thought ($t_k \in \mathcal{L}$). Note: This is in contrast to Yao et al. (2023) where $\hat a_k$ is a thought.

Under ReAct, a thought ($t_k \in \mathcal{L}$) is treated as an action. Its defining feature is what it doesn't do:

A thought does not touch the external environment and generates no output observation. Its only effect is appending text to the agent's context $c_k$ — everything the model conditions on at loop step $k$ — so $c_{k+1} = (c_k, t_k)$.

That is the entire idea. Thinking is not a phase before acting — it is an action whose only effect is to write to the context. Everything else this week does (the message list, statelessness, the context budget) follows from taking that literally.

In-Context Learning vs. Post-Training for ReAct¶

While the original ReAct paper (Yao et al., 2023) focused heavily on prompting foundation models via in-context learning, relying solely on few-shot prompts in production proved expensive, context-heavy, and fragile—especially for smaller or open-weights models. Consequently, the field transitioned toward Trajectory Tuning, fine-tuning models directly on multi-turn ReAct execution traces. By internalizing reasoning patterns and tool-use mechanics directly into the model's weights, this approach drastically reduces context overhead while boosting reliability. Furthermore, it enables agents to seamlessly transition between internal thought processing and tool execution without depending on rigid, hand-crafted prompt templates.

Dense vs. Sparse Thoughts¶

Because thinking consumes context and execution time, how often an agent should generate a thought is a design trade-off:

• Dense Reasoning: For complex problem-solving (e.g., multi-step math or search), the agent alternates $t_k \;\rightarrow\; a_k \;\rightarrow\; o_k $ at every step $k$.
• Sparse Reasoning: For environment-heavy decision tasks (e.g., web navigation), thoughts "only need to appear sparsely in the most relevant positions", and the model decides for itself when to think. Forcing a thought before every simple click in a 50-step sequence wastes tokens (Yao et al., ReAct 2023).

The Execution Loop¶

$$ \begin{array}{l} \textbf{Prompt-ToolLoop}(\textit{user\_input}) \\ \hline \textbf{Input: } \textit{user\_input} \text{; the tool set; a step cap } \text{MAX\_STEPS} \text{; the session } \textit{history} \\ \textbf{Output: } \text{the assistant's final text} \\ \hline 1: \quad \textit{history}.\text{append}(\textit{user\_input}) \\ 2: \quad \textbf{for } \textit{step} = 1 \dots \text{MAX\_STEPS} \textbf{ do} \\ 3: \quad \qquad \textit{response} \gets \text{LLM}(\textit{history}, \textit{tools}) \\ 4: \quad \qquad \textit{history}.\text{append}(\textit{response}) \\ 5: \quad \qquad \textbf{if } \textit{response}.\text{tool\_calls} = \emptyset \textbf{ then return } \textit{response}.\text{text} \qquad \text{// a text-only turn IS the answer} \\ 6: \quad \qquad \textbf{for each } \textit{call} \in \textit{response}.\text{tool\_calls} \textbf{ do} \\ 7: \quad \qquad\quad \textbf{try } \textit{result} \gets \text{execute}(\textit{call}) \; \textbf{catch } e: \textit{result} \gets \text{"ERROR: "} + \text{message}(e) \\ 8: \quad \qquad\quad \textit{history}.\text{append}(\text{tool\_result}(\textit{call}.\text{id}, \textit{result})) \\ 9: \quad \textbf{return } \text{"stopped: step budget exhausted"} \\ \hline \end{array} $$

The loop terminates when the model's response contains no tool calls. This serves as an implicit signal that the agent has finished reasoning and is presenting its final answer. In contrast, the original ReAct paper relied on an explicit Finish action (or IsFinal signal) to exit the loop.

Function Calling: Defining & Advertising Tools¶

The model has no built-in knowledge of available tools — the same base model powers many agents, each with a different toolset, so every call must tell it which tools exist right now (it can't call what it doesn't know).

To let the model use a tool you advertise a schema — name, description, and a JSON-Schema of its parameters — via the $\textit{tools}$ argument of the model call:

{ "type": "function",
  "function": {
    "name": "get_weather",
    "description": "Get the current weather for a city.",
    "parameters": {
      "type": "object",
      "properties": { "city": { "type": "string" } },
      "required": ["city"] } } }

What is fixed, and what is yours.

• The envelope — which fields exist and how they nest — is fixed by the API you call, not by the model, and it differs between providers. The parameters follow the JSON Schema standard.
• What you own is the name, the description, and the argument definitions — and the description is what the model reasons over when deciding whether to call.
• The description teaches the model when to use the tool; the schema constrains the arguments it may produce.
• The model replies with $\textit{response}.\text{tool\_calls}$ — a list of {id, name, arguments} (arguments is a JSON string).
• The runtime parses it, validates it, runs the function, and returns the result tagged with the matching tool_call_id, which pairs result ↔ request — needed when a turn issues several calls.

Validating Tool Calls: Parsing vs. Policy¶

Because the response is just a raw string generated by an LLM sampler, it must pass three mechanical checks before any tool code executes:

1. Valid JSON: The string can be parsed.
2. Schema Conformity: Arguments match the advertised signature (types, required fields, enums validated via Pydantic or jsonschema).
3. Tool Existence: The requested tool actually exists.

Note on Constrained Decoding: Even if your provider uses structured outputs or tool-masking, keep these checks active—do not rely solely on constrained decoding to prevent ill-formed inputs.

Error Recovery in the Loop¶

A validation failure should never throw a user-facing exception. Instead:

• Intercept the error.
• Return the detailed validation failure as the tool result back to the context.
• Allow the model to observe its mistake and attempt a self-repair on the next turn.

Validation vs. Policy: Passing structural validation only proves the payload is well-formed, not that it is permitted. Checking runtime permissions—such as path access, domain whitelists, or transactional limits—is a policy question (enforced separately at security boundaries).

Definition: Native Tool Calling¶

Native tool calling refers to an API-level mechanism where a fine-tuned model accepts a structured $\textit{tools}$ schema parameter and directly produces a structured response object (e.g. $\textit{response}.\text{tool\_calls}$) rather than raw text. Providers may back this with constrained decoding, making schema-invalid arguments unreachable during sampling (OpenAI's strict mode, Ollama's format); where they do not, the schema is advice — and a malformed call is still possible.

Specific Tools vs. a Code Tool¶

The tool comes in two styles:

• Specific, predefined tools — e.g. get_weather(city), send_email(to, body). One clear capability each; easy to validate and keep safe.
• A general code tool (code as the action — CodeAct; Wang et al., 2024) — a single tool like run_python(code) whose argument is code. Far more expressive (loop, compose several steps, use variables within one action) — but it runs arbitrary code, so it needs strong sandboxing (→ L-Gateway) and is harder to constrain.

Both are the same JSON tool call; the code tool just carries code in arguments. Same ReAct loop — only the tool differs.

Standardising Tools: MCP (Model Context Protocol)¶

Definition. MCP is an open protocol / contract by which a server exposes tools, data (resources), and prompt-templates, and an agent's runtime discovers and communicates with them — decoupling tool providers from agents so a tool written once works with any MCP-capable runtime.

So far each tool is hand-wired into our runtime. MCP (Model Context Protocol) is an open standard (2024) that decouples tools from the agent: tools, data, and prompts are exposed by an MCP server, and any MCP-capable client (the agent runtime) can discover and call them — without bespoke glue per tool.

What It Standardises:¶

• tools — callable functions the model invokes;
• resources — data/context the app reads;
• prompts — reusable
• prompt templates the user picks (slash-commands / menu items) — not a tool-call format.

Why It Matters:¶

• Write a tool once, reuse it across agents & providers;
• grow a large toolset by plugging in servers (files, web, databases, coding tools — the domains of an agent's toolset).

Under the Hood: Still Function Calling¶

• MCP standardises discovery & transport; the model still emits the same structured tool call as before (the tool_calls schema from the Function Calling slide).

Security:¶

• an MCP server is external, possibly untrusted code/data → the gateway's security boundary and prompt-injection defences apply (→ L-Gateway).

Takeaway. The model knows nothing about MCP — the runtime does. The model just sees a list of tools and emits an ordinary tool call; the runtime handles discovery, transport (JSON-RPC), routing, and security.

Designing Tools Well: The Agent–Computer Interface (ACI)¶

Definition. The Agent–Computer Interface (ACI) is the interface a tool presents to a model: the tool's name, its description, its argument names and types, and the shape of the result handed back. It is the agent-side counterpart of a human user interface (Schluntz & Zhang, 2024).

These are the fields the Function Calling slide attributed to the developer. The call's JSON envelope belongs to the endpoint and is fixed; the content of those fields is a design problem, and it is the whole of the ACI.

A human UI is judged by whether a person can operate it without a manual; an ACI is judged by whether a model can operate it without guessing. Anthropic report investing as much effort in the ACI as in a human UI — on SWE-bench, tuning the tools took more effort than tuning the prompt. They give design rules for the ACI, see Schluntz & Zhang, 2024.

Message Types (what the loop stores)¶

The conversation was introduced in as a list of role-tagged messages — system, user, assistant, tool. Those role strings are what goes on the wire. In code you want a type per role, because each carries different fields — this is what the lab's $\textit{history}$ holds:

Same three things, named for code rather than for JSON. The new part this week is the pairing: an AssistantMessage may carry several tool_calls, and each needs its own ToolResultMessage carrying the matching tool_call_id.

A Concrete Trace¶

User asks "What's the weather in Berlin?", with a get_weather tool available.

history: [ system, User("What's the weather in Berlin?") ]

turn 1   LLM(history) → Assistant(tool_calls=[get_weather(city="Berlin")])   # no text yet
         runtime runs get_weather("Berlin") → "12°C, rainy"
         history += Assistant(tool_call) , ToolResult("12°C, rainy")

turn 2   LLM(history) → Assistant("It's 12°C and rainy in Berlin.")          # no tool call → STOP

As typed messages (previous slide), $\textit{history}$ grew to:

1. UserMessage("What's the weather in Berlin?")
2. AssistantMessage(tool_calls=[{name:"get_weather", arguments:{city:"Berlin"}}])
3. ToolResultMessage(tool_call_id=…, content="12°C, rainy")
4. AssistantMessage("It's 12°C and rainy in Berlin.") ← returned

Two turns, one tool call — and every turn re-sent the whole history.

The next cell runs this same round-trip against a real model.

The Role of tool_call_id¶

In a single turn, the model may issue multiple tool calls in parallel (e.g., get_weather("Berlin") AND get_weather("Paris")). The runtime returns multiple ToolResultMessage objects. The model uses tool_call_id to match each result back to its original request.

System Prompt, Chat Template & Statelessness¶

The model has no memory: the entire conversation is re-sent every turn.

• System prompt = standing instructions, sent first every time.
• Messages are role-tagged; a chat template wraps them in special tokens:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
...<|im_end|>
<|im_start|>assistant

This format is ChatML (<|im_start|> … <|im_end|>), used by e.g. Qwen — our lab model qwen2.5 — and originally OpenAI. Other families use different delimiters (Llama, Mistral).

Tool use adds a tool role for results, and the model's tool request is carried in the assistant turn — the exact tokens are model-specific.

Actual Context Contents¶

Each LLM call sends more than the conversation. The loop maintains $\textit{history}$ — the running conversation (system prompt + all prior turns + this turn's tool calls/results). Around it, the runtime assembles extra context on every call:

context sent to the LLM =
    history              # system prompt + all turns + this turn's tool calls/results
  + tool schemas         # the advertised tools
  + retrieved documents  # RAG results for this query
  + long-term memory     # relevant remembered facts

• $\textit{history}$ only grows — so when it gets too large it is compacted (summarised).
• Tool schemas, RAG, and memory are not in $\textit{history}$; the runtime injects them each call — memory & RAG are fetched by similarity (embeddings).
• All of it must fit the context window — which is exactly why retrieval and compaction exist: you cannot append forever.

This is context engineering: deciding what to assemble into the context each call — and, under a finite budget, what to retrieve or compact — is the discipline itself.

$\textit{history}$ is a construction, not a transcript. The runtime chooses what to send: it can prune, merge, or summarise turns (drop a redundant reply, elide a consumed tool output) — as long as it keeps a paired tool_call / tool_result together. The messages you send ≠ the raw conversation.

The Context Budget¶

The context window is a fixed token allowance, and every part of the context spends from it: more tool schemas or retrieved documents ⇒ less room for $\textit{history}$, and vice-versa.

Because the model is stateless, the whole budget is re-assembled and re-sent on every call — so each token costs latency and money every turn, not once.

Deciding what to include, retrieve, or compact to stay within this budget is a central agent-design problem — hence embeddings / retrieval (→ L-Embed / → L-RAG) and compaction (→ L-Memory). It is context engineering in practice.

The same reasoning governs procedural instructions. Rather than keeping every how-to resident in the system prompt, an agent can file them as skills — directories whose one-line description is all that occupies the context until a task matches it, at which point the full instructions load (→ L-Memory). Tool schemas, retrieval and skills are three instances of one move: index first, load on demand.

In Practice / This Week's Lab¶

• CLI coding agents are this loop, with file/shell tools instead of get_weather. A while loop, a growing message list, a json.loads.
• Lab (lecture-02-ReAct-exercises.ipynb), three parts:
  • A — sampling & decoding: softmax, temperature and top-$k$/top-$p$ by hand, then a constrained decoder that masks the tokens a grammar forbids.
  • B — the ReAct loop & tools: implement the tool functions behind the schemas, run a function-calling round-trip, build your own run_agent() from scratch (no framework) — the loop at the heart of any agent runtime — and the prompt-based fallback for models without native tool calling.
  • C — reflection: temperature and the model's opinion.
• Keep ollama serve running; tool calling gets unreliable below ~3B.

Summary¶

• softmax + $T$ / top-$k$ / top-$p$ shape token sampling (determinism ↔ diversity); constrained decoding masks the distribution to a grammar — a guaranteed format, and the reason native tool calls parse.
• CoT makes reasoning explicit (a token scratchpad); ReAct = reason + act in a loop — just-in-time planning (deliberate planning:).
• Function calling: advertise a tool schema each call; the model returns tool_calls; the runtime executes and feeds results back. MCP standardises exposing tools.
• The loop stores UserMessage / AssistantMessage / ToolResultMessage in a persistent, growing history.
• Model is stateless → the full context budget is re-assembled & re-sent each turn (motivates memory / RAG / compaction).

Next week: breaking a task apart — task decomposition & a coding/web agent.

Recommended Readings¶

• Wei, Jason, et al. "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models." Advances in Neural Information Processing Systems. Vol. 35. 2022. arXiv:2201.11903.
• Yao, Shunyu, et al. "ReAct: Synergizing Reasoning and Acting in Language Models." International Conference on Learning Representations. Vol. 2023. 2023. arXiv:2210.03629.
  • Take a look at Appendix C for examples of in-context learning.

References¶

• Anthropic. "Model Context Protocol." Open standard, 2024. — the specification behind the MCP slide: tools · resources · prompts, JSON-RPC transport, runtime-side discovery. https://modelcontextprotocol.io
• Anthropic. "Prompting Best Practices." Claude Platform Documentation, 2026. — prompt-level steering (clarity, examples, roles, XML). https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices
• Anthropic. "Stop Reasons and Fallback." Claude Platform Documentation, 2026. — the seven stop_reason values: finished is reported, not inferred from the absence of a tool call. https://platform.claude.com/docs/en/api/handling-stop-reasons
• Casademunt, Helena, et al. "Censored LLMs as a Natural Testbed for Secret Knowledge Elicitation." arXiv preprint arXiv:2603.05494 (2026). — dropping the chat template entirely, and how much of the trained behaviour does not survive it.
• Chen, Charlie, et al. "Accelerating Large Language Model Decoding with Speculative Sampling." arXiv preprint arXiv:2302.01318 (2023). — concurrent formulation; Chinchilla 70 B.
• DeepSeek-AI, et al. "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning." arXiv preprint arXiv:2501.12948 (2025).
• Dziri, Nouha, et al. "Faith and Fate: Limits of Transformers on Compositionality." Advances in Neural Information Processing Systems. Vol. 36. 2023. arXiv:2305.18654. — computation graphs; linearized subgraph matching; exponential error compounding (the error bound the expressivity results omit).
• Feng, Guhao, et al. "Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective." Advances in Neural Information Processing Systems. Vol. 36. 2023. arXiv:2305.15408.
• Gulli, Antonio. Agentic Design Patterns: A Hands-On Guide to Building Intelligent Systems. Springer Nature, 2025. — the tool-use pattern (Ch. 5) and MCP (Ch. 10); see the further reading below.
• Holtzman, Ari, et al. "The Curious Case of Neural Text Degeneration." International Conference on Learning Representations. Vol. 2020. 2020. arXiv:1904.09751.
• Hugging Face. "Chat templates." Transformers Documentation, 2026. — control tokens, the end-of-turn token, generation prompts, prefilling, and the separate reasoning field. https://huggingface.co/docs/transformers/en/chat_templating
• Jacovi, Alon, and Yoav Goldberg. "Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?" Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020. arXiv:2004.03685. — the plausible / faithful distinction.
• Leviathan, Yaniv, et al. "Fast Inference from Transformers via Speculative Decoding." International Conference on Machine Learning. Vol. 202. 2023. arXiv:2211.17192. — faster decoding, unchanged output distribution.
• Merrill, William, and Ashish Sabharwal. "The Parallelism Tradeoff: Limitations of Log-Precision Transformers." Transactions of the Association for Computational Linguistics, vol. 11, 2023. arXiv:2207.00729. — the no-CoT ceiling: log-precision transformers sit inside TC⁰.
• Merrill, William, and Ashish Sabharwal. "The Expressive Power of Transformers with Chain of Thought." International Conference on Learning Representations. Vol. 2024. 2024. arXiv:2310.07923. — the ladder: what $t(n)$ intermediate tokens buy, up to exactly P.
• Nguyen, Minh Nhat, et al. "Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs." International Conference on Learning Representations. Vol. 2025. 2025. Oral. arXiv:2407.01082. — confidence-scaled truncation: keep tokens with $P \ge \texttt{min\_p}\cdot p_{\max}$.
• Nye, Maxwell, et al. "Show Your Work: Scratchpads for Intermediate Computation with Language Models." arXiv preprint arXiv:2112.00114 (2021). — the empirical origin of the scratchpad effect.
• Sanoja, Damaso. "What Are AI Agents?" JetBrains, 2026. — the agent control loop (reason → tool? → act → observe → goal?). https://www.jetbrains.com/pages/ai-agents/what-are-ai-agents/
• Schaeffer, Rylan, et al. "Min-p, Max Exaggeration: A Critical Analysis of Min-p Sampling in Language Models." arXiv preprint arXiv:2506.13681 (2025). — the refutation: no superiority once the hyper-parameter budget is equalised.
• Schick, Timo, et al. "Toolformer: Language Models Can Teach Themselves to Use Tools." Advances in Neural Information Processing Systems. Vol. 36. 2023. arXiv:2302.04761.
• Schluntz, Erik, and Barry Zhang. "Building Effective Agents." Anthropic Engineering, 2024. — tool design (ACI); workflows vs. agents.
• Sclar, Melanie, et al. "Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design, or: How I Learned to Start Worrying about Prompt Formatting." International Conference on Learning Representations. Vol. 2024. 2024. arXiv:2310.11324. — the grammar of semantically equivalent prompt formats, the performance spread metric, and the finding that meaning-preserving formatting choices move accuracy by up to 76 points regardless of model size, instruction tuning or shot count.
• Turpin, Miles, et al. "Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting." Advances in Neural Information Processing Systems. Vol. 36. 2023. arXiv:2305.04388.
• Wang, Xingyao, et al. "Executable Code Actions Elicit Better LLM Agents." International Conference on Machine Learning. Vol. 235. 2024. arXiv:2402.01030.
• Wei, Jason, et al. "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models." Advances in Neural Information Processing Systems. Vol. 35. 2022. arXiv:2201.11903.
• Willard, Brandon T., and Rémi Louf. "Efficient Guided Generation for Large Language Models." arXiv preprint arXiv:2307.09702 (2023). — FSM-indexed masking; $O(1)$-average constrained decoding (the Outlines library).
• Yao, Shunyu, et al. "ReAct: Synergizing Reasoning and Acting in Language Models." International Conference on Learning Representations. Vol. 2023. 2023. arXiv:2210.03629. — the primary source for this week's loop: the augmented action space $\hat{\mathcal{A}} = \mathcal{A} \cup \mathcal{L}$ and the thought that returns no observation (§2), dense vs. sparse thoughts, the HotpotQA/FEVER and ALFWorld/WebShop results, and the manual success/failure analysis (Table 2).
• Yao, Shunyu, et al. "ReAct Prompting." GitHub repository, 2023. https://github.com/ysymyth/ReAct. — the reference implementation of the paper above: finish[answer] as the only exit, think[…] as a no-op action, and stop sequences as turn boundaries.
• Zaman, Kerem, and Shashank Srivastava. "Is Chain-of-Thought Really Not Explainability? Chain-of-Thought Can Be Faithful without Hint Verbalization." arXiv preprint arXiv:2512.23032 (2026). — hint verbalisation measures reporting, not faithfulness; the omission may be incompleteness.
• Zhao, Jiachen, et al. "LLMs Encode Harmfulness and Refusal Separately." Advances in Neural Information Processing Systems. Vol. 38. 2025. arXiv:2507.11878. — refusal is not formed until the post-instruction template tokens arrive.

Further reading — tool use. Gulli, A. (2025). Agentic Design Patterns, Ch. 5 — Tool Use (Function Calling) (Springer): the tool-use process, agents as tools, code execution, and extensions vs. function calling.

Further reading — MCP. Gulli, A. (2025). Agentic Design Patterns, Ch. 10 — Model Context Protocol (Springer): MCP vs. function calling, client/server architecture, tools · resources · prompts, transport (STDIO / HTTP+SSE), and the agent-friendly-API caveat.