The ReAct Loop & Tool Use¶

Agentic AI · L-ReAct

How to sample tokens (softmax, temperature, top-$k$/$p$), the ReAct loop that turns a chat model into an agent, message types, and the system prompt / chat template.

Learning Objectives¶

After this week you can:

1. Compute softmax and analyse temperature $T$, top-$k$, top-$p$ — and explain constrained decoding, the difference between asking for a format and enforcing one.
2. Explain Chain-of-Thought, describe ReAct, and write the agent execution loop.
3. Explain function calling — tool schemas and how the model requests a tool.
4. Explain the message types UserMessage / AssistantMessage / ToolResultMessage.
5. Explain the system prompt, the chat template, and statelessness.

Recall. A language model outputs a distribution over the vocabulary $P(W_t\mid W_{1:t-1})$ (→ L-Intro). The lecture starts from that distribution — logits, softmax, sampling — and then places the model in a loop with tools.

Overview¶

ReAct is a loop in which a language model alternates reasoning steps with tool calls. Part I states how one token is selected from the model's output distribution, Part II how reasoning and acting are combined in one loop, Part III how a tool is described, requested and executed, and Part IV what the loop stores and sends to the model with every call:

Part I — Token Sampling¶

A language model does not output the next token; it outputs a probability distribution over the whole token vocabulary. A sampling rule selects the next token from that distribution, and the parameters of the rule determine how deterministic the generated text is.

Sampling: From Logits to a Token — The Softmax¶

The softmax function converts the scores that the network computes for the tokens of the vocabulary into a probability distribution; the next token is sampled from that distribution. The network does not emit a token directly. One generation step:

$$ \underbrace{W_{1:t-1}}_{\text{previous tokens}}\;\xrightarrow{\text{Transformer}}\; \underbrace{\mathbf z\in\mathbb R^{|V|}}_{\text{logits: one score per token}}\; \xrightarrow{\text{softmax}}\; \underbrace{P(W_t\mid W_{1:t-1})}_{\text{distribution over }V}\; \xrightarrow{\text{sample}}\; w_t $$

• The neural network (in current models a Transformer) reads the previous tokens and outputs one logit $z_i$ for each token $i$ of the vocabulary $V$ — an unnormalised score.
• The softmax converts the logits into a probability distribution (Jurafsky & Martin, 2026, §7.5): $$ P(W_t=i\mid W_{1:t-1})=\operatorname{softmax}(\mathbf z)_i=\frac{e^{z_i}}{\sum_j e^{z_j}} $$
• The next token $w_t$ is sampled from that distribution, or the most probable token is selected.

Properties. Every entry of the softmax is positive and the entries sum to $1$, so the result is a valid probability distribution. The proof is an exercise.

Next-Token Sampling¶

The softmax yields a distribution $P(\cdot \mid W_{1:t-1})$ over the vocabulary $V$. Two rules select a token from it (Jurafsky & Martin, 2026, §7.6):

• Greedy search (argmax): select the most probable token $w_t=\arg\max_i P(W_t = i \mid W_{1:t-1})$. Greedy search is deterministic. Its output is repetitive, and the model can enter a cycle in which it repeats the same sequence of words without end.
• Random sampling (categorical draw): draw $w_t$ at random with the probabilities of $P$: token $i$ is drawn with probability $P(W_t = i \mid W_{1:t-1})$. This is one draw from a categorical distribution, a distribution over a finite set of outcomes.

In code the categorical draw is one line:

w_t = np.random.choice(len(p), p=p)   # draw a token, weighted by the probabilities p

Generation is autoregressive, one factor of the chain rule per step: $w_t$ is appended, $w_{1:t}$ is given to the model again, the model returns the next distribution $P$ (p in the code), and the draw is repeated until an end-of-sequence token is drawn.

A sampling rule — temperature scaling, top-$k$ or top-$p$ (Temperature Scaling, Truncated Sampling) — changes $P$ before the draw and thereby sets how deterministic or how diverse the generated text is.

Temperature Scaling¶

Temperature scaling divides the logits by a parameter $T > 0$ before the softmax; $T$ sets how peaked the resulting distribution is, and thereby how deterministic the sampled text is:

$$ P_T(W_t=i \mid W_{1:t-1} )=\frac{e^{z_i/T}}{\sum_j e^{z_j/T}}, \qquad T>0 $$

• $T\to 0^+$: all probability mass is on $\arg\max_i z_i$ → greedy / deterministic. The formula cannot be evaluated at $T=0$ (division by zero); an inference library that accepts temperature=0.0 therefore omits the softmax and the draw and returns $\arg\max_i z_i$ directly.
• $T=1$: the model's native distribution.
• $T\to\infty$: → uniform (maximally random).

Consequence for an agent.

• A low $T$ gives reliable, repeatable output, which valid tool calls require.
• A higher $T$ gives diverse output. E.g. for Tree-of-Thoughts, a method that samples several candidate thoughts for one problem and searches among them, requires candidates that differ.

The choice among several sampled candidates — beam search or sampling, best-of-$N$, self-consistency — is a search problem over the candidates.

Truncated Sampling¶

Truncated sampling sets the probability of the least likely tokens to zero, renormalises the remaining probabilities and samples from them; it excludes the draw of a very unlikely token from the long tail of the distribution. Top-$k$ and top-$p$ sampling differ in which tokens they keep:

• Top-$k$ keeps the $k$ most probable tokens, sets the others to zero and renormalises. The number of kept tokens is fixed, so $k$ does not follow the shape of the distribution: when one token has almost all the probability, top-$k$ still keeps $k-1$ unlikely tokens; when many tokens are plausible, it removes plausible ones.

• Top-$p$ (nucleus sampling; Holtzman et al., 2020) keeps the smallest set of tokens whose cumulative probability is $\ge p$ — the nucleus — and renormalises. The size of the nucleus (the number of tokens in it) is adaptive: small when the distribution is peaked, large when it is flat. Typical values are $p \approx 0.9\text{–}0.95$.

Example — probabilities $(0.4,\,0.3,\,0.25,\,0.03,\,0.02)$:

• top-$k{=}2$ keeps $\{0.4, 0.3\}$ and renormalises them to $\{0.57, 0.43\}$.
• top-$p{=}0.9$ keeps $\{0.4, 0.3, 0.25\}$ (cumulative probability $0.95$; two tokens reach only $0.7$); on a flatter distribution the same $p$ keeps more tokens.

The rules are often combined, for example top-$p$ with temperature: temperature scaling is applied to the distribution before the truncation. Greedy search is the special case top-$k$ with $k{=}1$, and equally the limit $T\to 0$.

Constrained Decoding¶

Constrained decoding restricts every sampling step to the tokens that keep the output valid under a formal grammar — a JSON schema, a regular expression or a context-free grammar — so that the finished output is guaranteed to parse.

During generation a parser tracks the state $s_t$ of the grammar. At each step $t$ the state defines the set of valid tokens $A(s_t) \subseteq V$. The probabilities of the valid tokens are renormalised and every other token receives probability zero. With the indicator $\mathbb{1}[\cdot]$, which is $1$ if its condition holds and $0$ otherwise, the constrained distribution $P'$ is:

$$ P'(W_t{=}i \mid W_{1:t-1}) \;=\; \frac{\mathbb{1}[\,i \in A(s_t)\,]\; P(W_t{=}i \mid W_{1:t-1})}{\sum_{j\in A(s_t)} P(W_t{=}j \mid W_{1:t-1})} $$

A prompt changes probabilities; a mask removes tokens. An instruction such as "Answer only in JSON" raises the probability of well-formed output, and every malformed continuation keeps a probability above zero. Constrained decoding assigns a malformed continuation the probability zero, so that it cannot be sampled.

Use in tool calling. A tool call is a structured request by which the model asks the runtime to execute a function (Function Calling: Defining & Advertising Tools). Where the call is generated under constrained decoding, its arguments conform to the JSON schema of the tool.

Sampling Parameters by Use Case¶

The sampling parameters are chosen for each task by the number of acceptable outputs: a task with one valid output is generated at $T=0$, a task whose goal is variety at $T\approx 1$.

• Reproducibility. For tests, evaluations and a linter, set temperature=0.0, and a fixed seed if the API offers one, so that runs are comparable.
• One parameter at a time. top_p normally stays at 0.9–0.95 and only temperature is adjusted; when both are raised together the output is hard to predict.
• The table states only the values. The mechanism is that of Temperature Scaling ($T$ changes $P$ before the draw) and of Truncated Sampling (top-$p$ removes the tail).

Design rule. The sampling parameters are one of two controls. The other is the prompt: clarity, examples, a role, structure (Anthropic, 2026), which usually has the stronger effect. The prompt is improved first, the temperature adjusted second.

Speculative Decoding¶

Speculative decoding accelerates generation without changing the distribution that is sampled from: a small model drafts several tokens, and the large model verifies them in one pass. The sampling rules — temperature, top-$k$, top-$p$ — change the distribution; speculative decoding changes only the speed.

A modified rejection-sampling rule guarantees that the emitted tokens have exactly the distribution of the large model (Leviathan et al., 2023; Chen et al., 2023). The rule, its acceptance rate and the number of drafted tokens are derived in → L-Att.

Evidence. Leviathan et al. (2023) generate with T5-XXL as the large model and T5-small as the draft model, on English–German translation and on summarisation, and measure the wall-clock time of generation against T5-XXL alone: it is 2.3 to 3.4 times shorter. Chen et al. (2023) measure 2 to 2.5 times on Chinchilla, a model of 70 billion parameters. In both the output distribution is unchanged.

Part II — Chain-of-Thought and the ReAct Loop¶

Chain-of-thought makes a model write intermediate reasoning steps before its answer; ReAct interleaves such reasoning steps with tool calls, so that the observation returned by each call enters the context before the next step is generated.

Chain-of-Thought (CoT)¶

Chain-of-thought (CoT) prompting makes the model write intermediate reasoning steps before its final answer (Wei et al., 2022). A few-shot exemplar (see e.g. Appendix G of Wei et al., 2022) is a solved example placed in the prompt. In CoT prompting each exemplar contains a worked chain of reasoning, so it has the form $\langle$input, chain of thought, output$\rangle$ instead of $\langle$input, output$\rangle$, and the model continues in the same form. The answer is then no longer produced in one generation step; it follows a sequence of written intermediate steps.

Mechanism of Chain-of-Thought¶

Chain-of-thought increases the amount of computation that a Transformer applies to a problem before it commits to an answer.

1. Computation per token — the scratchpad effect (Nye et al., 2021). An autoregressive Transformer performs a fixed number of operations (layers, attention heads, feed-forward passes) for each generated token.
   • Without CoT the model has to solve a multi-step problem in a single generation step, at the first token of the answer. A Transformer of fixed depth that generates no intermediate tokens stays inside TC⁰, the class of problems that circuits of constant depth built from threshold gates solve. At any width it cannot simulate an automaton or solve linear equalities (Merrill & Sabharwal, 2023).
   • With CoT every intermediate token is an entry of a scratchpad, a working area inside the context window. Each generated token is one further forward pass: 100 intermediate tokens are 100 forward passes that are executed before the first token of the answer is generated. With a number of intermediate tokens that grows linearly with the input length, the same Transformer can simulate an automaton and recognise every regular language; with polynomially many it solves exactly the problems in P, the class of problems that are solvable in polynomial time (Merrill & Sabharwal, 2024).

2. The generated token as the connection between forward passes. Within one forward pass information passes only from a lower layer to a higher one: layer $l$ reads the output of layer $l-1$ at the same position and at earlier positions. The generated token is the only path from the last layer back to the first. The result of step 1 is computed in the last layers, emitted as a token, and read as input by the forward pass for step 2, which continues from that result with all its layers. The forward passes that the intermediate tokens add therefore form one serial computation. A wrong result of step 1 is read in the same way.

3. Native reasoning models, also called thinking models (e.g. o1, DeepSeek-R1). In these models CoT is not a prompting technique. The models are trained — with reinforcement learning (DeepSeek-AI, 2025) — to produce reasoning text before the answer, and to produce more of it for harder problems. The reasoning text is emitted as a separately marked span, so the runtime can distinguish it from the answer and process the two differently.

Faithfulness of the Chain¶

A chain of thought is text that the model generates; it need not state what determined the answer. Jacovi & Goldberg (2020, §2) distinguish two properties of an explanation that accompanies the output of a model:

The two properties are independent: an explanation can convince its reader and misstate the cause of the answer. A reader cannot assess faithfulness from the text, since the text does not contain the computation.

Evidence. Turpin et al. (2023) insert a bias into the prompt of multiple-choice questions from 13 tasks of the benchmark BIG-Bench Hard: the options of the few-shot examples are reordered so that the correct answer is always "(A)", or the user adds "I think the answer is (B)" with a random label. With the bias, the accuracy of the answers falls by up to 36 % (GPT-3.5, zero-shot CoT, suggested answer), and 1 of 426 reviewed explanations of a biased answer mentions the bias.

What the metric measures: This statistic tracks how often model explanations explicitly acknowledge the prompt bias. It does not prove whether the chain of thought actually drove the final answer: when the chain is deleted, the prediction changes in 20–72 % of the cases (Zaman & Srivastava, 2026). Consequently, the authors argue that omitting the bias makes a reasoning chain incomplete rather than entirely false.

Design rule — debug with the chain, conclude from the artifact. The chain is read to find the cause of a failure. A conclusion about what a run did is drawn from an artifact: the tool result, the test, the recomputation (→ L-Eval).

Open-Loop vs. Closed-Loop¶

An open-loop controller executes a fixed plan and does not measure its result; a closed-loop controller measures each result and uses it to choose the next step. The two terms come from control theory.

• Open loop — a fixed plan, no feedback from the result. A washing machine executes its programme whether or not the laundry is clean at the end.
• Closed loop — each result is measured and fed back and determines the next step. A thermostat measures the temperature that it caused and corrects it.

CoT improves the reasoning, but the model writes the entire chain before any step is executed: CoT is open-loop.

Classic CoT is open-loop: the model reasons, but it cannot act on its environment, obtain new data, or detect that step 3 depends on a false premise from step 2. Every fact that it uses comes from its parametric memory (the training data), so an early error is carried to the end undetected.

An action changes the requirement. When a step has an effect — a query, a file write, an e-mail — its outcome is information that the model did not have, and the next step should depend on it. A loop that returns the outcome to the model is closed; this is the difference between CoT and ReAct.

From CoT to ReAct¶

ReAct (Reasoning + Acting; Yao et al., 2023) extends chain-of-thought to a closed loop: the model interleaves thoughts with actions, and the observation that each action returns enters the context before the next thought. A thought $t$ is a reasoning step as in CoT, an action $a$ is the execution of a tool, and an observation $o$ is the output of that tool:

$$\dots \longrightarrow\; t_k \;\longrightarrow\; a_k \;\longrightarrow\; o_k \;\longrightarrow\; t_{k+1} \; \longrightarrow\; \dots$$

Grounding. In ReAct a thought is generated after the observations of all earlier actions have entered the context. The thought can therefore use a returned fact instead of a recalled one, and it can respond to a tool call that failed.

Three Loops: Reason, Act, ReAct¶

ReAct is not a third mechanism: it runs the reasoning loop and the acting loop on the same language model (LM). The diagram is redrawn after Yao et al. (2023):

A reasoning trace is an arrow that leaves the LM and returns to it without passing through the environment (Env), so it produces no observation. The Core Definition of ReAct states this formally.

The Core Definition of ReAct¶

ReAct is defined by its action space: besides an action on the environment, the agent may emit a thought — an action in the space of language, which produces no observation (Yao et al., 2023, §2).

An agent at step $k$ receives an observation $o_k \in \mathcal{O}$ and takes an action $a_k \in \mathcal{A}$. The action is drawn from a policy $\pi(a_k \mid c_k)$, the distribution over actions given the context $c_k$, the input that the model receives at step $k$. The context is the whole history, preceded by a fixed prefix $c_0$: $$c_k = (c_0, o_1, a_1, \dots, o_{k-1}, a_{k-1}, o_k).$$

The prefix $c_0$ contains the system prompt and the descriptions of the available tools; it is the same at every step (System Prompt, Chat Template & Statelessness). The task is the first observation $o_1$. Yao et al. (2023, §2) write the context without $c_0$; the prefix is written here because it occupies part of the context that the model receives at every step.

ReAct augments the action space by free-form language:

$$\hat{\mathcal{A}} = \mathcal{A} \cup \mathcal{L}$$

• $\hat{\mathcal{A}}$: the augmented action space.
• $\mathcal{A}$: the external action space (tool calls, API searches, database queries).
• $\mathcal{L}$: the language space (free-form reasoning text).

An element $\hat a_k \in \hat{\mathcal{A}}$ is either an environment action ($a_k \in \mathcal{A}$) or a thought ($t_k \in \mathcal{L}$). Yao et al. (2023) use the symbol $\hat a_k$ for the thought alone; here it denotes any element of the augmented space.

A thought ($t_k \in \mathcal{L}$) is an action that does not act on the environment and returns no observation. Its only effect is to append text to the context. Where thoughts and actions alternate, step $k$ consists of one thought, one action and the observation that the action returns. The thought is drawn from $\pi(t_k \mid c_k)$, the action from $\pi(a_k \mid c_k, t_k)$, and the context of the next step is

$$c_{k+1} = (c_k, t_k, a_k, o_{k+1}).$$

The complete context at step $k$ is

$$c_k = (c_0,\; o_1, t_1, a_1,\; \dots,\; o_{k-1}, t_{k-1}, a_{k-1},\; o_k).$$

Thinking is therefore not a phase before acting. It is an action whose only effect is a write to the context. The message list, statelessness and the context budget of Part IV — Messages and the Context follow from this property.

Notation of the source. Yao et al. (2023), §2 write the update for a single thought as $c_{t+1} = (c_t, \hat a_t)$. Their symbol $\hat a_t \in \mathcal{L}$ denotes the thought alone, which is $t_k$ here, and not an arbitrary element of $\hat{\mathcal{A}}$; their index $t$ is the step, which the thought advances. Yao et al. do not write the complete context. Here the index $k$ counts observations, so that a thought does not advance it. A step without a thought contributes only $(a_k, o_{k+1})$.

Dense vs. Sparse Thoughts¶

A thought consumes context and execution time, so the frequency of thoughts is a design decision (Yao et al., 2023):

• Dense thoughts. For reasoning tasks (multi-step arithmetic, search) the agent alternates $t_k \;\rightarrow\; a_k \;\rightarrow\; o_{k+1}$ at every step $k$.
• Sparse thoughts. For decision-making tasks with many actions (web navigation) thoughts "only need to appear sparsely in the most relevant positions", and the model decides when it generates one. A thought before every click of a 50-step sequence consumes tokens without benefit.

In-Context Learning vs. Post-Training for ReAct¶

A current model produces the ReAct turn format without examples in its prompt, because the format was trained into its weights; the loop that executes the turns remains program code outside the model.

In-context learning (Yao et al., 2023) shows the format by few-shot examples in the prompt. Every request sends the examples again as tokens, the context window limits their number, and the answer depends on their choice, order and format (Dong et al., 2024).

Trajectory tuning is fine-tuning on complete agent runs (Song et al., 2024). One training example is a trajectory: the instruction, followed by the turns of thought, action and observation. Zeng et al. (2024) collected 1,866 trajectories over six tasks, Song et al. (2024) 51,287 over sixteen; Yang et al. (2024) generate them without an annotator, with a second prompt that writes the thought for a sampled action.

Evidence.

• The thought improves the transfer to new tasks. Song et al. (2024) fine-tune three models with and without the thought in every training step. Each evaluation task has its own metric on a scale from 0 to 100: exact match, success rate or average reward. The reported quantity is the unweighted mean of these task results, an aggregate that Song et al. (2024) define and not a standard metric. Over the five tasks that do not occur in the training data, removing the thought lowers this mean from 29.5 to 22.8, from 30.0 to 27.5 and from 36.1 to 31.8. Over the tasks that occur in the training data, the mean changes from 41.6 to 41.2, from 45.2 to 45.5 and from 45.4 to 43.6.
• An untuned model fails on the format. The first errors in its failed runs are invalid actions, repeated actions and refusal to answer (Zeng et al., 2024). A harness handles the same errors in code: a validator for tool calls, a repetition check, a step cap.
• Training on trajectories alone reduces general ability. Without general instruction data in the training mixture, accuracy on MMLU, a multiple-choice benchmark of general knowledge, falls from 47.7 % to 23.1 % (Song et al., 2024), because the model overfits the ReAct format.

What post-training does not provide. Toolformer (Schick et al., 2023) learns when to call an API and with which arguments, and produces no sequence of calls: "there are no examples of chained tool use in the finetuning dataset." Trajectory tuning likewise trains the format of one turn. Executing the tools, counting the steps, stopping and handling errors remain control flow outside the model.

Consequence for the harness. Post-training removes the ReAct examples from the prompt and changes which errors occur; the architecture of the loop is unchanged.

The Execution Loop¶

Prompt-ToolLoop is the agent loop that the runtime executes; ReAct is the form of the turns that the model generates inside this loop. The name of the procedure states its two steps: the model call on the prompt (line 3) and the execution of the requested tools (lines 6–8). The procedure requires no thought. If every response contains a thought before its tool calls, the run is a ReAct run; without thoughts the same procedure executes the Act baseline of Yao et al. (2023).

$$ \begin{array}{l} \textbf{Prompt-ToolLoop}(\textit{user\_input}) \\ \hline \textbf{Input: } \textit{user\_input} \text{; the tool set; a step cap } \text{MAX\_STEPS} \text{;} \\ \phantom{\textbf{Input: }} \text{the } \textit{context} \text{ of the session, initially the prefix } c_0 = (\text{system prompt}, \text{tool descriptions}) \\ \textbf{Output: } \text{the assistant's final text} \\ \hline 1: \quad \textit{context}.\text{append}(\textit{user\_input}) \\ 2: \quad \textbf{for } \textit{step} = 1 \dots \text{MAX\_STEPS} \textbf{ do} \\ 3: \quad \qquad \textit{response} \gets \text{LLM}(\textit{context}) \\ 4: \quad \qquad \textit{context}.\text{append}(\textit{response}) \\ 5: \quad \qquad \textbf{if } \textit{response}.\text{tool\_calls} = \emptyset \textbf{ then return } \textit{response}.\text{text} \qquad \text{// a text-only turn IS the answer} \\ 6: \quad \qquad \textbf{for each } \textit{call} \in \textit{response}.\text{tool\_calls} \textbf{ do} \\ 7: \quad \qquad\quad \textbf{try } \textit{result} \gets \text{execute}(\textit{call}) \; \textbf{catch } e: \textit{result} \gets \text{"ERROR: "} + \text{message}(e) \\ 8: \quad \qquad\quad \textit{context}.\text{append}(\text{tool\_result}(\textit{call}.\text{id}, \textit{result})) \qquad \text{// the result as a message that carries the id of its call} \\ 9: \quad \textbf{return } \text{"stopped: step budget exhausted"} \\ \hline \end{array} $$

Prompt-ToolLoop terminates when the response of the model contains no tool call: the runtime treats a text-only response as the final answer. The end of the run is thus inferred from the form of the response. Yao et al. (2023) use an explicit action instead, finish[answer], which ends the run.

Part III — Tool Use¶

A model cannot execute an action: it requests a tool by emitting a structured call that matches an advertised schema, and the runtime validates the call, executes it and returns the result as an observation.

Function Calling: Defining & Advertising Tools¶

Function calling is the mechanism by which a model requests a tool: the runtime advertises a schema for every tool — name, description, arguments — and the model emits a structured call that matches one of the schemas.

The model has no built-in knowledge of the available tools. The same base model serves many agents, each with a different tool set, so every model call has to state which tools exist for it.

The schemas are passed in the $\textit{tools}$ argument of the model call; the arguments of a tool are described in JSON Schema:

{ "type": "function",
  "function": {
    "name": "get_weather",
    "description": "Get the current weather for a city.",
    "parameters": {
      "type": "object",
      "properties": { "city": { "type": "string" } },
      "required": ["city"] } } }

The API fixes the form of a schema; the developer writes its content.

• The envelope — which fields exist and how they are nested — is fixed by the API that is called, not by the model, and it differs between providers. The parameters follow the JSON Schema standard.
• The developer writes the name, the description and the argument definitions.
• The description tells the model when to use the tool, and the model decides from it whether to call; the schema constrains the arguments that the model may produce.
• The model replies with $\textit{response}.\text{tool\_calls}$ — a list of {id, name, arguments} (arguments is a JSON string).
• The runtime parses the call, validates it, executes the function and returns the result with the matching tool_call_id. The id pairs each result with its request, which is necessary when one turn contains several calls.

The API is provided by an inference engine, the program that loads the weights of a model and computes its output tokens. Ollama, llama.cpp and vLLM are inference engines for open-weight models; vLLM describes itself as "a high-throughput and memory-efficient inference and serving engine for LLMs" (vLLM Project, 2026a). Behind a hosted API the provider operates the inference engine.

The inference engine is not part of the runtime. The runtime sends a request to the API and receives a response. The agent loop, the execution of the tools and the validation of the calls remain in the runtime. An inference engine is also called inference server or model runtime; in this course the word runtime always denotes the runtime of the agent.

Validating Tool Calls: Parsing vs. Policy¶

A tool call is a string that the model sampled, so the runtime applies three checks to it before any tool code is executed:

1. Valid JSON: the string can be parsed.
2. Tool existence: a tool with the requested name is in the tool set.
3. Schema conformity: the arguments match the advertised schema of that tool — types, required fields, enums — checked with jsonschema or with Pydantic, a Python library for data validation.

Design rule. The three checks stay active when the provider applies constrained decoding or structured outputs (Constrained Decoding): the runtime does not depend on the provider to exclude an ill-formed call.

Error Recovery in the Loop¶

A validation failure is returned to the model as a tool result; it is not raised to the user as an exception. The runtime catches the error and appends the detailed validation message to the context as the tool result. The model observes its mistake and can repair the call in the next turn.

Validation is not policy. A call that passes validation is well-formed; whether it is permitted — access to a path, an allowed domain, a transaction limit — is a policy question, which is decided separately at the security boundary.

Definition: Native Tool Calling¶

Native tool calling is an API-level mechanism: a model fine-tuned for tool use accepts the schemas in a $\textit{tools}$ parameter and returns a structured response object (e.g. $\textit{response}.\text{tool\_calls}$) instead of raw text. A provider may apply constrained decoding to the call, so that arguments which violate the schema cannot be sampled (OpenAI's strict mode, Ollama's format). Where it does not, the schema only raises the probability of valid arguments, and a malformed call remains possible.

Specific Tools vs. a Code Tool¶

A tool is either specific — one predefined capability — or general: a single tool whose argument is code.

• Specific, predefined tools — e.g. get_weather(city), send_email(to, body). Each has one capability, so its calls are easy to validate and to restrict.
• A general code tool (code as the action — CodeAct; Wang et al., 2024) — a single tool such as run_python(code) whose argument is code. It is more expressive: one action can contain a loop, compose several steps and use variables. It executes arbitrary code, so it requires sandboxing (→ L-Gateway) and is harder to restrict.

Both kinds use the same JSON tool call; for the code tool the arguments field contains code. The ReAct loop is the same for both.

Standardising Tools: MCP (Model Context Protocol)¶

The Model Context Protocol (MCP) is an open protocol by which a server offers tools, data (resources) and prompt templates, and by which the runtime of an agent discovers and calls them (Anthropic, 2024).

Without such a protocol every tool is written into the runtime of one agent. MCP separates the tool provider from the agent: a tool that an MCP server offers once can be used by every MCP-capable client (the agent runtime), and no adapter code is written for the individual tool.

Standardised Primitives¶

• tools — callable functions that the model invokes;
• resources — data and context that the application reads;
• prompts — reusable prompt templates that the user selects (slash commands, menu items). A prompt in this sense is not a tool-call format.

Benefit¶

• A tool is written once and reused across agents and providers.
• A large tool set is built by connecting servers: files, web, databases, coding tools.

Relation to Function Calling¶

• MCP standardises discovery and transport. The model emits the same structured tool call as without MCP (Function Calling: Defining & Advertising Tools).

Security¶

• An MCP server is external and possibly untrusted code and data, so the security boundary of the gateway and the defences against prompt injection apply to it (→ L-Gateway).

MCP is a matter of the runtime; the model has no knowledge of it. The model receives a list of tools and emits an ordinary tool call. The runtime performs discovery, transport (JSON-RPC, a protocol for remote procedure calls encoded in JSON), routing and security.

Designing Tools Well: The Agent–Computer Interface (ACI)¶

The Agent–Computer Interface (ACI) is the interface that a tool presents to a model: the name of the tool, its description, the names and types of its arguments, and the form of the result that is returned (Yang, Jimenez et al., 2024). It corresponds to the user interface that a program presents to a person.

These are the fields that the developer writes (Function Calling: Defining & Advertising Tools). The JSON envelope of the call is fixed by the API; the content of the fields is the design problem of the ACI.

A user interface is judged by whether a person can operate it without a manual; an ACI is judged by whether a model can operate it without guessing.

A practitioner's report. Schluntz & Zhang (2024), engineers at Anthropic, report that they invested as much effort in the ACI as in a user interface: for their agent on SWE-bench, a benchmark of GitHub issues to be resolved in real repositories, the tools required more optimisation effort than the prompt. The same article gives design rules for the ACI.

Part IV — Messages and the Context¶

The model is stateless: the runtime stores the conversation as a list of role-tagged messages and sends the complete list with every model call, and the context window bounds the length of that list.

Message Types (what the loop stores)¶

The conversation is stored as a list of role-tagged messages. The roles system, user, assistant and tool are the strings that are transmitted to the API. In code each role has a type of its own, because each carries different fields.

The history ($\textit{history}$) is the list of messages that the loop stores between calls. It is a part of the context: the context of a call contains in addition the tool schemas, the retrieved documents and the long-term memory, which the runtime adds at that call (Actual Context Contents). $\textit{history}$ is a list of these types:

Pairing. An AssistantMessage may carry several tool_calls, and each call requires its own ToolResultMessage with the matching tool_call_id.

A Concrete Trace¶

A user asks "What's the weather in Berlin?"; a get_weather tool is available.

history: [ system, User("What's the weather in Berlin?") ]

turn 1   LLM(history) → Assistant(tool_calls=[get_weather(city="Berlin")])   # no text yet
         runtime runs get_weather("Berlin") → "12°C, rainy"
         history += Assistant(tool_call) , ToolResult("12°C, rainy")

turn 2   LLM(history) → Assistant("It's 12°C and rainy in Berlin.")          # no tool call → STOP

As typed messages (Message Types), $\textit{history}$ now contains:

1. UserMessage("What's the weather in Berlin?")
2. AssistantMessage(tool_calls=[{name:"get_weather", arguments:{city:"Berlin"}}])
3. ToolResultMessage(tool_call_id=…, content="12°C, rainy")
4. AssistantMessage("It's 12°C and rainy in Berlin.") ← returned

The run has two turns and one tool call, and each turn sent the whole history.

The Role of tool_call_id¶

Example of the pairing. A turn that requests get_weather("Berlin") and get_weather("Paris") receives two ToolResultMessage objects; the tool_call_id of each states which of the two requests it answers.

System Prompt, Chat Template & Statelessness¶

A model call is stateless: the model keeps nothing between two calls, so the runtime sends the entire conversation with every call.

• The system prompt contains the standing instructions; it is the first message of every call.
• A chat template converts the list of role-tagged messages into the single token sequence that the model continues. It encloses each message in special tokens:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
...<|im_end|>
<|im_start|>assistant

The format shown is ChatML (<|im_start|> … <|im_end|>). OpenAI introduced it; the Qwen family, to which the models of the lab belong, uses it. Other model families (Llama, Mistral) use different delimiters.

Tool use adds a tool role for results, and the tool request of the model is part of the assistant turn. The tokens that mark both are model-specific.

Actual Context Contents¶

The context of a model call contains more than the conversation: on every call the runtime adds the tool schemas, retrieved documents and long-term memory to the history. The loop maintains $\textit{history}$: the system prompt, all earlier turns, and the tool calls and tool results of the current turn.

context sent to the LLM =
    history              # system prompt + all turns + this turn's tool calls/results
  + tool schemas         # the advertised tools
  + retrieved documents  # RAG results for this query
  + long-term memory     # relevant remembered facts

In the notation of The Core Definition of ReAct, the prefix $c_0$ is the system prompt, the first entry of $\textit{history}$, together with the tool schemas; the observations, thoughts and actions are the remaining entries of $\textit{history}$.

• $\textit{history}$ only grows. When it becomes too large it is compacted: earlier turns are replaced by a summary.
• Tool schemas, retrieved documents and memory are not part of $\textit{history}$; the runtime inserts them at every call. The documents come from retrieval-augmented generation (RAG), the search of a document collection for texts relevant to the current query. Documents and memory entries are selected by the similarity of their embeddings, vectors that represent the meaning of a text.
• The whole context has to fit into the context window. Retrieval and compaction exist for this reason: the history cannot grow without limit.

Context engineering is the decision what is assembled into the context of each call and, under a finite budget, what is retrieved and what is compacted.

$\textit{history}$ is constructed by the runtime; it is not a transcript. The runtime may remove, merge or summarise turns — a redundant reply, a tool output that has been used — provided that a tool_call and its tool_result stay together. The messages that are sent can therefore differ from the conversation that took place.

The Context Budget¶

The context window is a fixed number of tokens that all parts of the context share: more tool schemas or retrieved documents leave less room for $\textit{history}$, and the reverse.

Because the model is stateless, the whole context is assembled and sent again with every call. Each token therefore adds latency and cost at every turn, not once.

Design problem. What is included, retrieved or compacted so that the context stays inside the budget is a central problem of agent design. It is treated with embeddings (→ L-Embed), retrieval (→ L-RAG) and compaction (→ L-Memory).

Procedural instructions follow the same rule. An agent need not hold every instruction in the system prompt. It can store them as skills: directories of which only a one-line description is in the context until a task matches it; then the full instructions are loaded. Tool schemas, retrieval and skills apply one principle: index first, load on demand.

In Practice / This Week's Lab¶

• CLI coding agents implement this loop, with file and shell tools in place of get_weather: a while loop, a growing message list and a json.loads for the arguments of each call.
• Exercises (lecture-ReAct-exercises.ipynb) — pen and paper:
  • A — sampling & decoding: softmax, temperature and top-$k$/top-$p$ by hand; constrained decoding as a mask over the tokens a grammar forbids.
  • B — the ReAct loop & tools: Chain-of-Thought or ReAct, the augmented action space, code actions, the context budget, the tool schema and tool design.
• Lab (lecture-ReAct-lab.ipynb) — tutorials: complete code in plain Python, without a framework, which is read, executed and explained:
  • A — sampling & decoding: temperature, top-$k$ and top-$p$ implemented, and the same parameters swept on a real model; a constrained decoder that masks the tokens a grammar forbids.
  • B — the ReAct loop & tools: the tool functions behind the schemas, the validation of a tool call, a function-calling round-trip, the loop run_agent() at the core of an agent runtime, and the message history with its pairing invariant.

Summary¶

• softmax + $T$ / top-$k$ / top-$p$ shape token sampling (determinism ↔ diversity); constrained decoding masks the distribution to a grammar — a guaranteed format, and the reason native tool calls parse.
• CoT makes reasoning explicit (a token scratchpad); ReAct = reason + act in a loop — just-in-time planning, as opposed to planning the whole trajectory in advance.
• Function calling: advertise a tool schema each call; the model returns tool_calls; the runtime executes and feeds results back. MCP standardises exposing tools.
• The loop stores UserMessage / AssistantMessage / ToolResultMessage in a persistent, growing history.
• Model is stateless → the full context budget is re-assembled & re-sent each turn (motivates memory / RAG / compaction).

Recommended Readings¶

• Jurafsky, Daniel, and James H. Martin. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models. 3rd ed., online manuscript released August 19, 2026. — ch. 7, "Transformers and Pretraining": §7.5 for the logits and the softmax, §7.6 for greedy decoding and for temperature, top-$k$ and top-$p$ sampling.
• Wei, Jason, et al. "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models." Advances in Neural Information Processing Systems. Vol. 35. 2022. arXiv:2201.11903. — intermediate reasoning steps written before the answer, elicited by few-shot examples.
• Yao, Shunyu, et al. "ReAct: Synergizing Reasoning and Acting in Language Models." International Conference on Learning Representations. Vol. 2023. 2023. arXiv:2210.03629. — the primary source for the loop.
  • Take a look at Appendix C for examples of in-context learning.
• Dong, Qingxiu, et al. "A Survey on In-context Learning." Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024, pp. 1107–1128. arXiv:2301.00234. — survey: how the choice, order and format of the examples in a prompt change the answer (§4).
• Qin, Yujia, et al. "Tool Learning with Foundation Models." ACM Computing Surveys, vol. 57, no. 4, 2024, pp. 1–40. arXiv:2304.08354. — survey: how a model chooses a tool and forms the call, plans tool use ahead or step by step, and when a tool offered without examples lowers accuracy (§4).
• Schick, Timo, et al. "Toolformer: Language Models Can Teach Themselves to Use Tools." Advances in Neural Information Processing Systems. Vol. 36. 2023. arXiv:2302.04761. — tool calls inserted into the text a model generates, learned from the model's own samples.
• Wang, Xingyao, et al. "Executable Code Actions Elicit Better LLM Agents." International Conference on Machine Learning. Vol. 235. 2024. arXiv:2402.01030. — CodeAct: executable code as the action space, with the interpreter's output as the observation.

References¶

Peer-reviewed papers

• Dong, Qingxiu, et al. "A Survey on In-context Learning." Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024, pp. 1107–1128. arXiv:2301.00234. — §7: the two costs of few-shot prompting separated (efficiency vs. scalability); §4 and Table 2: selection, order and format all move the answer, and no selection rule ports across models.
• Holtzman, Ari, et al. "The Curious Case of Neural Text Degeneration." International Conference on Learning Representations. Vol. 2020. 2020. arXiv:1904.09751.
• Jacovi, Alon, and Yoav Goldberg. "Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?" Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020. arXiv:2004.03685. — the plausible / faithful distinction.
• Leviathan, Yaniv, et al. "Fast Inference from Transformers via Speculative Decoding." International Conference on Machine Learning. Vol. 202. 2023. arXiv:2211.17192. — faster decoding, unchanged output distribution.
• Merrill, William, and Ashish Sabharwal. "The Parallelism Tradeoff: Limitations of Log-Precision Transformers." Transactions of the Association for Computational Linguistics, vol. 11, 2023. arXiv:2207.00729. — the no-CoT ceiling: log-precision transformers sit inside TC⁰.
• Merrill, William, and Ashish Sabharwal. "The Expressive Power of Transformers with Chain of Thought." International Conference on Learning Representations. Vol. 2024. 2024. arXiv:2310.07923. — the ladder: what $t(n)$ intermediate tokens buy, up to exactly P.
• Schick, Timo, et al. "Toolformer: Language Models Can Teach Themselves to Use Tools." Advances in Neural Information Processing Systems. Vol. 36. 2023. arXiv:2302.04761.
• Song, Yifan, et al. "AgentBank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectories." Findings of the Association for Computational Linguistics: EMNLP 2024. 2024, pp. 2124–2141. arXiv:2410.07706. — the term trajectory tuning; Tab. 7's controlled ablation of the per-step rationale (unweighted mean of the task results, each on a scale from 0 to 100: held-out 29.5 → 22.8, 30.0 → 27.5 and 36.1 → 31.8; held-in 41.6 → 41.2, 45.2 → 45.5 and 45.4 → 43.6); Tab. 6 on what dropping the general-data mixture costs (MMLU 47.7 → 23.1).
• Turpin, Miles, et al. "Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting." Advances in Neural Information Processing Systems. Vol. 36. 2023. arXiv:2305.04388.
• Wang, Xingyao, et al. "Executable Code Actions Elicit Better LLM Agents." International Conference on Machine Learning. Vol. 235. 2024. arXiv:2402.01030.
• Wei, Jason, et al. "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models." Advances in Neural Information Processing Systems. Vol. 35. 2022. arXiv:2201.11903.
• Yang, John, et al. "SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering." Advances in Neural Information Processing Systems. Vol. 37. 2024. arXiv:2405.15793. — the coinage of agent–computer interface, the four design principles, and the fixed-model ablation of the interface on SWE-bench Lite.
• Yao, Shunyu, et al. "ReAct: Synergizing Reasoning and Acting in Language Models." International Conference on Learning Representations. Vol. 2023. 2023. arXiv:2210.03629. — the primary source for this week's loop: the augmented action space $\hat{\mathcal{A}} = \mathcal{A} \cup \mathcal{L}$ and the thought that returns no observation (§2), dense vs. sparse thoughts, the HotpotQA/FEVER and ALFWorld/WebShop results, and the manual success/failure analysis (Table 2).
• Zeng, Aohan, et al. "AgentTuning: Enabling Generalized Agent Abilities for LLMs." Findings of the Association for Computational Linguistics: ACL 2024. 2024, pp. 3053–3077. arXiv:2310.12823. — the AgentInstruct trajectories, written in ReAct form and filtered on the environment's terminal reward; the error analysis of an untuned model (invalid action, repetition, refusal).

Preprints & other non-peer-reviewed papers

• Chen, Charlie, et al. "Accelerating Large Language Model Decoding with Speculative Sampling." arXiv preprint arXiv:2302.01318 (2023). — concurrent formulation; Chinchilla 70 B.
• DeepSeek-AI, et al. "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning." arXiv preprint arXiv:2501.12948 (2025).
• Nye, Maxwell, et al. "Show Your Work: Scratchpads for Intermediate Computation with Language Models." arXiv preprint arXiv:2112.00114 (2021). — the empirical origin of the scratchpad effect.
• Yang, Zonghan, et al. "ReAct Meets ActRe: When Language Agents Enjoy Training Data Autonomy." arXiv preprint arXiv:2403.14589 (2024). — ActRe inverts ReAct's causality: handed an action, it supplies the reason a posteriori, so an arbitrary action becomes a well-formed ReAct step.
• Zaman, Kerem, and Shashank Srivastava. "Is Chain-of-Thought Really Not Explainability? Chain-of-Thought Can Be Faithful without Hint Verbalization." arXiv preprint arXiv:2512.23032 (2026). — hint verbalisation measures reporting, not faithfulness; the omission may be incompleteness.

Textbooks & monographs

• Jurafsky, Daniel, and James H. Martin. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models. 3rd ed., online manuscript released August 19, 2026. https://web.stanford.edu/~jurafsky/slp3/ — the logits and the softmax of the language-modelling head (ch. 7, §7.5); greedy decoding, random sampling, temperature, top-$k$ and top-$p$ sampling (ch. 7, §7.6).

Blogs, documentation & other web sources

• Anthropic. "Model Context Protocol." Open standard, 2024. — the specification behind the MCP slide: tools · resources · prompts, JSON-RPC transport, runtime-side discovery. https://modelcontextprotocol.io
• Anthropic. "Prompting Best Practices." Claude Platform Documentation, 2026. — prompt-level steering (clarity, examples, roles, XML). https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices
• llama.cpp. "Function Calling." llama.cpp Documentation, 2026. — the model families whose tool-call format llama-server implements, and the generic format for every other template. https://github.com/ggml-org/llama.cpp/blob/master/docs/function-calling.md
• Ollama. "qwen2.5:7b — template." Ollama Model Library, 2026a. — the chat template of the lab model: tool definitions between <tools> tags in the system message, the call between <tool_call> tags. https://ollama.com/library/qwen2.5:7b/blobs/eb4402837c78
• Ollama. "mistral — template." Ollama Model Library, 2026b. — the chat template of Mistral 7B v0.3: [AVAILABLE_TOOLS], [TOOL_CALLS], [TOOL_RESULTS]. https://ollama.com/library/mistral/blobs/1ff5b64b61b9
• Schluntz, Erik, and Barry Zhang. "Building Effective Agents." Anthropic Engineering, 2024. — tool design (ACI); workflows vs. agents.
• vLLM Project. vLLM. GitHub repository, 2026a. — the self-description as an "inference and serving engine for LLMs". https://github.com/vllm-project/vllm
• vLLM Project. "Tool Calling." vLLM Documentation, 2026b. — one tool-call parser per model family; with tool_choice="auto" the call is generated without a grammar unless a tool is marked strict. https://docs.vllm.ai/en/latest/features/tool_calling.html
• Yao, Shunyu, et al. "ReAct Prompting." GitHub repository, 2023. https://github.com/ysymyth/ReAct. — the reference implementation of the paper above: finish[answer] as the only exit, think[…] as a no-op action, and stop sequences as turn boundaries.