Agentic AI · L-Intro
What an agent is, how a language model produces text (probabilistically), the training ladder, and a modular agent architecture we will build on for the whole term.
Prerequisite: the recap Probability.
After this lecture you can:
An agent is a language model placed in a loop by a runtime. Each of the four parts answers one question:
A language model is a probability distribution over token sequences. The probability that this part uses — sample space, random variables, joint, marginal and conditional distributions, conditional independence — is a prerequisite of the lecture, recapped in Probability.
Token: the atomic unit of text a language model reads and produces (Jurafsky & Martin, 2026, ch. 2)
"tokenization" might split into "token" + "ization".Vocabulary $V$: the fixed, finite set of all tokens the model knows (typically ~50k–200k).
Every prompt and every reply is a sequence of tokens from $V$.
How text is split into tokens (tokenization) is covered in → L-Att.
$W_t$ (capital) is the token at position $t$ as a random variable over $V$; a concrete (generic) token is lowercase $w_t$. Write $W_{1:T} := (W_1, W_2, \dots, W_T)$ for the random variable of sequence of $T$ tokens.
A concrete text is a token sequence
$$w_{1:T} = w_1 w_2 w_3 \dots w_T$$ (each $w_t \in V$).
Notation: $W_{1:t-1}$ is the prefix (token sequence) before position $t$.
Any joint factorises into conditionals: $$ P(W_{1:T}) = P(W_1)\,P(W_2\mid W_1)\cdots P(W_T\mid W_{1:T-1}) = \prod_{t=1}^{T} P(W_t\mid W_{1:t-1})$$
The identity behind language modelling:
A sequence is a (first-order) Markov chain if the next state (here token) depends only on the current: $$ P(W_t\mid W_{1:t-1}) = P(W_t\mid W_{t-1}), $$ captured by a transition matrix $P_{ij}=P(W_t{=}j\mid W_{t-1}{=}i)$, where the indices $i,j$ label the vocabulary tokens ($W_t{=}j$ means "$W_t$ is the $j$-th token"; a generic token value is $w_t$).
As conditional independence. The Markov assumption says that given the present $W_{t-1}$, the next token $W_t$ is independent of the older past $W_{1:t-2}$ — i.e. $W_t \perp W_{1:t-2}\mid W_{t-1}$ (cf. Conditional Independence above). In the bigram model the total probability is: $$P(W_{1:T}) = P(W_1) \prod_{t=2}^T P(W_t \mid W_{t-1})$$
# A first-order Markov (bigram) language model, estimated by counting.
from collections import Counter, defaultdict
import numpy as np
text = "the cat sat on the mat . the cat ate the fish . the dog sat on the log .".split()
# Count bigram transitions: P(next_token | prev_token)
counts = defaultdict(Counter)
for prev_token, next_token in zip(text[:-1], text[1:]):
counts[prev_token][next_token] += 1
def get_conditional_dist(prev_token: str) -> dict[str, float]:
token_counts = counts[prev_token]
total_count = sum(token_counts.values())
return {word: count / total_count for word, count in token_counts.items()}
print("P(next | 'the') =", get_conditional_dist("the"))
print("P(next | 'cat') =", get_conditional_dist("cat"))
# Sample a sequence using the Markov conditionals
rng = np.random.default_rng(seed=0)
current_token = "the"
generated_sequence = ["the"]
for _ in range(8):
dist = get_conditional_dist(current_token)
if not dist:
break
words, probabilities = list(dist.keys()), list(dist.values())
current_token = rng.choice(words, p=probabilities)
generated_sequence.append(current_token)
print("Sampled text:", " ".join(generated_sequence))
The sampled text loses coherence after a few tokens, because the bigram model conditions each token on its predecessor alone.
So far the model is just an abstract distribution $P(W_t \mid W_{1:t-1})$. In modern LLMs this conditional is computed by a neural network — a Transformer — trained on huge text corpora. Its internals (tokenization, attention, positional encoding, the context window, and the KV-cache) are the subject of Inside the Model.
For now we only need the interface: prompt in → a probability distribution over the next token out.
Generation repeats this step: each sampled token is appended to the input, and the model is called again. The vocabulary contains an end token. The program that calls the model stops the repetition when the end token is sampled, or when a set limit on the number of generated tokens is reached.
An agent needs a model that follows instructions and requests tools, and a base model does neither: two training stages beyond next-token prediction install these capabilities.
An AI agent is an aligned LLM (large language model) placed in a loop where it can use tools to act and observe the results, keeps memory across steps, and works toward a goal — often proactively — instead of only replying once.
Compactly: Agent = aligned LLM + tools + memory (+ planning & proactivity), run in a loop.
The standard AI-Book of Russell & Norvig (2021) defines intelligence as rationality—doing the right thing. What counts as the right thing is defined by the objective that we provide to the agent, e.g. to maximize an expected performance measure given percepts and prior knowledge.
Rational agent (Russell & Norvig, 2021, §1.1.4). A rational agent is one that acts so as to achieve the best outcome or, when there is uncertainty, the best expected outcome — i.e. it selects the action expected to maximise its performance measure, given its percepts and prior knowledge (R&N, §2.2).
"The problem of achieving agreement between our true preferences and the objective we put into the machine is called the value alignment problem: the values or objectives put into the machine must be aligned with those of the human." (Russell & Norvig, 2021, §1.1.5, p. 23)
Training on large corpora results in a base model (aka foundation model):
Two further training stages turn the base model into a model which can be used for an agent:
Note. The chain rule $P(W_{1:T})=\prod_t P(W_t\mid W_{1:t-1})$ is just the autoregressive factorization — it holds for every rung (base, aligned, tool-calling). Alignment doesn't change the maths; it reshapes the learned conditional $P(W_t\mid W_{1:t-1})$ itself, shifting probability mass toward helpful, instruction-following continuations.
A base model is trained to predict the next token, not to follow the user's instruction (Ouyang et al., 2022, §1; the base model itself: Brown et al., 2020). The two objectives give different outputs:
Example. Ask a base model "What is the distance from Earth to the Moon?" and a very typical continuation is more questions — "What is the distance from Earth to the Sun? To Venus? …" — because on the web a question is often followed by more questions. Answering is just one possible completion, not a preferred one.
Alignment is the fine-tuning of a base model so that it follows instructions and behaves as a helpful chatbot (Jurafsky & Martin, 2026, ch. 8):
Result: an aligned chat model = a chatbot — reactive and talk-only.
The three rungs say what a model can do. The field's own vocabulary for when each capability is installed has meanwhile settled on three named stages (Tu et al., 2025):
A tool-calling model only proposes actions; the runtime turns it into an agent by running it in a loop that executes each requested tool call and returns the observation to the model.
Unlike single-turn chatbots that process one input and pause, an AI agent runs in a continuous loop:
It repeats this sequence until it completes the goal or requests human assistance.
Transitioning from a simple chatbot to an autonomous agent exposes key challenges that a single LLM call cannot solve alone:
The runtime is the program that hosts the agent loop: every part of the agent that is neither
the model nor the inference engine, the program that loads the weights of the model and
computes its output tokens (Ollama, llama.cpp, vLLM; behind a hosted API it is operated by the
provider). Its jobs:
This is the tool gateway + agent loop of the agent architecture (Agent Control and Its Boundaries). The model cannot act on its own, so the component of the runtime that executes a tool call, the tool gateway, is the trust boundary.
The model is called with a list of role-tagged messages, not a single string:
system-prompt: standing instructions / persona, sent first; sets how the model should
behave (tone, rules, role). Treated as higher priority than user text — a behaviour learned during alignment (the chat template), not a hard guarantee (cf. prompt injection).user: the human's input.assistant: the model's replies (and, later, its tool-call requests).tool (see below): results handed back from a tool.Chat-Completions API Format:
[
{"role": "system", "content": "You are a terse assistant; answer in one sentence."},
{"role": "user", "content": "What is a token?"},
...
]
The system prompt is your main knob for behaviour — you'll change it in the lab.
The system prompt can also be found by search. With the weights frozen, candidate prompts are generated, scored with a metric on example inputs, and the better ones kept — automatic prompt optimization. The candidates come from bootstrapped demonstrations (Khattab et al. (2024)), from crossover and mutation of a population of prompts carried out by an LLM (Guo et al. (2024)), or from an LLM that reads execution traces and the evaluator's feedback and rewrites the instruction (Agrawal et al. (2026)). It optimizes the same objective as fine-tuning, over the prompt instead of the weights — which is the mechanism behind Level 2's self-improvement below.
The model does not remember previous turns. On every new turn, the runtime re-sends the entire array of history messages (system, user,
assistant, tool). If you leave past messages out, the model forgets them instantly.
A tool is a function/capability the runtime exposes to the model — a way to act on or observe the world beyond generating text:
Tool-calling training (also called function-calling training) is the fine-tuning step in which a model learns when a tool is needed and how to request one. During instruction tuning (SFT) and the reinforcement-learning phase the model is exposed to special chat formats, so that it learns to stop generating text and to emit a structured tool request.
The agent loop is the repetition of three steps — the model requests a tool call, the runtime executes the call, the model reads the result — until the task is done. Over the coming lectures, you'll learn different decision strategies to power this loop.
Written with the deliberation factored out, the loop commits to nothing — it is the classical agent program (Russell & Norvig, 2021, §2.4): a persistent internal state, updated from percepts, with one action selected per cycle.
$$ \begin{array}{l} \textbf{Agent-Loop}(\textit{goal}) \\ \hline \textbf{Input: } \textit{goal} \text{; a runtime providing the tools; a step budget } B \\ \textbf{Output: } \text{the result} \\ \hline 1: \quad \textit{state} \gets \text{Init}(\textit{goal}) \\ 2: \quad \textbf{while } \neg\,\text{Done}(\textit{state}) \textbf{ and } \text{steps} < B \textbf{ do} \\ 3: \quad \qquad a \gets \text{Decide}(\textit{state}) \qquad \text{// the only line a strategy changes} \\ 4: \quad \qquad o \gets \text{Execute}(a) \qquad \text{// the runtime acts; the model never does} \\ 5: \quad \qquad \textit{state} \gets \text{Update}(\textit{state}, a, o) \\ 6: \quad \textbf{return } \text{Result}(\textit{state}) \\ \hline \end{array} $$
where:
Three names carry everything a deliberation strategy decides:
Line 4 is the trust boundary. The step budget $B$ is there because line 3 may never choose to stop.
Planning is the selection of the sequence of actions that leads to a goal; the planning paradigms differ in when that sequence is fixed — one action at a time, or as a plan before the first action.
The model requests a tool by emitting a structured tool call. Commonly a JSON object naming the tool and its arguments, matching a schema the runtime advertised (JSON is the native format; some systems use other encodings):
{ "name": "read_file", "arguments": { "path": "notes.md" } }
Note: Some model provider (e.g. OpenAI) emit an escaped JSON string in arguments.
Memory is the information that the runtime stores and places back into the context, so that an agent can use it across turns and across restarts:
Because the model is stateless (Statelessness), memory must be managed by the runtime — stored, retrieved, and, when it grows too large, summarised (compaction). (Details: retrieval and compaction, later in the course.)
Proactivity is the ability of an agent to start a run without a new user message; the event that starts the run then comes from the runtime:
Sometimes agents without proactivity are called assistants.
Four capabilities turn a static language model into an autonomous actor — each with a concrete algorithmic form:
| Capability | What it provides | Mathematical / algorithmic form | Course |
|---|---|---|---|
| Tools | hands — interact with the outside world | function schemas & an execution API | → L-ReAct |
| Memory | a past — context survives across turns / time | vector spaces $\mathbb{R}^d$, KV-cache, JSONL logs | → L-Embed, → L-Att, → L-RAG, → L-Memory |
| explicit Planning (optional) | a future — decompose goals, search trajectories, replan | plan schemas, DAGs, $A^*$ search, MDP policy | → L-Decomp, → L-ToT, → L-MDP |
| Proactivity (optional) | initiative — act unprompted on timers / events | heartbeat daemons, background schedulers | → L-Orch |
Putting the pieces together:
| Capability | Chatbot | Agent |
|---|---|---|
| Converse | ✅ | ✅ |
| Call tools (act) | ❌ | ✅ |
| Multi-step loop | ❌ | ✅ |
| Memory across turns | limited | ✅ |
| Plan / decompose (goals → sub-tasks) | ❌ | ✅ (often) |
| Act unprompted (proactivity) | ❌ | ✅ (often) |
Agents built on the same loop differ in how much the system around the model does: which capabilities it adds, what it places in the context, and which component holds the control and the trust boundary.
Beyond the training ladder (how the model is made), agents differ in how much the system around the model does. A useful capability ladder (Gulli, 2025):
An LLM only "knows" what is in its context for this call (it is stateless, Statelessness). Context engineering is the discipline of deciding what to put there: selecting, packaging and trimming the most relevant information — the system prompt, tool descriptions, retrieved documents, tool outputs, memory / interaction history, and implicit data (user identity, session & environment state) — so the model's limited attention is spent on what matters.
Output quality often depends more on the context you assemble than on the model itself (Gulli, 2025) — a strong model still underperforms with a poor informational environment. The finite window is why the agent system must also select, not only add.
Context engineering is the job of the agent runtime and its developer. The context is assembled automatically at each call, around the user's message; the end-user only states the request.
Agents are powerful but not infallible — keep the limits in view from day one:
One component holds the control and every dependency on something outside the runtime is reached through a component of its own — a common, representative division, though simple agents discharge several of these duties in one place:
This division is not the design of one product; most agent runtimes converge on it.
Production practice describes an agent as a model plus a harness, and separates workflows, whose control flow is fixed in code, from agents, whose control flow the model decides.
aligned LLM = the model,
produced by the training ladder; tools + memory + loop (+ proactivity) = the harness.if/else conditions.The hands-on work is in two notebooks, and both are worked through in the lab session:
lecture-Intro-exercises.ipynb) — pen and paper:lecture-Intro-lab.ipynb) — a tutorial whose cells are executed in order: set up
Ollama, call a local tool-calling model via the OpenAI client, see that the model is
stateless — memory exists only if you resend the history, and the token bill grows
quadratically when you do — and inspect a structured tool call.Peer-reviewed papers
Preprints & other non-peer-reviewed papers
<|startofthought|> / <|endofthought|> tokens; the benefit as a skewed tail over hard tokens.Textbooks & monographs
Blogs, documentation & other web sources