Agentic AI · L-Intro
What an agent is, how a language model produces text (probabilistically), the training ladder, and a modular agent architecture we will build on for the whole term.
After this lecture you can:
A language model is a probability distribution over token sequences, so we start with the probability we need.
📖 More in detailed review: http://christianherta.de/lehre/algorithmic-fundations-of-robotics-and-artifical-intelligence/math/probabilities.html
A random variable is a function that assigns a numerical value to each outcome in the sample space. Random variables are usually denoted by capital letters, e.g. $X$ or $Y$.
Its set of possible values is written $\mathrm{Val}(X)$.
Die example: let $X=1$ if the face is odd $\{1,3,5\}$, else $X=0$ (even).
Notation ("abuse of notation"): we identify a specific distribution by its argument:
$P(X{=}x)$ is a single number.
Short notation: $P(x)$ for $P(X{=}x)$ if not ambiguous.
A discrete probability distribution defines the likelihood of all possible outcomes for a discrete random variable.
Formally it assigns a probability to every value with $$P(X{=}x)\ge 0 \qquad\text{and}\qquad \sum_{x\in \mathrm{Val}(X)} P(X{=}x)=1.$$
A joint distribution gives the probability of combinations of two (or more) random variables: $$ P(X=x, Y=y) \ge 0, \qquad \sum_{x}\sum_{y} P(X{=}x,\, Y{=}y) = 1. $$
Die example ($X$ odd/even, $Y=\mathbb{1}[\text{face}>3]$ — the indicator: $1$ if the face is greater than $3$, else $0$):
| $P(X,Y)$ | $Y{=}0\ (\le 3)$ | $Y{=}1\ (>3)$ |
|---|---|---|
| $X{=}1$ (odd) | $2/6$ | $1/6$ |
| $X{=}0$ (even) | $1/6$ | $2/6$ |
It generalises to many variables: $P(X_1,\dots,X_n)$ — e.g. the probability of a whole sequence of outcomes (e.g. tokens).
$$ P(X\mid Y) = \frac{P(X, Y)}{P(Y)}, \qquad P(Y)>0 $$
"Probability of $X$ given $Y$."
$X$ is independent of $Y$ given $Z$ if $$P(X,Y \mid Z)=P(X \mid Z)\,P(Y\mid Z)$$ or, equivalently, $$P(X \mid Y,Z)=P(X \mid Z).$$
Notation: $X \perp Y \mid Z$
Token: the atomic unit of text a language model reads and produces
"tokenization" might split into "token" + "ization".Vocabulary $V$: the fixed, finite set of all tokens the model knows (typically ~50k–200k).
Every prompt and every reply is a sequence of tokens from $V$.
How text is split into tokens (tokenization) is covered in → L-Att.
Notation: $W_t$ (capital) is the token at position $t$ as a random variable over $V$; a concrete (generic) token is lowercase $w_t$. Write $W_{1:T} := (W_1, W_2, \dots, W_T)$ for the random variable of sequence of $T$ tokens.
A concrete text is a token sequence
$$w_{1:T} = w_1 w_2 w_3 \dots w_T$$ (each $w_t \in V$).
Notation: $W_{1:t-1}$ is the prefix (token sequence) before position $t$.
Any joint factorises into conditionals: $$ P(W_{1:T}) = P(W_1)\,P(W_2\mid W_1)\cdots P(W_T\mid W_{1:T-1}) = \prod_{t=1}^{T} P(W_t\mid W_{1:t-1})$$
The identity behind language modelling:
A sequence is a (first-order) Markov chain if the next state (here token) depends only on the current: $$ P(W_t\mid W_{1:t-1}) = P(W_t\mid W_{t-1}), $$ captured by a transition matrix $P_{ij}=P(W_t{=}j\mid W_{t-1}{=}i)$, where the indices $i,j$ label the vocabulary tokens ($W_t{=}j$ means "$W_t$ is the $j$-th token"; a generic token value is $w_t$).
As conditional independence. The Markov assumption says that given the present $W_{t-1}$, the next token $W_t$ is independent of the older past $W_{1:t-2}$ — i.e. $W_t \perp W_{1:t-2}\mid W_{t-1}$ (cf. Conditional Independence above). In the bigram model the total probability is: $$P(W_{1:T}) = P(W_1) \prod_{t=2}^T P(W_t \mid W_{t-1})$$
# A first-order Markov (bigram) language model, estimated by counting.
from collections import Counter, defaultdict
import numpy as np
text = "the cat sat on the mat . the cat ate the fish . the dog sat on the log .".split()
# Count bigram transitions: P(next_token | prev_token)
counts = defaultdict(Counter)
for prev_token, next_token in zip(text[:-1], text[1:]):
counts[prev_token][next_token] += 1
def get_conditional_dist(prev_token: str) -> dict[str, float]:
token_counts = counts[prev_token]
total_count = sum(token_counts.values())
return {word: count / total_count for word, count in token_counts.items()}
print("P(next | 'the') =", get_conditional_dist("the"))
print("P(next | 'cat') =", get_conditional_dist("cat"))
# Sample a sequence using the Markov conditionals
rng = np.random.default_rng(seed=0)
current_token = "the"
generated_sequence = ["the"]
for _ in range(8):
dist = get_conditional_dist(current_token)
if not dist:
break
words, probabilities = list(dist.keys()), list(dist.values())
current_token = rng.choice(words, p=probabilities)
generated_sequence.append(current_token)
print("Sampled text:", " ".join(generated_sequence))
The bigram model is Markov — it uses only the last word, so it quickly loses coherence. An LLM conditions on the whole prefix, which is why it stays on topic. Same chain rule, no Markov shortcut.
So far the model is just an abstract distribution $P(W_t \mid W_{1:t-1})$. In modern LLMs this conditional is computed by a neural network — a Transformer — trained on huge text corpora. Its internals (tokenization, attention, positional encoding, the context window, and the KV-cache) are the subject of Inside the Model.
For now we only need the interface: prompt in → a probability distribution over the next token out.
An AI agent is an aligned LLM (large language model) placed in a loop where it can use tools to act and observe the results, keeps memory across steps, and works toward a goal — often proactively — instead of only replying once.
Compactly: Agent = aligned LLM + tools + memory (+ planning & proactivity), run in a loop.
The standard AI-Book of Russell & Norvig (2021) defines intelligence as rationality—doing the right thing. What counts as the right thing is defined by the objective that we provide to the agent, e.g. to maximize an expected performance measure given percepts and prior knowledge.
Rational agent (Russell & Norvig, 2021, §1.1.4). A rational agent is one that acts so as to achieve the best outcome or, when there is uncertainty, the best expected outcome — i.e. it selects the action expected to maximise its performance measure, given its percepts and prior knowledge (R&N, §2.2).
"The problem of achieving agreement between our true preferences and the objective we put into the machine is called the value alignment problem: the values or objectives put into the machine must be aligned with those of the human." (Russell & Norvig, 2021, §1.1.5, p. 23)
Training on large corpora results in a base model (aka foundation model):
Two further training stages turn the base model into a model which can be used for an agent:
Note. The chain rule $P(W_{1:T})=\prod_t P(W_t\mid W_{1:t-1})$ is just the autoregressive factorization — it holds for every rung (base, aligned, tool-calling). Alignment doesn't change the maths; it reshapes the learned conditional $P(W_t\mid W_{1:t-1})$ itself, shifting probability mass toward helpful, instruction-following continuations.
A base model continues text; it doesn't "answer" — its (standard) training objective is next-token prediction, not "follow the user's instructions", so the two come apart (Ouyang et al., 2022, §1; the base model itself: Brown et al., 2020).
Example. Ask a base model "What is the distance from Earth to the Moon?" and a very typical continuation is more questions — "What is the distance from Earth to the Sun? To Venus? …" — because on the web a question is often followed by more questions. Answering is just one possible completion, not a preferred one.
Alignment fine-tunes it to behave like a helpful chatbot:
Result: an aligned chat model = a chatbot — it converses and follows instructions, but it is reactive and talk-only.
The three rungs say what a model can do. The field's own vocabulary for when each capability is installed has meanwhile settled on three named stages (Tu et al., 2025):
Unlike single-turn chatbots that process one input and pause, an AI agent runs in a continuous loop:
It repeats this sequence until it completes the goal or requests human assistance.
Transitioning from a simple chatbot to an autonomous agent exposes key challenges that a single LLM call cannot solve alone:
The runtime is the program hosting the model — everything that is not the model itself. It is everything wrapped around the LLM — model plus runtime is the agent. Its jobs:
This is the tool gateway + agent loop of the agent architecture (Agent Control and Its Boundaries in the lecture note). Because the model cannot act on its own, the runtime is also the security boundary.
The model is called with a list of role-tagged messages, not a single string:
system-prompt: standing instructions / persona, sent first; sets how the model should
behave (tone, rules, role). Treated as higher priority than user text — a behaviour learned during alignment (the chat template), not a hard guarantee (cf. prompt injection).user: the human's input.assistant: the model's replies (and, later, its tool-call requests).tool (see below): results handed back from a tool.[
{"role": "system", "content": "You are a terse assistant; answer in one sentence."},
{"role": "user", "content": "What is a token?"},
...
]
The system prompt is your main knob for behaviour — you'll change it in the lab.
The model does not remember previous turns. On every new turn, the runtime re-sends the entire array of history messages (system, user,
assistant, tool). If you leave past messages out, the model forgets them instantly.
A tool is a function/capability the runtime exposes to the model — a way to act on or observe the world beyond generating text:
To use tools reliably, the model gets a fine-tuning step: tool- / function-calling training, i.e. it learns when a tool is needed and how to request one:
The model never runs the tool itself; it requests it.
A chatbot only talks. Turn it into an agent by adding a loop to tool-use to solve a task in several steps: call a tool, read the result, decide the next move, until the task is done — the agent loop. Over the coming lectures, you'll learn different decision strategies to power this loop.
Written with the deliberation factored out, the loop commits to nothing — it is the classical agent program (Russell & Norvig, 2021, §2.4): a persistent internal state, updated from percepts, with one action selected per cycle.
$$ \begin{array}{l} \textbf{Agent-Loop}(\textit{goal}) \\ \hline \textbf{Input: } \textit{goal} \text{; a runtime providing the tools; a step budget } B \\ \textbf{Output: } \text{the result} \\ \hline 1: \quad \textit{state} \gets \text{Init}(\textit{goal}) \\ 2: \quad \textbf{while } \neg\,\text{Done}(\textit{state}) \textbf{ and } \text{steps} < B \textbf{ do} \\ 3: \quad \qquad a \gets \text{Decide}(\textit{state}) \qquad \text{// the only line a strategy changes} \\ 4: \quad \qquad o \gets \text{Execute}(a) \qquad \text{// the runtime acts; the model never does} \\ 5: \quad \qquad \textit{state} \gets \text{Update}(\textit{state}, a, o) \\ 6: \quad \textbf{return } \text{Result}(\textit{state}) \\ \hline \end{array} $$
where:
Three names carry everything a deliberation strategy decides:
Line 4 is the security boundary — the model requests, the runtime executes. The step budget $B$ is there because line 3 may never choose to stop.
To use external tools effectively, an LLM must decide how to sequence its actions. Systems generally fall into three planning paradigms:
The model requests a tool by emitting a structured tool call. Commonly a JSON object naming the tool and its arguments, matching a schema the runtime advertised (JSON is the native format; some systems use other encodings):
{ "name": "read_file", "arguments": { "path": "notes.md" } }
The runtime parses the JSON, runs the function, and feeds the result back to the model. Chaining request → execute → result is the agent loop.
Note: Some model provider (e.g. OpenAI) emit an escaped JSON string in arguments.
So far each request is independent. Memory lets an agent carry information across turns — and even across restarts:
Because the model is stateless (Part I), memory must be managed by the runtime — stored, retrieved, and, when it grows too large, summarised (compaction). (Details: retrieval and compaction, later in the course.)
Some agents can act on their own, without a fresh prompt:
Sometimes agents without proactivity are called assistants.
Four capabilities turn a static language model into an autonomous actor — each with a concrete algorithmic form:
| Capability | What it provides | Mathematical / algorithmic form | Course |
|---|---|---|---|
| Tools | hands — interact with the outside world | function schemas & an execution API | → L-ReAct |
| Memory | a past — context survives across turns / time | vector spaces $\mathbb{R}^d$, KV-cache, JSONL logs | → L-Embed, → L-Att, → L-RAG, → L-Memory |
| explicit Planning (optional) | a future — decompose goals, search trajectories, replan | plan schemas, DAGs, $A^*$ search, MDP policy | → L-Decomp, → L-ToT, → L-MDP |
| Proactivity (optional) | initiative — act unprompted on timers / events | heartbeat daemons, background schedulers | → L-Orch |
Putting the pieces together:
| Capability | Chatbot | Agent |
|---|---|---|
| Converse | ✅ | ✅ |
| Call tools (act) | ❌ | ✅ |
| Multi-step loop | ❌ | ✅ |
| Memory across turns | limited | ✅ |
| Plan / decompose (goals → sub-tasks) | ❌ | ✅ (often) |
| Act unprompted (proactivity) | ❌ | ✅ (often) |
Agent = aligned LLM + tools + memory (+ planning & proactivity), run in a loop
Beyond the training ladder (how the model is made), agents differ in how much the system around the model does. A useful capability ladder (Gulli, 2025):
Two different ladders. The training ladder (base → aligned → tool-calling) is about the model; these capability levels are about the system built around it.
An LLM only "knows" what is in its context for this call (it is stateless, see above). Context engineering is the discipline of deciding what to put there: selecting, packaging and trimming the most relevant information — the system prompt, retrieved documents, tool outputs, memory / interaction history, and implicit data (user identity, session & environment state) — so the model's limited attention is spent on what matters.
Output quality often depends more on the context you assemble than on the model itself — a strong model still underperforms with a poor informational environment. The finite window is why the agent system must also select, not only add.
Who does it? Context engineering is the job of the agent runtime and its developer — the system prompt, retrieval, memory and compaction are assembled automatically each call, around the user's message. The end-user just asks; the agent engineers the context.
Agents are powerful but not infallible — keep the limits in view from day one:
One component holds the control and every dependency on something outside the agent is reached through a component of its own — a common, representative division, though simple agents discharge several of these duties in one place:
*This division is not one product's design — it is the shape most agent runtimes converge on.
(The runtime = everything inside the box — Agent Control and its boundary components; *not the model.)*
Two key insights from production systems reinforce why agency requires both training and software architecture:
aligned LLM = the model;
tools + memory + loop (+ proactivity) = the harness.if/else conditions.All hands-on work is in the exercises notebook (lecture-01-Intro-exercises.ipynb):
Work through all of it in the lab session.
Peer-reviewed papers
Preprints & other non-peer-reviewed papers
Textbooks & monographs
Blogs, documentation & other web sources