Agentic AI · B.Sc. Applied Computer Science
What an agent is, how a language model produces text (probabilistically), the training ladder, and a 3-layer architecture we will build on for the whole term.
After this week you can:
Prerequisites: basic probability; a prior data science course and ML course. Transformers are covered in Week 7.
A language model is a probability distribution over token sequences, so we start with the probability we need.
📖 More in detailed review: http://christianherta.de/lehre/algorithmic-fundations-of-robotics-and-artifical-intelligence/math/probabilities.html
A random variable is a function that assigns a numerical value to each outcome in the sample space. Random variables are usually denoted by capital letters, e.g. $X$ or $Y$.
Its set of possible values is written $\mathrm{Val}(X)$.
Die example: let $X=1$ if the face is odd $\{1,3,5\}$, else $X=0$ (even).
Notation ("abuse of notation"): we identify a specific distribution by its argument:
$P(X{=}x)$ is a single number.
Short notation: $P(x)$ for $P(X{=}x)$ if not ambiguous.
A discrete probability distribution defines the likelihood of all possible outcomes for a discrete random variable.
Formally it assigns a probability to every value with $$P(X{=}x)\ge 0 \qquad\text{and}\qquad \sum_{x\in \mathrm{Val}(X)} P(X{=}x)=1.$$
A joint distribution gives the probability of combinations of two (or more) random variables: $$ P(X=x, Y=y) \ge 0, \qquad \sum_{x}\sum_{y} P(X{=}x,\, Y{=}y) = 1. $$
Die example ($X$ odd/even, $Y=\mathbb{1}[\text{face}>3]$ — the indicator: $1$ if the face is greater than $3$, else $0$):
| $P(X,Y)$ | $Y{=}0\ (\le 3)$ | $Y{=}1\ (>3)$ |
|---|---|---|
| $X{=}1$ (odd) | $2/6$ | $1/6$ |
| $X{=}0$ (even) | $1/6$ | $2/6$ |
It generalises to many variables: $P(X_1,\dots,X_n)$ — e.g. the probability of a whole sequence of outcomes (e.g. tokens).
Token: the atomic unit of text a language model reads and produces
"tokenization" might split into "token" + "ization".Vocabulary $V$: the fixed, finite set of all tokens the model knows (typically ~50k–200k).
Every prompt and every reply is a sequence of tokens from $V$.
How text is split into tokens (tokenization) is covered in Week 7.
Here we just need: each token is a random variable $W_t$ over $V$; a concrete text is a token sequence
$$w_{1:T} = w_1 w_2 w_3 \dots w_T$$ (each $w_t \in V$).
$$ P(X\mid Y) = \frac{P(X, Y)}{P(Y)}, \qquad P(Y)>0 $$
"Probability of $X$ given $Y$."
$X$ is independent of $Y$ given $Z$ if $$P(X,Y \mid Z)=P(X \mid Z)\,P(Y\mid Z)$$ or, equivalently, $$P(X \mid Y,Z)=P(X \mid Z).$$
Notation: $X \perp Y \mid Z$
Notation: $W_t$ (capital) is the token at position $t$ as a random variable over $V$; a concrete token is lowercase $w_t$. Write $W_{1:T} := (W_1, W_2, \dots, W_T)$ for a sequence of $T$ tokens, and $W_{1:t-1}$ is the prefix before position $t$.
Any joint factorises into conditionals: $$ P(W_{1:T}) = P(W_1)\,P(W_2\mid W_1)\cdots P(W_T\mid W_{1:T-1}) = \prod_{t=1}^{T} P(W_t\mid W_{1:t-1})$$
The identity behind language modelling:
A sequence is a (first-order) Markov chain if the next state depends only on the current: $$ P(W_t\mid W_{1:t-1}) = P(W_t\mid W_{t-1}), $$ captured by a transition matrix $P_{ij}=P(W_t{=}j\mid W_{t-1}{=}i)$, where the indices $i,j$ label the vocabulary tokens ($W_t{=}j$ means "$W_t$ is the $j$-th token"; a generic token value is $w_t$).
As conditional independence. The Markov assumption says that given the present $W_{t-1}$, the next token $W_t$ is independent of the older past $W_{1:t-2}$ — i.e. $W_t \perp W_{1:t-2}\mid W_{t-1}$ (cf. Conditional Independence above). In the bigram model the total probability is: $$P(W_{1:T}) = P(W_1) \prod_{t=2}^T P(W_t \mid W_{t-1})$$
# A first-order Markov (bigram) language model, estimated by counting.
from collections import Counter, defaultdict
import numpy as np
text = "the cat sat on the mat . the cat ate the fish . the dog sat on the log .".split()
# Count bigram transitions: P(next_token | prev_token)
counts = defaultdict(Counter)
for prev_token, next_token in zip(text[:-1], text[1:]):
counts[prev_token][next_token] += 1
def get_conditional_dist(prev_token: str) -> dict[str, float]:
token_counts = counts[prev_token]
total_count = sum(token_counts.values())
return {word: count / total_count for word, count in token_counts.items()}
print("P(next | 'the') =", get_conditional_dist("the"))
print("P(next | 'cat') =", get_conditional_dist("cat"))
# Sample a sequence using the Markov conditionals
rng = np.random.default_rng(seed=0)
current_token = "the"
generated_sequence = ["the"]
for _ in range(8):
dist = get_conditional_dist(current_token)
if not dist:
break
words, probabilities = list(dist.keys()), list(dist.values())
current_token = rng.choice(words, p=probabilities)
generated_sequence.append(current_token)
print("Sampled text:", " ".join(generated_sequence))
The bigram model is Markov — it uses only the last word, so it quickly loses coherence. An LLM conditions on the whole prefix, which is why it stays on topic. Same chain rule, no Markov shortcut.
So far the model is just an abstract distribution $P(W_t \mid W_{1:t-1})$. In modern LLMs this conditional is computed by a neural network — a Transformer — trained on huge text corpora. Its internals (tokenization, attention, positional encoding, the context window, and the KV-cache) are the subject of Week 7 — Inside the Model.
For now we only need the interface: prompt in → a probability distribution over the next token out.
AI agent — working definition. An AI agent is an aligned LLM (large language model) placed in a loop where it can use tools to act and observe the results, keeps memory across steps, and works toward a goal — often proactively — instead of only replying once. Compactly: Agent = aligned LLM + tools + memory (+ planning & proactivity), run in a loop.
Part II builds up to this:
Classical AI (Russell & Norvig, 2021) defines intelligence as rationality—doing the right thing. What counts as the right thing is defined by the objective that we provide to the agent, e.g. to maximize an expected performance measure given percepts and prior knowledge.
Rational agent (Russell & Norvig, 2021, §1.1.4). A rational agent is one that acts so as to achieve the best outcome or, when there is uncertainty, the best expected outcome — i.e. it selects the action expected to maximise its performance measure, given its percepts and prior knowledge (R&N, §2.2).
"The problem of achieving agreement between our true preferences and the objective we put into the machine is called the value alignment problem: the values or objectives put into the machine must be aligned with those of the human." (Russell & Norvig, 2021, §1.1.5, p. 23)
Training on large corpora results in a base model:
Note. The chain rule $P(W_{1:T})=\prod_t P(W_t\mid W_{1:t-1})$ is just the autoregressive factorization — it holds for every rung (base, aligned, tool-calling). Alignment doesn't change the maths; it reshapes the learned conditional $P(W_t\mid W_{1:t-1})$ itself, shifting probability mass toward helpful, instruction-following continuations.
Two further training stages turn the base model into a model which can be used for an agent:
A base model continues text; it doesn't "answer" — its training objective is next-token prediction, not "follow the user's instructions", so the two come apart (Ouyang et al., 2022, §1; the base model itself: Brown et al., 2020).
Example. Ask a base model "What is the distance from Earth to the Moon?" and a very typical continuation is more questions — "What is the distance from Earth to the Sun? To Venus? …" — because on the web a question is often followed by more questions. Answering is just one possible completion, not a preferred one.
Alignment fine-tunes it to behave like a helpful chatbot:
Result: an aligned chat model = a chatbot — it converses and follows instructions, but it is reactive and talk-only.
You call a chat model with a list of role-tagged messages, not a single string:
system-prompt: standing instructions / persona, sent first; sets how the model should
behave (tone, rules, role). Treated as higher priority than user text — a behaviour learned during alignment (the chat template), not a hard guarantee (cf. prompt injection, W12).user: the human's input.assistant: the model's replies (and, later, its tool-call requests).tool (see below): results handed back from a tool (Week 2).messages = [
{"role": "system", "content": "You are a terse assistant; answer in one sentence."},
{"role": "user", "content": "What is a token?"},
]
The system prompt is your main knob for behaviour — you'll change it in the lab.
The model does not remember previous turns. On every new turn, the runtime re-sends the entire array of history messages (system, user,
assistant, tool). If you leave past messages out, the model forgets them instantly.
A tool is a function/capability the runtime exposes to the model — a way to act on or observe the world beyond generating text:
To use tools reliably, the model gets a third fine-tuning step (rung 3): tool- / function-calling training, i.e. it learns when a tool is needed and how to request one:
(A self-supervised variant needs no hand-written tool dialogues at all: Toolformer lets the model insert candidate API calls into ordinary text and keeps only those that reduce its own next-token loss — Schick et al., 2023.)
The model never runs the tool itself; it requests it (see below).
A chatbot only talks. Turn it into an agent by adding a loop to tool-use to solve a task in several steps: call a tool, read the result, decide the next move, until the task is done (the ReAct loop, Week 2).
The runtime is the program hosting the model — everything that is not the model itself. It is everything wrapped around the LLM — model plus runtime is the agent. Its jobs:
In our case study this is the Gateway (control plane) + toolkit of the 3-layer architecture (see below). Because the model cannot act on its own, the runtime is also the security boundary (Week 12).
The model requests a tool by emitting a structured tool call. Commonly a JSON object naming the tool and its arguments, matching a schema the runtime advertised (JSON is the native format; some systems use other encodings — prompt-based tool calling, W2):
{ "name": "read_file", "arguments": { "path": "notes.md" } }
The runtime parses the JSON, runs the function, and feeds the result back to the model. Chaining request → execute → result is the ReAct loop — the subject of Week 2.
So far each request is independent. Memory lets an agent carry information across turns — and even across restarts:
Because the model is stateless (Part I), memory must be managed by the runtime — stored, retrieved, and, when it grows too large, summarised (compaction). (Details: Weeks 8–9.)
Some agents can act on their own, without a fresh prompt:
Sometimes agents without proactivity are called assistants.
While a ReAct loop decides actions step-by-step (just-in-time selection), planning gives an agent deliberate look-ahead capabilities:
(Task decomposition: W3. The search itself: W4.)
Four capabilities turn a static language model into an autonomous actor — each with a concrete algorithmic form (this course's focus):
| Capability | What it provides | Mathematical / algorithmic form | Course |
|---|---|---|---|
| Tools | hands — interact with the outside world | function schemas & an execution API | W2 |
| Memory | a past — context survives across turns / time | vector spaces $\mathbb{R}^d$, KV-cache, JSONL logs | W6–9 |
| Planning (optional) | a future — decompose goals, search trajectories, replan | plan schemas, DAGs, $A^*$ search, MDP policy | W3–5 |
| Proactivity (optional) | initiative — act unprompted on timers / events | heartbeat daemons, background schedulers | W10 |
Putting the pieces together:
| Capability | Chatbot | Agent |
|---|---|---|
| Converse | ✅ | ✅ |
| Call tools (act) | ❌ | ✅ |
| Multi-step loop | ❌ | ✅ |
| Memory across turns | limited | ✅ |
| Plan / decompose (goals → sub-tasks) | ❌ | ✅ (often) |
| Act unprompted (proactivity) | ❌ | ✅ (often) |
Agent = aligned LLM + tools + memory (+ planning & proactivity), run in a loop
Beyond the training ladder (how the model is made), agents differ in how much the system around the model does. A useful capability ladder (Gulli, 2025):
Two different ladders. The training ladder (base → aligned → tool-calling) is about the model; these capability levels are about the system built around it. This course climbs Levels 0 → 3.
An LLM only "knows" what is in its context for this call (it is stateless, see above). Context engineering is the discipline of deciding what to put there: selecting, packaging and trimming the most relevant information — the system prompt, retrieved documents, tool outputs, memory / interaction history, and implicit data (user identity, session & environment state) — so the model's limited attention is spent on what matters.
Richness and budget (Gulli, 2025). Output quality often depends more on the context you assemble than on the model itself — a strong model still underperforms with a poor informational environment. The finite window (W2/W7) is why the agent system must also select, not only add.
It is a core skill of a Level 2 agent. We make it precise as the context budget (W2), and return to its main techniques — retrieval (W8) and compaction (W9).
Prompt engineering vs. context engineering. Prompt engineering is the narrower sibling: crafting the wording and structure of a single instruction — role, examples (few-shot), output format, "think step by step" — to steer one call. Context engineering is the broader discipline of choosing what information fills the whole context across steps. You practise prompt engineering hands-on from W2 (system prompts, CoT, tool descriptions).
Who does it? Context engineering is the job of the agent runtime and its developer — the system prompt, retrieval, memory and compaction are assembled automatically each call, around the user's message. The end-user just asks; the agent engineers the context.
Agents are powerful but not infallible — keep the limits in view from day one:
This is the course's first ⚖️ responsible-AI hook — capabilities and their limits — a thread we revisit all term (reward mis-specification W5, memory & privacy W9, multi-agent risks W11, security W12, synthesis W15).
Most agents share a similar three-layer shape — a useful "building" to keep in mind (a common, representative structure though simple agents may collapse these layers):
Our lab agent, Selma (an OpenClaw clone), implements exactly this pattern — you build on it in the exercises.
(The runtime = toolkit + control plane together — everything hosting the model.)
Two key insights from production systems reinforce why agency requires both training and software architecture:
Model + Harness — harness is the practitioner's name for what we called the runtime (toolkit + control plane).
aligned LLM = the model;
tools + memory + loop (+ proactivity) = the harness.if/else conditions.Workflows vs. Autonomous Agents
All hands-on work is in the Week 1 exercises notebook (week-01-exercises.ipynb):
Work through all of it in the lab session.
Next week (W2): the ReAct loop and building tool use from scratch.