{
"cells": [
{
"cell_type": "markdown",
"id": "a769b3cf",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": ""
},
"tags": []
},
"source": [
"# Building a SWE Agent from Scratch\n",
"\n",
"**Agentic AI** · playground\n",
"\n",
"*A hands-on walkthrough: we build a small software-engineering agent in plain Python (model call, action parsing, a sandboxed shell, the loop) and grow it step by step until it is, almost line by line, mini-swe-agent.*"
]
},
{
"cell_type": "markdown",
"id": "9297f90a",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": ""
},
"tags": []
},
"source": [
"## What We Build\n",
"\n",
"[mini-swe-agent](https://github.com/SWE-agent/mini-swe-agent) (Lieret & Jimenez, 2026) is the deliberately minimal coding agent of the team behind **SWE-agent** — a separate, much larger agent with a hand-designed set of file-editing and search tools ([Yang et al., 2024](https://arxiv.org/html/2405.15793)). The two are related by their authors, not by their code. Its core loop (`agents/default.py`) is about 100 lines; with serialisation and templating the file has about 190. It has **one tool** (bash), a **linear message history**, and it runs every command in a fresh subshell. We rebuild it ourselves in small pieces, then run the real package on the same task and compare.\n",
"\n",
"The plan:\n",
"\n",
"1. talk to a model through **LiteLLM** (here: a local model served by Ollama),\n",
"2. write the **input** (system prompt + task) and parse the **output** (one bash command),\n",
"3. build a **sandboxed environment** (a podman container),\n",
"4. glue it together into a **minimal loop**,\n",
"5. extend the loop: **termination**, **errors as observations**, **observation shaping**,\n",
"6. look at **tool calling** as the alternative to text parsing,\n",
"7. refactor into mini's three classes: **Model, Environment, Agent**,\n",
"8. save and read **trajectories**, add a **human in the loop**,\n",
"9. run the real **mini-swe-agent** on the same bug.\n",
"\n",
"At the end of most sections, a short **Concept** paragraph ties the code back to the lectures."
]
},
{
"cell_type": "markdown",
"id": "d3d31db2",
"metadata": {},
"source": [
"## Setup\n",
"\n",
"**What you need.**\n",
"\n",
"- **Ollama** running locally (`ollama serve`) with a model that is decent at code and follows formats, e.g.\n",
" ```bash\n",
" ollama pull qwen2.5-coder:7b\n",
" ```\n",
" Any LiteLLM model string works: set `AGENT_MODEL` before starting Jupyter (e.g. `AGENT_MODEL=ollama_chat/qwen2.5:3b`). Small models are fine for the plumbing sections but often fail the actual bug fix. That's instructive too.\n",
"- **podman** (rootless is fine). The notebook pulls `python:3.12-slim` once.\n",
"- Python packages: `litellm`, `jinja2`, `pyyaml` and, for the final section only, `mini-swe-agent`.\n",
"\n",
"**Safety first.** The model never executes anything itself; it only writes text. The harness, our code, extracts a shell command from that text and executes it. So the harness executes commands whose content a language model chose. In this notebook, **every model-chosen command** runs inside a podman container with no network, which can only see the toy repository in `./sandbox`. The only commands that run on your machine are the ones *we* wrote into the harness (starting the container, `git` on the sandbox). Keep it that way when you experiment."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4a16ef2c",
"metadata": {},
"outputs": [],
"source": [
"# %pip install litellm jinja2 pyyaml\n",
"import json\n",
"import os\n",
"import re\n",
"import subprocess\n",
"import time\n",
"from pathlib import Path\n",
"\n",
"import litellm\n",
"from jinja2 import StrictUndefined, Template\n",
"\n",
"MODEL = os.getenv(\"AGENT_MODEL\", \"ollama_chat/qwen2.5-coder:7b\") # \"ollama_chat/\" = Ollama's chat endpoint\n",
"IMAGE = \"docker.io/library/python:3.12-slim\"\n",
"SANDBOX = Path(\"sandbox\").resolve() # all that model-chosen commands can see; mounted at /repo in the container\n",
"\n",
"litellm.suppress_debug_info = True\n",
"print(\"model: \", MODEL)\n",
"print(\"sandbox:\", SANDBOX)"
]
},
{
"cell_type": "markdown",
"id": "9b1405b5",
"metadata": {},
"source": [
"### The Toy Repository\n",
"\n",
"Our agent needs something to fix. A tiny package `stats` with a classic bug: `median` ignores the even-length case. The tests catch it.\n",
"\n",
"`make_toy_repo()` creates the repo once and commits it; afterwards it **resets** it with `git reset --hard && git clean -fdx`. It never deletes the directory itself, because the container bind-mounts it, and a deleted-and-recreated directory would be a *different* directory from the container's point of view.\n",
"\n",
"`git` here is **harness code running on the host**, not an action: we use it to reset the repo and, later, to see what the model's commands changed (`git diff`). The container image has no git, so the model's commands cannot use it, and the prompt says so."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "502d5003",
"metadata": {},
"outputs": [],
"source": [
"TOY_FILES = {\n",
" \"stats/__init__.py\": \"\",\n",
" \"stats/core.py\": '''\\\n",
"def mean(xs):\n",
" return sum(xs) / len(xs)\n",
"\n",
"\n",
"def median(xs):\n",
" \"\"\"Return the median of a non-empty list of numbers.\"\"\"\n",
" xs = sorted(xs)\n",
" return xs[len(xs) // 2]\n",
"''',\n",
" \"tests/__init__.py\": \"\",\n",
" \"tests/test_core.py\": '''\\\n",
"import unittest\n",
"\n",
"from stats.core import mean, median\n",
"\n",
"\n",
"class TestStats(unittest.TestCase):\n",
" def test_mean(self):\n",
" self.assertEqual(mean([1, 2, 3]), 2)\n",
"\n",
" def test_median_odd(self):\n",
" self.assertEqual(median([3, 1, 2]), 2)\n",
"\n",
" def test_median_even(self):\n",
" self.assertEqual(median([4, 1, 3, 2]), 2.5)\n",
"''',\n",
"}\n",
"\n",
"\n",
"def git(*args):\n",
" return subprocess.run([\"git\", *args], cwd=SANDBOX, check=True, capture_output=True, text=True).stdout\n",
"\n",
"\n",
"def make_toy_repo():\n",
" \"\"\"Create the sandbox repo on first use; afterwards reset it to the initial commit.\"\"\"\n",
" if (SANDBOX / \".git\").exists():\n",
" git(\"reset\", \"--hard\", \"-q\")\n",
" git(\"clean\", \"-fdxq\")\n",
" return\n",
" if SANDBOX.exists() and any(SANDBOX.iterdir()):\n",
" raise RuntimeError(f\"{SANDBOX} exists and is not our toy repo; refusing to touch it\")\n",
" for name, text in TOY_FILES.items():\n",
" (SANDBOX / name).parent.mkdir(parents=True, exist_ok=True)\n",
" (SANDBOX / name).write_text(text)\n",
" git(\"init\", \"-q\")\n",
" git(\"add\", \".\")\n",
" git(\"-c\", \"user.name=tutorial\", \"-c\", \"user.email=tutorial@example.invalid\", \"commit\", \"-qm\", \"toy repo\")\n",
"\n",
"\n",
"make_toy_repo()\n",
"print(git(\"ls-files\"))"
]
},
{
"cell_type": "markdown",
"id": "a29e45de",
"metadata": {},
"source": [
"## The Big Picture: Model, Loop, Environment\n",
"\n",
"The authors of mini-swe-agent define the whole thing in one sentence:\n",
"\n",
"> *\"An AI agent is just a big loop: You start with a prompt, the agent proposes an action, you execute the action, tell the LM the output, and then repeat.\"* (Lieret et al., 2026)\n",
"\n",
"Three components take part in that loop, and mini-swe-agent gives each one its own class:\n",
"\n",
"| Component | Job | mini-swe-agent |\n",
"|---|---|---|\n",
"| **Model** | messages in → one assistant message (+ the parsed action) out | `LitellmModel`, `LitellmTextbasedModel` |\n",
"| **Environment** | one action in → one observation (output, return code) out | `LocalEnvironment`, `DockerEnvironment`, … |\n",
"| **Agent** | the loop: history, budgets, termination | `DefaultAgent`, `InteractiveAgent` |\n",
"\n",
"\n",
"\n",
"**Concept.** This is the **ReAct** loop (→ [L-ReAct](http://christianherta.de/lehre/ai-agents/lecture-02/lecture-02-ReAct.html)): the model's THOUGHT is the reasoning trace, the bash command is the action, and the command's output is the observation ([Yao et al., 2023](https://arxiv.org/html/2210.03629)). Everything around the model call is the **harness** (→ [L-Intro](http://christianherta.de/lehre/ai-agents/lecture-01/lecture-01-Intro.html)). The model is a given; this notebook is entirely about the harness."
]
},
{
"cell_type": "markdown",
"id": "b2b7cee5",
"metadata": {},
"source": [
"## Talking to a Model\n",
"\n",
"**LiteLLM** exposes one function, `completion`, in the OpenAI chat format, and translates it to 100+ providers. The provider is chosen by a prefix in the model string: `ollama_chat/…`, `anthropic/…`, `openai/…` (BerriAI, 2026). A request is a list of **messages**, each a dict with a `role` (`system`, `user`, `assistant`) and a `content`."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5141cb80",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": ""
},
"tags": []
},
"outputs": [],
"source": [
"messages = [\n",
" {\"role\": \"system\", \"content\": \"You are a terse assistant.\"},\n",
" {\"role\": \"user\", \"content\": \"What is 17 * 23? Reply with the number only.\"},\n",
"]\n",
"response = litellm.completion(model=MODEL, messages=messages)\n",
"print(response.choices[0].message.content)"
]
},
{
"cell_type": "markdown",
"id": "39773326",
"metadata": {},
"source": [
"The response carries more than the text. Three fields matter for an agent:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "778bb93e",
"metadata": {},
"outputs": [],
"source": [
"choice = response.choices[0]\n",
"print(\"role: \", choice.message.role)\n",
"print(\"finish_reason:\", choice.finish_reason) # \"stop\" = turn ended normally, \"length\" = hit max_tokens\n",
"print(\"usage: \", response.usage) # prompt + completion tokens\n",
"print(\"cost ($): \", litellm.completion_cost(response))"
]
},
{
"cell_type": "markdown",
"id": "81a60b8c",
"metadata": {},
"source": [
"- `finish_reason == \"length\"` means the reply was **cut off**. An agent should treat that differently from a badly formatted reply (the format-error template below does exactly that).\n",
"- `usage` is what you pay for with an API model. mini-swe-agent's default budget is **money** ($3 per task), computed from `usage` and LiteLLM's price table. For a local model the cost is 0, so we will budget **tokens** instead.\n",
"\n",
"**The API is stateless.** Ask a follow-up question without the history:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9d8b1965",
"metadata": {},
"outputs": [],
"source": [
"followup = {\"role\": \"user\", \"content\": \"What did I just ask you?\"}\n",
"\n",
"forgetful = litellm.completion(model=MODEL, messages=[followup])\n",
"print(\"without history:\", forgetful.choices[0].message.content[:200])\n",
"\n",
"history = messages + [{\"role\": \"assistant\", \"content\": choice.message.content}, followup]\n",
"print(\"with history: \", litellm.completion(model=MODEL, messages=history).choices[0].message.content[:200])"
]
},
{
"cell_type": "markdown",
"id": "93eca6fa",
"metadata": {},
"source": [
"The model remembers nothing between calls. Everything it \"knows\" about the run is in the list we send, so **the message list is the agent's entire state**. The agent appends to it at every step and sends all of it again with every call.\n",
"\n",
"**Concept.** Every message costs tokens and has to fit into the **context window** (→ [L-Att](http://christianherta.de/lehre/ai-agents/lecture-07/lecture-07-Att.html)). Keep this in mind: our history will only ever grow."
]
},
{
"cell_type": "markdown",
"id": "29ca434c",
"metadata": {},
"source": [
"## Input: System Prompt and Task Template\n",
"\n",
"Every run starts with two messages:\n",
"\n",
"- **system**: who the model is and the **format contract**: exactly one bash block, preceded by a THOUGHT,\n",
"- **user**: the concrete task, plus workflow advice and a few command recipes.\n",
"\n",
"mini-swe-agent keeps both as **Jinja** templates in a YAML config (`config/default.yaml`). Here is a shortened version:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b3cc1731",
"metadata": {},
"outputs": [],
"source": [
"SYSTEM_TEMPLATE = \"\"\"\\\n",
"You are a helpful assistant that can interact with a computer.\n",
"\n",
"Your response must contain exactly ONE bash code block with ONE command.\n",
"A pipeline (|) or commands joined with && or || count as one command.\n",
"Include a THOUGHT section before your command where you explain your reasoning process.\n",
"\n",
"\n",
"Your reasoning and analysis here. Explain why you want to perform the action.\n",
"\n",
"```mswea_bash_command\n",
"your_command_here\n",
"```\n",
"\n",
"\n",
"Failure to follow these rules will cause your response to be rejected.\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e332ab1b",
"metadata": {},
"outputs": [],
"source": [
"INSTANCE_TEMPLATE = \"\"\"\\\n",
"Please solve this issue: {{task}}\n",
"\n",
"You are in a Python repository at /repo (no git available).\n",
"You can execute bash commands and edit files to implement the necessary changes.\n",
"\n",
"## Recommended Workflow\n",
"\n",
"This workflow should be done step-by-step so that you can iterate on your changes and any possible problems.\n",
"\n",
"1. Analyze the codebase by finding and reading relevant files\n",
"2. Create a script to reproduce the issue\n",
"3. Edit the source code to resolve the issue\n",
"4. Verify your fix works by running your script again\n",
"5. Test edge cases to ensure your fix is robust\n",
"6. Submit your changes and finish your work by issuing the following command: `echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT`.\n",
" Do not combine it with any other command. After this command, you cannot continue working on this task.\n",
"\n",
"## Important Rules\n",
"\n",
"1. Every response must contain exactly one action\n",
"2. The action must be enclosed in triple backticks\n",
"3. Directory or environment variable changes are not persistent. Every action is executed in a new subshell.\n",
" However, you can prefix any action with `MY_ENV_VAR=MY_VALUE cd /path/to/working/dir && ...` or write/load environment variables from files\n",
"\n",
"## Useful command examples\n",
"Create a file:\n",
"```mswea_bash_command\n",
"cat <<'EOF' > newfile.py\n",
"print(\"hello\")\n",
"EOF\n",
"```\n",
"Change more than one line of a file: first read the file (`cat -n filename.py`), then write\n",
"its COMPLETE new content. Do not use sed for multi-line changes.\n",
"```mswea_bash_command\n",
"cat <<'EOF' > filename.py\n",
"\n",
"EOF\n",
"```\n",
"Replace text within a single line:\n",
"```mswea_bash_command\n",
"sed -i 's/old_string/new_string/g' filename.py\n",
"```\n",
"Show lines 10-20 with line numbers:\n",
"```mswea_bash_command\n",
"nl -ba filename.py | sed -n '10,20p'\n",
"```\n",
"\"\"\"\n",
"\n",
"\n",
"def render(template, **variables):\n",
" return Template(template, undefined=StrictUndefined).render(**variables)\n",
"\n",
"\n",
"print(render(INSTANCE_TEMPLATE, task=\"Make the tests pass.\")[:300], \"…\")"
]
},
{
"cell_type": "markdown",
"id": "5da7878d",
"metadata": {},
"source": [
"**Where the templates come from.** Both are adapted from `agent.system_template` and `agent.instance_template` in mini-swe-agent's `src/minisweagent/config/default.yaml`; the instance template in `config/mini_textbased.yaml`, which the final section loads, is nearly identical. They are not verbatim copies:\n",
"\n",
"- **Taken over verbatim:** the task line (*\"Please solve this issue: {{task}}\"*), the six-step recommended workflow with the sentinel instruction, and the three important rules (one action per response, the fence, a new subshell per command). The workflow steps build on each other: the script created in step 2 is the one re-run in step 4, so these lines are kept word for word.\n",
"- **Added:** the instance template gets *\"You are in a Python repository at /repo (no git available)\"*, which describes our container: `python:3.12-slim` has no git. A recipe for **multi-line edits** is added as well: read the file, then write its complete new content with `cat <<'EOF'`. Without it, a 7B model tends to put a multi-line fix into one `sed` command, gets the quoting wrong, and retries the same broken command until the step limit is reached.\n",
"- **Clarified:** mini's system template asks for *\"ONE command (or commands connected with && or ||)\"*. It does not mention pipes, although its own recipes use them (`nl -ba … | sed -n …`). We changed the rule to say that a pipeline counts as one command. Note that the harness does not check this rule at all: `parse_action` only counts fenced blocks, and whatever the block contains goes to `bash -lc` unchanged. The rule asks the model for one logical step per turn.\n",
"- **Shortened:** the command examples: the `sed` recipes are cut from four variants to one.\n",
"- **Removed:** the `{{system}} {{release}} …` block, the *\"Formatting your response\"* example (the system template already carries one), the macOS hint (`sed -i ''`) and the catch-all *\"any other command\"* example.\n",
"\n",
"**The template syntax.** Both strings are **Jinja2** templates, the templating language mini keeps its prompts in. `render` above is the whole API we use of it: `Template(source).render(**variables)`.\n",
"\n",
"| Syntax | Meaning |\n",
"|---|---|\n",
"| `{{task}}` | insert the value of the variable `task` |\n",
"| `{{actions \\| length}}` | insert a value through a **filter**; `length` yields the number of elements |\n",
"| `{% if … %} … {% else %} … {% endif %}` | a control block: it selects what is rendered and produces no text of its own |\n",
"| `{%- …` and `… -%}` | the same tags, with the whitespace on that side removed |\n",
"\n",
"Whitespace control earns its place because the rendered string reaches the model verbatim: a control tag on a line of its own would otherwise leave an empty line where it stood. The two templates above use `{{…}}` alone; the observation template and the format-error template below also use the conditional form.\n",
"\n",
"Three design decisions are hidden in these two strings:\n",
"\n",
"- **`StrictUndefined`**: a misspelled template variable raises an error instead of silently leaving a hole in the prompt. (Try `render(INSTANCE_TEMPLATE)` without `task=`.)\n",
"- **The odd fence name `mswea_bash_command`.** Models write ```` ```bash ```` blocks all the time, in their reasoning and in examples. A tag nobody uses by accident separates *\"this is my action\"* from *\"here is an example\"*.\n",
"- **The recommended workflow** (analyse → write a reproduction script → fix → re-run the script → test edge cases → submit) is a fixed plan for the task, written into the prompt instead of into code.\n",
"\n",
"**Concept.** The system prompt defines the agent's **action space** (one shell command per step) and its **output format**. The workflow list is task decomposition by prompt (→ [L-Decomp](http://christianherta.de/lehre/ai-agents/lecture-03/lecture-03-Decomp.html)): the model is *asked* to follow the plan, nothing *enforces* it."
]
},
{
"cell_type": "markdown",
"id": "4e42cbcb",
"metadata": {},
"source": [
"## Output: Parsing the Action\n",
"\n",
"The model returns free text. The harness has to find exactly one action in it. mini's text-based model uses a regular expression over the fenced block:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "eca74c17",
"metadata": {},
"outputs": [],
"source": [
"ACTION_REGEX = r\"```mswea_bash_command\\s*\\n(.*?)\\n```\"\n",
"\n",
"\n",
"class FormatError(Exception):\n",
" \"\"\"The model's reply does not contain exactly one action.\"\"\"\n",
"\n",
"\n",
"def parse_action(text):\n",
" actions = re.findall(ACTION_REGEX, text, re.DOTALL)\n",
" if len(actions) != 1:\n",
" raise FormatError(f\"Expected exactly 1 action, found {len(actions)}.\")\n",
" return actions[0].strip()"
]
},
{
"cell_type": "markdown",
"id": "28493e45",
"metadata": {},
"source": [
"**The regex, piece by piece** (it is mini's default `action_regex`):\n",
"\n",
"| Part | Matches |\n",
"|---|---|\n",
"| ```` ```mswea_bash_command ```` | the opening fence with its tag; plain ```` ```bash ```` blocks are ignored |\n",
"| `\\s*\\n` | optional whitespace, then a newline: the command starts on the next line |\n",
"| `(.*?)` | the command, the only capture group, so `findall` returns just the command |\n",
"| ```` \\n``` ```` | the closing fence at the start of a line |\n",
"\n",
"- **`re.DOTALL`** lets `.` match newlines, so multi-line commands (heredocs) are captured.\n",
"- **The lazy `.*?`** stops at the *first* closing fence. A greedy `.*` would merge two blocks into one match, including the text between them.\n",
"- **Edge cases:** a line starting with ```` ``` ```` inside the command (e.g. a heredoc that writes Markdown) ends the capture early; an empty block yields the empty command `\"\"`; a one-line block ```` ```mswea_bash_command ls``` ```` does not match at all."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "53e6df90",
"metadata": {},
"outputs": [],
"source": [
"good = \"Let me look around first.\\n\\n```mswea_bash_command\\nls -la\\n```\"\n",
"print(\"parsed:\", repr(parse_action(good)))\n",
"\n",
"for bad in [\"I think we should run ls.\", good + \"\\n\\n\" + good]:\n",
" try:\n",
" parse_action(bad)\n",
" except FormatError as e:\n",
" print(\"FormatError:\", e)"
]
},
{
"cell_type": "markdown",
"id": "6beb543a",
"metadata": {},
"source": [
"Why **exactly one**? One action gives one observation. The model sees the result of every command before it chooses the next, which is the interleaving that makes ReAct work. What to do *with* a `FormatError` comes later; the first loops simply stop on it.\n",
"\n",
"**Blocks, not commands.** `parse_action` counts **blocks**, not the commands inside one. A block holding a whole script (several lines, `;`, a loop) passes, and `bash -lc` runs all of it; mini's parser behaves the same. The prompt's \"ONE command\" rule is therefore a request to the model, not a check. A real check would need a bash parser: a heredoc spans several lines but is one command, a `;` inside quotes separates nothing, and `bash -c \"…\"` hides a whole script in one command. It would also reject the multi-line heredocs the prompt itself recommends. The rule serves the model's checking: with one command per turn, the model sees each result before it decides the next step. A script is executed in full anyway: later commands run even when an earlier one failed (unless joined with `&&`), the return code reports only the last command, and all outputs arrive mixed in one observation. The submit command `echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT` (the **sentinel**, see the Termination section below) has an extra constraint: it is only recognised on the first line of output, so it must not follow other commands."
]
},
{
"cell_type": "markdown",
"id": "74d09b92",
"metadata": {},
"source": [
"## Environment: A Sandboxed Shell\n",
"\n",
"The environment turns a command into an **observation**. First, who runs what: the model only produces text, and **the harness executes every command**. What differs is *who wrote the command*, and that decides *where* it runs:\n",
"\n",
"| | Who writes the command | Where the harness runs it |\n",
"|---|---|---|\n",
"| **Actions** | the model (parsed from its reply) | inside the container, via `podman exec` |\n",
"| **The harness's own commands** | us, as fixed code: `podman run`, `git reset`, `git diff` | on the host |\n",
"\n",
"The sandbox exists because of the first row: the content of those commands comes from a language model, so the harness runs them only where they cannot do damage. The second row is ordinary program code and runs like any other script.\n",
"\n",
"mini's `DockerEnvironment` starts one long-lived container (`sleep 2h`) and runs each command with `docker exec`. It accepts any compatible executable (`executable=\"podman\"`, or the environment variable `MSWEA_DOCKER_EXECUTABLE`), so podman works the same way.\n",
"\n",
"**The container is not mini's default.** The `mini` command and the `hello_world.py` example use `LocalEnvironment`, which runs each model-chosen command **directly on the host** with `subprocess.Popen(command, shell=True, …)`, without any sandbox. And even `DockerEnvironment` sets no security flags of its own: its `run_args` default to just `[\"--rm\"]`. Isolation in mini is something you configure, not something you get.\n",
"\n",
"We start the container the same way, and lock it down:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5a1cbd81",
"metadata": {},
"outputs": [],
"source": [
"def start_container(image=IMAGE, workdir=SANDBOX):\n",
" cmd = [\n",
" \"podman\", \"run\", \"-d\", \"--rm\",\n",
" \"--network=none\", # no internet: nothing to download, nothing to leak\n",
" \"--cap-drop=ALL\", # no Linux capabilities, even as root in the container\n",
" \"--memory=1g\", \"--pids-limit=256\", # a fork bomb or a memory leak stays inside\n",
" \"-v\", f\"{workdir}:/repo:Z\", # the toy repo is the only host directory it sees\n",
" \"-w\", \"/repo\",\n",
" image, \"sleep\", \"2h\", # keep it alive; we exec commands into it\n",
" ]\n",
" return subprocess.run(cmd, capture_output=True, text=True, check=True).stdout.strip()\n",
"\n",
"\n",
"CONTAINER = start_container()\n",
"print(\"container:\", CONTAINER[:12])"
]
},
{
"cell_type": "markdown",
"id": "3a2cffc9",
"metadata": {},
"source": [
"Each command runs as `bash -lc ` in a **new** process. The observation is a dict: the combined stdout/stderr, the return code, and an error note if something went wrong on *our* side (a timeout)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "bb71859a",
"metadata": {},
"outputs": [],
"source": [
"ENV_VARS = {\"PAGER\": \"cat\", \"MANPAGER\": \"cat\", \"LESS\": \"-R\", \"PIP_PROGRESS_BAR\": \"off\", \"TQDM_DISABLE\": \"1\"}\n",
"\n",
"def execute(command, timeout=30, container=None):\n",
" \"\"\"Run one bash command in the container and return the observation dict.\"\"\"\n",
" env_flags = [f\"--env={key}={value}\" for key, value in ENV_VARS.items()]\n",
" cmd = [\"podman\", \"exec\", \"-w\", \"/repo\", *env_flags, container or CONTAINER, \"bash\", \"-lc\", command]\n",
" try:\n",
" result = subprocess.run(cmd, text=True, timeout=timeout, encoding=\"utf-8\", errors=\"replace\",\n",
" stdout=subprocess.PIPE, stderr=subprocess.STDOUT)\n",
" return {\"output\": result.stdout, \"returncode\": result.returncode, \"exception_info\": \"\"}\n",
" except subprocess.TimeoutExpired as e:\n",
" output = e.output.decode(\"utf-8\", errors=\"replace\") if isinstance(e.output, bytes) else (e.output or \"\")\n",
" return {\"output\": output, \"returncode\": -1, \"exception_info\": f\"Command timed out after {timeout}s.\"}"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cf6ebf8c",
"metadata": {},
"outputs": [],
"source": [
"print(execute(\"python --version && ls\"))\n",
"print(execute(\"cd stats && pwd\"))\n",
"print(execute(\"pwd\")) # back in /repo: the cd above is gone\n",
"print(execute(\"python -m unittest -q 2>&1 | tail -3\"))\n",
"print(execute(\"sleep 5\", timeout=2))"
]
},
{
"cell_type": "markdown",
"id": "b85b0c50",
"metadata": {},
"source": [
"Things to notice:\n",
"\n",
"- **stderr is merged into stdout.** The model has to see error messages; they are often the most useful part of an observation.\n",
"- **A non-zero return code is not a harness error.** A failing test run is exactly the information the model is looking for. Only *our* problems (timeouts) go into `exception_info`.\n",
"- **Every command gets a fresh subshell**, so `cd` and `export` do not persist. That is mini's deliberate choice: simple, reproducible, and no shell can end up stuck in a broken state. SWE-agent, the team's earlier agent, kept a persistent shell session instead ([Yang et al., 2024](https://arxiv.org/html/2405.15793)). The prompt tells the model about the choice and how to work with it (prefix the action with `cd /path/to/working/dir && …`).\n",
"- **`PAGER=cat` and friends.** A command that opens an interactive pager (`git log`, `man`) would otherwise wait forever for a key press that never comes (Lieret et al., 2026).\n",
"\n",
"(The timeout kills `podman exec`, but a process inside the container may keep running. The container's own lifetime, `sleep 2h`, is the final limit. mini has the same property.)"
]
},
{
"cell_type": "markdown",
"id": "bff06e6a",
"metadata": {},
"source": [
"## A Minimal Agent Loop\n",
"\n",
"Now we glue the pieces together: query → parse → execute → append the observation, and repeat."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f2291a75",
"metadata": {},
"outputs": [],
"source": [
"def render_observation(output):\n",
" return f\"{output['returncode']}\\n\"\n",
"\n",
"\n",
"def minimal_agent(task, max_steps=6):\n",
" messages = [\n",
" {\"role\": \"system\", \"content\": render(SYSTEM_TEMPLATE)},\n",
" {\"role\": \"user\", \"content\": render(INSTANCE_TEMPLATE, task=task)},\n",
" ]\n",
" for _ in range(max_steps):\n",
" reply = litellm.completion(model=MODEL, messages=messages).choices[0].message.content # think + choose\n",
" messages.append({\"role\": \"assistant\", \"content\": reply})\n",
" try:\n",
" output = execute(parse_action(reply)) # act\n",
" except FormatError as e: # not handled yet: stop and show where\n",
" print(f\"STOPPED by FormatError: {e}\\n\")\n",
" break\n",
" messages.append({\"role\": \"user\", \"content\": render_observation(output)}) # observe\n",
" return messages"
]
},
{
"cell_type": "markdown",
"id": "e2e0d585",
"metadata": {},
"source": [
"That's the whole idea in a dozen lines; the `try` only makes a format error stop the run visibly. A small printer to read the trace:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "118ab57b",
"metadata": {},
"outputs": [],
"source": [
"def show(messages, width=1200):\n",
" for m in messages:\n",
" print(f\"── {m['role'].upper()} \" + \"─\" * 60)\n",
" text = m[\"content\"] or \"\"\n",
" print(text if len(text) <= width else text[:width] + f\"\\n… [{len(text) - width} more characters]\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c78f80b7",
"metadata": {},
"outputs": [],
"source": [
"make_toy_repo()\n",
"trace = minimal_agent(\"Create a file hello.py that prints 'Hello from the sandbox', then run it.\")\n",
"show(trace[2:]) # skip system prompt and task"
]
},
{
"cell_type": "markdown",
"id": "e85b5b21",
"metadata": {},
"source": [
"Read the trace as a ReAct trace: THOUGHT, action, observation, THOUGHT, …. Note that the observation arrives as a **`user`** message: to the model, the environment speaks in the user's voice. (With tool calling it gets a role of its own, see below.)\n",
"\n",
"What is wrong with this loop:\n",
"\n",
"1. **It never stops by itself.** It runs `max_steps` times, even when the task was finished after two steps, and it has no way to know whether the task was finished at all.\n",
"2. **One badly formatted reply ends the run.** The loop catches `FormatError` only to stop; it does not recover.\n",
"3. **Any output goes into the history unfiltered.** One `cat` of a large file fills the context window.\n",
"4. **No budget** except the hard-coded step count.\n",
"\n",
"We fix these one at a time."
]
},
{
"cell_type": "markdown",
"id": "e768c59a",
"metadata": {},
"source": [
"## Termination: Knowing When to Stop\n",
"\n",
"A run can end in three ways: the agent says it is **done**, a **budget** is exhausted, or something **breaks**. mini implements all of them the same way: an exception that carries an `exit` message. The loop then needs only one check: *is the last message an exit message?*"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "430a388d",
"metadata": {},
"outputs": [],
"source": [
"class InterruptAgentFlow(Exception):\n",
" \"\"\"Interrupts the normal step; carries the messages to append to the history.\"\"\"\n",
"\n",
" def __init__(self, *messages):\n",
" super().__init__()\n",
" self.messages = messages\n",
"\n",
"\n",
"class Submitted(InterruptAgentFlow):\n",
" \"\"\"The agent says it is done.\"\"\"\n",
"\n",
"\n",
"class LimitsExceeded(InterruptAgentFlow):\n",
" \"\"\"A budget is exhausted.\"\"\"\n",
"\n",
"\n",
"def exit_message(status, submission=\"\"):\n",
" return {\"role\": \"exit\", \"content\": status, \"extra\": {\"exit_status\": status, \"submission\": submission}}"
]
},
{
"cell_type": "markdown",
"id": "0e1cf50b",
"metadata": {},
"source": [
"**How does the agent say \"done\"?** By running a **sentinel command**, `echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT`. The *environment* recognises the sentinel as the first line of a successful command's output. Anything printed after it becomes the **submission**, so `echo SENTINEL && cat answer.txt` hands in a result together with the signal. No extra tool and no special parsing are needed."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7f8df495",
"metadata": {},
"outputs": [],
"source": [
"SENTINEL = \"COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT\"\n",
"\n",
"\n",
"def check_finished(output):\n",
" lines = output[\"output\"].lstrip().splitlines(keepends=True)\n",
" if lines and lines[0].strip() == SENTINEL and output[\"returncode\"] == 0:\n",
" submission = \"\".join(lines[1:])\n",
" raise Submitted({**exit_message(\"Submitted\", submission), \"content\": submission})\n",
"\n",
"\n",
"def check_limits(n_calls, n_tokens, step_limit, token_limit):\n",
" if 0 < step_limit <= n_calls or 0 < token_limit <= n_tokens: # 0 = no limit, as in mini\n",
" raise LimitsExceeded(exit_message(\"LimitsExceeded\"))"
]
},
{
"cell_type": "markdown",
"id": "fb4864d7",
"metadata": {},
"source": [
"The loop is now `while` rather than `for`. The body is a single `try`, and every interruption just appends its messages:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8e0e5388",
"metadata": {},
"outputs": [],
"source": [
"def agent_v2(task, step_limit=15, token_limit=200_000):\n",
" messages = [\n",
" {\"role\": \"system\", \"content\": render(SYSTEM_TEMPLATE)},\n",
" {\"role\": \"user\", \"content\": render(INSTANCE_TEMPLATE, task=task)},\n",
" ]\n",
" n_calls = n_tokens = 0\n",
" while messages[-1][\"role\"] != \"exit\":\n",
" try:\n",
" check_limits(n_calls, n_tokens, step_limit, token_limit)\n",
" response = litellm.completion(model=MODEL, messages=messages)\n",
" n_calls += 1\n",
" n_tokens += response.usage.total_tokens\n",
" reply = response.choices[0].message.content or \"\"\n",
" messages.append({\"role\": \"assistant\", \"content\": reply})\n",
" output = execute(parse_action(reply))\n",
" check_finished(output) # may raise Submitted\n",
" messages.append({\"role\": \"user\", \"content\": render_observation(output)})\n",
" except FormatError as e: # still problem 2: stop, fixed in the next section\n",
" print(f\"STOPPED by FormatError: {e}\\n\")\n",
" break\n",
" except InterruptAgentFlow as e:\n",
" messages.extend(e.messages)\n",
" print(f\"{n_calls} model calls, {n_tokens} tokens processed\")\n",
" return messages"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "da90b5c7",
"metadata": {},
"outputs": [],
"source": [
"make_toy_repo()\n",
"trace = agent_v2(\"Create a file hello.py that prints 'Hello from the sandbox', run it, then submit.\")\n",
"show(trace[2:])\n",
"print(trace[-1].get(\"extra\"))"
]
},
{
"cell_type": "markdown",
"id": "94cebe33",
"metadata": {},
"source": [
"The `exit` role is not a role the API knows. It never reaches the model, because the loop ends as soon as it is appended.\n",
"\n",
"**A note on the token budget.** `total_tokens` counts the *whole prompt* again on every call. With a history that grows by roughly $\\ell$ tokens per step, call $t$ processes about $t\\ell$ tokens, so $n$ steps cost about $\\ell\\,n^2/2$ in total: **quadratic** in the number of steps. That is why long agent runs get expensive, and why history management matters (→ [L-Memory](http://christianherta.de/lehre/ai-agents/lecture-09/lecture-09-Memory.html)).\n",
"\n",
"mini has four limits, two of them on by default: steps and wall-clock time are `0`, meaning no limit; **money** is $3 per task; and consecutive format errors are capped at three in a row (next section). Adding a wall-clock limit is one more `if`.\n",
"\n",
"**Concept.** The exit message is the **terminal state** of an episode; the step limit is a **horizon** (→ [L-MDP](http://christianherta.de/lehre/ai-agents/lecture-05/lecture-05-MDP.html)). mini ends an episode only on the agent's own signal or on a budget, never on a judgement of *whether the work is correct*. That judgement belongs to the evaluation, after the run."
]
},
{
"cell_type": "markdown",
"id": "abe1baff",
"metadata": {},
"source": [
"## Errors Become Observations\n",
"\n",
"The mini-swe-agent team's rule for known failures:\n",
"\n",
"> *\"When known exceptions arise (timeouts, format errors, etc.), let's just tell the LM and let it handle it itself.\"* (Lieret et al., 2026)\n",
"\n",
"So a `FormatError` no longer ends the run: it becomes a **message to the model** that explains what went wrong. We redefine it as an interruption that carries that message:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2bd4a25b",
"metadata": {},
"outputs": [],
"source": [
"FORMAT_ERROR_TEMPLATE = \"\"\"\\\n",
"{% if finish_reason in [\"length\", \"tool_calls\"] -%}\n",
"Your previous response reached the output token limit (finish_reason={{finish_reason}}) before you\n",
"produced a complete action, so it was cut off. Respond more concisely and provide exactly one action\n",
"in the required format. If you need to think more, do so briefly.\n",
"{%- else -%}\n",
"Format error:\n",
"\n",
"\n",
"{{error}}\n",
"\n",
"\n",
"Here is general guidance on how to format your response:\n",
"\n",
"Please always provide EXACTLY ONE action in triple backticks, found {{actions|length}} actions.\n",
"\n",
"Please format your action in triple backticks as shown in .\n",
"\n",
"\n",
"Here are some thoughts about why you want to perform the action.\n",
"\n",
"```mswea_bash_command\n",
"\n",
"```\n",
"\n",
"\n",
"If you have completed your assignment, please consult the first message about how to\n",
"submit your solution (you will not be able to continue working on this task after that).\n",
"{%- endif %}\"\"\"\n",
"\n",
"\n",
"class FormatError(InterruptAgentFlow):\n",
" \"\"\"The model's reply does not contain exactly one action.\"\"\"\n",
"\n",
"\n",
"def parse_action(text, finish_reason=\"\"):\n",
" actions = re.findall(ACTION_REGEX, text, re.DOTALL)\n",
" if len(actions) != 1:\n",
" error = f\"Expected exactly 1 action, found {len(actions)}.\"\n",
" content = render(FORMAT_ERROR_TEMPLATE, error=error, actions=actions, finish_reason=finish_reason)\n",
" raise FormatError({\"role\": \"user\", \"content\": content,\n",
" \"extra\": {\"n_actions\": len(actions), \"model_response\": text}})\n",
" return actions[0].strip()\n",
"\n",
"\n",
"for text, reason in [(\"I would run ls now.\", \"stop\"), (\"Let me begin by carefully considering\", \"length\")]:\n",
" try:\n",
" parse_action(text, reason)\n",
" except FormatError as e:\n",
" print(f\"── finish_reason={reason} \" + \"─\" * 40)\n",
" print(e.messages[0][\"content\"])"
]
},
{
"cell_type": "markdown",
"id": "a8e60c87",
"metadata": {},
"source": [
"Two more robustness pieces:\n",
"\n",
"- **A cap on consecutive format errors.** Three bad replies *in a row* end the run with `RepeatedFormatError`; any clean step resets the counter. So this is not really a budget. It detects a model that is **stuck**.\n",
"- **Retries for API errors**, with backoff, except for errors that a retry cannot fix (wrong key, unknown model, context window exceeded) — mini's `abort_exceptions`.\n",
"\n",
"Note that the two kinds of failure have nothing in common. A format error means the reply **arrived** and says the wrong thing; retrying the same request would only produce it again, so it is answered with a message. An API error means the reply **did not arrive**; the model never learns that it happened.\n",
"\n",
"`prepare_messages_for_api` in the same cell belongs to neither. It is the boundary between our own bookkeeping and the request: our messages carry an `extra` field of things the chat API has no vocabulary for, and only `role` and `content` are sent. The name is mini's own, where it is a method of the model class."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "982a0b08",
"metadata": {},
"outputs": [],
"source": [
"def prepare_messages_for_api(messages):\n",
" \"\"\"Our messages carry an \"extra\" field; the chat API knows role and content.\"\"\"\n",
" return [{\"role\": m[\"role\"], \"content\": m[\"content\"]} for m in messages]\n",
"\n",
"\n",
"ABORT_EXCEPTIONS = (litellm.exceptions.AuthenticationError, litellm.exceptions.NotFoundError,\n",
" litellm.exceptions.ContextWindowExceededError)\n",
"\n",
"\n",
"# Retries for model call API errors: the reply did not arrive. A format error is not an API error —\n",
"# the reply arrived and says the wrong thing — and is never retried.\n",
"def query_with_retry(messages, model=MODEL, attempts=3, **kwargs):\n",
" for attempt in range(attempts):\n",
" try:\n",
" return litellm.completion(model=model, messages=messages, **kwargs)\n",
" except ABORT_EXCEPTIONS:\n",
" raise\n",
" except Exception:\n",
" if attempt == attempts - 1:\n",
" raise\n",
" time.sleep(2 ** attempt)"
]
},
{
"cell_type": "markdown",
"id": "49efffb2",
"metadata": {},
"source": [
"The loop, with the new lines marked `# NEW`. Since `FormatError` is a subclass of `InterruptAgentFlow`, its `except` clause has to come first:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1f78c596-2b10-40d2-8566-2ca9fba6d614",
"metadata": {},
"outputs": [],
"source": [
"def agent_v3(\n",
" task: str,\n",
" step_limit: int = 15,\n",
" token_limit: int = 200_000,\n",
" max_consecutive_format_errors: int = 3, # NEW\n",
") -> list[dict]:\n",
" \"\"\"Runs the main agent loop to solve a given task.\n",
"\n",
" Calls & Exception Mapping:\n",
" -------------------------\n",
" - check_limits(...) -> raises LimitsExceeded\n",
" - query_with_retry(...) -> raises API/Network errors (e.g., ContextWindowExceededError)\n",
" - parse_action(...) -> raises FormatError\n",
" - execute(...) -> executes action inside container environment\n",
" - check_finished(...) -> raises Submitted\n",
" \"\"\"\n",
" messages = [\n",
" {\"role\": \"system\", \"content\": render(SYSTEM_TEMPLATE)},\n",
" {\"role\": \"user\", \"content\": render(INSTANCE_TEMPLATE, task=task)},\n",
" ]\n",
"\n",
" n_calls = 0\n",
" n_tokens = 0\n",
" n_consecutive_format_errors = 0 # NEW\n",
"\n",
" while messages[-1][\"role\"] != \"exit\":\n",
" try:\n",
" # 1. Budget enforcement (Raises: LimitsExceeded)\n",
" check_limits(n_calls, n_tokens, step_limit, token_limit)\n",
"\n",
" # 2. Model call (Raises: API errors)\n",
" api_messages = prepare_messages_for_api(messages) # NEW\n",
" response = query_with_retry(api_messages) # NEW\n",
"\n",
" n_calls += 1\n",
" n_tokens += response.usage.total_tokens\n",
" reply = response.choices[0].message.content or \"\"\n",
"\n",
" # 3. Format validation (Raises: FormatError)\n",
" # Validated BEFORE appending assistant message to keep invalid turns out of context history.\n",
" action = parse_action(reply, response.choices[0].finish_reason) # NEW\n",
" messages.append({\"role\": \"assistant\", \"content\": reply})\n",
"\n",
" # 4. Action execution & termination check (Raises: Submitted)\n",
" output = execute(action)\n",
" check_finished(output)\n",
"\n",
" # Successful step: record observation and reset format error counter\n",
" messages.append({\"role\": \"user\", \"content\": render_observation(output)})\n",
" n_consecutive_format_errors = 0 # NEW\n",
"\n",
" except FormatError as e: # NEW: tell the model, go on\n",
" # Tell the model what went wrong via prompt feedback and continue\n",
" n_consecutive_format_errors += 1\n",
" messages.extend(e.messages)\n",
"\n",
" # Check if consecutive format errors exceed the max threshold (0 = no limit)\n",
" if 0 < max_consecutive_format_errors <= n_consecutive_format_errors:\n",
" messages.append(exit_message(\"RepeatedFormatError\"))\n",
"\n",
" except InterruptAgentFlow as e:\n",
" # Catch flow control interrupts (e.g., LimitsExceeded, Submitted)\n",
" messages.extend(e.messages)\n",
"\n",
" print(f\"{n_calls} model calls, {n_tokens} tokens processed\")\n",
" return messages"
]
},
{
"cell_type": "markdown",
"id": "5b32d5ef",
"metadata": {},
"source": [
"The malformed reply never enters the history, though the failure is not concealed: `parse_action` raises before the assistant message is appended, and the correction that takes its place states what went wrong (*Expected exactly 1 action, found 0*) and how to write the action. A reply declared wrong is still an example of the wrong format in the model's own context, and a stuck model would contribute three of them to a history that is paid for on every later call. The text is recorded rather than lost: `extra[\"model_response\"]` holds it, and `prepare_messages_for_api` drops `extra` before sending, so the trajectory keeps what the model wrote without showing it again.\n",
"\n",
"A failed command is treated the opposite way: the command and its error both stay, as the pair the next step is derived from. The line is not *successful* against *failed* but **observation** against **protocol violation** — a failing test run is a fact about the repository, a reply the harness cannot read is a fact about the reply alone. That decides what counts as information, not what it costs: the failure is still paid for in the context window, and shaping the observation is the only thing mini does about it.\n",
"\n",
"The template is mini's, including the branch on `finish_reason`: a reply cut off at `max_tokens` is told to *\"respond more concisely\"* rather than to *\"use the right format\"* — same symptom, different cause, different advice. The rest restates the contract and shows a ``, because the reply that broke it is no longer there to be corrected."
]
},
{
"cell_type": "markdown",
"id": "2b339811",
"metadata": {},
"source": [
"## Observation Shaping: Protecting the Context Window\n",
"\n",
"History is linear: whatever enters it stays and is sent again on every call. One `cat` of a large log would fill the context window for the rest of the run. mini's observation template therefore **truncates**: up to 10,000 characters are shown in full; beyond that the model sees the first and last 5,000 characters and a warning that tells it how to do better."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "bd72072d",
"metadata": {},
"outputs": [],
"source": [
"OBSERVATION_TEMPLATE = \"\"\"\\\n",
"{% if output.exception_info -%}\n",
"{{output.exception_info}}\n",
"{% endif -%}\n",
"{{output.returncode}}\n",
"{% if output.output | length < 10000 -%}\n",
"\n",
"{%- else -%}\n",
"\n",
"The output of your last command was too long. Please try a command that produces less output,\n",
"e.g. use head, tail, grep or sed -n to view a part of a file.\n",
"\n",
"\n",
"{{ output.output[:5000] }}\n",
"\n",
"\n",
"{{ output.output | length - 10000 }} characters elided\n",
"\n",
"\n",
"{{ output.output[-5000:] }}\n",
"\n",
"{%- endif %}\"\"\"\n",
"\n",
"\n",
"def render_observation(output):\n",
" return render(OBSERVATION_TEMPLATE, output=output)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "57a3acd1",
"metadata": {},
"outputs": [],
"source": [
"big = execute(\"seq 1 100000\")\n",
"obs = render_observation(big)\n",
"print(f\"raw output: {len(big['output']):,} characters -> observation: {len(obs):,} characters\")\n",
"print(obs[:241], \"\\n…\")"
]
},
{
"cell_type": "markdown",
"id": "bd75a6af",
"metadata": {},
"source": [
"We just *redefined* `render_observation`. `agent_v3` looks the name up each time it is called, so it now uses the new template without being rewritten. The same holds for `parse_action` above. Convenient in a notebook; in real code, this is what configuration objects are for (see the refactoring below).\n",
"\n",
"**Concept.** Every token of an observation is paid for again on every later step and occupies the finite **context window** (→ [L-Att](http://christianherta.de/lehre/ai-agents/lecture-07/lecture-07-Att.html)). mini never summarises or drops old messages: the history is the full trajectory. That makes it the baseline against which **compaction** and memory strategies are measured (→ [L-Memory](http://christianherta.de/lehre/ai-agents/lecture-09/lecture-09-Memory.html)). The warning text is a small piece of **agent–computer interface** design: the observation tells the model how to act better, which is the idea SWE-agent is built on ([Yang et al., 2024](https://arxiv.org/html/2405.15793)).\n",
"\n",
"**Truncation bounds the size of an observation, not its position.** Every failed attempt that is kept remains in the middle of the history for the rest of the run, and the middle is where a long input is used least reliably: accuracy on material placed there falls well below accuracy on material at the beginning or the end of the context ([Liu et al., 2024](https://arxiv.org/html/2307.03172)). A traceback that was the most useful thing in the context at step 4 describes, by step 20, a state the repository is no longer in — it has become a distractor that is still paid for on every call. mini accepts this deliberately, in exchange for a trajectory that is exactly what the model saw; removing superseded attempts is what compaction does instead, and it gives that property up."
]
},
{
"cell_type": "markdown",
"id": "ed26f05c",
"metadata": {},
"source": [
"## A Real Task: Fixing the Bug\n",
"\n",
"The loop is now complete enough for a real (if tiny) software-engineering task:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "118ece74",
"metadata": {},
"outputs": [],
"source": [
"TASK = (\n",
" \"The unit tests fail. Run them with `python -m unittest -v`, find the bug in the `stats` package \"\n",
" \"and fix it. Do not modify the tests.\"\n",
")\n",
"\n",
"\n",
"def verify():\n",
" \"\"\"Our check, outside the loop: do the tests pass, and what did the model's commands change?\"\"\"\n",
" tests = execute(\"python -m unittest -q 2>&1 | tail -1\")\n",
" print(\"tests:\", tests[\"output\"].strip())\n",
" print(git(\"diff\") or \"(no changes)\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5c68ceaa",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": ""
},
"tags": []
},
"outputs": [],
"source": [
"make_toy_repo()\n",
"trace = agent_v3(TASK)\n",
"show(trace[2:])\n",
"print(trace[-1][\"extra\"][\"exit_status\"])\n",
"verify()"
]
},
{
"cell_type": "markdown",
"id": "d8b403d9",
"metadata": {},
"source": [
"Whether this succeeds depends a lot on the model. Typical failure modes of small models are worth reading in the trace: forgetting the fence, editing the test instead of the code, a `sed` expression with broken quoting, submitting without re-running the tests, or repeating one ineffective edit until the budget is exhausted — an import appended to `stats/__init__.py`, which the test does not import, followed by the same test run, again and again. Note the separation: the model **claims** success by sending the sentinel; `verify()` **checks** it. The two can disagree."
]
},
{
"cell_type": "markdown",
"id": "d155dc16",
"metadata": {},
"source": [
"## From Text Actions to Tool Calling\n",
"\n",
"Most model APIs today can **call tools**: the request declares functions as JSON Schema, and the model answers with a structured `tool_calls` field instead of a fenced block. No regex is needed. mini-swe-agent v2 uses this by default (`LitellmModel`); the text-based variant we built is `LitellmTextbasedModel`, reached by an explicit `-c mini_textbased.yaml`. mini's own recommendation for practice is unambiguous: text parsing *\"was the method used for mini-swe-agent v1.0 and the original SWE-agent. As of mini-swe-agent v2.0, we strongly recommend to use toolcalls instead\"* (`models/utils/actions_text.py`). The tool-calling model's only tool is `bash`, with a four-word description. The system prompt below is a single sentence, because the format contract has moved into the schema: there is no fence to name and no block to count."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8ce4bb00",
"metadata": {},
"outputs": [],
"source": [
"BASH_TOOL = {\n",
" \"type\": \"function\",\n",
" \"function\": {\n",
" \"name\": \"bash\",\n",
" \"description\": \"Execute a bash command\",\n",
" \"parameters\": {\n",
" \"type\": \"object\",\n",
" \"properties\": {\"command\": {\"type\": \"string\", \"description\": \"The bash command to execute\"}},\n",
" \"required\": [\"command\"],\n",
" },\n",
" },\n",
"}\n",
"\n",
"tool_messages = [\n",
" {\"role\": \"system\", \"content\": \"You can run bash commands in a Linux container. The repository is at /repo.\"},\n",
" {\"role\": \"user\", \"content\": \"How many Python files are there under /repo? Use the bash tool.\"},\n",
"]\n",
"response = litellm.completion(model=MODEL, messages=tool_messages, tools=[BASH_TOOL])\n",
"msg = response.choices[0].message\n",
"print(\"content: \", msg.content)\n",
"print(\"tool_calls:\", msg.tool_calls)"
]
},
{
"cell_type": "markdown",
"id": "84c9e510",
"metadata": {},
"source": [
"The observation goes back with its own role, `tool`, and the `tool_call_id` it answers, so a model that issues several calls can match results to calls:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0f8fc205",
"metadata": {},
"outputs": [],
"source": [
"if not msg.tool_calls:\n",
" print(\"No tool call: the model answered in plain text. In mini's tool-calling model this is a FormatError.\")\n",
"else:\n",
" call = msg.tool_calls[0]\n",
" command = json.loads(call.function.arguments)[\"command\"]\n",
" print(\"command:\", command)\n",
" output = execute(command)\n",
" tool_messages += [\n",
" msg.model_dump(exclude_none=True),\n",
" {\"role\": \"tool\", \"tool_call_id\": call.id, \"content\": render_observation(output)},\n",
" ]\n",
" final = litellm.completion(model=MODEL, messages=tool_messages, tools=[BASH_TOOL])\n",
" print(\"answer: \", final.choices[0].message.content)"
]
},
{
"cell_type": "markdown",
"id": "0228237a",
"metadata": {},
"source": [
"| | Text actions (what we built) | Tool calling |\n",
"|---|---|---|\n",
"| Action extraction | regex over the reply | `message.tool_calls`, JSON arguments |\n",
"| Typical format errors | no block, two blocks | no tool call, invalid JSON, unknown tool |\n",
"| Observation role | `user` | `tool` + `tool_call_id` |\n",
"| Reasoning | the THOUGHT in the text | `content` next to the call, or a separate thinking field |\n",
"| Actions per turn | exactly one, enforced by the parser | one or more, and mini's prompt asks for **at least one** |\n",
"| Requirements | any chat model | model *and* provider must support tools |\n",
"\n",
"**The one-action rule is reversed, not carried over.** In text mode the rule is enforced where the reply is read: `parse_action` counts blocks and rejects anything but one. In tool-calling mode mini asks for the opposite — *\"Your response MUST include AT LEAST ONE bash tool call\"* (`config/mini.yaml`) — and `parse_toolcall_actions` returns every call it finds, which `execute_actions` then runs in order, one `tool` observation per call. The `tool_call_id` pairing is what makes a turn with several actions unambiguous. The request for a THOUGHT survives the move as prose (*\"your response SHOULD include reasoning text\"*), since no schema can ask for reasoning.\n",
"\n",
"What is given up is the interleaving. A model that issues three calls in one turn chooses the second and the third before it has seen the result of the first, which is precisely the property *exactly one* protects — traded for fewer round trips.\n",
"\n",
"One caution: when an Ollama model has no native tool calling, LiteLLM falls back to a JSON mode (BerriAI, 2026). The code looks the same, but the mechanism underneath is different.\n",
"\n",
"**Concept.** Both variants give the agent **one generic tool**. The alternative is many specialised tools (`open_file`, `edit_lines`, `search_dir`, …). SWE-agent designed such an interface carefully ([Yang et al., 2024](https://arxiv.org/html/2405.15793)); CodeAct argues for code itself as the unified action space ([Wang et al., 2024](https://arxiv.org/html/2402.01030)). With mini-swe-agent the SWE-agent team went back to bash alone: *\"as LMs have become more capable, a lot of this is not needed at all to build a useful agent!\"* (Lieret & Jimenez, 2026). Which design wins depends on the model, which is why the question is still open (→ [L-ReAct](http://christianherta.de/lehre/ai-agents/lecture-02/lecture-02-ReAct.html))."
]
},
{
"cell_type": "markdown",
"id": "1f348a9a",
"metadata": {},
"source": [
"## Refactoring: Model, Environment, Agent\n",
"\n",
"Our functions share global state (`MODEL`, `CONTAINER`) and the loop does everything itself. mini-swe-agent splits the same code into three classes with small interfaces, so that each part can be **swapped** independently: another model class, a Docker/Singularity/remote environment, an interactive agent. Here is our code in that shape. The model class below is the **text-based** one, which mini keeps for providers and local models without native tool calling — and which is what this notebook can teach from, since prompt, parser and correction message are all visible in it. The tool-calling class is the same interface with a different body: `query` reads `message.tool_calls` instead of matching a regular expression, and nothing in `Agent` changes.\n",
"\n",
"**Model:** messages in, one assistant message with the parsed action out."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2a54b533",
"metadata": {},
"outputs": [],
"source": [
"class TextModel:\n",
" def __init__(self, model_name=MODEL, **model_kwargs):\n",
" self.model_name, self.model_kwargs = model_name, model_kwargs\n",
"\n",
" def query(self, messages):\n",
" api_messages = prepare_messages_for_api(messages)\n",
" response = query_with_retry(api_messages, model=self.model_name, **self.model_kwargs)\n",
" reply = response.choices[0].message.content or \"\"\n",
" extra = {\"tokens\": response.usage.total_tokens}\n",
" try:\n",
" action = parse_action(reply, response.choices[0].finish_reason)\n",
" except FormatError as e: # the reply is already in e.messages[0][\"extra\"]; add what it cost\n",
" e.messages[0][\"extra\"] |= extra\n",
" raise\n",
" return {\"role\": \"assistant\", \"content\": reply, \"extra\": {**extra, \"action\": action}}"
]
},
{
"cell_type": "markdown",
"id": "eca1ca12",
"metadata": {},
"source": [
"**Environment:** one command in, one observation out. The environment detects the sentinel, as `LocalEnvironment._check_finished` does in mini."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ed91d3c5",
"metadata": {},
"outputs": [],
"source": [
"class PodmanEnvironment:\n",
" def __init__(self, image=IMAGE, workdir=SANDBOX, timeout=30):\n",
" self.timeout = timeout\n",
" self.container = start_container(image, workdir)\n",
"\n",
" def execute(self, command):\n",
" output = execute(command, self.timeout, container=self.container)\n",
" check_finished(output)\n",
" return output\n",
"\n",
" def close(self):\n",
" subprocess.run([\"podman\", \"rm\", \"-f\", self.container], capture_output=True)"
]
},
{
"cell_type": "markdown",
"id": "1f2e045e",
"metadata": {},
"source": [
"**Agent:** the loop, split the way mini splits it: `run` loops over `step`, and `step` = `execute_actions(query())`. Each method is a place to hook in: a subclass overrides one method and inherits the rest. The trajectory is saved in `finally`, i.e. after **every** step, so a crashed run still leaves a file."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e1ee829a",
"metadata": {},
"outputs": [],
"source": [
"class Agent:\n",
" def __init__(self, model, env, step_limit=15, token_limit=200_000,\n",
" max_consecutive_format_errors=3, output_path=None):\n",
" self.model, self.env = model, env\n",
" self.step_limit, self.token_limit = step_limit, token_limit\n",
" self.max_consecutive_format_errors = max_consecutive_format_errors\n",
" self.output_path = output_path\n",
" self.messages = []\n",
" self.n_calls = self.n_tokens = self.n_consecutive_format_errors = 0\n",
"\n",
" def add_messages(self, *messages):\n",
" self.messages.extend(messages)\n",
" return list(messages)\n",
"\n",
" def run(self, task):\n",
" self.messages = []\n",
" self.add_messages({\"role\": \"system\", \"content\": render(SYSTEM_TEMPLATE)},\n",
" {\"role\": \"user\", \"content\": render(INSTANCE_TEMPLATE, task=task)})\n",
" while self.messages[-1][\"role\"] != \"exit\":\n",
" try:\n",
" self.step()\n",
" self.n_consecutive_format_errors = 0\n",
" except FormatError as e:\n",
" # the call was billed, but query() never returned to count it\n",
" self.n_tokens += e.messages[0][\"extra\"][\"tokens\"]\n",
" self.n_consecutive_format_errors += 1\n",
" self.add_messages(*e.messages)\n",
" if 0 < self.max_consecutive_format_errors <= self.n_consecutive_format_errors:\n",
" self.add_messages(exit_message(\"RepeatedFormatError\"))\n",
" except InterruptAgentFlow as e:\n",
" self.add_messages(*e.messages)\n",
" finally:\n",
" self.save()\n",
" return self.messages[-1][\"extra\"]"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6cc9992d",
"metadata": {},
"outputs": [],
"source": [
"class Agent(Agent): # continued: the three methods one loop iteration consists of\n",
" def step(self):\n",
" return self.execute_actions(self.query())\n",
"\n",
" def query(self):\n",
" check_limits(self.n_calls, self.n_tokens, self.step_limit, self.token_limit)\n",
" self.n_calls += 1\n",
" message = self.model.query(self.messages)\n",
" self.n_tokens += message[\"extra\"][\"tokens\"]\n",
" return self.add_messages(message)[0]\n",
"\n",
" def execute_actions(self, message):\n",
" output = self.env.execute(message[\"extra\"][\"action\"])\n",
" return self.add_messages({\"role\": \"user\", \"content\": render_observation(output),\n",
" \"extra\": {\"returncode\": output[\"returncode\"]}})\n",
"\n",
" def save(self):\n",
" if not self.output_path:\n",
" return\n",
" last = self.messages[-1].get(\"extra\", {})\n",
" data = {\"info\": {\"model\": self.model.model_name, \"api_calls\": self.n_calls, \"tokens\": self.n_tokens,\n",
" \"exit_status\": last.get(\"exit_status\", \"\"), \"submission\": last.get(\"submission\", \"\")},\n",
" \"messages\": self.messages}\n",
" self.output_path.parent.mkdir(parents=True, exist_ok=True)\n",
" self.output_path.write_text(json.dumps(data, indent=2))"
]
},
{
"cell_type": "markdown",
"id": "bbe2cc4c",
"metadata": {},
"source": [
"This is now essentially mini's `DefaultAgent`. The control flow:\n",
"\n",
"\n",
"\n",
"| Ours | mini-swe-agent | Differences in mini |\n",
"|---|---|---|\n",
"| `TextModel.query` | `LitellmTextbasedModel.query` | cost instead of tokens; stores the whole response object in `extra` |\n",
"| `PodmanEnvironment` | `DockerEnvironment(executable=\"podman\")` | config object; forwards env vars; `execute` takes the action as a dict |\n",
"| `Agent.run / step / query / execute_actions` | `DefaultAgent`, same four methods | wall-clock limit; templates get *all* config values as variables |\n",
"| `Agent.save` | `DefaultAgent.save` / `serialize` | also stores the full config of all three parts |\n",
"\n",
"**Concept.** The loop is a small **state machine**: states *query* and *execute*, transitions on the exception type, one accepting state *exit* (→ [L-Orch](http://christianherta.de/lehre/ai-agents/lecture-11/lecture-11-Orch.html)). Writing it as a state machine is what lets frameworks add checkpoints, branches and human approval steps."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b16c1ecd",
"metadata": {},
"outputs": [],
"source": [
"make_toy_repo()\n",
"env = PodmanEnvironment()\n",
"agent = Agent(TextModel(), env, output_path=Path(\"trajectories/ours-fix-median.traj.json\"))\n",
"result = agent.run(TASK)\n",
"print(result[\"exit_status\"], f\"after {agent.n_calls} model calls, {agent.n_tokens} tokens\")\n",
"verify()"
]
},
{
"cell_type": "markdown",
"id": "e76a3779",
"metadata": {},
"source": [
"## Trajectories\n",
"\n",
"The saved file is the **trajectory**: every message of the run plus some metadata. mini's authors point out that, with a linear history, there is *no difference* between what the model saw and what is logged, which makes debugging unusually easy. The one exception is a malformed reply: it is logged in the format-error message's metadata without ever entering the history. One line per step:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "28073d63",
"metadata": {},
"outputs": [],
"source": [
"data = json.loads(Path(\"trajectories/ours-fix-median.traj.json\").read_text())\n",
"print(data[\"info\"])\n",
"steps = [m for m in data[\"messages\"] if m.get(\"extra\", {}).keys() & {\"action\", \"model_response\"}]\n",
"for i, m in enumerate(steps, 1):\n",
" extra = m[\"extra\"]\n",
" text = extra.get(\"model_response\", m[\"content\"]).strip() # the reply, wherever it was kept\n",
" thought = text.splitlines()[0][:70] if text else \"\"\n",
" action = extra[\"action\"].replace(\"\\n\", \" ⏎ \")[:60] if \"action\" in extra else \"\"\n",
" print(f\"{i:>2}. {thought:<70} | $ {action}\")"
]
},
{
"cell_type": "markdown",
"id": "ae8da573",
"metadata": {},
"source": [
"**Concept.** The trajectory is one **episode** of an MDP (→ [L-MDP](http://christianherta.de/lehre/ai-agents/lecture-05/lecture-05-MDP.html)):\n",
"\n",
"| MDP | Our agent |\n",
"|---|---|\n",
"| state $s_t$ | the files in `/repo` plus the message history |\n",
"| observation $o_t$ | return code + output of the last command; the model sees only the history, so the problem is **partially observable** |\n",
"| action $a_t$ | one bash command (with its THOUGHT) |\n",
"| transition | the container executing the command |\n",
"| reward | none during the run; at the end: *do the tests pass?* A sparse, terminal reward that the agent never sees |\n",
"| terminal state / horizon | the `exit` message / `step_limit` |\n",
"\n",
"Trajectories are also what **evaluation** works on (→ [L-Eval](http://christianherta.de/lehre/ai-agents/lecture-10/lecture-10-Eval.html)). SWE-bench scores an agent by running tests on the final patch it produces for real GitHub issues ([Jimenez et al., 2024](https://arxiv.org/html/2310.06770)). That is our `verify()` at scale, and it ignores how the agent got there. Reading trajectories is how you find out *why* it failed."
]
},
{
"cell_type": "markdown",
"id": "a27aea08",
"metadata": {},
"source": [
"## Safety: A Human in the Loop\n",
"\n",
"The container limits the **damage** a bad command can do. A human in the loop decides **whether** it runs. mini's `InteractiveAgent` has three modes: `confirm` (ask before each command), `yolo` (never ask) and `human` (the human types the commands), plus a whitelist of regexes for commands that run without asking. In our class structure this is a single overridden method:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "65a67ddf",
"metadata": {},
"outputs": [],
"source": [
"class ConfirmingAgent(Agent):\n",
" def __init__(self, *args, mode=\"confirm\", whitelist=(r\"^(ls|cat|nl|grep|head|tail)\\b\",), ask=input, **kwargs):\n",
" super().__init__(*args, **kwargs)\n",
" self.mode, self.whitelist, self.ask = mode, whitelist, ask\n",
"\n",
" def execute_actions(self, message):\n",
" command = message[\"extra\"][\"action\"]\n",
" if self.mode == \"confirm\" and not any(re.match(p, command) for p in self.whitelist):\n",
" answer = self.ask(f\"Execute?\\n {command}\\n[y = run, anything else = reject with that comment] \")\n",
" if answer.strip().lower() not in (\"y\", \"yes\"):\n",
" raise InterruptAgentFlow({\"role\": \"user\", \"content\": f\"The user rejected this command: {answer}\"})\n",
" return super().execute_actions(message)"
]
},
{
"cell_type": "markdown",
"id": "477d971a",
"metadata": {},
"source": [
"A rejection is, again, just an observation: the model reads the user's comment and tries something else.\n",
"\n",
"Look closely at the whitelist, though: `re.match(r\"^(ls|…)\\b\", \"ls; rm -rf stats\")` **matches**. A prefix check on a shell string says nothing about what the string does. The whitelist is a convenience, not a security boundary; the container is the boundary (→ [L-Gateway](http://christianherta.de/lehre/ai-agents/lecture-13/lecture-13-Gateway.html))."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1e056bf3",
"metadata": {},
"outputs": [],
"source": [
"print(bool(re.match(r\"^(ls|cat|nl|grep|head|tail)\\b\", \"ls; rm -rf stats\")))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "11fd5485",
"metadata": {},
"outputs": [],
"source": [
"# Interactive: answer the prompts below. For a non-interactive run pass ask=lambda prompt: \"y\".\n",
"make_toy_repo()\n",
"agent = ConfirmingAgent(TextModel(), env, step_limit=10)\n",
"print(agent.run(TASK)[\"exit_status\"])\n",
"verify()"
]
},
{
"cell_type": "markdown",
"id": "c4d3a995",
"metadata": {},
"source": [
"## The Real Thing: mini-swe-agent\n",
"\n",
"Now the same task with the original package. We use its text-based config (`mini_textbased.yaml`), its text-based LiteLLM model and its `DockerEnvironment` with podman as the executable and our mount and security flags. Two local-model adjustments: cost tracking is switched off (Ollama has no prices), and since the default money budget can therefore never trigger, we set a step limit."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "12511035",
"metadata": {},
"outputs": [],
"source": [
"# %pip install 'mini-swe-agent==2.4.6' # or, from a clone: %pip install -e path/to/mini-swe-agent\n",
"import yaml\n",
"\n",
"os.environ[\"MSWEA_SILENT_STARTUP\"] = \"1\" # must be set before the import\n",
"from minisweagent import package_dir\n",
"from minisweagent.agents.default import DefaultAgent\n",
"from minisweagent.environments.docker import DockerEnvironment\n",
"from minisweagent.models.litellm_textbased_model import LitellmTextbasedModel\n",
"\n",
"import logging\n",
"logging.getLogger(\"minisweagent\").setLevel(logging.WARNING) # mini logs every container command at DEBUG\n",
"\n",
"config = yaml.safe_load((package_dir / \"config\" / \"mini_textbased.yaml\").read_text())\n",
"config"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3a6dc0ec",
"metadata": {},
"outputs": [],
"source": [
"make_toy_repo()\n",
"mini_env = DockerEnvironment(\n",
" image=IMAGE, executable=\"podman\", cwd=\"/repo\",\n",
" run_args=[\"--rm\", \"--network=none\", \"--cap-drop=ALL\", \"-v\", f\"{SANDBOX}:/repo:Z\"],\n",
" **config[\"environment\"],\n",
")\n",
"mini_model = LitellmTextbasedModel(model_name=MODEL, cost_tracking=\"ignore_errors\", **config[\"model\"])\n",
"mini_agent = DefaultAgent(\n",
" mini_model, mini_env,\n",
" **(config[\"agent\"] | {\"step_limit\": 15, \"output_path\": Path(\"trajectories/mini-fix-median.traj.json\")}),\n",
")\n",
"mini_result = mini_agent.run(TASK)\n",
"print(mini_result[\"exit_status\"], f\"after {mini_agent.n_calls} model calls\")\n",
"verify()"
]
},
{
"cell_type": "markdown",
"id": "79ac5cd3",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": ""
},
"tags": []
},
"source": [
"Compare the two trajectory files side by side: same message structure, same roles, same observation format. The differences are mini's much longer instance template (more command recipes, platform-specific hints for macOS `sed`) and the bookkeeping in `info`. Now `agents/default.py` should read like something you have written yourself.\n",
"\n",
"Clean up the containers:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c93890f8",
"metadata": {},
"outputs": [],
"source": [
"env.close()\n",
"subprocess.run([\"podman\", \"rm\", \"-f\", CONTAINER, mini_env.container_id], capture_output=True)\n",
"print(\"containers removed\")"
]
},
{
"cell_type": "markdown",
"id": "e4555919",
"metadata": {},
"source": [
"## What mini-swe-agent Leaves Out\n",
"\n",
"Every one of these omissions is deliberate, and each is a topic of its own:\n",
"\n",
"- **A persistent shell.** Every command runs in a fresh subshell, so a `cd`, an `export` or an activated virtual environment is gone by the next step: the prompt has to ask for `cd /repo && …` instead, and no long-running or interactive process is possible. SWE-agent keeps one shell session open for the whole run ([Yang et al., 2024](https://arxiv.org/html/2405.15793)). The session makes that state persist; it also introduces state the model cannot see unless it asks, and a shell that can hang on a pager or on a command waiting for input.\n",
"- **History management.** The history grows until the budget or the context window runs out; nothing is summarised or dropped (→ [L-Memory](http://christianherta.de/lehre/ai-agents/lecture-09/lecture-09-Memory.html)).\n",
"- **Search over alternatives.** One trajectory, no backtracking and no comparison of candidate fixes (→ [L-ToT](http://christianherta.de/lehre/ai-agents/lecture-04/lecture-04-ToT.html)).\n",
"- **Detection of a repetition.** Nothing compares a step with the steps before it, so a model alternating the same edit and the same test run is stopped only by a budget. The comparison itself is a set lookup; what to do on a match — warn the model, refuse the command, end the run — is the part that needs deciding.\n",
"- **An enforced workflow.** Reproduce → fix → verify is a suggestion in the prompt, not a state machine in code (→ [L-Orch](http://christianherta.de/lehre/ai-agents/lecture-11/lecture-11-Orch.html)).\n",
"- **Several agents.** No planner/worker split, no reviewer (→ [L-MAS](http://christianherta.de/lehre/ai-agents/lecture-12/lecture-12-MAS.html)).\n",
"- **Evaluation at scale.** One task, one run, no statistics. Models are stochastic, and one success proves little (→ [L-Eval](http://christianherta.de/lehre/ai-agents/lecture-10/lecture-10-Eval.html)).\n",
"\n",
"**Things to try.**\n",
"\n",
"1. Turn `TextModel` into a `ToolCallModel` using `BASH_TOOL`; what changes in `Agent`? (Answer: almost nothing. That is the point of the split.)\n",
"2. Add a wall-clock budget to `check_limits`.\n",
"3. Add naive compaction: above a token threshold, replace the content of old observations with `[elided]`. Does the agent still finish?\n",
"4. Write three commands that pass the whitelist in `ConfirmingAgent` but do something else.\n",
"5. Run the bug fix five times per model with two models. How often does each succeed, and how different are the trajectories?\n",
"6. Give the toy repo a harder bug (e.g. `mean([])` should raise `ValueError` with a clear message, and a test that checks the message).\n",
"7. Replace `PodmanEnvironment` with one that keeps a single shell open (`podman exec -i … bash`, commands written to its stdin) and reads the output up to a marker echoed after each command. Which commands behave differently now, what does the harness have to do when one of them never returns, and what does each of the two designs cost?"
]
},
{
"cell_type": "markdown",
"id": "2018621d",
"metadata": {},
"source": [
"## References\n",
"\n",
"**Peer-reviewed papers**\n",
"\n",
"- Jimenez, Carlos E., et al. \"SWE-bench: Can Language Models Resolve Real-World GitHub Issues?\" International Conference on Learning Representations. Vol. 2024. 2024. [arXiv:2310.06770](https://arxiv.org/html/2310.06770).\n",
"- Liu, Nelson F., et al. \"Lost in the Middle: How Language Models Use Long Contexts.\" Transactions of the Association for Computational Linguistics, vol. 12, 2024. [arXiv:2307.03172](https://arxiv.org/html/2307.03172).\n",
"- Wang, Xingyao, et al. \"Executable Code Actions Elicit Better LLM Agents.\" International Conference on Machine Learning. Vol. 235. 2024. [arXiv:2402.01030](https://arxiv.org/html/2402.01030).\n",
"- Yang, John, et al. \"SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering.\" Advances in Neural Information Processing Systems. Vol. 37. 2024. [arXiv:2405.15793](https://arxiv.org/html/2405.15793).\n",
"- Yao, Shunyu, et al. \"ReAct: Synergizing Reasoning and Acting in Language Models.\" International Conference on Learning Representations. Vol. 2023. 2023. [arXiv:2210.03629](https://arxiv.org/html/2210.03629).\n",
"\n",
"**Blogs, documentation & other web sources**\n",
"\n",
"- BerriAI. \"LiteLLM: Call 100+ LLM APIs in OpenAI Format — Documentation.\" 2026. \n",
"- Lieret, Kilian A., and Carlos E. Jimenez. *mini-swe-agent: The Minimal AI Software Engineering Agent*. GitHub repository, v2.4.6, 2026. \n",
"- Lieret, Kilian, et al. \"Minimal AI agent tutorial.\" 2026. "
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}