Building AI Agents from Scratch

Most agent tutorials start with a framework. That is fine for shipping, but it hides the part worth understanding: the loop.

Building an agent from scratch is not about replacing frameworks. It is about learning what they wrap. Once the loop is clear, tool calls, memory, stop conditions, and model switching stop feeling like magic and start looking like normal software decisions.

What is an AI Agent?#

An AI agent is a program that receives a goal, asks an LLM what to do next, executes that action (either a tool call or a final output), feeds the result back to the LLM, and repeats until done. The LLM is the brain, the tools are the hands, and the loop is what makes it autonomous.

Every framework in this space, whether it's LangGraph, LangChain, or Smolagents, is a variation of that same pattern. The differences are in how they handle structured outputs, error recovery, and multi-agent coordination, but underneath, it's always the same loop. That's the only part that matters. Everything else is a tool definition or a message format.

The Loop#

Imagine you hired someone to research a topic and write a report. They don't do it all in one shot. They search for something, read what comes back, decide what to search next, maybe open a document, take notes, and keep going until they have enough to write the report. They're running a loop in their head: observe, decide, act, repeat.

An agent does exactly the same thing. The difference is that the "deciding" step is done by an LLM, and the "acting" step is calling a function in your code. Every iteration, the agent gets a little more context about the problem, and that context informs what it does next.

Here's what that looks like as code:

1# Adapted from the OpenAI function calling guide
2# https://platform.openai.com/docs/guides/function-calling
3
4def run_agent(goal: str, tools: dict, tool_schemas: list, max_steps: int = 8):
5    messages = [{"role": "user", "content": goal}]
6
7    for _ in range(max_steps):
8        message = llm_call(messages, tool_schemas)
9        messages.append(message)
10
11        if not message.tool_calls:
12            return message.content
13
14        for tool_call in message.tool_calls:
15            name = tool_call.function.name
16            args = json.loads(tool_call.function.arguments)
17            result = tools[name](**args)
18            messages.append({
19                "role": "tool",
20                "tool_call_id": tool_call.id,
21                "content": str(result),
22            })
23
24    return messages[-1].content

The interesting part is what the model does with accumulated context. By step 5, it's reading its own previous tool calls and results, and that history is what stops it from calling the same tool with the same arguments twice. The messages list is doing the work here, not the loop. The loop just keeps feeding that history back in.

max_steps is just a stop condition, the same kind of guard used in any loop or if statement. Eight steps is enough for a small example and cheap enough to run safely. Without a limit, a confused model can call the same tool again and again, each call costing tokens. When the limit is hit, messages[-1].content is returned, which is whatever the model last said. Real output, not a hardcoded error string.

Everything else in this post is just filling in the details of that skeleton.

The LLM Call#

The model needs to return a structured decision about which tool to call and with what arguments. There are three useful layers to understand here: raw function calling, a model adapter like OpenRouter, and an Agent SDK that owns the loop for you.

First, the raw function-calling pattern. Rather than parsing free text to infer intent, the model signals intent through a structured channel. Free text parsing is fragile. Function calling is not.

1# From the OpenAI function calling guide
2# https://platform.openai.com/docs/guides/function-calling
3
4from openai import OpenAI
5import json
6
7client = OpenAI()
8
9messages = [{"role": "user", "content": "What is the latest Python release?"}]
10
11# 1. Send the request with tools defined
12response = client.chat.completions.create(
13    model="gpt-4o-mini",
14    messages=messages,
15    tools=TOOL_SCHEMAS,
16    tool_choice="auto",
17)
18
19messages.append(response.choices[0].message)
20
21# 2. Execute each tool call and feed the result back
22for tool_call in response.choices[0].message.tool_calls or []:
23    name = tool_call.function.name
24    args = json.loads(tool_call.function.arguments)
25    messages.append({
26        "role": "tool",
27        "tool_call_id": tool_call.id,  # required: links result to call
28        "content": str(TOOLS[name](**args)),
29    })
30
31# 3. Get the final response
32final = client.chat.completions.create(
33    model="gpt-4o-mini",
34    messages=messages,
35    tools=TOOL_SCHEMAS,
36)
37print(final.choices[0].message.content)

The API returns a list of tool_calls, not a single call. Most tutorials iterate over one. That's a bug. The tool_call_id field is how the model matches each result back to the specific call it made, and without it the API will reject the message.

Second, the model adapter. OpenRouter gives access to many models through a single OpenAI-compatible endpoint. One client, one base URL, and the model is just a string passed in per call:

1# From the OpenRouter quickstart
2# https://openrouter.ai/docs/quickstart
3
4import os
5from openai import OpenAI
6
7client = OpenAI(
8    base_url="https://openrouter.ai/api/v1",
9    api_key=os.environ["OPENROUTER_API_KEY"],
10)
11
12MODEL_FAST = "google/gemini-3.5-flash"
13MODEL_SLOW = "anthropic/claude-sonnet-4.5"
14
15def llm_call(messages: list, tools: list, model: str = MODEL_FAST):
16    response = client.chat.completions.create(
17        model=model,
18        messages=messages,
19        tools=tools,
20        tool_choice="auto",
21    )
22    return response.choices[0].message

MODEL_FAST for routine tool calls, MODEL_SLOW when a step needs deeper reasoning. The tradeoff is latency. OpenRouter adds a hop. For most agent workloads that hop is negligible because the bottleneck is the model thinking, not the network. For latency-critical applications, calling the provider directly is the better choice.

Third, the Agent SDK. This is the production-shaped version of the same idea. Instead of manually wiring the message loop, tool execution, and stop conditions, callModel owns the loop. Tools are defined with tool(), and stop conditions like stepCountIs and maxCost replace the manual max_steps check:

1// From the OpenRouter Agent SDK documentation
2// https://openrouter.ai/docs/agent-sdk/overview
3
4import { callModel, tool, stepCountIs, maxCost } from "@openrouter/agent";
5import { z } from "zod";
6
7const searchTool = tool({
8  name: "search_web",
9  description: "Search the web for current information.",
10  inputSchema: z.object({ query: z.string() }),
11  execute: async ({ query }) => {
12    return { results: ["..."] };  // plug in Tavily, Serper, etc.
13  },
14});
15
16const result = await callModel({
17  model: "anthropic/claude-sonnet-4.6",
18  messages: [
19    {
20      role: "user",
21      content: "What is the latest Python release?",
22    },
23  ],
24  tools: [searchTool],
25  stopWhen: [stepCountIs(8), maxCost(0.50)],
26});
27
28const text = await result.getText();

The SDK sends the prompt, receives tool calls, executes them, feeds results back, and loops until a stop condition is met. stepCountIs(8) is the same idea as max_steps; maxCost(0.50) adds a spend cap. Less control, much less wiring. The manual loop stays in this post because the goal is to see what the SDK is abstracting away.

The Tools#

Tools are just functions. They can call external APIs, query databases, read files, trigger workflows, or hit services like Firecrawl, Tavily, or Serper. You define the function, describe it in a schema, and pass that schema to the model. From there, the model picks which tool to call and the result goes back into the message history.

1# Tool schema format from the OpenAI function calling reference
2# https://platform.openai.com/docs/guides/function-calling#defining-functions
3
4def search_web(query: str) -> str:
5    # Swap this with Firecrawl, Tavily, Serper, or an internal API.
6    return f"Search results for: {query}"
7
8TOOLS = {"search_web": search_web}  # name -> callable registry
9
10TOOL_SCHEMAS = [
11    {
12        "type": "function",
13        "function": {
14            "name": "search_web",
15            "description": (
16                "Search the web for current information. "
17                "Use when the goal needs facts not in training data."
18            ),
19            "parameters": {
20                "type": "object",
21                "properties": {
22                    "query": {
23                        "type": "string",
24                        "description": "A concise, specific search query",
25                    }
26                },
27                "required": ["query"],
28                "additionalProperties": False,
29            },
30            "strict": True,
31        },
32    },
33    # add more tools following the same shape
34]

The description field is the only thing the model sees when deciding which tool to call. A vague description like "search the web" is weaker than "search the web for current facts not available in the model's training data." OpenAI's own function calling guide puts it well: write descriptions like API documentation for an intern who has never seen your codebase.

Memory#

Short-term memory is already solved. The messages list is the agent's memory for the current run, and every tool call and its result gets appended so the model sees the full history on every step.

Long-term memory, meaning remembering things across separate runs, is a different problem. It requires embeddings.

1# From the OpenAI embeddings reference
2# https://platform.openai.com/docs/guides/embeddings
3
4def embed(text: str) -> list[float]:
5    response = client.embeddings.create(
6        model="text-embedding-3-small",
7        input=text,
8    )
9    return response.data[0].embedding

The pattern is to embed important observations from a run and store them in a vector database like Pinecone, Qdrant, or Chroma. At the start of the next run, the current goal is embedded, the closest past observations are retrieved, and they're injected into the system prompt. The temptation is to inject everything. Don't. A system prompt stuffed with 50 retrieved observations is worse than an empty one, because the model spends its attention budget reading context instead of solving the problem. Three relevant observations beat thirty irrelevant ones.

Putting It Together#

1# Full example: loop skeleton + OpenRouter adapter + tools
2# OpenAI function calling:
3# https://platform.openai.com/docs/guides/function-calling
4# OpenRouter quickstart: https://openrouter.ai/docs/quickstart
5
6import json
7import os
8from openai import OpenAI
9
10client = OpenAI(
11    base_url="https://openrouter.ai/api/v1",
12    api_key=os.environ["OPENROUTER_API_KEY"],
13)
14
15MODEL_FAST = "google/gemini-3.5-flash"
16
17def run_agent(goal: str, max_steps: int = 8) -> str:
18    messages = [
19        {
20            "role": "system",
21            "content": (
22                "Use tools to accomplish the goal. "
23                "When done, respond without calling a tool."
24            ),
25        },
26        {"role": "user", "content": goal},
27    ]
28
29    for step in range(max_steps):
30        message = client.chat.completions.create(
31            model=MODEL_FAST,
32            messages=messages,
33            tools=TOOL_SCHEMAS,
34            tool_choice="auto",
35        ).choices[0].message
36        messages.append(message)
37
38        if not message.tool_calls:
39            return message.content
40
41        for tool_call in message.tool_calls:
42            name = tool_call.function.name
43            args = json.loads(tool_call.function.arguments)
44            try:
45                result = TOOLS[name](**args)
46            except Exception as e:
47                result = f"Error: {e}"
48            messages.append({
49                "role": "tool",
50                "tool_call_id": tool_call.id,
51                "content": str(result),
52            })
53
54    return messages[-1].content
55
56
57if __name__ == "__main__":
58    print(run_agent(
59        "Search for the latest Python release "
60        "and summarise it in one paragraph."
61    ))

This is the whole agent. The system prompt is one sentence. The loop is six lines. The tool execution is eight lines. Everything else is message plumbing. Run it and it searches, reads, writes, recovers from errors, and decides when it's done, all driven by the model.

When to Use a Framework#

Building an agent from scratch is the best way to learn what is happening underneath. It is also useful when you want to experiment with new features, build a custom workflow, or plug in your own tools and APIs without fighting an abstraction. The search_web tool in this post could call Firecrawl, Tavily, Serper, an internal search service, or anything else. The loop does not care. It only needs a function, a schema, and a result to feed back to the model.

Production agents are much more complex than this example. They need tracing, retries, budgets, permissions, evaluations, queues, human checkpoints, observability, and safe deployment paths. That is why agent SDKs and frameworks exist, and why so many people are actively working on them. If the goal is to ship a production system, using an SDK like the OpenRouter Agent SDK, LangGraph, Smolagents, or another mature framework is usually the better choice.

The point of this post is not that every agent should be written from scratch. The point is that writing one from scratch makes the abstraction obvious. Once the loop is clear, frameworks stop feeling like magic and start feeling like what they are: production-ready infrastructure around the same core idea.

And this is only the beginning. Tools can search the web, read files, manipulate the DOM, generate custom UI based on user interactions, call internal APIs, or retrieve information from a knowledge base. Agents can become retrieval-based systems, multi-agent workflows, browser automation layers, or product interfaces that adapt in real time. This post is the tip of the iceberg. If you want the deeper version, sign up for the newsletter.