The Architecture of an AI System

When you use an AI application, the mental model is usually pretty simple. You type something, the AI thinks, and you get an answer.

1You → AI → Answer

That model is useful, but it is also wildly incomplete. A modern AI system is not just a model sitting on a GPU waiting for your prompt. There are applications, context builders, agent loops, tools, retrieval systems, model-serving infrastructure, GPUs, storage systems, monitoring, and an entire lifecycle that happened before you ever typed the first character.

If you are building AI applications, understanding these boundaries is more useful than memorizing a list of AI buzzwords.

The naive mental model#

Start with the simplest possible architecture:

You send a prompt to an AI, it processes it, and it returns some text. At the center of this system is a model, the thing that takes an input and produces an output based on the patterns it learned during training.

That raises a few questions worth sitting with. Who actually sends your prompt to the model? What exactly gets sent to it? And if you are using a coding agent, how does the model read files, run commands, search the web, call APIs, and modify your code? The model itself does not magically have access to any of these things. There is a lot more software surrounding it.

Let's open the box#

A more useful mental model looks something like this:

This is not one universal implementation. Different companies combine, split, rename, or hide these components. They are still useful architectural boundaries for understanding how modern AI systems work.

There is another major part of the picture that does not sit underneath the GPU: the model lifecycle. Serving infrastructure runs a trained model. The lifecycle is how you get that model in the first place and keep improving it. Let's go through the stack from the top.

The AI application#

At the top is the thing you actually interact with. It could be a chat interface, an API, a coding environment, a customer support product, or an internal company tool. In every case, the application's job is to turn a user's interaction into something the rest of the AI system can process.

Suppose you ask:

What is the capital of France?

The application receives that message, but the model might not receive exactly those words. There may be system instructions, conversation history, retrieved documents, tool definitions, and information about the current user or task. Someone has to construct the actual context that will eventually reach the model. That is where things start getting interesting.

The agent and the harness#

If you are building a simple chatbot, you might not need a complicated agent layer. You can essentially do User message → Model → Answer and stop there.

A coding agent is a different problem. You say "fix the authentication bug in my application," and the system now has to inspect the repository, find the relevant files, read the code, understand the error, decide what to change, modify files, run tests, read the output, and repeat until the task is done. The model can reason about those actions, but something has to orchestrate the process. That surrounding software is often described as an agent harness.

The exact implementation differs between systems. Anthropic's Claude Code and OpenAI's Codex are sophisticated coding-agent systems, and their internal architectures are proprietary, so it would be wrong to assume either one literally implements this diagram. The point is the boundary.

The model is not the entire agent. The harness is the software around the model that manages the interaction between the model, the environment, and the task. If you want this layer in more detail, I wrote about building AI agents from scratch.

What does the model actually see?#

This is one of the most important concepts in AI engineering. Suppose you type "fix the login bug." The model does not see your entire computer. It does not automatically see every file in your repository, know which tools exist, or magically inspect your database. Something has to construct the model's input.

This is why context engineering matters so much. The model can only reason over the information that is actually made available to it. And that brings us to one of the most common pieces of AI application architecture.

Retrieval and RAG#

Suppose your company has 50,000 internal documents and you ask, "What is our refund policy for enterprise customers?" You probably don't want to put all 50,000 documents into the model's context. Instead, the system can retrieve the pieces of information that are relevant to the question. That is the basic idea behind Retrieval-Augmented Generation, or RAG.

There are two different flows to understand.

Before anyone asks a question, documents are split into smaller chunks, each chunk is converted into a numerical representation called an embedding, and those embeddings are stored alongside the original content and metadata.

1Document chunk:  "Enterprise refunds are..."
2Embedding:       [0.12, -0.44, 0.83, ...]
3Metadata:        document_id=123, department=billing

A vector database can then search for vectors that are close to the vector representing a user's query. The important thing to understand is that RAG does not retrain the model. The model remains the model. RAG changes the information that is supplied to it at request time.

It also helps to separate a vector database from a key-value store or a graph database. A key-value database answers key → value. A vector database answers "find things similar to this vector." A graph database is optimized around relationships: Alice works at a company, the company owns a product, the product depends on a service. Different storage models exist because different questions require different retrieval mechanisms.

Tool calling#

Retrieval lets an AI system find information. But what if it needs to do something?

Suppose you ask, "What is the weather in New York?" The model does not need today's weather baked into its weights. The application can expose a weather tool, the model can request it, and the surrounding software can actually call the API.

1User
2  → Model: "I need weather data"
3  → Tool call → Weather API
4  → Tool result → Model
5  → Answer

The model generates a structured request representing the tool it wants to use. The surrounding software interprets that request and executes it. That distinction matters. The model can request an action. The surrounding system performs the action.

The tool could be a weather API, a database query, a web search, a shell command, a file operation, a payment API, a Git operation, or an internal company service. The model does not need to implement any of those capabilities itself. It needs a way to express that it wants one of them.

MCP, tools, and the harness#

This is also where people often mix up several different concepts. A tool is an action the AI system can invoke. A harness is the surrounding orchestration layer that manages the model interaction, context, tools, state, permissions, and execution. MCP, the Model Context Protocol, is a standard way to plug tools and other capabilities into that harness. They are related, but they are not interchangeable terms.

Without a standard, every tool integration is custom glue written for one specific app. With MCP, someone writes a GitHub server or a Postgres server once, and any MCP-compatible harness can use it. The tool doesn't care which harness is calling it, and the harness doesn't care who wrote the tool.

1                HARNESS
2                   |
3         +---------+---------+
4         |                   |
5       MODEL               TOOLS
6                             |
7              +--------------+--------------+
8              |                             |
9          Built-in                     MCP servers
10       (files, shell)                       |
11                                  +---------+---------+
12                                  |         |         |
13                                GitHub    Slack    Postgres

MCP is one way tools reach the harness. It is not itself an agent, it is not the model, and it is not another word for a harness.

The agent loop#

Now we can finally explain what makes something feel like an agent. A simple model interaction is User → Model → Answer. An agentic interaction has a loop: the model can call a tool, receive the result, reason again, and either call another tool or stop.

Every pass through the loop adds another tool call and another result to the context, so the model knows a little more on each step. The stop condition is what keeps it from going forever. A useful engineering abstraction is:

1Agent = Model + Tools + State + Loop + Stop Condition

This isn't the only possible definition of an agent, but it is a useful way to reason about agentic systems as an engineer. The loop is the part that matters. Take it away and you are back to Input → Model → Output. That is the basic machinery behind many systems that people casually call "AI agents."

The AI runtime#

So far we've mostly talked about what happens around the model. Eventually the system still has to actually run it. Call that layer the AI runtime, or the inference platform. It can own request management, routing, authentication, rate limiting, guardrails, policy enforcement, caching, observability, model selection, and scheduling. The exact mix depends on the architecture.

This distinction matters because inference and the platform around inference are not the same thing. Inference means running the trained model to produce an output. The platform around it can include a lot of other responsibilities.

Model serving#

Now we are getting closer to the actual computation. A model-serving system is responsible for running inference efficiently at scale. If thousands of users send requests at the same time, you don't want each request to grab an entire GPU and run independently. Modern serving systems have to manage resources carefully, which is where batching, scheduling, GPU memory, KV cache, and the tradeoff between throughput and latency start to matter.

There is another useful distinction inside inference itself. When a prompt first arrives, the system processes all the input tokens at once. That is commonly called prefill. Then the model generates output tokens one at a time, each one depending on the last. That is the decode phase.

The two phases stress the hardware in different ways. Prefill is parallel, so it is limited by how much compute the GPU has. Decode is sequential, and every new token means reading the model's weights from memory again, so it is limited by memory bandwidth. That is why time to first token and tokens per second are two separate numbers, and why serving systems tune for them separately. It is also why batching matters so much. One read of the weights can produce the next token for many users at once.

The KV cache is what keeps decode from being far worse. For every token it processes, the model computes intermediate values called keys and values inside each attention layer. Instead of recomputing those for the whole sequence on every new token, the server keeps them in GPU memory and reuses them. The cost is memory. Long conversations and many concurrent users add up to a lot of cache, and managing that memory is one of the main jobs of a serving engine.

The model repeatedly predicts the next token. That sounds simple. The hardware and systems engineering required to do this efficiently at large scale is not.

What actually happens on the GPU?#

At the lowest level of this request path, we eventually reach hardware. A GPU performs the enormous amount of numerical computation that modern neural networks require. The model contains parameters, those parameters are numerical values, and inference is the process of applying them to the input through many layers of computation.

1Tokens
2  → Embeddings
3  → Transformer layers
4      → Attention
5      → Feed-forward computation
6      → Normalization
7  → Output probabilities
8  → Next token

For a language model, this process happens repeatedly as new tokens are generated. The important thing is that the GPU is not the model. The GPU is hardware. The model is the learned mathematical structure represented by its architecture and parameters. The serving system loads that model onto the available hardware and executes it. The model sits at the bottom of the stack diagram not because it lives underneath the GPU, but because everything above it exists to run it.

The model itself#

So what is the model? At a high level, you can think of it as an architecture plus a set of trained parameters, usually called weights. For a modern language model, the architecture defines how information flows through the network, and the parameters are the learned numerical values used during inference to turn input into predictions.

1Input tokens → Model → Probability distribution → Next token → Repeat

The model does not need a file system, a browser, a Git repository, or your company's database to predict the next token. Those capabilities come from the software surrounding it. That distinction becomes extremely important once you start building AI systems.

Where did the model come from?#

There is another entire system behind the model. Before it can serve your request, someone had to create it.

Models need enormous amounts of training data, and that data has to be collected, processed, filtered, transformed, and prepared. During training, the model's parameters are optimized using large amounts of computation, which is where massive GPU clusters and distributed training systems become important. Evaluation is how you measure whether any of that worked, through general benchmarks, task-specific tests, safety evaluations, human review, and internal suites.

A base model can then be further trained or optimized for particular behaviors. That post-training step includes techniques such as supervised fine-tuning and preference-based optimization. Once a model is ready, it has to be packaged and deployed into a serving environment. After users start sending requests, you monitor latency, errors, throughput, resource utilization, quality, safety, and cost, and eventually you learn enough to improve the next version.

This entire loop is part of the broader MLOps and model lifecycle.

Putting the layers together#

We can now return to the original question. What actually happens when you send a prompt to an AI system? Let's say you type, "Fix the authentication bug in my application." A simplified request path looks like this:

The original User → AI → Answer abstraction makes a lot more sense from here. It isn't wrong. It's just hiding a lot of boxes.

Run the authentication example through it and it is the same agent loop from earlier, with the rest of the stack visible. The model reads files, requests edits, and runs tests, but only because the harness executes each of those tool calls and feeds the results back. Every one of those model calls then travels down through the runtime, the serving layer, and the GPU before a single token comes back.

This is why calling the model itself "the coding agent" is misleading. The model is a critical component, but the useful behavior emerges from the system around the model.

The bigger lesson#

This way of looking at AI systems changes how you think about AI engineering. Instead of asking which framework to use, you can start asking which part of the system you are actually building.

If the problem is retrieval, you are thinking about chunking, embeddings, vector search, reranking, metadata, and context construction. If it is tool use, you are thinking about schemas, execution, permissions, failures, and structured outputs. If it is agentic behavior, you are thinking about state, loops, stop conditions, context management, and recovery. If it is serving, you are in batching, scheduling, GPU memory, KV cache, throughput, latency, and inference engines. If it is the model itself, you are talking about architecture, training, evaluation, post-training, and weights.

These are very different engineering problems, and they live at different layers.

You don't need to build every layer#

There is one final thing worth emphasizing. You do not need to build all of this yourself. Most AI engineers are not training a foundation model from scratch, most application developers are not writing GPU kernels, and most teams are not building their own inference engine. You can consume these layers as services.

1Your Application → Agent / Harness → Model API → Someone else's AI infrastructure

Or you might control more of the stack:

1Your Application → Your Agent → Your Serving Layer → Your GPU Cluster → Your Model

The abstraction boundary is a choice. That is exactly how software engineering has always worked. You don't need to build your own database to build a backend, you don't need to build your own operating system to build a web server, and you don't need to train your own foundation model to build an AI application. What matters is understanding where the boundary is and what happens across it.

The next time someone says RAG, agents, MCP, inference, model serving, harness, or MLOps, don't just memorize the definition. Ask which box it is. Once you can place the box, the rest of the system starts making a lot more sense.