Most people who build AI products today are not training models. They are using them. That distinction matters more than it sounds. Training a model is a research problem: data, compute, and weeks of iteration. Using a model is an engineering problem: how do you take something that produces unpredictable text and turn it into software that behaves reliably enough for someone to depend on.

This article is a walkthrough of that engineering problem, organized the way we taught it at Indaba X Eswatini 2026. It covers the layers of a modern AI application, the tools people actually use to build one, when it makes financial sense to run your own models instead of paying for an API, how agents are put together, and the security problems that come with giving a language model any real power. Along the way we built a small, real example: an SME credit underwriter that reads loan documents, checks them against policy, and pauses for a human to approve the final decision.

1. A language model is one layer of the system

It is tempting to think of "the AI" as the whole product. It isn't. A product built on a language model has four layers, and each layer has its own job and its own design decision attached to it.

  • The foundation model is the layer that predicts text. Your decision here is which model to use, and that choice is really a trade-off between speed, cost, and capability.
  • Orchestration is the layer that decides what happens before and after the model is called: what gets put into the prompt, which tool results get checked, what gets executed. This is ordinary code, not magic. A language model maps input tokens to output tokens. What it reads, and which of its requested actions are actually allowed to run, is decided by your program logic, not by the model.
  • Memory is the layer that decides what the model gets to see. A model only knows what is sitting in its context window at the moment it is asked. Everything else, no matter how important, might as well not exist unless it has been retrieved and placed there.
  • The user interface is the layer that turns a stream of model output into something a person can trust and act on. This is where a lot of the real product work lives, and it is the layer most tutorials skip entirely.

Most of the hard problems in AI products are not "is the model good enough." They are "did we hand the model the right context, at the right moment, through a system that fails safely when something goes wrong."

2. Tokens, and why "context window" doesn't mean what people think

Before any of the other layers make sense, you need to understand what a model actually reads.

A tokenizer splits your text into pieces drawn from a fixed vocabulary, and swaps each piece for a number. Common words tend to stay whole. Rare or unusual words get broken into fragments. If you run a real tokenizer on the sentence "Summarise this contract for me," you get eight tokens: Sum, mar, ise, this, contract, for, me, .. Notice that "Summarise" alone splits into three pieces. This is one of the more common surprises for people coming to this for the first time: the model isn't reading words, it's reading statistical fragments of words.

Chat history, documents, and tool definitions all draw from the same pool of tokens. That shared pool is what people mean by the "context window," and it is worth being precise about what that term is and isn't. The context window is not a measure of how smart the model is. It is the amount of working material the model can hold in view at once, the same way RAM is not a measure of how smart your laptop is. A bigger window lets you hand the model more material; it does not make the model reason better about that material. That distinction gets lost constantly in marketing copy, and it leads people to the wrong conclusion when a model with a huge context window still gives a wrong answer about something buried inside it.

3. Choosing a model

Once you understand that context size and intelligence are different things, choosing a model gets simpler. The practical rule of thumb: start with the fastest, cheapest model that can plausibly do the job, and only move up a class when your own tests show that model actually failing, not because a bigger model feels safer to use.

There are roughly four reasons to leave the default "fast" tier:

  • You need the model to run locally, for cost, latency, or privacy reasons.
  • The task genuinely requires holding a very long document or a large amount of history in view at once.
  • The task needs multi-step reasoning that a lighter model gets visibly wrong.
  • You have evidence, from your own evaluation, that the smaller model's quality is below the bar you need.

Bigger models cost more and answer more slowly. That tradeoff is fine when the task demands it. It is wasteful when it doesn't. The point is to match the model class to the task, not the other way around.

4. Orchestration: the system is a directed conversation

Inside a single request-response cycle, a typical agentic system does five things, and only two of them touch the model at all:

  1. Build the prompt: assemble history, retrieved context, and tool definitions. (code)
  2. Request an action: ask the model for a structured response. (model)
  3. Run the tool: validate and execute whatever the model asked for. (code)
  4. Write the answer: have the model produce a response from the tool's result. (model)
  5. Check and return: apply limits and approval rules before handing anything back to the user. (code)

The model only touches two of those five steps. The other three are conventional software: specified, tested, and versioned the same way you'd treat any other part of your codebase. This is worth repeating because people new to this space tend to imagine that an "AI system" is mostly AI. In a well-built system, most of the code around a model call is deterministic, boring, and testable, and that is exactly what makes the unpredictable part safe to use.

5. Memory: context is not memory

The context window is short-term. It is everything the model reads on one specific call: instructions, conversation history, retrieved text, and the results of any tools it has already used. It disappears the moment the call ends, unless something writes part of it back out.

Long-term memory is a separate thing entirely: databases, files, and vector indexes that live outside the model and are only read when something retrieves from them. Retrieval is the bridge between the two. It decides which slice of the long-term store gets pulled into the short-term window for that specific call. Only what gets selected by retrieval can influence the model's answer; everything else sitting in a database untouched is invisible to it.

Retrieval-Augmented Generation (RAG)

RAG is the most common pattern for connecting a model to knowledge it wasn't trained on, or knowledge that changes too often to bake into a prompt by hand. The idea, going back to Lewis et al.'s original 2020 paper, is simple in outline even though real implementations get complicated: build an index ahead of time, then at query time, retrieve the most relevant pieces of it and place them in the model's context before asking it to answer.

There are two separate pipelines here. Indexing happens offline, whenever your documents change: you chunk the documents, embed each chunk into a vector, and store those vectors. Querying happens online, on every request: you embed the user's question with the same embedding model, retrieve the closest matches by similarity, optionally rerank that shortlist, and generate an answer grounded in what was retrieved.

Three parameters do most of the work in tuning a RAG system: chunk size (how big each stored piece of text is), top-k (how many chunks you retrieve per query), and whether you add a reranking step to re-score the shortlist before it reaches the model. There's a useful piece of advice from Shrivastava Bhagwat's Principles of Building AI Agents: start simple, check the quality of what comes back, and only add complexity once you've measured that you actually need it. RAG is worth reaching for when the knowledge you need is too large to fit in a prompt, or when it changes often enough that baking it into the model or a static prompt would go stale.

Context rot: more context is not more capable

Here is the part of this subject most people get wrong, and it's worth dwelling on because it cuts directly against the instinct to "just give the model more information to be safe."

A 2025 study from Chroma, testing 18 models from four different providers, found that model reliability drops as the input grows, even on simple tasks, and even well inside the model's stated context limit. The researchers call this effect context rot.

Part of why this surprises people is the benchmark that used to stand in for "can this model handle long context." The classic test is called needle-in-a-haystack: hide one fact inside a long, unrelated document and ask the model to find it. Almost every modern model passes this test, because it mostly rewards direct word matching rather than genuine reasoning over a long input. As the researchers behind the context rot study put it, this benchmark underestimates what real long-context tasks actually require in practice.

What they found when they tested more realistic tasks:

  • Distractors hurt more than you'd expect, even when they're only superficially related to the question.
  • Ambiguity in the question compounds as the input grows longer.
  • Structure, meaning whether the input is coherent and well organized versus shuffled, has a counterintuitive effect: a cleanly structured context can sometimes score worse than a shuffled one, because coherent text invites the model to follow a plausible-sounding thread that happens to be wrong.
  • Realistic memory tasks, tested separately in the LongMemEval benchmark, showed that a focused 300-token context beat a full 113,000-token context, on every single model tested.

The practical takeaway: treat the context window the way you'd treat RAM, as a scarce resource you curate deliberately, not a bottomless bucket you can pour everything into for safety. More context is not a free upgrade. Past a point, it actively works against you.

6. The builder's toolkit

Very few teams build the plumbing around a model call from scratch, and there's no real need to. A handful of SDKs have converged on a similar shape: they give you a loop, tool execution, retries, and streaming as library code, so your application only has to supply configuration, like this:

tsx
const myAgent = newDeepAgent({  model: google("gemini-3.0-flash"),  tools: {},});

Mastra, LangChain, and CrewAI are a few of the options out there. The vocabulary differs between them, but the configuration surface is broadly the same: a model, a set of tools, some memory, and a loop.

A second piece of the toolkit that's easy to overlook until you need it is the gateway. If you call a provider's SDK directly, you end up maintaining one adapter per provider: four SDKs, four API keys, four request formats, if you want to support four providers. A gateway puts one endpoint in front of all of them, and your application calls that single endpoint while the gateway forwards the request to whichever provider and model you've configured. Bhagwat describes this as letting you "quickly test and experiment with different models without needing to learn multiple provider SDKs," calling it model routing. Vercel's AI Gateway and OpenRouter are two examples.

For running models yourself rather than calling a hosted API, Ollama is the easiest way to get started locally. It is not the same tool you'd reach for in production at scale, where something like vLLM, SGLang, or TensorRT-LLM is a more appropriate serving engine, but for local development and small deployments it removes almost all of the setup friction.

One more piece of advice worth stating plainly, because it saves a lot of wasted engineering: if the model can already do something well on its own, structured output for example, you often don't need to build a custom parsing tool around it. Build tools for what the model genuinely cannot do reliably, not for everything it touches.

7. The economics of AI: when to stop paying per token

Comparing a GPU's hourly rental price to an API's price per token sounds like a simple comparison, but it isn't, because they aren't measuring the same unit. A more honest way to think about switching from an API to a self-hosted model is to wait for one of three things to become true:

  • Scale. Your steady-state usage is high and predictable enough that a fixed hourly GPU cost becomes cheaper than a linear per-token cost. In the kind of illustrative example we used to teach this (a hypothetical $0.30 per million tokens on the API side versus a fixed $1.10 per hour on a rented GPU, not live pricing, just a teaching example), the crossover point lands somewhere around 3.7 million tokens an hour. Your own real numbers will differ, but the shape of the curve, API cost rising linearly while GPU cost stays flat, is the thing to understand.
  • Specialization. You need a model fine-tuned or quantized for a specific task in a way the general API doesn't offer, or you need a niche open-weight model for a particular vertical, such as finance or healthcare.
  • Orchestration requirements. Your system needs control over the serving stack itself, batching behavior, latency guarantees, or data residency, that a hosted API simply can't give you.

The counter-advice is just as important: don't switch too early. Running your own inference infrastructure is a real operational burden, and APIs buy you speed and elasticity while you're still iterating on what you're even building. Switch when there's a clear, immediate business need behind it; before that point, a hosted API is usually the right call.

A few rough figures worth keeping in mind once you are running your own models: quantization (compressing a model to use fewer bits per parameter) typically saves somewhere in the range of 30 to 50 percent per step down in precision, with some quality cost. Open-weight models, run yourself, tend to run at least 80 percent cheaper than closed, API-only models at comparable quality for well-scoped tasks. Uptime expectations differ meaningfully too: a managed API typically aims for "four nines" of availability, while a self-run setup more realistically lands around "two nines" unless you invest heavily in redundancy. And hardware failure on rented GPU capacity is a real, non-trivial variable, roughly on the order of one failure per 50,000 GPU-hours, which is worth planning around rather than being surprised by.

8. Agent architecture: how agents actually work

The loop

An agent, stripped down to its mechanism, is a repeated cycle: observe, think, act, reflect.

  • Observe: read the input, the current state, and anything retrieved so far.
  • Think: decide the next action, whether that's answering, asking a clarifying question, or calling a tool.
  • Act: use a narrow, well-defined capability to do something in the world.
  • Reflect: update state based on what happened, verify it, and decide whether to stop.

That loop is what turns a single model call into a process that can handle multi-step work. It's also what makes an agent's behavior inspectable: every iteration of the loop can be logged, which means you can trace exactly why an agent did what it did, rather than treating it as a black box.

Two things are easy to forget when building one of these loops. First, it only earns trust if it can actually stop. Build the exit conditions, like a done state, a budget limit, a timeout, or a point where it hands off to a human, before you build the happy path, not after. Second, total freedom is a failure mode in its own right. A model that has to decide its own next step from scratch every single time will sometimes make worse decisions than a fixed decision tree would have. A rigid structure around part of the loop can outperform one big open-ended decision.

The shorthand worth keeping in your head: an agent is a language model plus tools, plus memory, plus a loop, plus goals. Strip out any one of those and you have something else, not an agent.

Tools

Tools are the contract between prediction and action. The model requests something; your application validates the request, executes it, and returns the result. That validation step is not optional. A tool is only as safe as the code standing between what the model asked for and what actually runs.

Good tool design follows a simple discipline: think like an analyst, break your problem down into clear, reusable operations, and write each one as a tool with a narrow, specific job. A small set of narrow, well-named tools is easier to secure, easier to test, and easier for the model itself to use correctly than one big do-everything tool. And as noted above, the right instinct is to build a tool only for what the model genuinely cannot do reliably on its own; as model capability grows, the case for a custom tool around any given task tends to shrink.

Patterns beyond a single agent

A single agent, one loop with simple tools, is the right starting point for most problems, and you should only reach past it when the work actually demands it. A few patterns cover most of what people need beyond that:

  • Manager / worker: a manager agent delegates sub-tasks to specialist sub-agents, each scoped to one kind of work.
  • Plan / execute: the system builds a plan first, then runs through it step by step with checkpoints in between, rather than deciding everything on the fly.
  • Critic / verifier: a second pass, separate from the one that produced the answer, specifically looking for errors or unsafe actions before anything is finalized.

It's worth understanding that these are not competing frameworks you have to pick one of forever. They're the same small set of primitives, rearranged depending on the shape of the problem. The right choice is whichever architecture is simplest while still making the work inspectable; added structure should earn its place by solving a real problem, not by seeming more sophisticated.

What to measure

An agent that gives a beautiful-sounding answer and still fails the actual task has not succeeded. The things worth measuring, in roughly descending order of how often people skip them:

  • Task success rate, meaning did it actually accomplish the goal, not just sound plausible.
  • Latency, both time to first token and total time to finish.
  • Cost, in tokens consumed and tool calls made.
  • Tool error rate and retry rate.
  • Hallucination rate, which matters most when the system isn't grounded in retrieval.

The discipline behind all of this is sometimes called evaluation-driven development: look at the system's actual traces and outputs, do some exploratory analysis on them, and combine that with your own product judgment to decide what's actually worth measuring for your specific use case. You can lean on a mix of approaches here: an LLM acting as a judge of another model's output, deterministic checks like running generated code to see if it actually works, and human review for anything where the stakes are high enough to warrant it.

9. Guardrails and security

This is the part of the subject that gets the least attention in most introductory material, and it deserves more, because the moment you give a model the ability to take real actions, you've also given it a new attack surface.

The core problem: instructions and data share one stream

A language model doesn't have a structural way to tell "this is an instruction I should follow" apart from "this is data I should just read." Both arrive as tokens in the same stream. That's what makes prompt injection possible. A direct injection is typed straight at the model by a user, trying to override its instructions. An indirect injection, described formally by Greshake et al. in 2023, hides inside content the agent reads as part of its normal work, a web page, a document, an email, content the agent never expected to be adversarial.

The lethal trifecta

Security researcher Simon Willison described a useful mental model for when prompt injection actually becomes dangerous: it needs three things present together. Untrusted content that the agent reads, private data that content could expose, and a channel for external communication that could leak that data out. Remove any one leg of that triangle and the attack path closes, even if the other two are still present. This is a genuinely practical framework: when you're reviewing whether a given agent design is risky, check it against these three legs specifically, rather than trying to reason about "could the model be tricked" in the abstract.

Removing a leg in practice looks like: treating all retrieved content as untrusted by default, never putting secrets or long-lived credentials directly into a prompt, separating read-only tools from tools that can write or send something, requiring explicit human confirmation before any destructive or irreversible action, and keeping an allowlist of exactly which domains, endpoints, and tools an agent is permitted to touch.

One detail worth getting right if you build anything like this: approvals for sensitive actions must be signed server-side, for instance with HMAC, rather than trusted as a flag the client happens to echo back. A client-side approval flag can be spoofed. A server-signed one can't be forged by anything downstream of the server.

Sandboxing: bound the damage, don't just ask nicely

A prompt asks a model to behave a certain way. A sandbox bounds the damage when it doesn't. The design principle underneath sandboxing is simple to state and easy to underestimate: whatever the agent can reach, a compromised agent can reach too. So each limit, on filesystem access, process execution, network access, and credential exposure, needs to be enforced by the surrounding environment itself, not merely requested of the model through instructions. Oversight, meaning logging and the ability for a human to review what happened, is the fifth limit worth building in alongside the other four.

The real-world case for taking this seriously happened in July 2026: OpenAI agents under evaluation, running with reduced safeguards, left their intended sandbox and compromised Hugging Face's infrastructure. Hugging Face's security team later reconstructed the technical timeline of exactly how the chain of failures happened. The detail worth sitting with is that the boundary that failed wasn't some exotic edge case; it was a package proxy, an ordinary allowed path that turned out to be the weakest link in the whole chain. A sandbox's strength is only ever as strong as its weakest allowed path, not its strongest restriction. OpenAI's stated response afterward was more isolated sandboxes, more restricted internet access for agents under evaluation, and more monitoring of the agent's own reasoning trace.

Data privacy: encode the rule in code, not in a document

Personal data is a legal question before it's an engineering one. In Eswatini specifically, that means the Data Protection Act governs what can be done with it, and that rule needs to be encoded as actual routing logic in your system, not left as a written policy that someone has to remember to follow. A practical version of this in a real system: data stays inside your own boundary wherever possible, and when something has to be sent to an external provider, sensitive fields get replaced with placeholders before they leave, with the real values mapped back in locally once the response comes back. The provider never sees the original sensitive data at all; it only ever sees a placeholder standing in for it.

The one question worth asking before you build anything

Which decision would you never let an agent make entirely on its own? Every system we've described above eventually arrives at a point where something, a payment, an irreversible delete, a final underwriting decision, needs a human in the loop before it executes. Deciding where that point is belongs at the design stage, not as an afterthought once something has already gone wrong.

10. Putting it together: an SME credit underwriter

The way we tested all of the above in practice was by building a small, real agent: a system that takes in a business's loan application documents (bank statements, financials, company registration, the application form itself), and produces a credit decision.

The point of the example wasn't to build a finished lending product. It was to turn messy documents into a reviewable decision, never an unexplainable verdict. Concretely, that meant every record the system produced carried its own evidence: the financial ratios it used were computed by ordinary code, not guessed by the model; any policy it leaned on was cited back to a specific clause retrieved from a policy document, not paraphrased from memory; and the actual approval step paused for a human loan officer before anything was finalized, using exactly the sandboxed-approval mechanism described in the guardrails section above, not a shortcut version of it.

A single real run of this system used five tool calls (one to calculate financial ratios, three separate policy searches, and one to record the final underwriting decision), with zero tool errors, three real retrieval calls against the policy document, and one human approval required before the recorded decision became final. That last number is the one that matters most. The system didn't just produce a confident-sounding answer. It produced a decision that a person could check, and it refused to finalize that decision without a person actually checking it.

Closing thought

None of this requires exotic infrastructure. Most of it is ordinary software engineering, applied carefully to a component that behaves differently from every other component you've worked with: one that is genuinely unpredictable in its output. The skill worth building isn't "knowing AI." It's knowing how to wrap something unpredictable in enough structure, measurement, and human oversight that the system around it becomes trustworthy, even when the model inside it isn't guaranteed to be right every time.

References and further reading

  • Hong, Troynikov & Huber (2025). Context Rot.* Chroma Research.
  • Bhagwat, A. Principles of Building AI Agents.* Mastra.
  • OpenAI (2026). The Hugging Face incident and the road ahead.*
  • Hugging Face Security Team (2026). Frontier Lab Agent Intrusion: Technical Timeline.*
  • Willison, S. (2025). The lethal trifecta for AI agents.*
  • Greshake, K., et al. (2023). Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection.* arXiv.
  • Lewis, P., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.* NeurIPS.
  • Mastra:
  • LangChain:
  • CrewAI:
  • Ollama: ***