Everyone wants an AI agent running. Few people understand the anatomy. Without that clarity, the team builds a chatbot with a big prompt, calls it an agent, and finds out in three months that the system answers fast, makes up facts, ignores new data and executes no action outside the conversation.
This post opens the hood. You will understand what a real agent is, why it needs RAG (Retrieval-Augmented Generation), how a vector database makes semantic search happen, and how it all connects in the execution flow. No mysticism, just engineering.
What an agent is: the loop behind the hype
An agent is not a model. An agent is a structured loop around the model. The LLM reasons and decides; the rest of the system (memory, tools, validation, flow control) executes. Without that framework, you have a pretty text generator, not an agent.
The basic loop has four phases: (1) perception (it receives input from the user or from another system), (2) reasoning (the LLM analyzes context, retrieves memory, chooses the next action), (3) execution (it calls a tool, queries a database, writes to an API), (4) observation (it reads the result and decides whether to continue, finish or ask for help). The loop repeats until the task closes or the budget runs out.
Three variations hold up in production: the reactive agent (one step, simple decision), the planning agent (generates a plan, executes in stages, replans when something fails) and the multi-agent setup (several specialized roles talking to each other, with an orchestrator). Start with the reactive one. Most real cases are solved at the first level and teach you what matters before you complicate things.
RAG: why the agent needs external memory
The LLM has knowledge frozen at training time. It does not know your customer's name, the ticket history, the content of the PDF you uploaded yesterday. All of that is external data. To enter the agent's reasoning, that data has to be retrieved and injected into the prompt at execution time. That is the heart of RAG.
RAG means Retrieval-Augmented Generation. The agent does not try to remember, it searches. On every turn, before reasoning, the system queries a knowledge base, brings in the relevant excerpts and adds them to the context sent to the LLM. The model then generates the answer with grounding, citing or referencing the source, instead of making things up.
RAG solves three problems that a prompt alone does not. Updating: you index new documents at any time, without retraining the model. Scale: the LLM does not need to have seen your entire corpus, it only needs to read the relevant excerpt. Auditing: the answer cites the source, so a human validates it and the team trusts it.
Vector database: how semantic search works
The vector database is the infrastructure that lets RAG scale. Instead of searching by keyword (which ignores synonyms and context), you search by semantic similarity between what the user asked and what is indexed.
It works like this. Each piece of text (chunk) is converted into a high-dimensional numeric vector (768, 1024 or 1536 dimensions, depending on the embedding model). That vector represents the meaning of the text in a mathematical space. When the user asks something, the question also becomes a vector, and the database returns the k closest vectors by distance (cosine, euclidean or inner product).
Choices that matter: the embedding model (OpenAI text-embedding-3-large, Cohere embed v4, BGE for self-hosted), the vector database (Qdrant for self-hosted, Pinecone for managed, pgvector if you already have Postgres, Weaviate when you need hybrid filtering) and the indexing strategy (HNSW for speed, IVF for huge volume, exact for small datasets). There is no universal choice, there is a choice justified by latency, volume and cost requirements.
Ingestion pipeline: from the document to the indexable chunk
Ingestion is where most RAG agents die before they even go live. The raw dataset is rarely what goes into the database. You need a pipeline with clear stages.
(1) Parsing: extract clean text from PDFs, spreadsheets, HTML, audio transcripts. Tools like Unstructured, LlamaParse and Docling do the heavy lifting, but always require review per document type. (2) Chunking: break it into pieces of useful size (300 to 800 tokens works for most cases), respecting semantic boundaries (paragraph, section, never cut mid-sentence). (3) Enrichment: add metadata (source, date, author, tags, ACL) that the agent can filter later. (4) Embedding: run each chunk through the chosen model and store the vector along with the text and the metadata. (5) Indexing: save it to the vector DB with the appropriate index.
One detail that separates amateur from professional: continuous reindexing. Documents change, embedding models evolve, the chunking strategy needs to be revisited. Treat your index like code: versioning, tests, rollback. Without that, you get stuck with the first wrong choice and find out too late.
Tool calling: what sets an agent apart from a chatbot
A chatbot talks. An agent acts. Tool calling is the mechanism that gives the LLM hands: you define a set of functions (each with a name, a description and an input schema), the model chooses which one to call and with what arguments, your code actually executes it, returns the result, and the cycle continues.
Common tools in a serious agent: search_knowledge_base (does the RAG retrieval), query_database (queries structured data), call_external_api (integrates with CRM, ERP, internal systems), execute_code (a safe sandbox for calculation or transformation), write_action (creates a ticket, sends an email, schedules a meeting). Each tool is validated by schema and has its own retry, timeout and logging.
Modern protocols like MCP (Model Context Protocol) standardize how the agent discovers and calls tools across systems. Adopting MCP early avoids redoing the integration for every new model and opens the door to using third-party tools without rewriting the client.
From the prompt to the action: the end-to-end flow
Look at the full path from a real question to the action. A user asks in the internal chat: what is the margin on contract 4421 in the last quarter?. The agent receives it, passes it through the planner, and identifies that it needs two pieces of data: the contract (structured, in the SQL database) and the margin calculation rule (unstructured, in the finance manual in PDF).
Tool 1: query_database(contract_id=4421, period=Q3) brings the gross value. Tool 2: search_knowledge_base(query="cálculo de margem para contratos do tipo X") embeds the question, queries the vector DB, returns the three most relevant chunks from the manual. The LLM now has the raw data and the calculation rule in context, does the math, formats the answer with the source cited (manual page, contract ID) and responds. All in seconds, with an audit of the reasoning recorded.
This is the pattern that scales: the agent decides when to use RAG, when to use a structured query and when to just answer directly. RAG is not the answer to everything, it is one tool on the belt.
Mistakes that kill RAG agents in production
Chunks that are too large: the irrelevant passage pollutes the context and the model gets confused. Chunks that are too small: you lose context, you retrieve a meaningless fragment. A single embedding for everything: legal, technical and conversational domains call for different models (or embedding fine-tuning). No reranker: the first retrieval is rarely optimal, a reranker (Cohere Rerank, BGE-reranker) raises precision a lot. No feedback loop: you do not know when the agent got it wrong if you do not log the query, the retrieved context, the generated answer and a sample of human judgment.
And the most common one: lack of evaluation. An agent with no golden set, no retrieval metric (recall@k, MRR, NDCG), no regression test, becomes a gamble. You push a new prompt in the dark, break something, and find out when the customer complains. Evaluation is not optional, it is what separates an agent that scales from a pretty demo that dies in the second month.
An agent is architecture, not a model
The LLM has become a commodity. It costs cents per million tokens and improves with every release. What sets a working agent apart from a failing one is not the chosen model, it is the architecture around it: a solid ingestion pipeline, a well-configured vector DB, tools with a disciplined schema, observability from day zero and continuous feedback.
Building an agent is distributed systems engineering with a new piece in the middle. Treat it as engineering, not as prompt engineering with a new face, and your agent will last.