RAG (Retrieval-Augmented Generation) is the architecture that connects an LLM to your company's private knowledge at runtime, without training a model, without exposing data to the provider, and with the ability to cite the source of every answer. In 2026, it is the standard approach for internal chatbots, customer service, product copilots, engineering support, and any case that needs the AI to answer based on your content, not the model's generic knowledge.
This guide explains how RAG really works, which decisions matter, which pitfalls to avoid, and how to measure quality to move from "it seems to work" to "it works in production with metrics".
Why RAG and not fine-tuning
There are three ways to make an LLM know something it did not know: (1) put it in the prompt (in-context), (2) train the model (fine-tuning), (3) retrieve dynamically (RAG). RAG combines the advantages of all three: fresh, updatable context like (1), the capacity to absorb a large corpus like (2), and low operational cost like (3).
Fine-tuning shines when the response style matters a lot (voice, format, tone) or when there are specific linguistic patterns. Factual knowledge, however, is the natural territory of RAG: you can update the base without retraining, cite the source, and revoke access to specific documents. For most corporate cases, RAG wins.
Anatomy of a complete RAG pipeline
Seven stages. 1. Ingestion: content extraction from sources (PDF, HTML, Word, Confluence, Notion, repositories, databases). It includes OCR for images, table parsing, hierarchy preservation. 2. Cleaning and normalization: removing boilerplate, deduplication, anonymization where applicable. 3. Chunking: dividing into pieces with an appropriate size and structure. 4. Embedding generation: each chunk becomes a vector. 5. Indexing: vectors + metadata are stored in the vector DB. 6. Retrieval: given a query, retrieve the top-K most relevant chunks via similarity + filters + re-ranking. 7. Generation: the retrieved chunks enter the prompt and the LLM generates the answer, citing sources.
Chunking: the decision that impacts quality the most
Bad chunking sabotages any model. Principles. Respect structure: divide by section, paragraph, or code block, not by a blind token count. Size coherent with the domain: 256 to 512 tokens for a short FAQ; 800 to 1200 for technical documentation; 2000+ for code that needs context. Overlap between chunks of 10 to 20% to preserve continuity. Enrich with context: each chunk carries the document title, the section breadcrumb, the date, the author, the source.
Advanced pattern: contextual retrieval, in which each chunk receives a short sentence generated by an LLM describing the context of the chunk within the larger document. That small enrichment dramatically raises recall on ambiguous queries.
Hybrid retrieval and re-ranking
Purely vector search loses on queries with an exact term (product name, acronym, error code). Purely keyword search loses on queries with paraphrasing. The modern path is hybrid: combining BM25 with vector similarity and merging the results with a scheme like RRF (Reciprocal Rank Fusion).
Above the fusion comes the re-ranker. Taking the top 50 from retrieval and reordering with a specialized model (Cohere Rerank, Voyage Reranker, BGE Reranker), you double the precision in many cases, at an acceptable marginal cost. In serious RAG, a re-ranker is almost always worth it.
The generation prompt: the last mile
The generation prompt makes or breaks the answer. Best practices. Inject the retrieved context in an identifiable block, with the source. Instruct it to cite the source in every statement. Instruct it to admit it does not know when the context does not cover the question. Limit creativity in sensitive domains (medical, legal, financial): "answer only based on the provided context". Ask for a structured format where applicable (JSON, markdown, bullets).
RAG evaluation: the metrics that matter
Without evaluation, RAG becomes a feeling. Necessary metrics. Recall@k and Precision@k: from retrieval, how many relevant chunks appeared in the top-k. MRR (Mean Reciprocal Rank): the average position of the right chunk. Faithfulness: the generated answer actually uses the context, without making things up. Answer relevancy: the answer responds to the question. Context precision: whether the retrieved context is mostly useful or has noise. Frameworks like Ragas, TruLens, Phoenix, and DeepEval automate these metrics.
Build an evaluation dataset early with 50 to 200 pairs of question/expected answer, and run it before every change to the pipeline. Without this, a good change sometimes worsens average quality and no one notices.
Permissions and multi-tenant
A vector DB does not have native ACL in many cases. You need to model permissions at the chunk level. Patterns. Metadata filter: each chunk carries a list of groups/users that can see it; the query filters by the user's group. Collection per tenant: each client in a separate collection (more isolation, more operational cost). Post-processing layer: filters the result before sending it to the LLM based on the ACL of the source system.
Treating permissions as an afterthought is the most expensive mistake in corporate RAG: a leak between clients or between departments becomes a serious incident quickly.
Updating and continuous ingestion
Data changes. Your knowledge base needs to be continuously incremented, not reindexed every time. Pattern: an event queue from sources (Confluence webhook, Drive polling, database CDC) that triggers incremental ingestion, with document versioning and chunk deduplication. A document removed at the source must be removed (or marked) in the base; otherwise, the AI cites a dead source.
Where RAG fails (and how to work around it)
RAG struggles in some scenarios. Aggregative questions ("how many clients requested a cancellation this month?"): a vector DB does not sum; combine RAG with SQL or an agent that delegates to a structured query. Questions that require multi-hop reasoning ("if A depends on B and B changed in January, what is the impact on C?"): simple RAG does not navigate the chain; use an agent that performs multiple guided searches. Questions about code with a reference to a specific symbol: chunking needs to preserve the AST; pure embedding search loses. Use a combination with a symbol index.
RAG as a product, not as a project
Good RAG is a living product: it has an owner, metrics, an improvement cycle, continuous evals, observability. Companies that treat it as a project that "is done" see quality degrade over time as data changes and usage patterns evolve. Companies that treat it as a product reap compound gains.
At Steply, we set up corporate RAG in short cycles: from diagnosing the available knowledge to the first use case in production, with observability and evals, in a few weeks. From there, it evolves with real usage, connecting new sources, refining re-ranking, and expanding to more cases.