Key takeaways
- RAG makes AI look up your documents before answering, so answers are grounded in your data and can cite sources.
- Answer quality depends mostly on your documents and the retrieval step, not on picking the biggest model.
- Permissions must carry through: people should only get answers from documents they're allowed to see.
The problem with asking a model directly
A language model knows a lot about the world and nothing about your refund policy, your contracts or last quarter's decisions. Ask it anyway and it may produce a confident, wrong answer.
Retrieval-augmented generation (RAG) fixes this. Before the model answers, the system finds the most relevant passages in your own documents and gives them to the model with the question. The model answers from that material and points to where it came from.
How it works, in five steps
- Collect the sources: policies, manuals, tickets, contracts, wiki pages, database records.
- Split documents into passages small enough to be precise but large enough to keep context.
- Index each passage by meaning (using embeddings) and usually by keywords too, in a search index or vector database.
- Retrieve the best passages for each question, filtered by what the person is allowed to see.
- Generate an answer from those passages, with citations back to the source.
RAG or fine-tuning?
Fine-tuning changes how a model behaves: its tone, format or specialized skills. RAG changes what it knows at the moment of answering. For company knowledge that changes every week, RAG is almost always the right starting point: update a document and the next answer reflects it, no retraining needed.
What makes a RAG system good
- Clean sources. Outdated and duplicate documents produce outdated and conflicting answers. A cleanup pass pays for itself.
- Good retrieval. Combining meaning-based and keyword search, then re-ranking results, fixes most "it couldn't find it" failures.
- Permissions end to end. The retrieval step must respect the same access rules as the source systems.
- Citations. People trust answers they can check, and wrong answers become easy to spot and fix.
- Evaluation. A set of real questions with known answers, run on every change, tells you whether quality is going up or down.
Questions to ask before you build
- Which documents are the source of truth, and who keeps them current?
- Does any of this data need to stay on private infrastructure?
- Who can see what, and how is that enforced today?
- How will we measure whether answers are right?
RAG powers internal knowledge assistants, support agents and customer portals. We build and run RAG pipelines as part of managed AI infrastructure and inside AI apps.