RAG (retrieval-augmented generation) is AI that answers from your documents: it first retrieves the relevant passages and only then builds an answer from them. You won't find out whether it works from a demo on three questions. You find out from a gold question set that measures both retrieval and answers.

This is how I measured Construction Graph RAG, a private AI over a construction company's documents that runs on a single computer with a 4B model.

What is a gold question set?

A gold set is a list of questions where you know the right document, page and answer. In my project it has 128 questions over 12 linked documents in Slovak and English1: The corpus is synthetic but deliberately hard: two projects with similar numbers and the same subcontractors across contracts. All companies and people are made up.. It includes simple factual questions, questions about tables, questions across documents, and questions the documents can't answer.

The last group matters more than it looks. RAG for a company has to be able to say "it's not in the documents", otherwise it makes the answer up.

What do I measure?

  • MRR (mean reciprocal rank): how high the correct passage ranks among the retrieved results.
  • Exact facts in the answer: whether the answer contains the expected numbers, names and dates.
  • Valid citations: whether the cited source really contains what the answer claims.
  • Refusals: whether the AI refuses questions without an answer and doesn't refuse those that have one.

What helped

I started with hybrid search (vectors + BM25) at an MRR of 76.1 %. Two changes brought almost all of the improvement:

  1. Contextual retrieval. At indexing time I add one sentence of context to each chunk of a document. It costs nothing at query time and MRR rose from 76 to 85 %.
  2. Reranker. A model that re-sorts 20 candidates raised MRR from 85 to 96.5 % for about 0.8 seconds per question.

The final setup gives 99.2 % of answers with exactly the expected facts, 100 % valid citations, and correctly refuses 90 % of questions that have no answer.

What hurt

A knowledge graph made results worse on every question: MRR dropped from 76.1 to 54.8 %, because candidates from the graph outvoted the rest of the search. The graph only helps with questions about relations between documents, so that's the only time it runs.

Without measurement I would have left the graph on because it "makes sense". The number showed the opposite.

Why the AI must not compute numbers

Small models make up numbers, times and dates. So in this project the model owns no number: amounts and deadlines are computed by code, and every value the model reads from a document is checked against the source. If it doesn't match, it's dropped.

I tested protection against prompt injection the same way, meaning instructions hidden in documents. Without protection 16.7 % of attacks succeeded; with it, 0 %.

How to start in your company

If you're wondering whether to use a fine-tuned model instead of RAG, read RAG vs. fine-tuning.

All you need is 30 to 50 real questions your people ask and the documents the answers should come from. Build a prototype on them and measure it before you invest more. I'm happy to help if you want.

Related