RAG (retrieval-augmented generation) gives the model the relevant passages from your documents with every question, so it answers from current facts and cites the source. Fine-tuning trains the model itself, so it changes its behaviour: format, style or skill at a narrow task. For questions over company documents you almost always need RAG. Fine-tuning makes sense when the model has the facts but does the task unreliably.

I measured both on Slovak data. Here are the numbers.

What is the difference between RAG and fine-tuning?

RAGFine-tuning
What it changeswhat the model knows when answeringhow the model behaves
New or changed factsjust add a documentyou have to retrain
Source citationyes, the exact passageno
Where an error is when it happenstraceable in the retrieved passageshidden in the model's weights
Best forquestions over contracts, manuals, policiesa fixed format, data extraction, a narrow task at high volume

A simple rule: facts belong in documents, behaviour belongs in weights.

When to choose RAG?

When the model needs facts that are private or change: price lists, contracts, internal procedures, insurance terms.

On Slovak insurance terms I measured how often the model answers correctly. The language model alone, without retrieval, answered 26 % of 77 test questions correctly. With retrieval and a reranker, 83.1 %. The same model, it just got the right passages. The reranker made the difference, raising retrieval accuracy (MRR@10) from 0.511 to 0.776. Neither chunking nor query expansion moved the numbers more than their own noise.

RAG has one more advantage fine-tuning lacks: it can say "it's not in the documents". In the same project, after setting a refusal threshold, the system answered none of the questions the policy didn't cover.

Construction Graph RAG came out similarly: 99.2 % of answers with exact facts on a local 4B model, without any fine-tuning.

When to choose fine-tuning?

When the model can basically do the task but does it unreliably: it returns the wrong format, ignores conventions or is slow at high volume.

In step 13 of my repository I taught a small 1.7B model to extract data from Slovak insurance claim reports into JSON1: The reports are synthetic, so the numbers measure the method, not a product. Expect worse results on real texts with typos and attachments.. All the answers were right there in the report text; the problem was behaviour:

  • Valid JSON: 24 % without examples → 100 % after fine-tuning
  • Field match (F1): 0.812 without examples, 0.885 with three examples in the prompt, 0.968 after fine-tuning
  • Speed: 0.83 s per report versus 1.02 s, at the same quantisation

Typical errors before fine-tuning: an amount returned as the string "1950" instead of a number, a city in the wrong grammatical case. Weights learn exactly this kind of thing well. I do fine-tuning and custom models as part of the ML engineering service.

How much data and money does fine-tuning need?

Less than people usually claim. Just 25 examples beat the best few-shot prompt (0.916 versus 0.885), and the curve flattened at around 400 examples. The whole training ran locally on a Mac mini, with no cloud costs.

LoRA helps: instead of the whole model, only a small add-on matrix is trained. On one layer of a large model (4096 × 4096) that's 0.39 % of the parameters. In step 09 I measured why that's enough: LoRA with rank 8 reached 95.9 % versus 96.9 % for full fine-tuning with three times as many trained parameters.

What fine-tuning won't fix

New knowledge. In the same experiment the weakest field was the city: cities the model had seen in training it got right in 97 of 99 cases, new cities in only 22 of 36. Part of the improvement was memorised, not learned. If you used fine-tuning to teach the model your price list, every price change means retraining, and the model still won't say where the answer came from.

Preference tuning on a task with one correct answer. The next step, DPO, added nothing measurable (0.964 versus 0.968). After fine-tuning the model made so few mistakes that there was nothing left to learn from.

Can they be combined?

Yes, and on larger projects it makes sense. RAG supplies facts from documents and a small fine-tuned model turns them into an exact format, for example for a CRM or accounting. But start with RAG. Add fine-tuning only once you've measured that format or speed is a problem.

How to decide

  1. Prompt and a few examples first. It costs nothing and changes in a minute.
  2. Is the model missing facts that are private or change? Add RAG.
  3. Does the model have the facts but do the task unreliably and at high volume? Consider fine-tuning, starting with a few dozen examples.

Measure every step on a set of real questions, not three demos. I explain how in How to measure RAG quality.

Related