RAG vs fine-tuning: which approach makes sense?

What each one actually changes

Retrieval-augmented generation fetches relevant text at question time and gives it to the model as context. The model's weights are untouched. It knows your facts because you just handed them to it.

Fine-tuning adjusts the model's weights on examples of desired behaviour. It shifts style, format and task-specific behaviour. It is a poor and expensive way to install facts, and the facts it does absorb cannot be corrected without training again.

Choose retrieval when

• The knowledge changes — prices, policies, inventory, documentation
• Answers must be traceable to a source, because someone will ask where a figure came from
• Different users may see different subsets of the content
• The corpus is large and only a fraction is relevant to any one question
• You need to be able to fix a wrong answer today rather than at the next training run

That list covers the large majority of business applications.

Consider fine-tuning when

• Output must follow a rigid structure that prompting keeps drifting away from
• A specialised tone or vocabulary matters and examples communicate it better than instructions
• A narrow, high-volume classification or extraction task justifies a smaller, cheaper, faster model
• Prompt length has become a material cost at your volume and examples can be moved into the weights

Note what these have in common: they are all about behaviour, not knowledge.

When both apply

A common mature pattern is a fine-tuned smaller model handling a narrow task with tight output requirements, retrieving facts at question time. You get the format reliability from tuning and the factual currency from retrieval, usually at lower inference cost than a large general model.

This is an optimisation, not a starting point. Build with retrieval and a capable general model first, measure, and only then consider whether tuning a smaller model earns its keep.

The maintenance question

The decision people underweight is not build cost but change cost. A retrieval system is corrected by editing a document and re-indexing — minutes, by a non-engineer. A fine-tuned model is corrected by curating new examples, retraining, evaluating and redeploying — days, by a specialist.

Ask how often the underlying material changes. If the answer is monthly or faster, retrieval is almost certainly the right instrument.