RAG vs fine-tuning vs long context: which should you use?

One of the first questions on almost any AI project is some version of: "How do we get the model to know about our business?" Three answers come up repeatedly. Retrieval-augmented generation (RAG) fetches relevant documents at query time. Fine-tuning trains a model further on your examples. Long context simply puts everything into the prompt. Teams often pick one based on what they read last, then discover it was the wrong tool.

These are not competing answers to the same question. They solve different problems, and the clearest way to choose is to separate two things: what the model needs to know, and how the model needs to behave.

Knowledge vs behaviour

Knowledge is facts: your pricing, policies, product docs, contracts, a customer's order history. Knowledge changes, needs to be accurate and often needs to be cited. Behaviour is how the model responds: the format of its output, its tone, which fields it extracts, how it classifies an edge case, how terse it is.

As a rule of thumb, knowledge belongs in the context window, delivered by retrieval or by loading it directly. Behaviour is shaped first by prompting and examples, and only if that fails by fine-tuning. Most disappointing fine-tuning projects are attempts to teach knowledge through a mechanism built for behaviour.

Long context: the simplest thing that might work

Modern models accept very large prompts. If your entire knowledge base is a product manual, a handful of policies and an FAQ, the simplest architecture is to put all of it in the prompt on every request. No vector database, no chunking strategy, no retrieval failures. The model sees everything and can connect information across documents.

The trade-offs are cost and latency, since you pay for every input token on every call, and attention: models can miss details buried in very long prompts, especially when many documents are similar. Prompt caching changes the economics significantly when the large block of context is identical across requests, because repeated reads of a cached prefix are billed at a reduced rate by most providers.

Long context is the right default when the corpus is small and stable, when questions need reasoning across the whole set, or when you are prototyping and want to learn what users ask before building retrieval.

RAG: when knowledge is large, changing or permissioned

Retrieval earns its complexity when the corpus is too large to send each time, when it changes frequently, or when different users may see different documents. Updating knowledge is as simple as re-indexing a document. Access control can be applied at query time, so a user never retrieves what they are not allowed to read. And because each answer is built from specific passages, citations come naturally.

The cost is engineering: ingestion, chunking, embeddings, search quality, reranking and monitoring. Retrieval also introduces a new failure mode. If the right passage is not retrieved, the model cannot use it, and it may answer anyway. Good RAG systems measure retrieval quality separately from answer quality.

Fine-tuning: for behaviour you cannot prompt

Fine-tuning adjusts a model's weights using example inputs and the outputs you want. It is good at making behaviour consistent: always returning a specific structure, matching a house style, classifying according to your internal taxonomy, or getting a smaller, cheaper model to perform a narrow task as well as a larger one does with a long prompt.

It is poor at reliably adding facts. The model may absorb some information from training examples, but it will not reliably recall it, cannot cite it, and will blend it with what it already believed. Every time the facts change, you retrain. Fine-tuning also needs a curated dataset, an evaluation process and ongoing maintenance as base models are updated or retired. Training examples typically look like this:

{
  "messages": [
    { "role": "system", "content": "Classify the support ticket. Reply with JSON only." },
    { "role": "user", "content": "Charged twice for my March invoice, please fix." },
    { "role": "assistant", "content": "{\"category\":\"billing\",\"subcategory\":\"duplicate_charge\",\"urgency\":\"high\"}" }
  ]
}

Notice what the example teaches: a mapping and a format, not a fact. That is where fine-tuning shines. Before committing, try few-shot prompting with a dozen of these examples in the prompt. If that gets you most of the way, you may not need to train at all.

A practical decision path

1. Start with a strong model, a clear prompt and examples. Measure it on a small eval set of real tasks.
2. If it lacks knowledge and the corpus is small, load the documents into the context and use prompt caching.
3. If the corpus is large, changes often or has per-user permissions, add retrieval, and measure retrieval quality on its own.
4. If behaviour is still inconsistent after good prompts and examples, or you need a cheaper model to match a bigger one on a narrow task, consider fine-tuning.
5. Re-run the same evals at every step, so each added piece of complexity has to justify itself with results.

The order matters. Each step adds cost and maintenance, and it is common to find the problem is solved at step one or two. Teams that start with fine-tuning often spend weeks preparing data for a problem that a well-structured prompt with the right documents would have solved.

Key takeaways

• Separate knowledge (facts) from behaviour (format, style, decisions) before choosing an approach.
• Put knowledge in context: long context for small, stable corpora; RAG for large, changing or permissioned ones.
• Use fine-tuning for consistent behaviour or to make a smaller model good at a narrow task, not to teach facts.
• Hybrids are normal: retrieve generously, read with a long-context model, fine-tune only where needed.
• Let an eval set decide each step, starting with the simplest approach.

Not sure which architecture fits your data and budget? Our AI consulting engagements start with exactly this question. Get in touch to talk it through.