RAG vs Fine-Tuning vs Prompt Engineering: Which Do You Need?

Quick answer: Start with prompt engineering. If the model still gets facts wrong or does not know your data, you have a knowledge problem and you need RAG. If it knows the facts but will not behave consistently, in your format or your tone, you have a behaviour problem and that is what fine-tuning is for. The single most expensive mistake in this area is reaching for fine-tuning to teach a model new information. Fine-tuning changes how a model behaves, not what it knows. OpenAI's own optimisation guide puts it plainly: "Prompt engineering is typically the best place to start," and you should "squeeze as much accuracy out of basic methods as you can before reaching for more complex RAG or fine-tuning."

The mistake almost everyone makes first

A team builds something on an LLM. It gets answers wrong about their own products, their own policies, their own data. Someone says the obvious-sounding thing: we should fine-tune it on our documents.

That instinct is wrong, and it is expensive. Fine-tuning is further training on additional data so the model "starts representing and mimicking the patterns and characteristics of the fine-tuning dataset" (Claude docs). It teaches patterns, style and format. It is a poor and costly way to teach facts, and the facts it does absorb are frozen at training time and cannot be updated without training again.

If your model does not know something, the fix is almost always to give it the information at the moment you ask. That is RAG.

What each one actually does

Prompt engineering

Changing what you ask and how you ask it: instructions, structure, examples, reasoning steps, and the settings that control variability. Nothing about the model changes. It is the cheapest and fastest of the three, it requires no infrastructure, and it is where every project should begin, because it is also how you find out what is actually broken.

RAG (Retrieval-Augmented Generation)

Fetching relevant information at query time and passing it to the model along with the question. Claude's documentation describes it as combining retrieval with generation "to improve the accuracy and relevance of the generated text, and to better ground the model's response in evidence", which lets the model "access and use information beyond its training data" (Claude docs). This is the answer to a knowledge gap, and it has a property fine-tuning does not: update the documents and the answers update immediately.

Fine-tuning

Further training the model on your examples so it internalises a pattern of behaviour. This is the right tool when you need a specific format or voice on every single response, and prompting gets you there only most of the time. It is the most expensive of the three in money, data and effort, and it is worth noting that Anthropic does not offer it for Claude at all: "The Claude API does not currently offer fine-tuning" (Claude docs).

The diagnostic that settles it

OpenAI's optimisation guide frames the choice as two different kinds of memory problem, and it is the most useful way to think about this. After you have a baseline from prompting, ask which of these you are looking at:

Read it twice, because it is the whole article: knowledge problems go to RAG, behaviour problems go to fine-tuning, and everything starts at prompting.

Side by side

The order to actually work in

1. Write a decent prompt and measure it

   Not a guess at a prompt: a proper one, with clear instructions, examples where they help, and a reasoning step for anything involving multi-step logic. Then build a small evaluation set so you can tell whether a change helped. Without that baseline you cannot diagnose anything, and you will not know whether RAG or fine-tuning improved matters or not.

2. Diagnose the failures, do not guess at them

   Read the wrong answers. Are they wrong because the model lacked information, or because it had the information and behaved badly with it? That one question determines everything that follows.

3. Add retrieval if it is a knowledge problem

   Most production failures are here. The model cannot know your contracts, tickets or product documentation unless you give them to it. This is also where most of the engineering work lives: chunking, retrieval quality, re-ranking and evaluation.

4. Consider fine-tuning only for stubborn behaviour

   If after good prompting and good retrieval the remaining problem is that the output format or voice still drifts, fine-tuning is the right tool. Arriving here is much rarer than the internet suggests.

Can you combine them? Yes, carefully

These are not mutually exclusive, and serious systems often use prompting plus RAG together. Fine-tuning plus RAG is also possible, but it deserves a warning that comes from OpenAI's own worked example: in their Icelandic language case study, adding RAG to an already fine-tuned model reduced accuracy, from 87 to 83. Their conclusion is the lesson, which is to use "the right optimization tool for the right job" rather than stacking techniques and hoping.

More tooling is not more accuracy. Each addition brings its own failure modes, and retrieval that surfaces irrelevant context actively damages an answer the model would otherwise have produced correctly.

Where agents fit into this

A fourth option has become common enough to mention. Instead of retrieving a fixed set of documents, you can let the model decide what it needs, call a tool to fetch it, check the result and try again. That is an agentic system, and it is what people mean by autonomous retrieval or research agents.

It is more capable and considerably less predictable, and it belongs after you have retrieval working rather than instead of it. If your retrieval is unreliable, an agent calling it repeatedly produces unreliable results more expensively. Our Agentic AI course covers that layer, including self-healing retrieval and multi-agent supervision.

The part nobody budgets for: evaluation

Whichever route you take, the work that decides whether it succeeds is evaluation, and it is consistently the piece teams skip. Without a measurable way to tell whether an answer is good, you cannot know whether your new chunking strategy helped, whether a prompt change broke something else, or whether fine-tuning was worth the money.

This is also why "it works" is such a weak claim in AI engineering, and why the first thing a serious interviewer asks is how you knew your output was right. Build the evaluation set before you build the clever part.

Which should you learn first?

If you are deciding where to put your own study time rather than choosing a technique for a project, the order is the same as the engineering order, for the same reasons.

• Prompt engineering first. It needs no programming, it applies to every AI tool you already use, and it teaches you how models behave, which the other two assume. Our self-paced prompt engineering course covers the mechanism rather than a list of prompts to copy.
• RAG next, once you are comfortable in Python. It is the backbone of most applied AI work in production, and it is where you learn to measure quality properly. Our RAG course runs from embeddings through hybrid search, re-ranking and RAGAS evaluation.
• Agents after that. They extend retrieval with planning, memory and tools, and they assume you already understand what they are extending.
• Fine-tuning last, if at all. For most engineers it never becomes the daily job. Knowing precisely when it is the right answer is more valuable than being able to run one.

If you would rather have the full career path than individual skills, our Artificial Intelligence course and the Applied AI Forward Deployment Engineering Program cover this ground with live teaching and placement assistance.

Sources

• OpenAI, Optimizing LLM accuracy: the recommended order, the context-versus-behaviour diagnostic, and the Icelandic case study where RAG reduced the accuracy of a fine-tuned model from 87 to 83: developers.openai.com.
• Anthropic, Claude documentation glossary: definitions of fine-tuning, RAG, context windows, tokens and temperature, and the statement that the Claude API does not currently offer fine-tuning: platform.claude.com.
• Both sources were read directly at the time of writing. Model capabilities and vendor offerings in this area change quickly, so check the current documentation before making an architecture decision on the strength of any article, including this one. Written 3 October 2026.