Fine-tuning

Fine-tuning takes a pre-trained foundation model (Llama, Mistral, GPT-3.5, etc.) and continues training it on a much smaller, task-specific dataset of input/output pairs. The result is a model that's specialized for your domain — your tone of voice, your output format, your edge cases.

The mental model: foundation models know how to write English, code, and reason. Fine-tuning teaches them how to write your English, in your format, for your problem.

The big shift in the last two years: fine-tuning used to be the default for "make GPT do my thing." Today, for most teams, prompting + RAG outperforms fine-tuning at a fraction of the cost. Reasons:

• Cost — a fine-tune costs $50-5000 per run depending on size; a prompt costs nothing to iterate on.
• Speed — you can change a prompt in seconds; you can't change a fine-tune in seconds.
• Capability ceiling — modern frontier models out-of-the-box are better than a fine-tuned mid-tier model from 2023.

That said, fine-tuning still wins for: tightly constrained output formats (structured JSON, function calls), domain-specific tone that prompts can't reliably elicit, latency-critical paths where you can't afford a giant system prompt, and on-prem deployments where you can't call an API.

If you're choosing between fine-tuning and RAG: pick RAG if you have data that changes; pick fine-tuning if you have style that's hard to describe.