Today's concept: fine-tuning vs prompting - when to use which

You probably don't need to fine-tune. Most people reach for it the second a model misbehaves, when a sharper prompt (or a third option most people forget) would fix it for free.

Plain-English version: prompting is talking to the model - instructions, a few examples, some context to steer it. Instant, no training, you only pay for tokens. Fine-tuning retrains the model's weights on your data so the new behavior is baked in. That costs compute, time, and a real dataset. Configuring a tool vs rebuilding the tool.

What the "fine-tune vs prompt" framing hides: there are three levers, not two.

1) Prompting - change behavior by talking. Start here, always. 2) Retrieval (RAG) - the model isn't dumb, it's missing knowledge. Pull the right docs into the prompt at query time. This is how you feed it your data and today's facts. 3) Fine-tuning - you need one specific behavior, tone, or format, consistently and at scale, and prompting has plateaued.

The rule I keep in my head: RAG fixes what the model KNOWS. Fine-tuning fixes how the model ACTS. Swapping those two is the classic mistake, and fine-tuning to teach facts mostly buys you confident hallucinations.

This bites hardest in agentic loops, where every step is a prompt. One sloppy instruction that triggers retries multiplies your bill, and that's usually when people panic and fine-tune, when fixing (or caching) the prompt was the cheaper move. Fine-tuning does earn its keep when you have real signal (hundreds of clean examples, not a handful) and want to distill that behavior into a small, cheap, fast model. It's lighter than it used to be too: LoRA-style tuning nudges a tiny set of added weights, not the whole model.

One more reason to prompt first: this year tilts even further that way. Long-context windows let you drop dozens of examples straight into the prompt (many-shot). Prompt caching (KV-cache reuse) makes a big, stable system prompt cheap on repeat calls. Reasoning models think longer before answering (test-time compute), so a good prompt now clears bars that used to need training. Prompt first, measure, then spend.

Quick check before you scroll: You want your LLM to always format outputs as valid JSON. Should you fine-tune or prompt?

Full breakdown + the answer: frankduah.me/learnings/2026-06-23-fine-tuning-vs-prompting-when-to-use-which

New here? Why I post these: https://lnkd.in/gK8knHDH

#FineTuningVsPrompting #RAG #AI #LLM #AIAgents #MachineLearning