RAG vs Fine-Tuning: An Ablation Study From Production

The System

A production market-analysis assistant built on a tuned pgvector/HNSW store (131,881 vectors) feeding a fine-tuned 4B model served locally. The retrieval layer provides grounding in five years of market data; the fine-tuned model locks behaviour, tone and output schema.

The Ablation

We ran a controlled ablation — same model, same eval harness, same held-out cases — with and without the retrieval layer:

Schema-valid output (750 held-out cases)	100%
Directional accuracy (fine-tuned)	97.6%
RAG ablation lift (base model)	+2.7pts direction · ECE 0.335→0.306
kNN query latency (tuned HNSW)	0.93 ms
Regression suite	29 passing tests incl. index verification

What the Ablation Proved

Remove retrieval and directional accuracy drops 2.7 points while calibration degrades (ECE 0.306→0.335). The ablation proved the architecture — not the demo. RAG isn't decoration; it's the grounding that makes fine-tuning predictions reliable.

When to Use RAG vs Fine-Tuning

• Default to RAG for knowledge that changes — it's the right first choice for roughly 80% of enterprise LLM applications.
• Fine-tune to lock behaviour, tone and output schema that prompting can't hold, or to distil a frontier model into a cheaper one for cost and latency.
• In production we usually combine both: retrieval for facts, fine-tuning for behaviour.

Key Takeaway

Without an ablation study, you're guessing about your architecture. With one, you know whether each component earns its place. The 2.7-point lift and 0.029 ECE improvement aren't vanity metrics — they're the difference between a system that works and one that's confidently wrong.