System Design Cases
Fine-tuning vs RAG vs Prompting
Concept page: Fine-tuning vs RAG vs Prompting. Decision framework for choosing between prompt engineering (cheap, instant), RAG (dynamic facts + citations), and fine-tuning (LoRA/QLoRA/DPO for behavior/style/format). Shows hybrid production pattern combining all three. 6 scenarios: prompting ladder (zero-shot to CoT), RAG for knowledge, LoRA SFT for format consistency, hybrid (fine-tune+RAG+prompt), anti-pattern fine-tune-for-facts, LoRA hot-swap for multi-domain inference. Includes 2 ADRs covering the full decision ladder and LoRA/QLoRA/DPO/full fine-tune tradeoffs.
Prompting, retrieval, and fine-tuning solve different failure classes
Prompting changes inference context, retrieval supplies external evidence, and fine-tuning changes model parameters or behavior. None guarantees correctness, freshness, safety, or deterministic output, and combinations introduce separate data and lifecycle risks.
Mental model
- RAG supplies retrieved non-parametric memory at inference time in addition to model parameters. Use retrieval for evidence with provenance, not as a factuality guarantee.
- Instruction fine-tuning with human feedback changed behavior in the reported model and prompt distribution, while errors remained. Do not promise general alignment or correctness from fine-tuning.
- OpenAI's model optimization workflow couples evals, prompting, and fine-tuning. Adopt an eval-driven sequence instead of a fixed vendor-independent recipe.
- Fine-tuning jobs produce new model artifacts from explicit training data and configuration. Treat tuned models as versioned deployable artifacts with rollback.
Guarantees and boundaries
- Each approach changes a different part of the system and has a separate version.
- A held-out test set is not training data.
- Fresh retrieved evidence does not force the generator to use it correctly.
Diagram scenarios
Prompting baseline first. A template and examples are the lowest-lifecycle-cost intervention, but still require regression evaluation.
Retrieval for changing evidence. RAG can update attributable knowledge without changing weights, while adding retrieval failures and injection risk.
Fine-tuning as a model lifecycle. Training requires curated data, held-out evaluation, provenance, release gates, and rollback.
Combination requires ablation. Prompt, retrieval, and tuned weights can mask each other's regressions and compound cost.
Architecture decision
Start with a versioned baseline and task evaluation. Use prompting for interface and examples, retrieval for attributable changing knowledge, and fine-tuning only when a curated dataset and stable behavioral objective justify training operations. Combine methods only when an ablation proves each component's value.
Failure modes
- Fine-tuning can memorize sensitive or erroneous examples.
- Retrieval can return stale, unauthorized, or injected passages.
- Prompt changes can regress previously passing cases without an API error.
Operational checklist
- Name the observed failure before selecting a technique.
- Keep data provenance, consent, licensing, deletion, and contamination controls.
- Run ablations under the same model, corpus, prompt, and eval versions.
- Budget training, indexing, serving, review, rollback, and monitoring—not only token charges.