System Design Cases
AI Cost Optimization
AI cost optimization concept page covering prompt caching, model routing, batch API, distillation, semantic response cache, per-user budgets, self-host vs API break-even.
AI cost optimization: measured unit economics under quality and risk gates
AI cost includes input and output processing, retrieval, tools, storage, training, idle capacity, retries, evaluation, review, and incidents. Provider prices and model efficiency change, while cheaper routing can reduce quality or safety on important slices.
Mental model
- FrugalGPT studies prompt adaptation, model approximation, and cascades with workload-specific cost and performance results. Use cascades only after evaluating routing errors and accepted task outcomes.
- PagedAttention and vLLM report serving efficiency under specific models, traces, and baselines. Treat serving optimizations as benchmark candidates, not universal savings.
- OpenAI pricing and platform billing dimensions are current product data and can change. Fetch current prices for calculations and preserve the effective date; do not hardcode course prices.
- OpenAI documents prompt caching and batch behavior as provider-specific features. Model cache and batch eligibility, keys, and failure states explicitly.
Guarantees and boundaries
- Every request has a maximum work budget and typed termination state.
- Cost reports include all attempts for an accepted task.
- Cache authorization and version dimensions cannot be dropped for hit rate.
Diagram scenarios
Measured model cascade. A cheaper candidate handles eligible tasks and escalates uncertainty under evaluated policy.
Authorization-safe exact cache. A cache hit is valid only when every behavior-affecting and access-control dimension matches.
Batch only delay-tolerant work. Batching can change price or utilization but introduces queue, cancellation, and partial-result semantics.
Budget exhaustion and quality gate. A cheaper system is unacceptable when it violates a declared quality, safety, or fairness gate.
Architecture decision
Measure cost per completed acceptable task using current provider and infrastructure data. Bound work before execution, cache only when the full semantic and authorization key is safe, batch only delay-tolerant jobs, and route or cascade models under quality, safety, latency, and fairness gates with explicit fallback.
Failure modes
- Retries, judge calls, long outputs, and agent loops multiply cost.
- Semantic caches can leak cross-tenant or stale answers.
- A cheap route can disproportionately fail tail or high-risk slices.
Operational checklist
- Measure cost per accepted task and per failure class.
- Pin effective prices, hardware utilization assumptions, and workload distribution.
- Evaluate routing, caching, batching, compression, and quantization separately.
- Test budget exhaustion, provider timeout, duplicate jobs, cache poisoning, and safe fallback.