Model serving: admission, scheduling, versioning, and uncertain outcomes

Serving turns a model artifact into a concurrent network service. Latency and throughput depend on hardware, model shape, input/output length, batching, scheduler, cache state, load, and failure policy; benchmark results do not transfer automatically.

Mental model

• TensorFlow Serving describes versioned model serving and lifecycle mechanisms for production deployments. Teach model serving as a versioned service rather than a single predict call.
• PagedAttention and vLLM report workload-specific KV-cache and throughput results under evaluated models and traces. Explain scheduling and memory design without universal throughput multipliers.
• KServe documents inference services, revisions, autoscaling, and serving runtimes. Separate platform controls from model quality guarantees.
• NIST AI RMF calls for measured risk management across the AI lifecycle. Gate serving changes by use-case risk and evidence.

Guarantees and boundaries

• Every request carries an immutable model and runtime version.
• Admission occurs before expensive allocation.
• A timeout is represented as UNKNOWN until an authoritative outcome is found.

Diagram scenarios

Admission before batching. The gateway validates and budgets work before the scheduler allocates model resources.

Canary a compatible model bundle. A model or runtime update is released to a bounded cohort and compared with the champion.

Overload and cancellation. Queue growth and client cancellation are explicit states, not hidden retries.

Timeout leaves an uncertain outcome. The client can lose the response after work ran; request identity supports status lookup and safe retry policy.

Architecture decision

Pin model and runtime digests, enforce admission and token budgets before allocation, measure time-to-first-output and completion percentiles under representative concurrency, and make overload and cancellation explicit. Canary releases by compatible request cohort and preserve rollback plus request identity.

Failure modes

• Dynamic batching can improve utilization while increasing queue delay.
• Cancellation may arrive after generation or billing work occurred.
• A healthy HTTP response says nothing about model quality, calibration, or safety.

Operational checklist

• Load-test representative prompt and output lengths at realistic concurrency.
• Measure queue, prefill, first-output, decode, and end-to-end percentiles.
• Test OOM, overload, cancellation, worker loss, retry, and rollback.
• Keep tenant fairness and sensitive prompt logging policies explicit.

Related concept

Primary sources

1. https://arxiv.org/abs/1712.06139
2. https://arxiv.org/abs/2309.06180
3. https://kserve.github.io/website/docs/concepts/resources
4. https://www.nist.gov/itl/ai-risk-management-framework