A model is fast alone but slow when several learners arrive together. Where did the extra time go? Separate queueing, input processing, token generation, and network delivery before choosing an optimization.
Before you begin: You understand autoregressive generation, context windows, and quantization.
Serving turns a checkpoint into a dependable request-processing service. It includes input limits, scheduling, memory management, streaming, cancellation, and failure reporting.
Separate the two main phases
During prefill, the model processes the input context and builds the state needed for generation. During decoding, it produces new tokens step by step. Their hardware behavior differs, so optimizing one phase does not automatically optimize the other.
Time to first token includes more than model prefill: queueing, retrieval, network time, and application work can all contribute. Inter-token latency describes the pace after generation begins. Users notice both.
| Measure | What it helps answer |
|---|---|
| Time to first token | How long does the user wait for a response to begin? |
| Inter-token latency | How smoothly does the answer arrive? |
| End-to-end latency | When is the usable result complete? |
| Throughput | How much work can the service sustain? |
| Tail latency | What happens to slower requests under load? |
Batching trades efficiency against waiting
Batching lets hardware process several requests together. Continuous batching can add and remove active sequences as requests arrive and finish. It can improve utilization, but the scheduling policy still affects fairness and waiting time.
Long inputs and long outputs consume capacity differently. Set separate limits and avoid letting one oversized request monopolize the service. A queue needs a maximum size or admission policy; unlimited waiting is not a reliable overload strategy.
The key/value cache grows with cached tokens, layers, key/value heads, and active sequences. Paged cache management can reduce waste from allocation patterns, but it does not make context storage free.
Put a long request ahead of a short one
Imagine one request with a long document arrives just before a short factual question. Under one scheduling policy, the short request waits behind the long prefill. Under another, work can be scheduled differently. The exact behavior depends on the serving engine and configuration.
What measurements would distinguish faster model computation from reduced waiting?
Split the latency trace
Record admission time, queue wait, prefill duration, first visible output, and completion. If queue wait falls while model phases stay similar, scheduling improved the journey. A single end-to-end number would show an improvement without explaining its cause. Keep the same load mix when comparing configurations.
Add cancellation and overload cases. A service that rejects excess work promptly with a useful retry response may serve users better than one that accepts unlimited work and times out much later. Measure tail latency and accepted-task completion, not only peak throughput.
Choose a serving path you can operate
A hosted API shifts much infrastructure work to a provider, while introducing provider limits and data-handling choices. Self-hosting provides more control but requires capacity planning, patching, monitoring, and hardware operations. Compare complete costs and constraints rather than only model-token prices.
For local experiments, use the serving project's official quickstart and a compatible model. Keep the endpoint private until authentication, request limits, and operational controls are in place. A development server listening on every interface is not a finished deployment.
Record model revision, tokenizer, runtime version, quantization, generation settings, and hardware. Without them, a reported tokens-per-second number is difficult to interpret or reproduce.
Test the workload users actually create
Mix short and long prompts, short and long outputs, cancellations, and concurrent requests. Measure cold starts separately from warm requests. Inspect latency percentiles and errors as load grows, not only the fastest single response.
On mobile, stop server-side generation when cancellation is supported and the user no longer needs the result. Reconnecting should recover durable task state where the product promises continuity, rather than launch a duplicate costly request.
Exercise: throughput increases after larger batching, but users wait longer for the first token. Is the change an improvement?
Compare your reasoning
It is a tradeoff. Judge it against the product's latency and capacity targets, including tail behavior. Higher aggregate throughput alone does not establish a better user experience.
Next, learn which repeated work can be cached safely.
Sources
The current vLLM documentation describes serving and scheduling capabilities. PagedAttention explains one approach to managing attention-cache memory efficiently.
Continue: Prompt Caching and KV-Cache Strategies.