Model Serving: vLLM, TGI, SGLang
Serving an LLM means turning model weights into a reliable API.
The hard part is not just loading the model. The hard part is keeping latency, throughput, cost, and reliability under control.
Serving architecture
text
client -> API gateway -> scheduler -> model workers -> tokenizer -> GPU inference -> streaming response
Core serving concepts
| Concept | Meaning |
|---|---|
| prefill | process the prompt before generation |
| decode | generate new tokens one at a time |
| KV cache | stored attention keys/values for prior tokens |
| continuous batching | combine active requests dynamically |
| quantization | reduce memory and compute with lower precision |
| tensor parallelism | split model computation across GPUs |
Why KV cache matters
Every active request consumes KV cache memory. Long prompts and long generations can reduce how many users fit on the same hardware.
That is why serving engines care about:
- cache paging
- prefix caching
- max context length
- batching policy
- memory fragmentation
Choosing a serving path
| Option | Best for |
|---|---|
| hosted API | fastest product development |
| vLLM/TGI/SGLang | open-weight production serving |
| llama.cpp-style local serving | edge, desktop, small models |
| gateway plus multiple providers | reliability and routing |
Production checklist
- health checks
- request timeouts
- streaming support
- token limits
- rate limits
- model versioning
- logs and traces
- rollback plan
- capacity tests
Knowledge check
Q1: What is the KV cache?
Stored attention state that lets generation continue without recomputing all prior tokens.
Q2: Why use continuous batching?
To keep accelerators busy while serving many requests with different lengths.