Choosing a serving engine

vLLM, SGLang, TensorRT-LLM and others, by fit

Published: 8 November 2026
Last reviewed: 8 November 2026
Re-checked every: 90 days
Reviewed by: Ales Analytics engineering
Used in: Private AI infrastructure

In one paragraph

A serving engine runs model weights on GPUs and answers many requests at once through an API. The choice depends on the hardware, the model architecture, the workload's shape (prompt lengths, prefix sharing, concurrency) and who will operate it. Published comparisons differ in all of those, so the engine is chosen by testing on the client's own workload rather than from a ranking.

How it works

Engine	Typical fit
vLLM	General-purpose GPU serving, broad model support, OpenAI-compatible API
SGLang	Heavy prefix sharing and structured generation (radix-tree cache)
TensorRT-LLM	NVIDIA hardware where compiled, hardware-specific engines justify a build step
LMDeploy	Alternative GPU engine with strong quantisation support
llama.cpp, Ollama	CPUs, laptops, edge devices and small deployments
NVIDIA Dynamo, llm-d	Cluster layers for routing, disaggregation and scaling across many GPUs

Design choices and trade-offs

Support for the model. New architectures reach engines at different speeds; check support for the exact model and quantisation format.
Operations. An engine the team can upgrade and debug is worth more than a few per cent of throughput.
Features. Prefix caching, speculative decoding, multi-LoRA, structured outputs and metrics differ in maturity between engines.

When not to use it

When a single user runs occasional jobs on a workstation, a simple local runtime is enough.

How we evaluate it

Run each candidate on the target hardware with the client's prompts at the expected concurrency, and report TTFT, inter-token latency and goodput at the agreed targets, with versions and settings stated.

Sources

Should you have any questions regarding this page, please contact us at solutions@alesanalytics.com.