# Choosing a serving engine

> How to choose between serving engines by hardware, model architecture, workload and operations, tested on your own prompts.

- Web page: https://www.alesanalytics.com/docs/choosing-a-serving-engine
- Markdown: https://www.alesanalytics.com/docs/choosing-a-serving-engine.md
- Last updated: 2026-11-08
- Publisher: Ales Analytics

Technical page. vLLM, SGLang, TensorRT-LLM and others, by fit. Published 2026-11-08. Last reviewed 2026-11-08. Reviewed by Ales Analytics engineering.

## In one paragraph

A serving engine runs model weights on GPUs and answers many requests at once through an API. The choice depends on the hardware, the model architecture, the workload's shape (prompt lengths, prefix sharing, concurrency) and who will operate it. Published comparisons differ in all of those, so the engine is chosen by testing on the client's own workload rather than from a ranking.

## How it works

| Engine               | Typical fit                                                                    |
| -------------------- | ------------------------------------------------------------------------------ |
| vLLM                 | General-purpose GPU serving, broad model support, OpenAI-compatible API        |
| SGLang               | Heavy prefix sharing and structured generation (radix-tree cache)              |
| TensorRT-LLM         | NVIDIA hardware where compiled, hardware-specific engines justify a build step |
| LMDeploy             | Alternative GPU engine with strong quantisation support                        |
| llama.cpp, Ollama    | CPUs, laptops, edge devices and small deployments                              |
| NVIDIA Dynamo, llm-d | Cluster layers for routing, disaggregation and scaling across many GPUs        |

## Design choices and trade-offs

- **Support for the model.** New architectures reach engines at different speeds; check support for the exact model and quantisation format.
- **Operations.** An engine the team can upgrade and debug is worth more than a few per cent of throughput.
- **Features.** Prefix caching, speculative decoding, multi-LoRA, structured outputs and metrics differ in maturity between engines.

## When not to use it

When a single user runs occasional jobs on a workstation, a simple local runtime is enough.

## How we evaluate it

Run each candidate on the target hardware with the client's prompts at the expected concurrency, and report TTFT, inter-token latency and goodput at the agreed targets, with versions and settings stated.

## Sources

- [vLLM](https://docs.vllm.ai/en/latest/features/), [SGLang](https://docs.sglang.io/), [TensorRT-LLM](https://nvidia.github.io/TensorRT-LLM/latest/overview.html), [LMDeploy](https://github.com/InternLM/lmdeploy), [llama.cpp](https://github.com/ggml-org/llama.cpp), [NVIDIA Dynamo](https://docs.nvidia.com/dynamo/latest/), [llm-d](https://llm-d.ai/)
- [Inference performance](https://www.alesanalytics.com/engineering/inference)

## Contact

solutions@alesanalytics.com, or https://www.alesanalytics.com/contact.

- Every page as Markdown: https://www.alesanalytics.com/llms.txt