TensorRT-LLM

What is TensorRT-LLM?

TensorRT-LLM is NVIDIA’s open-source library for maximizing large-language-model inference performance on NVIDIA GPUs. It takes a trained LLM and compiles it into a highly-optimized inference “engine” tailored to specific NVIDIA hardware, applying aggressive low-level optimizations — kernel fusion, quantization (including FP8 and INT8/INT4), optimized attention kernels, paged KV cache management, and in-flight (continuous) batching — to extract the highest possible throughput and lowest latency the hardware can deliver. It sits at the performance-focused end of the LLM-serving spectrum: where vLLM prioritizes ease and strong out-of-the-box throughput, TensorRT-LLM prioritizes squeezing the absolute maximum out of NVIDIA silicon.

The compilation approach — power and cost

The defining characteristic is that TensorRT-LLM is a compiler, not just a runtime, and this cuts both ways. On the upside, ahead-of-time compilation lets it do optimizations a general runtime can’t: it fuses operations, selects the best kernels for the exact GPU, bakes in quantization, and produces an engine finely tuned to a specific model, precision, and hardware configuration — yielding class-leading performance on NVIDIA GPUs, which is why it’s the backbone of many high-scale, latency-sensitive production LLM deployments. On the downside, that compilation is a heavier, less flexible build step: you must build an engine for your specific model + GPU + precision + batch/sequence configuration, and that engine is tied to those choices — change the GPU type, the quantization, or key parameters, and you generally rebuild. This is more operational friction than a library where you just point at model weights and serve. The optimizations (especially low-precision quantization) also require validation: FP8/INT8/INT4 quantization can dramatically speed inference and shrink memory, but can subtly affect output quality, so you must evaluate that the quantized engine still meets your accuracy bar rather than assuming the speedup is free.

Where it fits, and the honest tradeoffs

In practice TensorRT-LLM is frequently deployed underneath Triton Inference Server: TensorRT-LLM provides the maximally-optimized inference backend, and Triton wraps it with production serving infrastructure (batching orchestration, model management, HTTP/gRPC endpoints, scaling). Choosing it comes down to a clear tradeoff. Reach for TensorRT-LLM when you’re running LLMs at scale on NVIDIA GPUs and performance/cost-per-token is critical — the compilation effort pays for itself across billions of tokens. Prefer a simpler engine like vLLM when you value flexibility and speed of iteration (frequently swapping models, experimenting, avoiding rebuild friction) and its already-strong throughput is good enough — which for many teams it is. The two big caveats: NVIDIA lock-in — TensorRT-LLM is NVIDIA-specific by design, so building your serving stack around it ties you to NVIDIA hardware (a real strategic consideration as AMD and other accelerators mature); and complexity — the engine-build workflow and tuning surface are more involved than drop-in servers, so the performance win comes with genuine engineering investment. The recurring mistakes: adopting TensorRT-LLM’s complexity for a small-scale workload where vLLM would be far simpler and plenty fast; forgetting to re-validate model quality after aggressive quantization; and not accounting for the rebuild burden when models or hardware change.

What people get wrong

• Using it at small scale — the engine-build complexity only pays off at high volume where maximum throughput and cost-per-token matter; vLLM is simpler and often enough.
• Assuming quantization is free: FP8/INT8/INT4 speedups can subtly degrade output quality — you must evaluate the quantized engine against your accuracy requirements.
• Ignoring rebuild friction and lock-in — engines are tied to specific model/GPU/precision configs (change one, rebuild), and the whole approach commits you to NVIDIA hardware.

Primary source: NVIDIA TensorRT-LLM documentation