Vision Field Notes

TensorRT for YOLO: When and How to Use It

TensorRT is NVIDIA's inference optimiser and runtime. You give it a trained network, usually through ONNX, and it builds an optimised "engine" for your specific GPU. For NVIDIA hardware, including Jetson boards, it is one of the most effective ways to speed up a detector. It also adds some friction. This article explains what it does, when it is worth using and how to adopt it without surprises.

What TensorRT actually does

TensorRT takes a network graph and applies several optimisations:

None of this changes your training. It is a deployment step. The result is a binary engine file that is fast but tied to its environment.

Diagram: before fusion, a convolution kernel, a bias add kernel and an activation kernel run one after another with a write and read of memory between each. After TensorRT fusion, a single kernel does convolution, bias and activation.
Fusing small consecutive operations into one kernel removes the memory round-trips between them, which is one of the main sources of TensorRT speed-ups.

When it is worth it

TensorRT is a good idea when:

It is less useful when:

Always profile first. If model inference is a small part of the total time, optimising it gives little gain. The article on measuring FPS and latency correctly shows how to split end-to-end time into stages.

The safe workflow

A reliable path is:

  1. Train and validate the PyTorch model.
  2. Export to ONNX and confirm the ONNX output matches (see exporting YOLO to ONNX).
  3. Build a TensorRT engine on the target device or a matching environment.
  4. Validate the engine's accuracy on your validation set.
  5. Benchmark with realistic input, after warm-up.
  6. Deploy with the exact same library versions you built with.

Skipping step 4 is the most common mistake. Faster but less accurate is not a win.

Building an engine with Ultralytics

Ultralytics wraps the process:

from ultralytics import YOLO

model = YOLO("best.pt")
model.export(format="engine", half=True, imgsz=640)

This produces a .engine file you can load again with YOLO("best.engine"). The export needs TensorRT installed in your environment, and it needs an NVIDIA GPU. On Jetson, TensorRT comes with JetPack, so you usually do not install it separately.

Other export arguments exist, such as int8=True with calibration data, dynamic, batch and workspace. Names and defaults can change between versions, so check the current Ultralytics export documentation instead of copying flags from old posts.

You can also use NVIDIA's trtexec command-line tool, which ships with TensorRT, to build an engine directly from an ONNX file. It is useful when you want fine control or you work outside Ultralytics.

Precision modes

Mode Speed Accuracy risk Notes
FP32 Baseline None Largest and slowest
FP16 Usually faster Low for most detectors A sensible default on modern NVIDIA GPUs
INT8 Often fastest Higher, must be checked Needs calibration data

FP16 is the easy win. Most detectors lose very little accuracy, and the speed and memory benefit is usually clear. Still, verify on your own data.

INT8 needs more care. TensorRT must learn the value ranges of activations, which it does using a calibration dataset: a few hundred representative images is a common starting point, but check the guidance for your tool. If the calibration images do not look like production images, accuracy can drop in ways that a headline metric hides. The details are in INT8 quantization for detectors.

Portability limits

This is the part that surprises people. A TensorRT engine is not a portable model file. It is tied to:

So keep the ONNX file as your portable source of truth, and treat the engine as a build artefact. Rebuild it on each target device type, ideally as part of your deployment script or container build. Store the ONNX, record the TensorRT version and note the build options. If you deploy with containers, see Dockerize a computer vision app for how to keep the environment reproducible.

Diagram: PyTorch best.pt exports to best.onnx, the portable source of truth. From it, three arrows lead to separate builds on a Jetson device, a desktop RTX GPU and a cloud GPU, each producing its own engine file that is marked not portable and must be validated and benchmarked.
One ONNX file, one engine per target. Each engine is tied to its GPU type and TensorRT version, so rebuild and re-validate it on every target.

Input shape and batch size

Engines are optimised for specific input shapes. A fixed shape is the simplest and often the fastest. If you need several shapes or batch sizes, you can use dynamic shapes with optimisation profiles, but this adds complexity. For many camera systems, a fixed input size and a small fixed batch is enough.

Batching can raise throughput when you process many streams, but it also raises latency, because frames wait for a batch to fill. Decide which matters for your use case: total streams per device, or time from frame to alert.

Pre- and post-processing still matter

The engine only covers the network. Resizing, colour conversion, NMS and tracking happen elsewhere. If those run slowly on the CPU in Python, the end-to-end speed will not match the engine's speed. Options include optimising NumPy code, using GPU-based pre-processing, or including NMS in the engine, which some export paths support. Make changes only after profiling, because each one adds complexity.

Common problems

Licence note

Ultralytics YOLO models are AGPL-3.0, with a commercial licence available from Ultralytics. Converting the model to TensorRT does not change the licence of the underlying weights and code. Check the current licence terms for your product.

Key points

Next steps

Read INT8 Quantization for Detectors for the next level of optimisation, and Raspberry Pi vs Jetson Orin Nano if you are still choosing hardware.