Vision Field Notes

Measuring FPS and Latency Correctly

"How fast is it?" sounds like a simple question. In computer vision it rarely is. A single FPS number can mean the model's forward pass, the whole pipeline from camera to alert, or an average over a lucky run. Numbers shared online often leave out which one they mean, and that makes them hard to compare.

This article shows how to measure speed in a way you can trust, using your own model on your own hardware.

Define the terms

These are different. A system can have high throughput and poor latency, for instance when it batches frames or queues them. A model can run in a few milliseconds, while the full pipeline is slower because decoding and Python code dominate.

Decide which number answers your question. If you want to know whether a chosen board can handle your cameras, you need end-to-end throughput. If you want to compare two model sizes, model time is useful, but only if the rest of the setup is identical.

Why naive measurements mislead

No warm-up

The first inferences are slow. Libraries load kernels, allocate memory, build caches and, on GPUs, initialise contexts. Some runtimes compile or tune on first use. If you time the first few runs, you measure start-up, not steady state.

Fix: run a number of dummy inferences first (ten to fifty is typical), then start timing.

Line chart of latency per iteration on a log scale. The first run is about 420 milliseconds and the next runs fall quickly, levelling off near 21 milliseconds after about ten iterations. The first 20 iterations are shaded as warm-up to discard.
Simulated, illustrative data. Timing the first runs would report start-up cost, not the steady-state speed you will actually get.

Asynchronous GPU execution

GPU calls are often asynchronous. Your Python code returns before the GPU finishes. If you time with a simple timer around the call, you may measure only the time to queue the work.

In PyTorch, call torch.cuda.synchronize() before reading the end time. Other runtimes have their own ways to wait for completion; fetching the output to the CPU usually forces completion. If you do not synchronise, your results can look impossibly fast.

Wrong timer

Use time.perf_counter() in Python. It is a monotonic, high-resolution clock meant for timing intervals. time.time() can jump and has lower resolution on some systems.

Averaging hides problems

A mean can look fine while the system stutters. Report the median and a high percentile, such as the 95th or 99th, and the worst case. For alerting systems, the slow tail often matters more than the average.

Histogram of per-frame latency on a log count scale. Most frames sit near 25 milliseconds, a smaller group sits between 40 and 60, and a few reach up to 109. Dashed lines mark median 25, p95 31, p99 56 and max 109 milliseconds; the mean is 26.
Simulated, illustrative data. Mean and median are almost the same, but the p99 is more than twice the median, and those slow frames are what a live system feels.

Cold hardware

On small boards, a short test runs while the device is cool. After some minutes, heat can cause throttling and slower speed. Run long enough for temperatures to stabilise. See power, cooling and storage for an edge AI box.

Dynamic clocks and power modes

Many devices change clock speeds based on load and power settings. Record the power mode and note it with your results. A benchmark in one mode is not comparable to another.

A simple benchmark method

Here is a basic template for timing a model function:

import time
import numpy as np

def benchmark(run, warmup=20, iters=200):
    for _ in range(warmup):
        run()
    times = []
    for _ in range(iters):
        t0 = time.perf_counter()
        run()
        times.append((time.perf_counter() - t0) * 1000.0)
    t = np.array(times)
    return {
        "median_ms": float(np.median(t)),
        "p95_ms": float(np.percentile(t, 95)),
        "max_ms": float(t.max()),
    }

Here run is a function that performs the step you care about. For a GPU model, make sure run includes synchronisation or retrieval of results. Pass a real image, or a real frame, not a tensor of zeros, because input content rarely changes speed for a fixed network but can for post-processing (the number of boxes affects NMS time).

Repeat the test several times, on separate runs of the program, to see how much results vary.

Split the pipeline into stages

To find the bottleneck, time each stage separately:

  1. Capture or decode: reading and decoding the frame.
  2. Pre-processing: resize, colour conversion, normalisation.
  3. Inference: the forward pass.
  4. Post-processing: decode outputs, NMS, tracking, rules.

Ultralytics results include a speed dictionary with pre-process, inference and post-process times in milliseconds for each prediction. That is a convenient start for model-level timing, but it does not include video decoding or your own application code, so time those yourself.

With stage timings, you often find surprises. A common one: the model takes a few milliseconds, while pre-processing in NumPy or decoding a high-resolution stream takes longer. Optimising the network would then bring little gain. See reading RTSP streams with OpenCV and FFmpeg for stream-related delays.

Batch size effects

Batching groups several frames into one forward pass. On GPUs it usually raises throughput because the hardware is used more fully. But it also raises latency: the first frame waits until the batch fills, and the whole batch finishes together.

Timeline of twelve frame arrivals. With batch 1, each frame is processed right after it arrives. With batch 4, frames wait in a yellow band until four have arrived, then one orange inference block runs, so the first frame of each batch gets its result about four frame intervals later.
Schematic, not a measurement. A batch can process more frames per second, but each frame waits for the batch to fill, so latency rises.

So report batch size with every number. A result of "X FPS at batch 8" and one at batch 1 answer different questions. For a single live camera where fast alerts matter, batch 1 reflects reality. For processing many streams on one GPU, a larger batch may be right. Scaling decisions are discussed in scaling video analytics to hundreds of camera streams.

Measure under realistic conditions

Camera-to-alert latency

Sometimes you care about real wall-clock delay: from the event in the scene to the alert reaching a person. This includes camera encoding, network, buffering, decoding, analysis, rules and notification delivery. You cannot see all of it from inside your code.

One approach: show a clock or a visible change in front of the camera and compare when the alert fires. This is crude but honest. Also check buffers in your capture library, since a growing buffer can add seconds of delay without any change in measured FPS.

Report results properly

When you share or record a benchmark, include:

Without these, a number cannot be reproduced or compared. This also protects you from fooling yourself when you try an optimisation. Change one thing at a time and re-measure.

Common mistakes

Checklist

Next steps

Apply these methods in Raspberry Pi vs Jetson Orin Nano for Object Detection and Scaling Video Analytics to Hundreds of Camera Streams.