Vision Field Notes

IoU and Non-Maximum Suppression, Explained with Examples

Two small ideas sit behind almost every object detector: Intersection over Union (IoU) and non-maximum suppression (NMS). IoU measures how much two boxes overlap. NMS uses that measure to remove duplicate detections. Once you understand both, many odd behaviours become easy to explain: why two people standing together become one box, why a single object gets two boxes, and why a metric changes when you change a threshold.

Intersection over Union

IoU compares two boxes. Take the area where they overlap (the intersection) and divide it by the total area they cover together (the union).

IoU = area of overlap / area of union

The result is between 0 and 1.

A worked example

Take two boxes in pixel coordinates, written as (x1, y1, x2, y2) for the top-left and bottom-right corners.

The overlap is the square from (5, 5) to (10, 10). Its area is 5 x 5 = 25. The union is 100 + 100 - 25 = 175. So IoU = 25 / 175, about 0.14.

Subtracting the overlap once matters. If you add the two areas without subtracting it, you count the overlap twice.

Three pairs of equal-sized square boxes, blue A and orange B, with the overlap hatched. Left, the worked example offset diagonally by half: IoU 25 over 175 equals 0.14. Middle, shifted sideways by a quarter: IoU 75 over 125 equals 0.60. Right, shifted sideways by a tenth: IoU 90 over 110 equals 0.82.
The union grows as the overlap shrinks, so IoU falls faster than the overlap alone suggests. A sideways shift of a quarter of the box width already drops IoU to 0.60.

IoU in code

def iou(a, b):
    # boxes are (x1, y1, x2, y2)
    ix1, iy1 = max(a[0], b[0]), max(a[1], b[1])
    ix2, iy2 = min(a[2], b[2]), min(a[3], b[3])
    inter = max(0, ix2 - ix1) * max(0, iy2 - iy1)
    area_a = (a[2] - a[0]) * (a[3] - a[1])
    area_b = (b[2] - b[0]) * (b[3] - b[1])
    union = area_a + area_b - inter
    return inter / union if union > 0 else 0.0

print(iou((0, 0, 10, 10), (5, 5, 15, 15)))  # about 0.143

The max(0, ...) guards matter. Without them, boxes that do not overlap produce a negative width and a wrong answer.

Where IoU is used

IoU appears in at least three places, and it is easy to mix them up:

  1. Evaluation. A prediction counts as correct only if its IoU with a true box passes a threshold, such as 0.5. See mAP, Precision and Recall.
  2. Training. Detectors often use an IoU-based term in the loss, and some use it to decide which predictions are matched to which labels.
  3. NMS. IoU between predictions decides which are duplicates. This is the one we look at next.

The thresholds in these three places are separate settings. Changing one does not change the others.

Why NMS is needed

A detector produces many candidate boxes. Several nearby locations in the feature map can all notice the same object, so one dog can end up with five slightly different boxes, each with its own confidence. Showing all five would be wrong. You want one box per object.

NMS keeps the best box and removes the others that overlap it heavily.

Two copies of a store-aisle frame. Left, before NMS: 32 orange boxes above score 0.25, stacked on top of each other around three shoppers and two shelf bowls. Right, after NMS at IoU 0.45: five labelled boxes, three people scoring 0.79 to 0.82 and two bowls.
Before NMS, each shopper is covered by several near-identical boxes. NMS keeps the highest-scoring box for each object, leaving one box per shopper. Footage: Intel IoT DevKit sample videos (CC BY 4.0). Detections: YOLOv8n.

NMS step by step

For one class at a time:

  1. Drop all boxes below the confidence threshold.
  2. Sort the remaining boxes by confidence, highest first.
  3. Take the top box. Keep it.
  4. Compute its IoU with every other remaining box. Remove any box whose IoU is above the NMS IoU threshold.
  5. Repeat from step 3 with the boxes that are left, until none remain.

A small example

Suppose a model produces three boxes of class "person":

Box Confidence IoU with box 1
1 0.92 -
2 0.80 0.75
3 0.60 0.05

With an NMS threshold of 0.5, box 1 is kept first. Box 2 overlaps it by 0.75, which is above 0.5, so it is removed as a duplicate. Box 3 overlaps by only 0.05, so it survives and is treated as a separate person. Result: two detections.

If you had set the threshold to 0.8, box 2 would survive because 0.75 is below 0.8, and you would see a duplicate. This shows the direction of the setting.

Choosing the NMS IoU threshold

The NMS IoU threshold answers: how much overlap do I accept before calling two boxes the same object?

The same hallway crop with two people walking, processed twice. Left, NMS IoU threshold 0.45: two person boxes, scores 0.84 and 0.67. Right, NMS IoU threshold 0.9: four person boxes, because the second person also keeps two extra overlapping boxes scoring 0.35 and 0.26.
Same frame, same detections, only the NMS threshold changed. At 0.9 the overlapping lower-score boxes on the second person survive as duplicates. Footage: Intel IoT DevKit sample videos (CC BY 4.0). Detections: YOLOv8n.

Values in the range of about 0.45 to 0.7 are common starting points, and Ultralytics exposes the setting as iou in prediction and validation. Check the current documentation for the default in the version you use, because it can differ between prediction and validation and between releases.

The right value depends on your scene:

Scene Tendency
Sparse objects, such as a lone vehicle at a gate A lower or middle value works well
Dense crowds or stacked items A higher value, tested carefully
Large objects with a few close neighbours Middle value

Do not tune this on a hunch. Collect clips that show your worst crowding cases, vary the threshold, and count duplicates and merges by eye.

Running NMS in practice

With the Ultralytics package, NMS is already part of prediction:

from ultralytics import YOLO

model = YOLO("yolov8n.pt")
results = model.predict("crowd.jpg", conf=0.25, iou=0.5)
print(len(results[0].boxes))

If you build your own pipeline, for example after exporting to ONNX, you may need to apply NMS yourself. OpenCV has a function for it:

import cv2
import numpy as np

boxes = [[0, 0, 10, 10], [1, 1, 11, 11], [50, 50, 20, 20]]  # x, y, w, h
scores = [0.9, 0.8, 0.7]
keep = cv2.dnn.NMSBoxes(boxes, scores, score_threshold=0.25, nms_threshold=0.5)
print(np.array(keep).flatten())

Notice that cv2.dnn.NMSBoxes expects boxes as x, y, width, height, not corner coordinates. Passing the wrong format is a very common bug, and it produces boxes that are removed or kept for no obvious reason. PyTorch users can use torchvision.ops.nms, which takes boxes as (x1, y1, x2, y2), scores, and an IoU threshold.

Class-aware versus class-agnostic

Standard NMS runs per class, so a "car" box and a "truck" box on the same vehicle can both survive. Sometimes that is what you want. If your classes overlap in meaning, you may prefer class-agnostic NMS, which treats all boxes alike and keeps only the highest-scoring one. Ultralytics has an option for this, called agnostic_nms, in prediction. The choice should follow your definition of the classes.

Common mistakes

Key points

Next steps: Confidence Thresholds: Tuning a Detector to Cut False Alarms, mAP, Precision and Recall and Anchor-Free vs Anchor-Based Object Detectors.