Vision Field Notes

Annotation Mistakes That Quietly Ruin Your Model

When a detector underperforms, the instinct is to change the model, the learning rate or the architecture. In many projects the real problem is in the labels. A model learns exactly what the annotations tell it, including the mistakes. If the labels are inconsistent, the model learns inconsistency.

This article lists the annotation errors that do the most damage, shows how to find them, and gives a repeatable audit you can run before blaming the model.

Why label errors are hard to notice

Label noise does not crash training. Loss still goes down, metrics still come out, and the model still produces boxes. The symptoms are indirect: a plateau in validation results, boxes that wobble, false detections that look like they should be labelled, and confusion between two classes.

Another problem is that validation labels have the same errors as training labels. If both are noisy in the same way, your metrics look fine while real-world behaviour is poor. Auditing the data is the only way to see it.

The common mistakes

Missing labels

An object is clearly in the image but has no box. For the detector, an unlabelled object is a negative example, so it is taught to ignore something it should find. This is the most damaging mistake because it directly contradicts the true task.

It typically happens when labellers skip small, distant, partially hidden or crowded objects, or when they get tired late in a batch.

Loose or inconsistent boxes

Some boxes hug the object, others leave a wide margin, others cut off a limb. The model cannot learn a single rule for where the edge is, so predictions vary and overlap-based metrics suffer, especially at strict thresholds. Review IoU and Non-Maximum Suppression, Explained with Examples to see why a few pixels matter more than they look, particularly for small objects.

Inconsistent classes

One labeller marks a van as "car", another as "truck". The same helmet appears as "hard hat" in one batch and "helmet" in another, or two classes overlap in definition. The model sees contradictory targets for similar pixels and settles on a muddy average.

Inconsistent treatment of edge cases

Occluded objects, objects cut off by the frame border, reflections, objects in posters or on screens, crowds where individuals cannot be separated. If these cases are sometimes labelled and sometimes ignored, the model has no stable rule. Decide once and apply it everywhere.

Wrong class on a correct box

The box is right, the class is wrong. Often caused by similar-looking classes or by a labeller clicking the wrong hotkey. These errors show up as high-confidence mistakes in the confusion matrix.

Duplicate boxes

Two boxes on one object, usually from a slip while drawing or from merging datasets. The model is taught that one object can be two, which hurts duplicate suppression behaviour.

Four copies of the same store-aisle crop with three shoppers. Top left: each shopper has one tight green box. Top right: the left shopper has only a dashed red outline marked no label. Bottom left: the left shopper has a red box much larger than the person, marked loose. Bottom right: the left shopper has a green box and a second, slightly shifted red box, marked 2 boxes.
Three common mistakes made on the same person: a skipped box, a loose box and a duplicate. The starting boxes are YOLOv8n detections used as labels; the mistakes are simulated to show what to look for. Footage: Intel IoT DevKit sample videos (CC BY 4.0). Detections: YOLOv8n.

Format and conversion bugs

Not an annotation error as such, but common: swapped width and height, boxes in pixels where normalised values are expected, class ids shifted by one after merging datasets, or image resizing that was not applied to the labels. Everything looks plausible and everything is wrong. Always draw the converted labels on a handful of images before training.

Labels that do not match the task

Labelling the "person" when the business need is "person without a helmet" is a definition problem. A detector can only learn what is visually defined. If the target depends on context that is not visible in the image, the label will always look inconsistent.

How to audit a dataset

Run this audit before any serious training, and again whenever you add a large batch.

  1. Draw and look. Render boxes and class names on a random sample of images. Look at 100 to 200 of them yourself. This single step finds format bugs, missing labels and loose boxes faster than any metric.
  2. Count per class. Check instance counts and look for classes that are rare or unexpectedly absent. Rare classes need attention. See Class Imbalance and Small Objects: Practical Fixes.
  3. Check box statistics. Look at the distribution of box widths and heights. Boxes of near-zero size, boxes touching the whole image, or boxes extending outside the image are red flags.
  4. Look for duplicates. Flag pairs of boxes of the same class with very high overlap in one image.
  5. Use the model to find errors. Train a first model, run it on the training set, and list places where it confidently disagrees with the label. High-confidence predictions with no matching label point to missing labels. High-confidence predictions with a different class point to wrong labels. Review each by hand and do not auto-correct, because the model may be the one that is wrong.
  6. Review the confusion matrix. Ultralytics saves a confusion matrix after training. Large off-diagonal cells between two classes usually mean the classes are ill-defined or mislabelled, not that the model is weak.
  7. Measure agreement. Have two people label the same 50 images independently. Compare. Disagreement shows you which rules are unclear.
Scatter plot of box width against box height, both as fractions of the image. Most boxes form a dense diagonal cluster of narrow, taller-than-wide boxes. Red circles mark outliers: a few near-zero boxes at the origin, a few wide but very short boxes below the cluster, and a few boxes covering the whole image in the top right corner.
Illustrative data, not a real dataset. When you plot every box's width against its height, slips, swapped coordinates and whole-image boxes stand apart from the main cluster; open those images first.

How to fix them

Do you need perfect labels?

No. Some noise is normal and models tolerate a small amount of it. The goal is consistency, not perfection. Random, rare errors do little harm. Systematic errors, such as always skipping small objects, do real harm because the model learns the pattern.

A useful rule: fix errors that are systematic, concentrated in a rare class, or present in the validation set. Validation errors matter most because they distort every decision you make from the metrics.

Validate the validation set

Check your validation labels with extra care. If they miss objects, your recall will look worse than it is and you might chase a problem that does not exist. If they contain the same mistakes as the training set, your metrics will look better than the real world warrants. Keep the validation set small enough to review completely by hand.

Also make sure the split avoids leakage between near-identical frames. See Train/Validation Split for Video Frames: Avoid Data Leakage.

Checklist

Next steps: compare tools that support review workflows in CVAT vs Label Studio vs Roboflow: Choosing an Annotation Tool, and learn to read the results afterwards in mAP, Precision and Recall: How to Read Detection Results.