Vision Field Notes

Single-Class or Multi-Class: When to Merge or Split Detection Classes

Early in every detection project someone writes the class list, and it is rarely revisited. Yet the class list shapes everything after it: how long labelling takes, how much data each class needs, which errors the model makes and which numbers end up in the report. Too many classes and the model confuses near-identical ones; too few and you cannot make the decision the client needs.

This article gives a practical way to decide whether classes should be merged or split, and a design that keeps both options open.

Start from the decision

Classes exist to support decisions. For each proposed class, ask: does the system do something different for this class? If cars, vans and trucks all lead to the same action, such as "vehicle in the fire lane", they are one class for that system. If trucks trigger a different rule, such as a height restriction, trucks need to be distinguished.

Classes that change no decision cost labelling time, split the training data into smaller pieces, and create confusion errors that do not matter but still appear in the metrics.

Reasons to merge

Reasons to split

Detect broad, classify fine

A common and effective design separates the two jobs:

Diagram: a detector finds broad classes, vehicle and person. Vehicle crops go to a second stage that classifies car, van or truck; person crops go to a second stage that decides helmet yes or no. Notes: merge when classes look alike, are rare, or lead to the same action; split when the action differs and the difference is visible; a second stage can be trained, tested and changed without touching the detector.
A diagram, not data. The detector handles localisation with plenty of data per class; the finer decision is made on crops by a separate, smaller model or an attribute head.
  1. A detector finds broad classes, such as "person" and "vehicle". With fewer, larger classes, each has plenty of data and localisation is reliable.
  2. A second stage looks at each detected crop and makes the finer decision: vehicle type, helmet or not, vest colour.

The advantages are practical. The crop classifier sees the object at a higher effective resolution, because a crop can be resized up to the classifier's input. It can be retrained when the client adds a vehicle type without touching the detector. Its errors are easier to analyse, since localisation and classification errors are separated. The cost is an extra model in the pipeline and some extra compute per detection.

For attributes of people, such as PPE, this pattern is often more robust than training a detector with classes like "person with helmet" and "person without helmet". Those combined classes split the person data in half and make the detector learn two hard things at once.

Label fine, train coarse

The most useful rule of all: label at the finest level you might ever need, and merge at training time. Merging labels later is a one-line mapping. Splitting labels later means relabelling every image of that class.

If there is a reasonable chance the client will later want vans separated from cars, label "car" and "van" now, and train a "vehicle" detector by mapping both to one class. The extra labelling effort is small when the labeller is already drawing the box; the saving later can be weeks.

Do this within reason. Labelling twenty vehicle subtypes because someone might want them one day slows labelling and invites disagreement. Pick the distinctions with a real chance of being needed.

Watch what the metrics do

Changing classes changes the numbers, which can mislead:

A quick procedure

  1. List the decisions the system must make.
  2. For each candidate class, note which decision it supports and how many examples you expect.
  3. Check a sample of images: can people reliably tell the classes apart at your resolution?
  4. Merge classes that fail any of these tests; plan a second stage for distinctions that only some decisions need.
  5. Label one level finer than you train, where a split is plausible later.
  6. Write the class definitions and boundaries into the annotation guide. See Writing an Annotation Guide Labellers Can Follow.

Checklist

Next steps: Classification vs Detection vs Segmentation and Class Imbalance and Small Objects.