Vision Field Notes

Classification vs Detection vs Segmentation: Pick the Right Task

Before choosing a model, choose the task. Many projects start with "we need YOLO" when the real need is simpler, or harder, than detection. The task decides how much labelling you do, what the model can tell you, and how the system behaves in production. Picking wrongly wastes weeks.

This article compares four tasks and gives a way to choose.

The four tasks

Image classification

The model looks at the whole image and returns a label, for example "ripe" or "unripe", or "OK" or "defective". One image, one answer (or a score per class).

Object detection

The model finds each object and returns a bounding box, a class and a confidence. A single image can contain many objects.

Semantic segmentation

The model assigns a class to every pixel. The output is a mask in which each pixel is, say, road, sky, building or person. It does not separate two objects of the same class that touch each other.

Instance segmentation

The model finds each object and also gives its exact outline as a mask. Two people next to each other get two separate masks.

Pose estimation and oriented boxes are related tasks. The Ultralytics package supports several of these, which makes it easy to try neighbouring tasks.

Schematic of one scene, three round fruits on a belt with two of them touching, shown four ways. Classification: a single label, fruit 0.97, for the whole image. Detection: a box around each fruit, with the boxes of the two touching fruits overlapping. Semantic segmentation: all fruit pixels in one orange colour, so the two touching fruits form one region. Instance segmentation: each fruit filled with its own colour.
A schematic, not model output. Note the two touching fruits: detection and instance segmentation keep them apart, while semantic segmentation merges them into one region.

Compare them

Task Answers Label effort Typical compute Failure style
Classification What is this image? Low Lowest Wrong label for the whole image
Detection What and where? Medium Medium Missed or extra boxes
Semantic segmentation Which pixels are which class? High Higher Ragged or merged regions
Instance segmentation Which pixels belong to each object? Highest Highest Merged or split instances

Costs rise as you move down the table, both in labelling and in compute. So the guiding rule is: use the simplest task that answers your question.

A decision guide

Ask these questions in order.

  1. Is there one thing of interest per image, or is the decision about the whole image? Use classification. Examples: sorting parts that arrive one at a time on a fixture, or deciding whether a lens is dirty.
  2. Do you need to know where objects are, or how many? Use detection. Examples: counting people at a door, spotting a missing helmet, locating packages.
  3. Do you need area, outline or shape? Use segmentation. Examples: measuring the size of a spill, finding the extent of damage, estimating the covered fraction of a surface.
  4. Do touching objects of the same type need to be separated? Use instance segmentation. Semantic segmentation will merge them.
  5. Do you only need coarse location? A box is probably enough. Do not pay for masks you will never use.
Flow chart of four questions on the left, each with a yes arrow to a task on the right. Is the decision about the whole image: classification. If not, do you need where objects are or how many: detection. If not, do you need area, outline or shape: segmentation. If yes, must touching objects of the same type be told apart: instance segmentation. A note says semantic segmentation is enough when touching objects do not matter.
The first four questions as a flow chart. Stop at the first task that answers the business question; each step down costs more labelling and compute.

If you are unsure between two, start with the cheaper one and see whether it answers the business question. You can often upgrade later, and the data you labelled may still help. A dataset with masks can be converted into boxes, but boxes cannot be turned into masks.

Examples by situation

Situation Suitable task Why
Is the product label present on this bottle? One bottle per image Classification Whole-image decision
How many people are in the queue? Detection Need count and rough place
Is a worker inside a marked zone? Detection (plus zone logic) Needs position of a person
What fraction of the surface is corroded? Semantic segmentation Area measure
How large is each piece of fruit on a belt, with fruit touching? Instance segmentation Separate objects and size
Is this scene day or night? Classification Whole-image property

Many real systems combine tasks. A detector finds a region, then a small classifier checks the crop. A cascade like this can improve results because the classifier sees the object larger and cleaner than the detector did. It also adds one more model to maintain, so use it when it solves a real problem.

Pitfalls when choosing

Using detection when classification is enough

If your camera always frames one object in the same place, a classifier is simpler, faster to label and often more robust. A detector adds label work and extra failure modes: boxes that drift, duplicates, thresholds.

Using classification when objects vary in number

Classifying a frame as "contains a person" works until you need to know how many or where. When the requirement grows, you start over. Ask about future needs now, within reason.

Overpaying for segmentation

Masks cost many times more labelling time than boxes. If a box answers the question, choose the box. If a stakeholder says "we need exact shapes", ask what decision the shape supports. Often it is area or size, which might be approximated from a box when the camera is fixed and the object is regular.

Ignoring the business metric

Choose the task by the output you need, then choose metrics that measure it. Classification uses accuracy, precision and recall per class. Detection uses mAP, precision and recall. See mAP, Precision and Recall. Segmentation uses overlap measures between masks.

Consider data and hardware

Think about the cost of each task in practice.

Quick code comparison

With Ultralytics, the task is chosen by the weights you load:

from ultralytics import YOLO

cls_model = YOLO("yolov8n-cls.pt")   # classification
det_model = YOLO("yolov8n.pt")       # detection
seg_model = YOLO("yolov8n-seg.pt")   # instance segmentation

The calling pattern is similar, which lets you test a neighbouring task quickly on a few images before you commit to labelling a large dataset.

Checklist

Next steps: What Is YOLO?, Train YOLO on a Custom Dataset and Nano, Small, Medium: How to Choose a YOLO Model Size.