Vision Field Notes

Object Tracking Explained: SORT, ByteTrack and Friends

A detector looks at each frame alone. It can tell you there are three people in the picture, but not that the person at the left in frame 100 is the same person at the right in frame 130. A tracker supplies that continuity by giving each object an ID that persists over time. You need this for counting, dwell time, speed estimation, and any rule that depends on what an object did rather than where it is.

The tracking-by-detection idea

Most practical trackers follow one pattern, called tracking by detection:

  1. Run the detector on the new frame.
  2. Predict where each existing track should be in this frame.
  3. Match detections to predicted tracks.
  4. Update matched tracks, start new tracks for unmatched detections, and age out tracks that have not been matched for a while.

The detector does the hard visual work. The tracker is mostly bookkeeping and geometry, which is why it is cheap compared with the detector.

Cycle diagram: 1 Detect, 2 Predict with a Kalman filter, 3 Associate using IoU and one-to-one matching, 4 Update matched, new and lost tracks, then back to Detect for the next frame.
Caption: Tracking by detection repeats the same four steps every frame; only the first one looks at pixels.

Prediction: the Kalman filter

To predict where an object will be, SORT-style trackers use a Kalman filter. The state includes the box position, size, and their rates of change. Between frames, the filter assumes roughly constant velocity and projects the box forward. When a detection is matched, the filter corrects its estimate. In plain terms, it is a smoothed guess of motion that also tells the tracker how uncertain it is.

This prediction is why a tracker can keep following a person who is briefly missed by the detector: the track coasts on its predicted motion for a few frames and reconnects when the detection returns.

Association: matching boxes

Matching needs a cost between every track and every detection. SORT uses IoU (intersection over union) between the predicted box and the detected box. If you need a refresher on IoU, read IoU and Non-Maximum Suppression, Explained with Examples. A solver for the assignment problem (the Hungarian algorithm) then finds the best one-to-one matching, and pairs below a minimum IoU are rejected.

This works well when objects move modestly between frames and do not overlap much. It struggles when the camera frame rate is low (objects jump far between frames), when objects cross each other, and when detections flicker.

What ByteTrack changes

Older approaches threw away low-confidence detections before tracking. The ByteTrack idea is to keep them for a second round of matching. Low-score boxes often are real objects that are partly occluded or blurred. The method works in two steps:

  1. Match high-confidence detections to tracks first.
  2. Take the tracks that remain unmatched and try to match them to the low-confidence detections.

A low-confidence box that fits an existing track's predicted location is probably a real, partly hidden object, so it keeps the track alive. A low-confidence box that matches nothing is treated as noise and discarded. This recovers many tracks that would otherwise break during occlusion, without adding a separate appearance model.

Two rows of three crops from a store aisle at frames 254, 261 and 270. A shopper is partly hidden by another person in frame 261 and her confidence drops to 0.34. In the top row her track is lost and she returns as ID 4; in the bottom row she keeps ID 2 throughout.
Caption: When the shopper is half hidden, her score falls below 0.5. Matching only high-confidence boxes loses her and gives her a new ID; a second round with low-confidence boxes keeps ID 2. IDs come from a minimal tracker written for this figure that follows the ByteTrack idea, not from the Ultralytics implementation. Footage: Intel IoT DevKit sample videos (CC BY 4.0). Detections: YOLOv8n.

SORT and ByteTrack use motion and box overlap only. Other trackers, such as those in the DeepSORT family, also compare appearance using a small re-identification network. Appearance helps when objects cross paths or leave and re-enter, but it costs more compute and needs a re-ID model that suits your objects. Which is better depends on the scene, so test on your own footage.

Using a tracker with Ultralytics

Ultralytics ships tracker configurations, and tracking is a single call:

from ultralytics import YOLO

model = YOLO("yolo11n.pt")  # any detection weights you have

results = model.track(
    source="video.mp4",
    tracker="bytetrack.yaml",
    persist=True,
    stream=True,
)

for r in results:
    if r.boxes.id is None:
        continue
    ids = r.boxes.id.int().tolist()
    boxes = r.boxes.xyxy.tolist()
    # use ids and boxes

Notes:

The parameters you will typically see cover the confidence level that separates "high" from "low" detections, the threshold for starting a new track, how many frames a lost track is kept before removal, and the matching threshold. Their exact names and defaults depend on the version, so read the file rather than relying on memory.

Remember the licence point: Ultralytics YOLO models are AGPL-3.0, with a commercial licence available from Ultralytics. Check the current licence before using them in a product. See YOLO Licenses Explained: AGPL, GPL and Commercial Use.

Why IDs switch

An ID switch happens when one real object receives a new ID, or two objects swap IDs. Common causes:

Tuning approach

Tune on a short clip that contains your hard cases, and review it visually with IDs drawn on the frames. Do not tune only on a summary metric.

Problem Direction to try
IDs break during short occlusions Keep lost tracks longer
Many short-lived ghost tracks Raise the threshold to start a track
Two nearby objects merge or swap Stricter matching threshold, better detector
Tracks stay on after the object left Shorten the lost-track buffer
Detector misses objects for many frames Fix the detector first (more data, better threshold)

Keeping lost tracks longer helps with occlusion but raises the risk of linking a new object to an old track. Every setting trades one error for another.

The best improvement usually comes from a better detector and a sensible camera angle, not from tracker tuning. A camera that looks down at a steep angle has fewer occlusions than one at head height in a crowd.

What to do about it downstream

Design your logic to tolerate imperfect IDs. For counting, count crossings of a line by a track and de-duplicate within a short window rather than trusting that an ID never changes. For dwell time, allow short gaps. The tutorial People Counting with Line Crossing: A Practical Tutorial shows how to do this.

To evaluate a tracker, compare against hand-labelled sequences and look at identity-focused metrics such as IDF1 and the count of ID switches, along with the standard MOTA. Do not report a tracker as good because the demo video looks fine.

Checklist

Next steps: apply tracks to counting in People Counting with Line Crossing: A Practical Tutorial and to areas in Zone Intrusion and Dwell Time Alerts.