Vision Field Notes

Metrics and Logging for a Vision Service

A vision service can fail in many quiet ways: a camera delivers frames at half rate, inference latency creeps up, a model starts producing fewer detections after a site change, alerts stop being delivered. None of these crash the process. Without metrics you learn about them from the client.

This article lists the metrics worth collecting from a video analytics service, how to organise them by stage and by camera, and how to write logs that help rather than drown you.

Metrics and logs do different jobs

You need both. Metrics show the camera whose detection count fell to zero on Tuesday; logs show that its stream reconnected 40 times that morning.

The pipeline, stage by stage

Collect metrics at each stage, labelled by camera, so a problem can be placed precisely.

Stage Metrics What a change suggests
Stream Connected (yes/no), reconnects, frames received per second, seconds since last frame Network, camera or credential problems; frame-rate changes at night
Decode Decode errors, decode time Corrupt streams, CPU overload, codec issues
Queue Queue length, frames dropped Inference cannot keep up
Inference Latency per frame (histogram), frames processed per second, batch size Throttling, contention, model change
Detections Detections per minute per class, mean confidence Scene change, camera moved, model drift, blind camera
Rules and alerts Alerts raised per rule, alerts suppressed by cooldown Rule misconfiguration, false-alarm source
Delivery Alerts delivered, delivery failures, delivery delay Notification service or network problems
Health checks Sharpness ratio, brightness, view shift, day/night mode Dirty, blocked or moved camera
Device CPU, GPU, memory, temperature, throttle flags, disk free Overload, leaks, heat, full disks

Two derived numbers are particularly useful. Frames processed versus frames received per camera shows whether you are falling behind. End-to-end delay, from frame arrival to alert delivery, is the latency the client experiences.

Watch the absence of signal

The most important failures produce silence. A blind camera produces no detections. A broken notification path produces no alerts. A stuck process produces no metrics at all.

So alert on absence:

The detection-count check needs care, because a quiet scene is normal at night or at weekends. Compare with the same hour on previous days, not with a fixed number.

A line chart of detections per hour from one camera over four days. The count peaks around 60 each afternoon and falls to near zero every night. A dotted red line marks a fixed threshold of five, which the real counts drop below on every one of the quiet nights, 30 hours in total. A dashed green line shows the average for the same hour over the previous three days, and a shaded band marks a quarter of that value. Partway through day four the camera goes blind and the count drops to zero while the dashed baseline continues to rise toward its usual afternoon peak, so the count falls through the shaded band one hour later.
Simulated counts for a shop doorway, not a real camera. The fixed threshold and the baseline are given the same data; only the comparison differs.

The two lines get identical data. A fixed threshold of five fires on 30 ordinary quiet hours across those four days, which is enough to train everyone to ignore it, and when the camera actually fails at nine in the morning it says nothing, because nine in the morning is above five right up until it isn't.

The baseline catches the same failure within an hour, and stays silent through every night, because it is asking a different question. Not "is this number low?" but "is this number low for this hour on this camera?" That is the whole trick, and it costs you a rolling average rather than a model. Handling RTSP Dropouts and Detecting a Dirty, Blocked or Moved Camera cover the stream and image checks behind these numbers.

Using Prometheus-style metrics

A common setup is to expose metrics from each service with a Prometheus client library, collect them with Prometheus, and graph them in Grafana. Choose the right metric type:

from prometheus_client import Counter, Gauge, Histogram, start_http_server

FRAMES = Counter("frames_received_total", "Frames received", ["camera"])
DROPPED = Counter("frames_dropped_total", "Frames dropped before inference", ["camera"])
LAST_FRAME = Gauge("last_frame_timestamp_seconds", "Time of last frame", ["camera"])
INFER = Histogram("inference_seconds", "Inference latency", ["model"],
                  buckets=[0.005, 0.01, 0.02, 0.04, 0.08, 0.16, 0.32, 0.64])
DETS = Counter("detections_total", "Detections above threshold", ["camera", "cls"])

start_http_server(9100)

Be careful with labels. Each unique combination of label values becomes a separate time series. Labels for camera, class and model are fine, even across hundreds of cameras. Labels for track IDs, frame numbers or timestamps create an unbounded number of series and will overload the metrics system.

Logging that helps

Keep personal data out of logs where possible. Stream URLs often contain credentials; strip them before logging.

Dashboards and alerts

Build one overview dashboard that answers "is everything working?" at a glance: cameras connected, cameras behind, cameras with health problems, devices hot or full, alerts delivered in the last hour. Then per-camera dashboards for investigation.

Alert on the few conditions that need a person: a camera down for more than a few minutes, a device not reporting, delivery failing, a disk nearly full, a camera with zero detections when it should have some. Route them to whoever can fix them, with a cooldown. Everything else belongs on a dashboard, not in someone's phone.

Checklist

Next steps: Scaling Video Analytics to Hundreds of Camera Streams and Watchdogs and Self-Recovery for Unattended Edge Boxes.