Vision Field Notes

A Pre-Deployment Testing Checklist for Vision Models

A good validation score does not mean a model is ready for a real camera. Validation data is usually cleaner, closer to the training set and easier than what a live site produces. In production video analytics, a common pattern is that a model looks fine on the test set and then fails on things nobody sampled: a night shift, a dirty lens, a camera that moved a little, or a lorry parked in the same place all day.

This checklist is for the step between "the metrics look good" and "it is live". It is organised so you can work through it in order.

1. Test on footage from the real deployment

The most valuable test is footage from the actual camera, at the actual angle, in the actual place. Not a similar camera. Not stock video.

Review the output by eye. Watch the video with the boxes drawn. Ten minutes of watching will tell you things a metric will not.

Two overhead frames of a car park, looking straight down at the cars, with detection boxes from a pretrained model. On the left, a white car seen from above is boxed and labelled cell phone 0.94. On the right, a red car at the top is labelled bus 0.54, while a silver car at the bottom is correctly labelled car 0.74.
Caption: A general-purpose model watching from an unusual angle: cars seen from directly above come out as a confident cell phone and a bus. Only footage from the real camera position reveals this kind of error. Footage: Intel IoT DevKit sample videos (CC BY 4.0). Detections: YOLOv8n.

2. Measure what matters for the task

Choose metrics that match the real cost of errors.

Write down the target before you test, for example "at most N false alerts per camera per day" or "counts within an agreed error of manual counts". Then you are measuring against a decision, not looking for a reason to say yes.

3. Cover the edge cases

List the situations that are rare but plausible, and test each on purpose.

Edge case What to check
Night, dusk, dawn Does accuracy hold with low light, noise and infrared mode?
Glare, reflections, rain, fog Are there false detections on wet surfaces or lights?
Crowds and occlusion Do objects overlap, merge or lose IDs?
Unusual objects Costumes, large loads, carts, animals, reflections in glass
Small or distant objects Do they fall below the detectable size? See class imbalance and small objects
Camera changes Slight movement, a lens smudge, a changed exposure setting
Empty scenes Does the model hallucinate objects when nothing is there?
Static look-alikes Posters, mannequins, shadows, parked vehicles

Testing empty scenes deserves emphasis. Many false alarms come from long periods with nothing happening, and a test set made only of interesting frames will not show them.

4. Test performance under real load

5. Test failure handling

Real systems fail in boring ways. Cause each failure on purpose and see what happens.

A system that fails silently is worse than one that fails loudly. Silence looks like "nothing happened".

6. Test the alert logic

Alerts are what people actually experience. Check:

Design guidance is in designing alert rules, and ways to cut noise are in reducing false alarms in production.

7. Check privacy and compliance basics

Before going live, confirm what is stored, for how long, who can see it, and whether people on site have been informed as required where you work. Do not treat this as an afterthought. This is not legal advice. See privacy and ethics in camera analytics.

8. Prepare monitoring and rollback

Launch is the start of testing, not the end.

Line chart of detections per hour for one camera over seven days. For the first four days the count follows a clear daily cycle, rising to peaks of about 25 to 33 around midday and falling to a few per hour at night. A dashed red line on day 5 is labelled camera knocked out of position; after it the counts stay low, mostly under 10, and the daily peaks disappear.
Caption: A moved camera rarely raises an error. It shows up as a change in the counts, which is why per-camera detection counts are worth tracking. The data is illustrative.
Diagram of four stages in a row joined by arrows: offline tests on held-out footage, edge cases, soak runs and failure drills; shadow mode, which runs and logs without real alerts; one camera live, with real alerts and close monitoring; and wider rollout to more cameras and sites. A monitoring band runs under the last three stages, listing counts per camera, confidence, uptime and frame rate. A red rollback line runs from the end back to the first stage, labelled previous model and config, tested before launch.
Caption: Each step widens the audience only after the previous one has run cleanly, and monitoring covers every live stage. The rollback path is part of the plan, and it should be tested before launch.

9. Get sign-off with clear limits

Write a short note for the people who will use the system: what it detects, what it does not, the known weak spots, and the metrics you measured on real footage. Honest limits prevent disappointment and unfair blame later.

Checklist

Next steps: reducing false alarms in production video analytics, measuring FPS and latency correctly.