Vision Field Notes

Reading RTSP Streams with OpenCV and FFmpeg

Most IP cameras and NVRs expose video over RTSP. Reading that stream with OpenCV looks trivial: one call to cv2.VideoCapture and a loop. In production video analytics, that simple loop is also where many of the first problems show up: growing delay, frozen frames, green smears and silent disconnects. This article covers how to open a stream, why latency builds up, and how to debug the common errors.

What RTSP actually is

RTSP is a control protocol. It negotiates a session with the camera, and the video itself travels as RTP packets, either over UDP or interleaved inside the same TCP connection. The video is usually H.264 or H.265. OpenCV does not decode this itself. It hands the work to a backend, most often FFmpeg in the pip wheels, and sometimes GStreamer if your build includes it.

That matters because most behaviour you will fight (buffering, timeouts, transport choice) belongs to the backend, not to OpenCV's Python API.

Diagram of five boxes in a row: IP camera, network, FFmpeg, VideoCapture and your loop, with a bracket over network and FFmpeg labelled transport, buffering and timeouts live here.
Caption: Most of the behaviour you tune (transport, buffering, timeouts) lives in the network and FFmpeg stages, not in OpenCV's Python API.

The minimal reader

import cv2

url = "rtsp://user:password@192.168.1.20:554/stream1"
cap = cv2.VideoCapture(url, cv2.CAP_FFMPEG)

if not cap.isOpened():
    raise RuntimeError("Could not open stream")

while True:
    ok, frame = cap.read()
    if not ok:
        break
    # run your model on frame here
    cv2.imshow("stream", frame)
    if cv2.waitKey(1) & 0xFF == ord("q"):
        break

cap.release()
cv2.destroyAllWindows()

The URL path (/stream1 above) differs by manufacturer and model. Look in the camera's documentation or web interface for the exact RTSP path. Many cameras offer a main stream and a lower-resolution sub stream on different paths. For analytics, the sub stream is often enough and is much cheaper to decode.

Two details that trip people up:

Why latency keeps growing

The most common complaint is that the image on screen falls further behind real time the longer the program runs. The cause is almost always the same: your processing loop is slower than the camera's frame rate. The backend keeps receiving and queueing frames while you are busy with inference. cap.read() then returns old frames from that queue instead of the newest.

Suppose the camera sends 25 frames per second and your pipeline handles 10. Every second, 15 frames pile up. Nothing is dropped, so the delay only increases until memory or the backend's buffer limit is hit.

Line chart of delay behind real time over 60 seconds. The plain read loop rises in a straight line to 36 seconds; the latest-frame reader stays flat near zero.
Caption: The 25 versus 10 frames per second example worked out: a plain loop adds 0.6 s of delay every second, while a latest-frame reader stays bounded. Computed from the example numbers, not measured.

There are three honest fixes:

  1. Make the pipeline faster than the stream. Use a smaller model, a lower resolution sub stream, or skip frames deliberately (see below).
  2. Read in a separate thread and keep only the latest frame. The reader thread loops on cap.read() as fast as frames arrive and overwrites a single shared variable. The processing thread takes whatever is newest.
  3. Grab and discard. Call cap.grab() several times to drop stale frames, then cap.retrieve() for the one you want. grab() is cheaper than read() because it skips the conversion into an array, though the decoding still happens.

A latest-frame reader looks like this:

import threading
import cv2

class LatestFrameReader:
    def __init__(self, url):
        self.cap = cv2.VideoCapture(url, cv2.CAP_FFMPEG)
        self.frame = None
        self.ok = True
        self.lock = threading.Lock()
        self.thread = threading.Thread(target=self._run, daemon=True)
        self.thread.start()

    def _run(self):
        while self.ok:
            ok, frame = self.cap.read()
            if not ok:
                self.ok = False
                break
            with self.lock:
                self.frame = frame

    def read(self):
        with self.lock:
            return self.frame

    def release(self):
        self.ok = False
        self.thread.join(timeout=2)
        self.cap.release()

This keeps latency bounded, at the cost of dropping frames your model never sees. For most alerting use cases that is the right trade. If you must process every frame (for example, exact counting with a tracker), you need a pipeline that is fast enough instead, or you should batch and queue frames explicitly and monitor the queue length.

You may see advice to set cv2.CAP_PROP_BUFFERSIZE to 1. Support for that property depends on the backend and build, and with the FFmpeg backend it often has no effect. Test it on your own setup instead of assuming it works.

TCP versus UDP

By default, FFmpeg may try RTP over UDP. On a clean local network this works. On a busy or lossy network, UDP packets are dropped, and you see grey areas, smeared blocks, or errors about missing references in the log. Forcing TCP trades a little latency for reliability.

With the OpenCV FFmpeg backend you can pass options through an environment variable that must be set before the capture is opened:

import os
os.environ["OPENCV_FFMPEG_CAPTURE_OPTIONS"] = "rtsp_transport;tcp"

The format is key;value, and several options are separated by |. If you run in Docker or systemd, set it in the service environment rather than in code, so it is visible and easy to change.

Symptom Likely cause First thing to try
Smeared or grey patches UDP packet loss Force TCP
Stream opens, then stalls after some seconds Firewall or NAT dropping UDP Force TCP
Delay grows over time Processing slower than stream Latest-frame reader, sub stream
isOpened() is False Wrong path, credentials, or camera session limit Test the URL in VLC or ffplay
Works in VLC, not in OpenCV Codec or backend differences Check FFmpeg build, try the sub stream

Check the URL outside your code first

Before blaming OpenCV, open the same URL in VLC or with ffplay. If it does not play there, the problem is the camera, network or credentials. Cameras also limit how many simultaneous RTSP sessions they accept. If a colleague has the stream open in a viewer, your service may fail to connect. That is a frequent cause of "it worked yesterday".

Decode cost and frame skipping

Decoding a 1080p H.264 stream on a CPU is not free, and decoding several at once adds up. The actual cost depends on your CPU, resolution, bitrate and codec, so measure it on your own hardware. Ways to reduce it:

Camera and network choices

A wired connection is much more dependable than Wi-Fi for continuous streams. A PoE IP camera (RTSP) powered through a PoE network switch removes separate power adapters and keeps cabling simple. Reserve static addresses or DHCP leases for cameras so URLs do not change.

Checklist

Next steps: make the reader survive failures with Handling RTSP Dropouts: Reconnects and Health Checks, and learn to measure your pipeline with Measuring FPS and Latency Correctly.