DNN Filters – AI-Powered Video Processing

FFmpeg has had built‑in support for deep neural networks since 2019.[reference:0] You can upscale video with AI super‑resolution, remove noise, detect faces, classify objects, and more—all from the command line. Yet almost no one knows about these features.

The DNN filters are scattered across the documentation: dnn_processing for image processing, dnn_detect for object detection, and dnn_classify for classification.[reference:1] But the examples are sparse, the setup instructions are buried in mailing lists, and the model conversion process is almost undocumented.

This guide changes that. You'll learn exactly how to use FFmpeg's DNN filters—from setting up the backends to running real AI models on your video.

The Gap: AI Inside FFmpeg, But Almost No One Knows

The first DNN filter—dnn_processing—was added to FFmpeg in October 2019.[reference:8] Since then, dnn_detect and dnn_classify have been added.[reference:9][reference:10] Yet the official documentation is sparse, and practical examples are almost non‑existent.

As one user asked on the FFmpeg mailing list: "Does those filters support text/face recognition? Is there some command examples?"[reference:11] The answer, buried in developer threads, is yes—but finding the information requires digging through patches and mailing list archives.[reference:12]

This guide brings everything together in one place.

The Symptom: "I Need AI Video Processing—But How?"

You've seen what AI can do: upscale old footage, remove noise, detect faces, classify objects. You know FFmpeg can do it—but every search leads to dead ends. The documentation mentions the filters but doesn't show you how to use them. The examples online are incomplete or use models you can't find.

You need a practical guide. This is it.

Core Concepts: What Are DNN Filters?

FFmpeg's DNN (Deep Neural Network) filters allow you to run AI models on video frames. There are three main filters:

Each filter loads a pre‑trained model, runs inference on each frame, and produces output—either a modified frame (for dnn_processing) or metadata (for dnn_detect and dnn_classify).

The dnn_processing Filter

The dnn_processing filter is a generic image processing filter that accepts any DNN model that does image processing.[reference:13] It's the most versatile of the three DNN filters.

Syntax:

dnn_processing=dnn_backend=BACKEND:model=MODEL_PATH:input=INPUT_NAME:output=OUTPUT_NAME[:fmt=FORMAT]

Options:

• dnn_backend — native, tensorflow, openvino, or torch. Default is native.[reference:14]
• model — Path to the model file.[reference:15]
• input — Name of the input tensor in the model.[reference:16]
• output — Name of the output tensor in the model.[reference:17]
• fmt — Pixel format: rgb24 or bgr24. Default is rgb24.[reference:18]

Important limitations:

• Currently, only RGB24 and BGR24 formats are supported.[reference:19]
• The DNN network can accept data in float32 or uint8 format.[reference:20]
• The network can change frame size—useful for super‑resolution.[reference:21]

The dnn_detect Filter

The dnn_detect filter performs object detection using deep learning networks.[reference:22] It detects objects in each frame and adds bounding boxes as side data that can be used by other filters.

Syntax:

dnn_detect=dnn_backend=BACKEND:model=MODEL_PATH:input=INPUT_NAME:output=OUTPUT_NAME:confidence=THRESHOLD[:labels=LABELS_FILE]

Options:

• confidence — Confidence threshold (default: 0.5). Detections below this threshold are ignored.[reference:23]
• labels — Path to a label file mapping label IDs to names.[reference:24]

When a detection is made, the filter adds side data with:

• Bounding box coordinates
• Label name (if labels file is provided)
• Confidence score

As demonstrated in the FFmpeg developer mailing list, the output looks like:

[Parsed_showinfo_1 @ 0x...] side data - detection bounding boxes:
    index: 0, region: (1005, 813) -> (1086, 905), label: face, confidence: 10000/10000.
    index: 1, region: (888, 839) -> (967, 926), label: face, confidence: 6917/10000.[reference:25]

The dnn_classify Filter

The dnn_classify filter performs classification on video frames or audio using deep neural networks.[reference:26] It supports three classification modes:[reference:27]

• Standard image classification (OpenVINO backend)
• CLIP (Contrastive Language-Image Pre‑training) classification (Torch backend)
• CLAP (Contrastive Language-Audio Pre‑training) classification (Torch backend)

Syntax:

dnn_classify=dnn_backend=BACKEND:model=MODEL_PATH:input=INPUT_NAME:output=OUTPUT_NAME:confidence=THRESHOLD[:labels=LABELS_FILE][:categories=CATEGORIES_FILE][:tokenizer=TOKENIZER_PATH]

Additional options:

• categories — Path to a categories file for hierarchical classification (CLIP/CLAP only).[reference:28]
• tokenizer — Path to the text tokenizer for CLIP/CLAP.[reference:29]

DNN Backends: Native, TensorFlow, OpenVINO, and Torch

FFmpeg supports four DNN backends, each with different capabilities and requirements.[reference:30]

Backend Comparison

As noted in the FFmpeg documentation: "Different backends use different file formats. TensorFlow and native backend can load files for only its format".[reference:34]

Setting Up DNN Support in FFmpeg

To use DNN filters, you need to build FFmpeg from source with the appropriate backend enabled. Most pre‑built binaries do not include DNN support.[reference:35]

1. Native Backend (No Additional Dependencies)

The native backend is built‑in and requires no additional libraries. However, you need to convert models to the .model format.

2. TensorFlow Backend

# Install TensorFlow C library
# Download from: https://www.tensorflow.org/install/install_c

# Configure FFmpeg
./configure --enable-libtensorflow \
  --extra-cflags='-I/path/to/tensorflow/include' \
  --extra-ldflags='-L/path/to/tensorflow/lib'

3. OpenVINO Backend

As documented in the FFmpeg developer mailing list:[reference:36]

# 1. Download and install OpenVINO Toolkit
# https://software.intel.com/content/www/us/en/develop/tools/openvino-toolkit/download.html

# 2. Set environment variables
export LD_LIBRARY_PATH=.../deployment_tools/inference_engine/lib/intel64/:.../deployment_tools/inference_engine/external/tbb/lib/

# 3. Configure FFmpeg
./configure --enable-libopenvino \
  --extra-cflags='-I.../deployment_tools/inference_engine/include/' \
  --extra-ldflags='-L.../deployment_tools/inference_engine/lib/intel64'

4. Torch (LibTorch) Backend

For CLIP/CLAP support, you need the Torch backend.[reference:37]

# Download LibTorch from: https://pytorch.org/get-started/locally/
./configure --enable-libtorch \
  --extra-cflags='-I/path/to/libtorch/include' \
  --extra-ldflags='-L/path/to/libtorch/lib'

Model Conversion: From TensorFlow to Native

The native backend uses a .model file format that can be generated from a TensorFlow .pb file using FFmpeg's conversion script.[reference:38]

Here's a complete example from the FFmpeg commit that introduced dnn_processing:[reference:39]

Step 1: Create a TensorFlow Model

This Python script creates a simple model that halves the value of the first channel (e.g., reduces red channel intensity):[reference:40]

# Python script to create a TensorFlow model (halve_first_channel.py)
import tensorflow as tf
import numpy as np
import imageio

in_img = imageio.imread('in.bmp')
in_img = in_img.astype(np.float32)/255.0
in_data = in_img[np.newaxis, :]

# Filter that halves the first channel (R) and leaves G and B unchanged
filter_data = np.array([0.5, 0, 0, 0, 1., 0, 0, 0, 1.]).reshape(1,1,3,3).astype(np.float32)
filter = tf.Variable(filter_data)

x = tf.placeholder(tf.float32, shape=[1, None, None, 3], name='dnn_in')
y = tf.nn.conv2d(x, filter, strides=[1, 1, 1, 1], padding='VALID', name='dnn_out')

sess=tf.Session()
sess.run(tf.global_variables_initializer())
output = sess.run(y, feed_dict={x: in_data})

# Save as .pb file
graph_def = tf.graph_util.convert_variables_to_constants(sess, sess.graph_def, ['dnn_out'])
tf.train.write_graph(graph_def, '.', 'halve_first_channel.pb', as_text=False)

# Verify output
output = output * 255.0
output = output.astype(np.uint8)
imageio.imsave("out.bmp", np.squeeze(output))[reference:41]

Step 2: Convert to Native Format

Use FFmpeg's conversion script to generate the .model file for the native backend:[reference:42]

# Convert .pb to .model for native backend
python tools/python/convert.py halve_first_channel.pb halve_first_channel.model[reference:43]

Step 3: Run with FFmpeg

Now you can use the model with both the native and TensorFlow backends:[reference:44]

# Native backend
ffmpeg -i input.jpg -vf "dnn_processing=model=halve_first_channel.model:input=dnn_in:output=dnn_out:fmt=rgb24:dnn_backend=native" -y out.native.png

# TensorFlow backend
ffmpeg -i input.jpg -vf "dnn_processing=model=halve_first_channel.pb:input=dnn_in:output=dnn_out:fmt=rgb24:dnn_backend=tensorflow" -y out.tf.png[reference:45]

Practical Example 1: Super‑Resolution

Super‑resolution models upscale images while preserving detail. FFmpeg's dnn_processing filter can run super‑resolution models like SRCNN.[reference:46]

# Super‑resolution with SRCNN model (OpenVINO backend)
ffmpeg -i lowres.jpg -vf "dnn_processing=dnn_backend=openvino:model=srcnn.xml:input=data:output=srcnn" -y superres.jpg

As noted in the FFmpeg documentation: "Scale factor is necessary for SRCNN model, because it input upscaled using bicubic upscaling with proper scale factor".[reference:47]

For the native backend, you can find pre‑trained models at:

https://github.com/guoyejun/ffmpeg_dnn/tree/main/models/openvino/2021.1[reference:48]

Practical Example 2: Object Detection

Face detection with OpenVINO—a complete example from the FFmpeg developer mailing list:[reference:49]

# Face detection with OpenVINO
# Download models:
# wget https://github.com/guoyejun/ffmpeg_dnn/raw/main/models/openvino/2021.1/face-detection-adas-0001.bin
# wget https://github.com/guoyejun/ffmpeg_dnn/raw/main/models/openvino/2021.1/face-detection-adas-0001.xml
# wget https://github.com/guoyejun/ffmpeg_dnn/raw/main/models/openvino/2021.1/face-detection-adas-0001.label

ffmpeg -i cici.jpg -vf "dnn_detect=dnn_backend=openvino:model=face-detection-adas-0001.xml:input=data:output=detection_out:confidence=0.6:labels=face-detection-adas-0001.label,showinfo" -f null -[reference:50]

The output will show detected faces with confidence scores:

[Parsed_showinfo_1 @ 0x...] side data - detection bounding boxes:
    index: 0, region: (1005, 813) -> (1086, 905), label: face, confidence: 10000/10000.
    index: 1, region: (888, 839) -> (967, 926), label: face, confidence: 6917/10000.[reference:51]

Practical Example 3: Image Classification

Classify images with the dnn_classify filter. The filter supports standard classification with OpenVINO, and CLIP/CLAP with the Torch backend.[reference:52]

# Standard image classification with OpenVINO
ffmpeg -i image.jpg -vf "dnn_classify=dnn_backend=openvino:model=classification.xml:input=data:output=prob:confidence=0.5:labels=imagenet.txt,showinfo" -f null -

For CLIP‑based classification with the Torch backend:

# CLIP classification with Torch backend
ffmpeg -i image.jpg -vf "dnn_classify=dnn_backend=torch:model=clip.pt:labels=categories.txt:categories=units.txt:tokenizer=tokenizer.txt,showinfo" -f null -[reference:53]

Practical Example 4: Combined Detection + Classification

You can chain dnn_detect and dnn_classify together to first detect objects, then classify each detected object.[reference:54]

# Combined detection + classification
ffmpeg -i image.jpg -vf "dnn_detect=dnn_backend=openvino:model=face-detection-adas-0001.xml:input=data:output=detection_out:confidence=0.6:labels=face-detection-adas-0001.label,dnn_classify=dnn_backend=openvino:model=emotions-recognition-retail-0003.xml:input=data:output=prob_emotion:confidence=0.3:labels=emotions-recognition-retail-0003.label:backend_configs='async=0',showinfo" -f null -[reference:55]

This detects faces, then classifies the emotion of each detected face. As noted in the FFmpeg developer mailing list, "classification is done on every detection bounding box in frame's side data".[reference:56]

Performance & Hardware Considerations

DNN inference is computationally intensive. Here are some performance considerations:

• GPU acceleration is available through the OpenVINO backend (Intel GPUs) and the Torch backend (CUDA for NVIDIA GPUs).
• Async execution is supported. As noted in the FFmpeg docs: "To get full functionality (such as async execution), please use the dnn_processing filter".[reference:57]
• Model size matters—larger models are more accurate but slower. The `confidence` parameter can be adjusted to trade accuracy for speed.[reference:58]
• Resolution affects performance—higher resolution frames take longer to process.

Debugging Common DNN Issues

Here are the most common issues and how to fix them.

Visualize DNN Filters with the FFmpegLab IDE

DNN filters are much easier to debug with visual feedback. The FFmpegLab IDE lets you:

• Preview DNN processing in real time—see super‑resolution, denoising, and detection results instantly
• Adjust confidence thresholds with sliders and see which objects are detected
• Overlay bounding boxes on the preview for dnn_detect results
• Display classification labels directly on the frame
• Compare different backends side by side
• Generate the exact command with your tuned parameters

Here's how DNN object detection visualization looks in the FFmpegLab IDE.

# Visual DNN detection with bounding box overlay

The IDE shows you exactly what's happening—the original frame, the detected objects with bounding boxes, and the confidence scores. You can adjust the confidence threshold with a slider and see which detections appear or disappear in real time.

Frequently Asked Questions (FAQ)

Final Word

FFmpeg's DNN filters represent the future of video processing. With dnn_processing for super‑resolution and denoising, dnn_detect for object detection, and dnn_classify for classification, you can bring AI‑powered intelligence to your video workflows—all from the command line.

The key is understanding the four backends and their requirements: Native for simplicity, TensorFlow for existing models, OpenVINO for Intel hardware, and Torch for CLIP/CLAP classification. Each has its strengths, and with the FFmpegLab IDE's visual feedback, you can experiment, debug, and perfect your DNN pipelines without the guesswork.

Next time you need to upscale old footage, detect faces, or classify images, reach for FFmpeg's DNN filters. The power of AI is now at your fingertips.