AI Video Analysis: Best Free Tools & Open-Source Models (2026)

AI video analysis turns video into structured data that software can search, measure, classify, and act on. This guide focuses on free and self-hosted options for developers who need to choose between running models themselves and calling an external API. You will find current open-source models grouped by task, with their licences and hardware requirements, so you can compare what is practical for local testing, server deployment, and larger video workloads.

The main question is not only what a model can detect. Video adds time, which changes both the problem and the cost. A useful system may need to keep track of the same person or object across hundreds of frames, understand actions that unfold over several seconds, separate one shot from another, and process hours of footage without making compute costs impractical.

The sections below break those requirements into the main video analysis tasks and the open-source options available for each.

* Google publishes Apache-2.0 for the MediaPipe repository, but does not publish a separate licence statement for the Pose Landmarker model weights.

What is AI Video Analysis?

AI video analysis is the use of machine learning models to extract information from video frames, motion, audio, and changes over time. The field is also commonly called Video Content Analysis (VCA).

Video analysis vs. image analysis: why time changes everything

Image analysis treats each image as a separate input. Video analysis has to account for what happens between frames.

Identity is one example. Detecting a person in a single frame is an image task. Following that same person through a corridor, after partial occlusion, or across hundreds of frames requires the system to preserve identity over time. A new detection on every frame is not enough.

Motion also carries information that a still image cannot show. A frame may contain a person next to a bicycle, but a sequence can show whether the person is riding it, pushing it, approaching it, or walking away. Some events only make sense when several frames are considered together.

Cost changes too. Video may contain many frames every second, so compute scales with both footage duration and frame rate. A model that is inexpensive to run on one image can become costly when applied to every frame of an hour-long recording.

Shot and scene boundaries add another problem. A video can jump between cameras, locations, or subjects. Systems that ignore those boundaries can connect objects or actions that belong to unrelated parts of the footage.

The core tasks

Most video analysis systems combine several of these tasks:

• Object detection: Finds and classifies objects in individual video frames.
• Object tracking: Associates detections across frames so the same object keeps a consistent identity over time.
• Segmentation: Assigns pixels to objects or regions, often while following how those regions change across frames.
• Pose estimation: Detects body keypoints and can track how a person’s posture or movement changes through a sequence.
• Video captioning and understanding: Produces text or structured interpretations based on events, actions, and context across multiple frames.
• Video search and retrieval: Finds relevant clips or moments based on text, visual content, actions, or other indexed video features.
• Speech transcription from the audio track: Converts spoken audio inside a video into searchable or timestamped text.

How We Chose These Models

We selected models using criteria that matter for teams deciding whether self-hosting is practical.

First, the model had to be actively maintained, with a release or meaningful update within the last 18 months. We also checked the licence for commercial use, since some well-known research models restrict commercial deployment.

Hardware requirements were another filter. We focused on models that can run on hardware a normal engineering team can access, and we state VRAM requirements where that information is available. A model that only works on large research clusters is less useful for most product teams.

We also looked for published benchmark results on recognised datasets. We did not run our own benchmark suite or test every model in production. Instead, we compared published benchmark numbers, licence terms, maintenance activity, and stated hardware requirements. Where a performance figure comes from a model vendor or project team rather than independent testing, we say so.

Finally, the model had to be designed for video, not just an image model applied independently to each frame.

All benchmark numbers in this article are cited from the model’s own documentation or paper.

Video-Language Models: The Shift That Changed Video Analysis

The older approach to video analysis was built around narrow models. One model detected objects, another linked those detections across frames, and another classified actions. A production system often meant assembling several components into a pipeline, then handling the handoff between them.

Each stage also had its own training data and label space. If you wanted to detect delivery vans, track them, and identify when they stopped at a loading area, you might need separate datasets or tuning steps for detection, tracking, and action recognition. Adding a new question could mean changing part of the pipeline and collecting more labelled examples.

Video-language models change that interface. Instead of defining a fixed task first, you give the model a video and a text question. The response can be natural language, timestamps, or another structured output supported by the model. You can ask, “when does the delivery van arrive?” without first training a dedicated van detector for that question.

This opens several types of analysis that are awkward with fixed-label pipelines. Open-vocabulary queries let users ask about objects, actions, or situations that were not defined as a closed set of classes during deployment. Temporal grounding lets a system identify when an event occurs, rather than only reporting that an object or action appears somewhere in the video. Longer context also makes it possible to reason across much more footage, including workloads measured in hours rather than short clips.

The trade-off is cost and speed. A video-language model requires far more compute per minute of footage than a small detector built for one fixed task. It is also a poor fit for many 30 FPS real-time streams running on edge hardware. If the requirement is low latency, high throughput, and a stable set of labels, a task-specific detector, tracker, or classifier is still the better choice.

The model list ahead is grouped by task rather than treating every video model as interchangeable. Most production systems in 2026 combine both approaches: a cheap detector handles the always-on pass, while a video-language model handles the questions that need reasoning.

Video Understanding & Captioning

The two main general-purpose choices here are Qwen3-VL and InternVideo3, but they are not positioned the same way. Qwen3-VL has the broader multimodal ecosystem and supports context up to 1M tokens. InternVideo3 is more narrowly focused on video and currently leads Qwen3-VL-8B on 12 of 15 video benchmarks reported in the InternVideo3 paper. The comparison is not one-sided, since Qwen3-VL-8B still wins on LVBench, VideoMME-v2, and TempCompass.

The figures below come from the InternVideo3 paper, arXiv 2606.12195, where Qwen3-VL-8B was evaluated as a baseline. Qwen's own reported results may differ.

For context, the same Video-MME benchmark reports 88.6 for Gemini 3 Pro, 87.8 for Gemini 2.5 Pro, and 83.3 for GPT-5. Open-weight models remain behind those results on this benchmark, so self-hosting still comes with a measurable quality trade-off.

Qwen3-VL (Alibaba)

Released 15 and 21 October 2025, technical report 27 November 2025 · Apache-2.0, commercial use permitted

Qwen3-VL is the broader general-purpose video-language model in this group. It supports long-video understanding, video grounding, dense video captions, textual timestamps, and an Interleaved-MRoPE architecture. Dense variants come in 2B, 4B, 8B, and 32B sizes, each with Instruct and Thinking versions. MoE variants include 30B-A3B with 3B active parameters and 235B-A22B with 22B active. Native context is 256K tokens and can expand to 1M. In the InternVideo3 paper, Qwen3-VL-8B scores 71.4 on Video-MME, 62.4 on LongVideoBench, 68.7 on MVBench, and 58.0 on LVBench. Official documentation does not publish VRAM requirements.

Best for: general-purpose multimodal systems that need long context, timestamped video questions, and a broader model ecosystem rather than a video-only specialist.

InternVideo3 (OpenGVLab / Shanghai AI Lab)

Released June 2026 · Apache-2.0

InternVideo3 is the video specialist in this group. The current 8B instruct model leads Qwen3-VL-8B on most of the reported comparisons above and posts the best open-weight results in the supplied data on Video-MME, MLVU, VRBench, and EgoSchema. 

It scores 73.8 on Video-MME, 77.3 on MLVU, 69.4 on VRBench, and 76.6 on EgoSchema. It runs on a single H200. Its M2LA mechanism also targets long-context efficiency, delivering 1.84x decode throughput at 32K prefill tokens and 5.01x at 384K, while remaining executable at 512K prefill where the baseline runs out of memory.

The wider InternVideo family also shows sustained maintenance: InternVideo1 arrived in January 2023, InternVideo2 checkpoints in April 2024, InternVideo2.5 in January 2025, InternVideo-Next in December 2025, and InternVideo3 in June 2026.

Best for: self-hosted video QA, retrieval, temporal localisation, and long-context analysis when video benchmark performance matters more than having the broadest multimodal ecosystem.

VideoMAE (MCG-NJU)

NeurIPS 2022 Spotlight · CC-BY-NC 4.0, non-commercial only

VideoMAE is a video classification and representation backbone rather than a chat-style model. Its masked video autoencoder training masks 80% of tokens frame by frame, and the available variants are ViT-S, ViT-B, ViT-L, and ViT-H, pretrained on Kinetics-400. 

Reported Kinetics-400 top-1 accuracy is 79.0% for ViT-S, 81.5% for ViT-B, 85.2% for ViT-L, 86.6% for ViT-H, and 87.4% for ViT-H at 320x320 with 32 frames. On Something-Something V2, the correct figures are 66.8% for ViT-S, 70.8% for ViT-B, 74.3% for ViT-L, and 75.4% for ViT-L at 32 frames.

Best for: research or internal video classification and representation work where commercial use is not required.

Gemma 4 (Google)

Apache 2.0 per the model card.

Gemma 4 is better treated as a short-clip multimodal model than a long-video system. Video input is capped at approximately 60 seconds at 1 frame per second, while audio input is capped at 30 seconds. Audio support is limited to E2B, E4B, and 12B. Google does not publish video benchmark results for Gemma 4.

The model sizes and context windows vary by variant. E2B has 2.3B effective parameters, 5.1B including embeddings, with 128K context. E4B has 4.5B effective, 8B including embeddings, with 128K context. The 12B Unified model has 11.95B parameters and 256K context. The 26B-A4B MoE has 25.2B total parameters with 3.8B active and 256K context. The 31B Dense model has 30.7B parameters and 256K context. The “E” designation refers to effective parameters through Per-Layer Embeddings.

Memory figures below are community measurements from Unsloth rather than official Google specifications:

Best for: short-clip multimodal work on local hardware where video and audio can go through one model, and where approximately 60-second video clips are enough.

Object Detection

Object detectors still work frame by frame. In a video pipeline, the detector finds objects in each frame, while a tracker links those detections over time so the same person, vehicle, or item keeps a stable identity. The next subsection covers that tracking layer.

Latency also has a direct effect on video throughput. A detector that is acceptable for occasional images may fail to keep up with a live stream once it has to run dozens of times per second.

YOLO26 (Ultralytics)

Announced September 2025, released January 2026 · AGPL-3.0 or paid Enterprise licence. AGPL requires source disclosure, so closed-source products need the Enterprise licence.

YOLO26 is Ultralytics' edge-focused detector, aimed at robotics, drones, mobile, and embedded workloads. Its architecture adds native end-to-end NMS-free inference, DFL-free regression, Progressive Loss with STAL for small-target-aware label assignment, MuSGD, and task-specific heads. It supports detection, instance segmentation, semantic segmentation, depth estimation, classification, pose, and oriented bounding boxes.

At 640px, YOLO26n reaches 40.9 mAP50-95 with 2.4M parameters and 5.4B FLOPs. On the tested Intel Xeon CPU it takes 38.9 ms per frame, which works out to roughly 26 FPS. That is already below a 30 FPS video stream before adding decoding and tracking overhead. On a T4 with TensorRT10, the same model takes 1.7 ms per frame, or roughly 588 FPS, leaving much more headroom for concurrent streams.

YOLO26 also supports open-vocabulary detection. YOLOE-26x reports 40.6 AP on LVIS minival under text prompting, which makes it possible to search video for text-defined concepts without training a detector around a fixed class list.

Best for: video pipelines that need higher detection accuracy at a given model size, especially small CPU deployments or GPU workloads where open-vocabulary detection is useful.

YOLO11 (Ultralytics)

Released 10 September 2024 · AGPL-3.0 or paid Enterprise licence. AGPL requires source disclosure, so closed-source products need the Enterprise licence.

YOLO11 remains a widely deployed Ultralytics baseline with a large set of tutorials and integrations. It supports detection, instance segmentation, classification, pose, and oriented bounding boxes. At 640px, YOLO11n reports 39.5 mAP50-95 with 2.6M parameters and 6.5B FLOPs. It takes 56.1 ms on the tested Intel Xeon CPU and 1.5 ms on a T4 with TensorRT10. At the other end of the range, YOLO11x reaches 54.7 mAP50-95 with 56.9M parameters and 194.9B FLOPs.

On GPU, YOLO26 is not a throughput upgrade across the board. YOLO11n is faster on the tested T4 at 1.5 ms versus 1.7 ms for YOLO26n, while YOLO11x takes 11.3 ms versus 11.8 ms. The medium and large variants report the same T4 latency in both families.

Best for: existing Ultralytics production pipelines where compatibility, available integrations, and GPU throughput matter more than the accuracy gains in YOLO26.

YOLO26 vs. YOLO11 on CPU

The CPU story changes with model size. YOLO26 is faster only for the nano and small variants. From medium upward, YOLO11 is faster.

That reversal matters for video deployment. Choosing YOLO26m because the nano model is 31% faster on CPU would lead to the opposite result: YOLO26m takes 220.0 ms per frame versus 183.2 ms for YOLO11m, making it about 20% slower on the tested CPU.

The consistent YOLO26 advantage is accuracy. It gains 1.4 mAP50-95 at nano, 1.6 at small, medium, and large, and 2.8 at extra-large. CPU latency improves only at the two smallest sizes.

For video, detector selection should therefore be based on the exact model size and hardware you plan to run. Once detections are produced, a tracker is still needed to maintain identities across frames.

Object Tracking

Every modern tracker is detector-driven. The detector decides what objects exist in each frame, then the tracker tries to preserve their identities over time. A weak detector cannot be rescued by a clever association step, so detector quality often matters more than small differences between association methods.

The six Ultralytics trackers are easier to understand as a few related branches rather than six unrelated choices:

├── DeepSORT (2017, adds appearance)
├── ByteTrack (2022)
│   ├── BoT-SORT (+ camera-motion compensation, + optional ReID, improved Kalman state)
│   └── FastTracker (+ occlusion handling)
├── OC-SORT (2023, observation-centric)
│   └── Deep OC-SORT (+ ReID, + camera-motion compensation)
└── TrackTrack (multi-cue association)

‍

Starting with Ultralytics YOLO v8.4.63, OC-SORT, Deep OC-SORT, FastTracker, and TrackTrack are available alongside BoT-SORT and ByteTrack. On earlier versions, only BoT-SORT and ByteTrack are available.

BoT-SORT

Aharon, Orfaig & Bobrovsky · Submitted 29 June 2022 · MIT licence

BoT-SORT is the Ultralytics default and is built on ByteTrack and FastReID. It combines motion and appearance information with camera-motion compensation and a more accurate Kalman filter state vector. On MOT17 test, BoT-SORT reports 80.6 MOTA, 79.5 IDF1, and 64.6 HOTA. The ReID variant reports 80.5 MOTA, 80.2 IDF1, and 65.0 HOTA. On MOT20 test, BoT-SORT reports 77.7 MOTA, 76.3 IDF1, and 62.6 HOTA, while BoT-SORT-ReID reaches 77.8 MOTA, 77.5 IDF1, and 63.3 HOTA. It ranked first on the MOTChallenge MOT17 and MOT20 test sets at publication. No FPS figure was published.

The ReID results give a useful answer to whether appearance embeddings are worth enabling. On MOT17, adding ReID changes MOTA from 80.6 to 80.5 while increasing IDF1 from 79.5 to 80.2. On MOT20, MOTA changes from 77.7 to 77.8 while IDF1 rises from 76.3 to 77.5. For workflows where maintaining identity matters more than raw detection counts, that trade-off can justify the extra appearance stage.

Best for: general-purpose multi-object tracking where identity preservation, camera motion, and optional appearance matching matter.

ByteTrack

Zhang et al., ECCV 2022 · MIT licence

ByteTrack tracks objects using detection boxes alone, without an appearance or ReID model. On MOT17 it reports 80.3 MOTA, 77.3 IDF1, and 63.1 HOTA at 29.6 FPS on a single V100. On MOT20 it reports 77.8 MOTA, 75.2 IDF1, and 61.3 HOTA at 13.7 FPS on the same GPU.

Those end-to-end FPS figures are dominated by detector cost rather than association alone. They also cannot be compared directly with OC-SORT's 28 FPS result because OC-SORT was measured on an RTX 2080Ti rather than a V100. ByteTrack's throughput advantage comes from keeping the association stage lightweight, not from treating cross-paper FPS as a controlled comparison.

Best for: high-throughput video tracking where low association overhead matters more than appearance-based identity matching.

OC-SORT

Cao et al., CVPR 2023 · MIT licence

OC-SORT is an observation-centric tracker built around motion rather than ReID. On MOT17 using private detections, it reports 78.0 MOTA, 77.5 IDF1, and 63.2 HOTA. On MOT20 using private detections, the corresponding figures are 75.9, 76.4, and 62.4.

Its DanceTrack test results are 89.4 MOTA, 54.2 IDF1, and 55.1 HOTA. Those numbers tell a different story than MOTA alone. MOTA is driven heavily by detection misses and false positives, while IDF1 and HOTA put more weight on keeping the same identity across frames. On DanceTrack, objects are detected reliably, but identities are still swapped because dancers look similar and move unpredictably. That is why the 54.2 IDF1 should not be compared directly with 77.5 on MOT17 as though the lower number meant OC-SORT performs worse overall.

OC-SORT runs at about 28 FPS on an RTX 2080Ti end to end. Its association stage alone reaches 700 FPS on an i9 3.0 GHz CPU when detections are pre-supplied, which shows how little compute the tracker itself can add compared with the detector.

Best for: non-linear motion and scenes where identity association becomes difficult even when detections remain strong.

How to read the benchmark numbers

Do not treat the benchmark table below as a clean ranking of trackers. ByteTrack's 80.3 MOTA and OC-SORT's 78.0 MOTA, for example, come from different papers using different detectors. Since tracking metrics depend heavily on detector quality, a 2.3-point cross-paper gap says little about the association algorithm itself.

BoT-SORT and ByteTrack are somewhat closer in lineage because BoT-SORT is built directly on ByteTrack's codebase, but detector parity should still be confirmed before treating the numbers as a controlled comparison.

For counting unique objects, per-object analytics, or any downstream task tied to identity, IDF1 and HOTA matter more than MOTA. MOTA is quoted often because it is familiar, but it can hide identity switches that are costly in video analysis.

This also explains why BoT-SORT is a sensible Ultralytics default. In the reported MOT17 and MOT20 results above, its ReID variant posts the highest IDF1 and HOTA values, which are the metrics most directly tied to preserving identity across frames.

What happened to DeepSORT?

DeepSORT is the 2017 reference point, not a current recommendation. It introduced appearance embeddings into the SORT-style tracking pipeline, helping preserve identities when objects crossed or disappeared temporarily.

The older version of this article treated DeepSORT as a top tracking option. That is now outdated. ByteTrack, BoT-SORT, OC-SORT, Deep OC-SORT, FastTracker, and TrackTrack offer newer choices around motion modelling, ReID, camera compensation, and occlusion handling.

DeepSORT still matters historically because it established the appearance-assisted tracking pattern, but it should not be the default starting point for a new 2026 implementation.

Segmentation & Concept Tracking

SAM 3.1 (Meta)

Released 27 March 2026 · Meta SAM License, not a standard OSI licence. Read the licence terms before using it in a commercial product.

SAM 3.1 is Meta's current recommendation for promptable concept segmentation and is described as a drop-in replacement for SAM 3, released 19 November 2025. The foundation remains Promptable Concept Segmentation: give the model a noun phrase such as “yellow school bus” or an image exemplar, and it returns masks and identities for every matching instance. SAM 1 and SAM 2 predicted one object per prompt. SAM 3 introduced the detector-plus-tracker design with a presence head and shared vision encoder, while SAM 3.1 improves how that system handles multiple objects in video.

The main change is object multiplexing. SAM 3.1 can track up to 16 objects in one forward pass, while SAM 3 processed objects individually. On a single H100, throughput for videos with medium object counts increases from 16 FPS to 32 FPS. Global reasoning removes redundant computation and memory bottlenecks, improves accuracy in crowded scenes, and lowers GPU requirements so the model can run on smaller hardware.

The SAM 3 foundation has approximately 840M parameters and a 3.45 GB checkpoint. For images, Roboflow reports approximately 30 ms per image on an H200 GPU while handling more than 100 objects.

The benchmark figures below are from SAM 3, not SAM 3.1. SAM 3 reports 65.0 CGF1 on SA-Co/Gold versus a previous best of 34.3, 47.0 zero-shot Mask AP on LVIS versus 38.5, 60.1 J&F on MOSEv2 versus 47.9 for SAM 2.1, and 92.0 J&F on DAVIS 2017.

SAM 3 is also reported as stronger on thin, small, low-contrast, and occluded objects, with sharper boundaries and better separation between touching objects than SAM 2. SAM 3.1 builds on that foundation but does not yet have a separate published benchmark table or confirmed parameter count.

Best for: open-vocabulary video segmentation with SAM 3.1, especially multi-object workloads where object multiplexing makes production-rate tracking practical without training a class-specific segmentation model.

Pose & Motion Analysis

The deployment rule is straightforward: MediaPipe for the browser, RTMPose for CPU and edge, ViTPose for GPU servers.

Video adds two problems that still-image pose benchmarks do not show. First, per-frame keypoint estimates can jitter, so production video pipelines often need temporal smoothing. A pose model can look accurate on isolated frames and still produce unstable motion traces. Second, multi-person pose needs tracking if identities must stay consistent across frames. For that part of the pipeline, see the object tracking subsection above.

RTMPose (MMPose / OpenMMLab)

Apache-2.0, commercial use permitted

RTMPose combines a CSPNeXt backbone with a SimCC head, treating keypoint localisation as classification over discretised x and y coordinates instead of heatmaps. At 256x192 input, RTMPose-t reaches 68.5 COCO AP at 300+ FPS on an i7-11700 CPU and 940+ FPS on a GTX 1660Ti. RTMPose-s reaches 72.2 AP at 200+ FPS CPU and 710+ FPS GPU. RTMPose-m reaches 75.8 AP at 90+ FPS CPU and 430+ FPS GPU, while RTMPose-l reaches 76.5 AP at 50+ FPS CPU and 280+ FPS GPU.

On Snapdragon 865 with ncnn FP16, reported latency is 9.02 ms for RTMPose-t (~111 FPS), 13.89 ms for RTMPose-s (~72 FPS), 26.44 ms for RTMPose-m (~38 FPS), and 45.37 ms for RTMPose-l (~22 FPS). The paper also reports 70+ FPS for RTMPose-s and 35+ FPS for RTMPose-m on Snapdragon 865.

The video implication is concrete: on the tested i7-11700 CPU, every RTMPose variant from t through l exceeds 30 FPS. A standard video stream can therefore be processed in real time without a GPU, before accounting for decoding, tracking, and smoothing overhead.

RTMPose supports 17 COCO or 26 Halpe body keypoints, 133 COCO-WholeBody keypoints, 106 face keypoints, 21 hand keypoints, and 17 animal keypoints.

Best for: real-time pose estimation on CPU and edge hardware, especially when a 30 FPS video stream needs to run without a dedicated GPU.

ViTPose / ViTPose++ (ViTAE-Transformer)

NeurIPS 2022 / TPAMI 2023 · Apache-2.0, commercial use permitted

ViTPose uses a plain Vision Transformer backbone with minimal modification, while ViTPose++ adds multi-dataset training. Its role in this list is accuracy at the high end rather than CPU deployment. At 256x192, ViTPose-B reaches 75.8 COCO AP in single-task training and 77.1 with COCO, AIC, and MPII combined. ViTPose-L reaches 78.3 and 78.7, while ViTPose-H reaches 79.1 and 79.5. ViTPose-H also reports 76.3 AP on CrowdPose test, including 65.6 AP on the hard subset. MPII PCKh reaches 94.1 for H, while ViTPose-L reports 90.9 AP and 92.2 AR on OCHuman test.

The RTMPose comparison explains the deployment split. ViTPose-B and RTMPose-m both report 75.8 COCO AP, but RTMPose-m reaches that score at 90+ FPS on an i7-11700 CPU, while ViTPose-B is positioned for GPU use. ViTPose earns its place when you want to push further up the accuracy range, reaching 79.1 AP with the H backbone and 76.3 AP on CrowdPose.

Best for: GPU-server pose estimation where maximum published accuracy matters more than CPU throughput, especially in dense multi-person scenes.

MediaPipe Pose Landmarker / BlazePose GHUM 3D (Google)

MediaPipe framework and repository: Apache-2.0. Google does not publish an explicit licence statement for the Pose Landmarker model weights on the current model page. Documentation is CC-BY 4.0 and code samples are Apache-2.0.

MediaPipe Pose Landmarker, based on BlazePose GHUM 3D, is the browser-first option in this group. It ships in Lite, Full, and Heavy float16 variants and supports Android, iOS, Python, and Web through JavaScript. Input modes include IMAGE, VIDEO, and LIVE_STREAM, which makes it the most direct fit here for continuous pose estimation inside browser or mobile applications.

The model outputs 33 body landmarks from RGB input. Google's BlazePose GHUM 3D material describes these as 3D landmarks and also describes a background segmentation mask. The pose detector uses a 224x224x3 input and the landmarker uses 256x256x3. Google does not publish model sizes or latency figures for the three variants.

The weight licensing deserves separate treatment from the code licence. Google's repository is Apache-2.0, and third-party redistributions such as OpenCV Zoo and Qualcomm AI Hub also label the pose model Apache-2.0, but Google's own Pose Landmarker page does not state a licence for the .task model bundles.

Best for: browser and mobile video applications that need LIVE_STREAM support and 3D body landmarks from a single RGB stream, while teams that require explicit model-weight licensing should confirm the terms of the downloaded .task bundle before commercial deployment.

DWPose (IDEA Research)

ICCV 2023 CV4Metaverse Workshop · Apache-2.0, commercial use permitted

DWPose focuses on whole-body estimation with body, foot, face, and hand keypoints in one system. At 256x192, DWPose-t uses 0.5G FLOPs and reaches 48.5% Whole AP. DWPose-s uses 0.9G FLOPs and reaches 53.8%, DWPose-m uses 2.2G FLOPs and reaches 60.6%, and DWPose-l uses 4.5G FLOPs and reaches 63.1%. Increasing DWPose-l to 384x288 raises compute to 10.1G FLOPs and Whole AP to 66.5%.

Its best-model component scores are 72.2% for body, 70.4% for foot, 88.7% for face, and 62.1% for hand. The hand score is the weakest part of the published breakdown. Inference speed is not stated in the repository.

Do not compare these Whole AP figures directly with RTMPose or ViTPose COCO AP. DWPose is measured on COCO-WholeBody, while the headline RTMPose and ViTPose figures above use 17 body keypoints on standard COCO. Whole-body AP is a different and harder benchmark.

RTMPose-l also reports 67.0% AP on COCO-WholeBody at 130+ FPS, which puts it in the same general range as DWPose-l's 66.5%.

Best for: whole-body research where the face, hands, feet, and body need separate published component scores rather than only a body-pose metric.

OpenPose (CMU Perceptual Computing Lab)

Non-commercial only. OpenPose is free for non-commercial use, while commercial use requires a separate CMU licence through FlintBox.

OpenPose is the 2017 reference that helped establish modern multi-person pose estimation. It supports body and foot layouts with 15, 18, or 25 keypoints, including 6 foot keypoints, plus 21 keypoints per hand and 70 for the face, for 135 whole-body keypoints in total.

Supported platforms are Ubuntu, Windows, macOS, and Nvidia TX2. Available build types include CUDA, OpenCL, and CPU-only.

Its enduring technical property is that inference time stays constant as the number of detected people increases, while competing approaches can scale linearly with person count. MediaPipe and RTMPose now provide faster practical deployment paths, and the non-commercial licence rules OpenPose out for standard commercial products.

Best for: historical reference, research, or workloads where constant inference time across changing crowd sizes is specifically useful.

Best Speech & Audio Track

Video analysis that ignores the audio track misses information the pixels cannot provide. Base Whisper gives you what was said and when it was said. Speaker attribution is a separate problem and needs an added diarization layer.

Whisper (OpenAI)

MIT licence, commercial use permitted

Whisper turns a video's audio track into timestamped text. For video, those timestamps are the key feature: they let a viewer jump to the moment a phrase was spoken, align speech with visual detections on the same timeline, and make transcript search part of a larger video search system. Transcript plus detections plus timestamps is the basic pattern behind searchable video.

Whisper takes audio rather than video, so the audio track normally needs to be demuxed first with a tool such as ffmpeg.

Whisper has a 30-second input limit for the short-form path. Audio of 30 seconds or less can run as a single short-form transcription. Anything longer uses a sequential long-form algorithm based on timestamp predictions, with compression-ratio, log-probability, and temperature fallback heuristics described in section 3.8 of the Whisper paper. Almost every video audio track exceeds 30 seconds, so video workloads normally use this long-form path.

For audio longer than 30 seconds, return_timestamps=True is required. Timestamp precision defaults to 0.02 seconds, or about 20 ms. That is what makes it possible to align a spoken phrase with a specific point in the video rather than returning one untimed block of text.

Whisper also exposes a no_speech_threshold so segments containing only silence can be skipped. That matters for surveillance recordings, lectures, and other footage with long quiet stretches, and it helps reduce text hallucinated over silence.

The model family ranges from tiny at 39M parameters to large at 1550M. Turbo is an optimised version of large-v3 and remains the practical default among the reference models: 809M parameters, about 6 GB of VRAM, and roughly 8x the relative speed of large.

Whisper was trained on 680,000 hours of audio: 438,000 hours of English, 126,000 hours of non-English audio paired with English transcripts, and 117,000 hours of non-English audio paired with native-language transcripts. The training data covers 98 languages, but the model card itself reports strong ASR results in about 10 languages. Training coverage should therefore not be read as equal transcription quality across all 98.

Best for: extracting timestamped speech from video so spoken content can be searched and aligned with visual events.

faster-whisper

MIT licence

faster-whisper is the practical production runtime for Whisper when speed and memory matter. It is built on CTranslate2 and supports word_timestamps=True as well as batching through BatchedInferencePipeline. Its project claims up to 4x faster inference than openai/whisper at the same accuracy while using less memory.

On an RTX 3070 Ti 8GB with CUDA 12.4, using large-v2 and beam size 5, the published figures are:

On the same GPU, the int8 configuration cuts VRAM by 38% and runtime by 2.4x compared with the OpenAI reference implementation.

The CPU difference is also substantial. With the small model on an Intel i7-12700K using 8 threads:

Best for: production transcription pipelines that want Whisper compatibility with lower latency, lower memory use, batching, and word-level timestamps.

WhisperX

BSD-2-Clause

WhisperX adds the pieces base Whisper does not provide for speaker-aware video transcripts. It uses forced alignment with wav2vec2 for word-level timestamps and pyannote-audio for speaker diarization. Its project reports 70x realtime transcription with whisper large-v2 using batched inference.

This is the answer to the “who said what” gap, but it comes with deployment friction. Speaker diarization uses the speaker-diarization-community-1 model and requires a Hugging Face read token plus acceptance of that model's user agreement. Base Whisper gives you what was said and when. WhisperX with pyannote adds who said it.

Best for: meeting, interview, and support-call video where the transcript needs word-level timing and speaker attribution rather than timestamps alone.

For a deeper comparison, see Eden AI's article on free and open-source speech-to-text options.

Libraries and Frameworks You'll Actually Build With

OpenCV, PyTorch, and similar tools are often listed as “video analysis models” in articles like this one. They are not models. They are the layer you build with, and for a production video pipeline they matter just as much as the model choice itself.

Supervision (Roboflow) is the glue layer most lists omit. It is MIT-licensed, actively maintained, and has 49.4k GitHub stars. It provides multi-object tracking, customisable annotators, real-time zone counting, dwell-time analysis, and frame-by-frame video utilities. It integrates with Ultralytics, Transformers, MMDetection, Roboflow Inference, RF-DETR, and model-agnostic connectors. If you are combining detection, tracking, overlays, counters, and video I/O, this is often the layer that turns separate model outputs into an application.

OpenCV handles the lower-level mechanics: decoding video, reading and writing frames, and running classical computer vision operations around your model.

PyTorch is the training and inference framework behind nearly all of the models covered in this article. It is where you load weights, run inference, fine-tune models, and connect model code to the rest of your pipeline.

MMPose and MMAction2 are OpenMMLab toolkits for pose estimation and action recognition. RTMPose sits inside this ecosystem, alongside other video-focused research and deployment components.

Decord and PyAV handle video decoding. That can become a bottleneck once you process many files or long recordings, so decoding belongs in the architecture discussion, not as an afterthought.

How to Choose: Self-Host or API?

Start from your volume

Low and uneven usage usually favours an API. You pay for the video you process and do not need to keep GPU capacity running when nothing is happening. That is useful for prototypes, internal tools, and products where traffic changes sharply from one day to the next.

Sustained high volume changes the economics. A fixed GPU cost can become cheaper than paying per minute once usage passes your own break-even point. Do not rely on a generic threshold. Take your expected monthly video minutes, compare them with the per-minute API prices in the previous section, then compare that total with the full cost of running your own inference stack.

Check your latency requirement

Real-time video pushes you toward smaller local models. A 30 FPS stream gives you about one frame every 33 ms, and that budget also has to cover decoding, tracking, application logic, and any network overhead.

YOLO26n takes 1.7 ms per frame on a T4, while the tested CPU figure is 38.9 ms. The GPU result leaves room for real-time processing and concurrent streams. The CPU result does not keep pace with a 30 FPS stream on its own.

Batch and overnight workloads remove much of that constraint. If a recording can take minutes or hours to process, larger video-language models become practical because throughput matters more than per-frame latency.

Decide where your data can go

Privacy and infrastructure rules can override the cost calculation. Regulated data, on-premise requirements, or footage containing identifiable people may prevent you from sending raw video to a third-party service.

In those cases, self-hosting may be the only workable choice even if the API would otherwise cost less. If EU processing is sufficient for your requirements, Eden AI also provides an EU endpoint.

Be honest about your team

A downloadable model is not a finished video system. Self-hosting needs someone to own GPU inference, decoding, batching, monitoring, model updates, failure handling, and the surrounding tracking or indexing pipeline.

If nobody on the team is responsible for that infrastructure, the model can be free and the project can still fail. An API costs more per unit of usage, but it also removes part of that operational burden.

A common production pattern is hybrid: run a small local detector continuously, then send only selected frames or clips to a video-language model or API when deeper reasoning is needed. This keeps the always-on path cheap while reserving expensive analysis for the moments that justify it. In many 2026 deployments, that is the lowest-cost design that still meets the product requirement.

FAQs - Free AI Video Analysis