Tutorial
Video Processing
8 min reading

How to Detect Labels (Objects) in Videos Using Python

Summarize this article with:

summary
  • Label detection automatically identifies and categorizes objects, people, and scenes in videos or images.
  • Developers can Detect Labels (Objects) in Videos Using Python using Eden AI's unified REST API with a single API key across all supported providers.
  • Eden AI lets you experiment with various AI models before integration, making it easy to compare and choose the provider that best fits your needs.
  • Label detection in videos now goes through a multimodal LLM: you send the video URL and get the labels back as JSON in the same response.
  • Each label comes with the start and end times where it appears in the video.

Label Detection is a key task in video and image analysis that involves identifying and categorizing elements within each frame such as objects, people, animals, or landmarks.

This helps with content tagging, indexing, and search optimization. In this tutorial, we’ll show you how to implement label detection using Python and the Eden AI API.

With just a few lines of code, you can run label detection and extract meaningful insights from visual content.

‍

What is Label/Object Detection?

__wf_reserved_inherit

Label detection automatically identifies and categorizes objects, people, and scenes in videos or images. It assigns labels like animals, buildings, or landmarks to visual elements, making content easier to search, organize, and analyze.

‍

How to Detect Labels in Videos

‍

Get Access to Eden AI API

1. Sign up: Visit Eden AI and create an account for free to get started. Once registered, navigate to the API section to find your personal API key which gives you access to a wide range of AI services, including Label Detection.

__wf_reserved_inherit

2. Navigate to Video Technologies – Log in and head to the Video Technologies section.

3. Select Label Detection– From there, select the Label Detection option.

‍

Live Test Models on Eden AI

Eden AI lets you experiment with various AI models before integration, making it easy to compare and choose the provider that best fits your needs.

‍

Implementing Label Detection in Python

‍

Step 1: Install Python Requests Module

Before we start, make sure you have the requests module installed. This module allows your Python script to make HTTP requests to APIs like Eden AI.


pip install requests

This module is required to send HTTP requests to the Eden AI API.

‍

Code Explanation

Label detection in videos now goes through a multimodal LLM that accepts video as input, called with Eden AI’s OpenAI-compatible chat completions endpoint. It works in two steps:

  1. Sending the request: you send the video URL, your instruction and the JSON format you expect for the answer.
  2. Reading the results: the labels come back in the same response, as JSON, so there is no job to poll.

A 10-second clip is processed in a few seconds. For long videos, set a generous timeout on your HTTP client.

‍

Sending the Label Detection Request

This part of the code sends the video and the instruction to Eden AI.



import requests

# Replace with your actual API Key
headers = {"Authorization": "Bearer {API_KEY}"}

# The JSON format expected for the answer: each label with the time range where it appears
schema = {
"type": "object",
"properties": {
"labels": {
"type": "array",
"items": {
"type": "object",
"properties": {
"label": {"type": "string"},
"start_time": {"type": "number"},
"end_time": {"type": "number"},
},
"required": ["label", "start_time", "end_time"],
},
}
},
"required": ["labels"],
}

# Replace with the actual URL of your video
json_payload = {
"model": "google/gemini-2.5-flash", # Any model that accepts video input
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "List the objects, people and scenes in this video, with start and end times in seconds."},
{"type": "image_url", "image_url": {"url": "🔗 URL of your video"}},
],
}
],
"response_format": {"type": "json_schema", "json_schema": {"name": "labels", "schema": schema}},
}

# Eden AI's OpenAI-compatible chat endpoint
url = "https://api.edenai.run/v3/chat/completions"

# Send the POST request
response = requests.post(url, json=json_payload, headers=headers)

# Get the response and display it
result = response.json()
print(result)

Explanation:

  • headers: Contains your API key to authenticate with Eden AI.
  • url: Eden AI’s chat completions endpoint, https://api.edenai.run/v3/chat/completions.
  • json_payload:
    • model: The multimodal LLM to use, in the provider/model format. It must accept video: check for video in input_modalities at https://api.edenai.run/v3/models (for example google/gemini-2.5-flash or amazon/amazon.nova-2-lite-v1:0).
    • messages: Your instruction in a text block, and the public URL of your video in an image_url block.
    • response_format: The JSON schema the answer must follow, here a list of labels with start and end times.
  • response: This POST request sends the video and your instruction to Eden AI.
  • result: The full response. The answer is in choices[0].message.content, the cost of the request in cost and the provider that served it in provider.

‍

Reading the Results

The labels are in the same response. This part of the code reads them from the result of the previous step.



import json

# The labels are in the model's answer, as a JSON string
labels = json.loads(result["choices"][0]["message"]["content"])["labels"]
for item in labels:
print(f'{item["label"]}: {item["start_time"]}s to {item["end_time"]}s')

print("Cost:", result["cost"], "USD")

Explanation:

  • json.loads: The answer is a JSON string that follows your schema, so it is turned into a Python dictionary.
  • The loop prints each label with the time range where it appears, then the cost of the request.

‍

Interpreting the Results

Here’s an example of the model’s answer, read from choices[0].message.content:



{
"labels": [
{"label": "Snowy mountain landscape", "start_time": 0.0, "end_time": 9.0},
{"label": "Red-haired character", "start_time": 0.0, "end_time": 9.0},
{"label": "Large muscular character", "start_time": 1.0, "end_time": 9.0},
{"label": "Man with a knife", "start_time": 6.0, "end_time": 7.0}
]
}

Explanation of Output Fields:

  • label: Name of the detected object, person or scene.
  • start_time / end_time: When the label appears in the video, in seconds.

The timestamps are estimated by the model: check them on a few of your own videos before relying on them in production.

‍

Managing Your Video Files

To analyze a video stored on your machine, upload it first with POST https://api.edenai.run/v3/upload, then send a file block with the returned file_id instead of the image_url block. Eden AI provides additional optional endpoints to manage these files:

Video Files - List Files

GET Request:
https://api.edenai.run/v3/upload
This lists the files you uploaded, with their file_id and expiration date.

Video Files - Delete Files

POST Request:
https://api.edenai.run/v3/upload/delete
Send the list of file_ids you no longer need in the file_ids field to delete them.

‍

Why Eden AI is the Best Tool for Label Detection

Eden AI offers a robust Label Detection solution with several advantages.

__wf_reserved_inherit

‍

Multiple Provider Options

Eden AI lets you compare multimodal models from providers like Google and Amazon by changing a single parameter, model.

Ease of Integration

Eden AI simplifies the integration of label detection into your applications with clear and straightforward Python code examples.

Security and Trust

‍The Eden AI platform prioritizes data security and privacy, ensuring that your image data is handled securely.

‍

Conclusion

Label detection adds powerful visual intelligence to your applications, and Eden AI makes it easy.

With just a few lines of Python and a simple two-step process, you can analyze video content using top AI providers like Google and Amazon.

Eden AI’s unified API, OpenAI-compatible endpoint, and multi-provider flexibility make it an ideal solution for scalable video analysis, whether you're working on media apps, security systems, or content moderation tools.

FAQ — Detect Labels (Objects) in Videos

You need an API key from your chosen AI provider. Eden AI lets you access multiple providers with a single key, removing the need for separate vendor accounts.
Any language that supports HTTP requests works — Python, JavaScript, PHP, Ruby, Go, and more. Ready-to-use code snippets are available for the most common languages.
Most developers complete a basic integration in under an hour using standardized API endpoints and ready-to-use code examples.
Implement exponential backoff for rate limit errors and use try-catch blocks for network failures. Eden AI's built-in fallback routing automatically redirects requests if a provider is unavailable.
Eden AI supports GDPR-compliant provider filtering and does not store or reuse your data, ensuring compliance with European privacy regulations.

Similar articles

An operator steers a thread of prompt blocks toward the GPU tower whose cache blocks are still lit, away from a tower where they have faded.
Tutorial
Generative AI
Should You Switch to SageMaker KV Cache-Aware Routing?
11/17/2026
·
Written byClément Moreau
Fireworks AI and Notion logos connected by a simple data path, representing third-party AI data exposure.
Tutorial
Generative AI
Fireworks AI Breach: How to Limit Third-Party AI Data Exposure
11/12/2026
·
Written byClément Moreau
Tutorial
Generative AI
From Shadow AI to Governed AI in 30 Days: A Practical Playbook
10/28/2026
·
Written byTaha Zemmouri
let’s start

Start building with Eden AI

A single interface to integrate the best AI technologies into your products.