How to Detect and Track Objects in a Video with JavaScript?

In this tutorial, you will learn how to use Video Object Tracking API in 5 minutes using JavaScript. Eden AI provides an easy and developer-friendly API that allows you to track objects in videos.

Getting Started with Video Object Tracking API

The first step is to set Axios, a promise-based HTTP client for the browser and Node.js, that will allow you to call Eden AI API.

‍

Next, you'll need to initialize the File System module in order to access local files on your computer.

‍

Finally, you'll need to create your multipart/formdata parameters form:

‍

How to use Video Object Tracking API with JavaScript?

You are now ready to process your video files into Eden AI Object Tracking API.

‍

1. Get a Object Tracking API Key on Eden AI

To perform Object Tracking in videos, you just need to create an account on Eden AI for free. On the homepage, you will be able to get your API key with free credits offered by Eden AI.

‍

Let’s track objects in your videos

Now that you have initialized the required JavaScript modules and got your API key, you will be able to track objects in videos. Object tracking in videos now goes through a multimodal LLM that accepts video as input, called with Eden AI’s OpenAI-compatible chat completions endpoint: the model lists the objects with the time ranges where they appear, rather than frame-by-frame bounding boxes. You can use any model that has video in its input_modalities: see our documentation here.

‍

Here is the JavaScript code to add your video to the form data and set your API key:

form.append("file", fs.createReadStream("path/to/your/video.mp4"));
const apiKey = "Your_API_Key";

‍

Once the file is added, you can configure your requests: the video is first uploaded to Eden AI, which returns a file_id, then the request asks a multimodal LLM, here google/gemini-2.5-flash, to track the objects and to answer in the JSON format set by the schema:

// Step 1: upload your video to Eden AI, which returns a file_id
const uploadOptions = {
  method: "POST",
  url: "https://api.edenai.run/v3/upload",
  headers: {
    authorization: `Bearer ${apiKey}`,
    ...form.getHeaders(),
  },
  data: form,
};

// Step 2: ask a multimodal LLM to track the objects, with the JSON format of the answer
const schema = {
  type: "object",
  properties: {
    objects: {
      type: "array",
      items: {
        type: "object",
        properties: {
          object: { type: "string" },
          start_time: { type: "number" },
          end_time: { type: "number" },
        },
        required: ["object", "start_time", "end_time"],
      },
    },
  },
  required: ["objects"],
};

const options = (fileId) => ({
  method: "POST",
  url: "https://api.edenai.run/v3/chat/completions",
  headers: {
    authorization: `Bearer ${apiKey}`,
  },
  data: {
    model: "google/gemini-2.5-flash", // any model with video in its input_modalities
    messages: [
      {
        role: "user",
        content: [
          { type: "text", text: "List the main objects in this video, with the time range in seconds where each one appears." },
          { type: "file", file: { file_id: fileId } },
        ],
      },
    ],
    response_format: { type: "json_schema", json_schema: { name: "objects", schema: schema } },
  },
});

‍

Then, you need to create launchJob() function that will execute POST request:

‍

Finally, create the run() function that uploads the video, launches the request and reads the answer. There is no job to poll, as the result comes back in the same response:

const run = async () => {
  const upload = await axios.request(uploadOptions);
  const response = await launchJob(options(upload.data.file_id));
  console.log(response.data);

  // The answer is a JSON string that follows the schema
  const result = JSON.parse(response.data.choices[0].message.content);
  console.log(result.objects);
};

run().catch((error) => {
  console.error(error.response ? error.response.data : error);
});

‍

You will get this response, where the answer is a JSON string in choices[0].message.content:

{
  "status": "success",
  "model": "gemini-2.5-flash",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "{\"objects\": [{\"object\": \"Snowy Mountains\", \"start_time\": 0.6, \"end_time\": 8.5}, ...]}"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 15630,
    "completion_tokens": 2674,
    "total_tokens": 18304
  },
  "provider": "google",
  "cost": 0.0125437
}

‍

Once parsed, this JSON string gives the result for Object Tracking task (the time ranges are estimated by the model, so check them on a few of your own videos before relying on them in production):

[
  {
    "object": "Snowy Mountains",
    "start_time": 0.6,
    "end_time": 8.5
  },
  {
    "object": "Sintel",
    "start_time": 5.5,
    "end_time": 41
  },
  {
    "object": "Old Man",
    "start_time": 12,
    "end_time": 14
  },
  {
    "object": "Villain",
    "start_time": 16.8,
    "end_time": 17.5
  },
  ...
]

Benefits of using Video Object Tracking API with Eden AI

Using Video Object Tracking with Eden AI API is quick and easy.

‍

Save time and cost

We offer a unified API for all providers: simple and standard to use, with a quick switch between providers and an access to the specificic features of each provider.

‍

Easy to integrate

The JSON output format is the same for all suppliers thanks to Eden AI's standardisation work. The response elements are also standardised thanks to Eden AI's powerful matching algorithms.

‍

Customization

With Eden AI you have the possibility to integrate a third party platform: we can quickly develop connectors. To go further and customize your Video Object Tracking request with specific parameters, check out our documentation.

FAQ — Detect and Track Objects in