The Gap in Shared Understanding of LLM Capabilities: What "Supports" Means in Production

Ask four people whether a model "supports tool calling" and you can get four different answers, all of them technically true.

The model card says the model was trained for it. The provider's documentation says the endpoint accepts a tools field. A gateway catalogue sets a capability flag to true. And the model, on the day you ship, may or may not actually call your tool.

That is the gap in shared understanding of LLM capabilities. Developers read "supports" as "works the way I expect, every time". In production, it often means "the request will not be rejected". The difference only shows up when a feature breaks, usually without an error.

To measure it, we ran the same five requests against 44 endpoints: four popular open-weight model families, each available through 10 to 13 provider routes, all called through Eden AI's API on 16 October 2026. Eden AI does not host or train these models: it gives developers one API to the providers that serve them, which is exactly what makes a side-by-side test like this possible. This article walks through what we found and how to test for it before your users do.

Four sources of truth that rarely agree

When a team decides whether a model can handle a feature, it usually relies on one of four sources:

• The model card: what the lab trained and evaluated. It describes the weights, not a running service.
• The provider's documentation: what one inference service says its endpoint accepts. Two providers serving the same weights can document different parameters.
• A catalogue or gateway listing: a normalised view across providers, useful for discovery, but only as accurate as the declarations behind it.
• The observed behaviour: what actually comes back from a real request.

The first three are declarations. Only the fourth is evidence. They drift apart for ordinary reasons: providers serve different checkpoints or dated versions of the same model, use different quantisation, run different serving stacks, and enable or disable features such as constrained decoding or multimodal input independently of the model itself.

What we tested

We picked four model families that many providers serve: GLM-5.3, DeepSeek V4.1 Flash, gpt-oss-120b and Kimi K3. For each provider route, we sent five requests:

We classified each result in one of three ways. It worked. It failed with an explicit error. Or it failed silently: an HTTP 200 response that looked normal but did not do what was asked. Routes that were rate-limited or timed out during the test window are counted as not measured, not as failures. To make sure the gateway itself was not the cause, we reproduced two of these cases directly on the providers' own APIs, without Eden AI in the path, and got the same results.

The results

The headline is not that providers are unreliable. Most routes did what was asked, and streaming worked everywhere. The gap is concentrated in a few features, and in one risky pattern: requests that fail without saying so.

Five places where "supports" and "works" diverge

1. A forced tool call that never happens

This was the clearest gap. We sent one tool, set tool_choice to "required", and asked for a joke. A model that honours the parameter has to call the tool anyway.

On 31 of 40 routes, it did. On 8, the response was a normal HTTP 200 with a joke and no tool call. Only one provider returned an explicit error, stating that tool choice was required but the model did not call a tool.

The same weights behaved both ways. For gpt-oss-120b, seven routes forced the call, four silently ignored the parameter and one rejected the request. For an agent that relies on required to guarantee a structured action, the silent case is the worst one: the code that parses tool_calls receives nothing and has to decide what that means.

2. Structured output that is accepted, then ignored

Strict JSON schema worked on 36 of 42 routes. The other six split evenly. Three providers rejected the parameter with a clear message, one of them even suggesting json_object mode instead. Three accepted it and returned something else: a bare [1], an empty content field, or the model's reasoning written out as plain text.

An explicit refusal is easy to handle: you fall back to another mode or another route. An accepted request with non-conforming output is not, because nothing in the response tells you the schema was not applied.

3. Images sent to a text-only model

GLM-5.3 and gpt-oss-120b are text-only, so we expected every image request to fail. The question was how. Fourteen routes rejected the request with a useful message such as "This model does not support image input". Nine accepted it, and the model simply replied that it could not see any image.

For a chatbot, that answer is merely confusing. For a document pipeline that sends scans to a model and stores whatever comes back, it is a silent data loss.

4. A parameter one provider does not understand

reasoning_effort: "medium" was accepted on 41 of 42 routes. On one provider serving DeepSeek V4.1 Flash, it returned HTTP 400 with the message Invalid request and "param": null. The same request with "none", "low" or "high" worked.

Nothing in that error points to the parameter or lists the supported values. A developer who sees it in production logs has to bisect the request by hand.

5. The model you asked for is not always the model that answered

Two behaviours from the last few weeks complete the picture, both reproducible through public APIs:

• Retired model IDs that keep responding: when a provider retires a model, old IDs can keep returning HTTP 200 for a while, served by the replacement model, which is a sensible way to avoid breaking integrations overnight. The model field in the response still echoes the ID you sent, so the response alone does not tell you which model answered: the provider's deprecation notes do.
• Asking the model is not a check: when we asked several models which model they were, answers included "GPT-4" from a DeepSeek model and "Claude" from another. Self-identification reflects training data, not the serving stack.

Why the gap exists, and why it will not close by itself

None of this requires anyone to be careless. A capability is not a property of a model alone, it is a property of a model, a version, a serving configuration and a provider's request handling, on a given day. Each of those layers changes on its own schedule.

Catalogues help with discovery, but this is where the misunderstanding usually starts. Users read a capability flag as a promise about every request, when it only says what a route is declared to accept. A flag cannot tell you how a given provider handles an edge case, or what happens when a parameter is silently dropped. The only reliable reference is a test you run yourself, on the routes you actually use.

How to close the gap before your users find it

• Write capability tests, not just prompt tests: for each feature your product depends on, keep one small request with a machine-checkable expected result, like the five above. Run them in CI and on a schedule, not only when you first pick a model. Our guide on testing multiple LLMs without rewriting your code shows how to set this up.
• Assert on structure, not on status codes: a 200 is not success. Check that tool_calls exists when you required it, that the content parses against your schema, and that usage is present when you bill on it.
• Treat silent failures as errors in your own code: if tool_choice was required and no tool call came back, raise. If an image was sent and the answer says there is no image, raise.
• Pin dated model IDs where they exist: aliases are convenient, but they can move to a new version or a replacement model without notice.
• Only fall back to equivalent routes: a fallback that switches to a route without the capability you need turns an outage into a silent failure. Match fallbacks on the features you use, not just on the model name.
• Log what you sent and what came back: when behaviour changes, the request, the route and the raw response are what let you see it in minutes rather than days.

Here is a minimal version of the forced tool call test, run across every route that serves one model family:

import json, requests
from openai import OpenAI

client = OpenAI(base_url="https://api.edenai.run/v3", api_key="YOUR_EDENAI_API_KEY")

# Every route that serves the same model family, from the public catalogue
models = requests.get("https://api.edenai.run/v3/models").json()["data"]
routes = [m["id"] for m in models if m["id"].lower().endswith("glm-5.3")]

tools = [{"type": "function", "function": {
    "name": "get_weather", "description": "Get the weather for a city",
    "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}]

for route in routes:
    try:
        r = client.chat.completions.create(model=route, tools=tools, tool_choice="required",
                                           messages=[{"role": "user", "content": "Tell me a joke."}])
        calls = r.choices[0].message.tool_calls or []
        print(route, "works" if calls else "SILENT FAILURE: no tool call")
    except Exception as e:
        print(route, "explicit error:", str(e)[:80])

How Eden AI helps you measure the gap

Eden AI does not make the gap disappear, and we would not claim it does. It does not host the models either: it connects you to the providers that serve them, behind one API. What it changes is the cost of measuring the gap, and of acting on what you find.

• One API for every route: with one key and one request format, testing the same feature across ten providers is a loop over model strings, as in the snippet above, rather than ten integrations to build and maintain.
• Declared capabilities in one place: the public /v3/models endpoint lists every route with its declared capabilities, regions and data retention, so you can build your test matrix from it. The same information is filterable in the Eden AI model catalogue.
• Pin a route, or let Eden AI pick one: call provider/model to pin a route that passed your tests, or request a model by name alone and let Eden AI choose the provider for each call when the difference does not matter for that feature.
• Switch without a migration: when a route regresses, moving traffic to one that passed is a change of model string, and fallbacks keep requests flowing while you do it.

The declarations are a starting point. The tests are what you ship on. To run them on your own routes, get your Eden AI API key and start with the snippet above.

Frequently asked questions

What does "supports tool calling" actually mean?

It usually means the endpoint accepts a tools field without rejecting the request. It does not guarantee that the model will call a tool when asked, that it honours tool_choice, or that the arguments will be valid JSON. Those behaviours depend on the model, the version and the provider's serving stack.

Why does the same model behave differently across providers?

Providers can serve different versions or quantisations of the same weights, use different inference engines, and implement features such as constrained decoding, tool parsing or multimodal input differently. Two endpoints with the same model name are two different services.

What is a silent failure in an LLM API?

A response that returns HTTP 200 and looks normal but does not do what the request asked: a required tool call that is missing, a JSON schema that was not applied, or an image that was dropped before reaching the model. Silent failures are harder to catch than errors because nothing in the status code signals them.

How often should capability tests run?

At least whenever you change a model, a provider or a model alias, and on a regular schedule in production. Provider behaviour changes without a release note on your side, so a weekly or daily run catches regressions before users do.

Can I rely on a model catalogue for capabilities?

Use it to shortlist routes and build your test matrix, not as a guarantee. A capability flag says what a route is declared to accept, not how every request will behave. Confirm each feature you depend on with a test on the exact route you use.

Does Eden AI host the models it gives access to?

No. Eden AI is an API layer: it connects you to the providers that host and serve each model, and lets you call any of them with the same key and request format. Which provider serves a request is visible in the model string you send, or chosen by Eden AI when you request a model by name alone.