LLM Proxy vs OpenTelemetry: Do You Need Your Own AI Gateway?

Self-hosted gateways, managed AI gateways and OpenTelemetry solve different parts of the same problem. Here's how to decide what your stack actually needs.

On October 18, 2026, developer Jared Siegel published a short post arguing that anyone who wants to understand their AI workloads should run their own LLM proxy. Its title, softened on Hacker News to "If you don't run your own LLM proxy, how do you know what's going on?", is deliberately provocative. The frustration behind it is familiar: provider dashboards show how much you used a service, not the request-level detail you need to debug a production application or a coding agent.

Siegel lists five reasons to run a proxy: one endpoint for multiple providers and models, logs of which key called which model with which client and when, virtual API keys, infrastructure you control, and the ability to switch models or providers without rebuilding your setup. He also says he can pinpoint the cause of a cache miss during a coding session in about 20 seconds by reading his proxy's data.

He anticipated the main objection himself: coding agents can already export telemetry through OpenTelemetry. The Hacker News discussion pushed that objection further. One commenter argued that putting a proxy in the request path creates a single point of failure for everyday work. Others replied that telemetry covers only part of what a proxy does, since it cannot set hard limits or reroute traffic.

Both sides have a point. The useful question isn't whether every developer should run a proxy. It's which capabilities your application needs, which layer should provide them, and who is responsible for maintaining that layer.

What an LLM proxy actually does

An LLM proxy sits between your applications and the model providers they call. Instead of connecting every application to every provider, you send requests to one endpoint and the proxy forwards them. The terms "LLM proxy" and "LLM gateway" are often used interchangeably; "gateway" usually implies policy features such as budgets, access rules and routing.

That architecture simplifies integration, but the bigger benefit is what a well-configured gateway can do with the traffic passing through it. Consider a team running several coding agents against different models. Without a shared control layer, provider credentials, usage records and configuration end up scattered across laptops, CI jobs and agent sandboxes, and investigating a single failed request means stitching together several sources of information.

Siegel's five arguments overlap, but each solves a distinct problem.

A single endpoint reduces integration work. Applications use one interface to reach supported models from several providers. Switching models gets easier, but it is rarely fully transparent: tool calling, structured outputs and provider-specific parameters can behave differently from one model to another, a gap we covered in what "supports" really means in production.

Request-level logs make debugging practical. Instead of relying on aggregated usage charts, engineers can inspect individual requests: the key or client, the model, the timestamp, the status, token usage and cost. That helps explain why an application behaved differently than expected, though it won't reveal every decision a provider makes internally.

Virtual keys separate access from credentials. Each application or agent gets its own credential instead of the provider's main API key. Depending on the gateway, that key can be scoped, budgeted, monitored or revoked without touching anything else. This matters most for sandboxed coding agents: the sandbox only ever sees a key you can revoke. It also makes it easier to detect abnormal credential usage.

Owning the infrastructure allows deeper customization. A self-hosted gateway can plug into internal authentication, logging, networking and deployment policies. Siegel, for example, modified his to get better insight and custom MCP endpoints. Every customization, however, becomes something your team has to maintain.

Routing and fallback can improve resilience. A gateway can retry eligible requests or switch providers when a supported failure occurs. Whether that improves reliability depends on the failure mode, the retry policy and model compatibility. Blind retries can raise costs or duplicate side effects, so fallback should be designed rather than simply switched on. Our guide to LLM routing strategies goes deeper on this.

These benefits pay off when an organization has enough AI traffic, users and operational complexity to justify a shared control layer. For a developer making occasional calls to one model, the extra infrastructure is usually unnecessary.

LLM proxy vs OpenTelemetry: visibility is not control

The OpenTelemetry objection is the strongest counterargument in the debate, and it deserves more than a passing mention.

OpenTelemetry (OTel) is an open-source, vendor-neutral framework for generating, collecting and exporting traces, metrics and logs. It is not an observability backend: teams use it to instrument their applications, then send the data to a system that stores, queries and visualizes it. Its GenAI semantic conventions, still marked as in development, standardize how a model call is described, including the provider, the model and token usage.

Coding agents already emit useful signals. Claude Code exports metrics and events through OTel, with distributed tracing in beta. Codex supports OTel export too, but it is off by default and has to be enabled in its configuration file. Once configured, these signals help teams investigate failed requests, latency, tool execution and cost.

For a solo developer who mainly wants to understand a coding session, local logs and OTel may be enough. For a team with an established observability stack, adding a service only to collect data the application already emits is needless complexity.

But telemetry and traffic control are not interchangeable. Telemetry describes what happened. A gateway sits in the request path and can act before a request is forwarded: enforce a budget, block a model, revoke a virtual key or send the request to another provider. OTel helps you detect and investigate a problem; on its own, it doesn't enforce anything.

The line isn't absolute. Applications can enforce limits themselves through hooks or code, and many gateways emit OTel data, which combines traffic control with standardized observability. The real comparison is between the capabilities each architecture provides and the operational cost of providing them.

Telemetry also has blind spots, because it only shows what instrumented components expose. Claude Code, for example, does not pass its OTel settings to the subprocesses it launches, so a tool run through its shell won't report to the same pipeline unless it is configured separately. Provider-side cache decisions and network calls made by tools may not appear either. Teams concerned about broader agent behavior may need network monitoring, sandboxing or a gateway on top of telemetry.

The hidden cost of self-hosting an LLM gateway

A self-hosted proxy gives a team direct control, but it also adds a service to deploy, monitor, update and secure. If applications can't work without it and there is no redundancy or bypass, it becomes a single point of failure, which is exactly the risk raised on Hacker News.

Teams have to plan for availability, capacity, latency, credential storage, access control, backups, upgrades and incident response. Customization is valuable, but custom code and configuration also widen the surface that needs testing and maintenance.

Security is the other side of that responsibility. On March 24, 2026, two LiteLLM releases published to PyPI, versions 1.82.7 and 1.82.8, contained malicious code. According to LiteLLM's incident update, the packages were live for about 40 minutes before PyPI quarantined them, and the team believes the compromise originated from the Trivy security scanner used in its CI/CD pipeline. The payload was a credential stealer that searched for environment variables, SSH keys, cloud credentials, Kubernetes tokens and database passwords.

Users of the official LiteLLM Proxy Docker image were not affected, because that deployment path pins its dependencies. LiteLLM released a clean v1.83.0 through a rebuilt release pipeline on March 30, and its Docker images are now signed.

The incident matters because a gateway holds exactly what attackers want: provider keys and configuration. A compromised component in that position can expose far more than the requests it routes, a risk we explored in when your LLM router turns against you.

The wrong lesson would be that self-hosting LiteLLM is inherently unsafe, or that a managed gateway can't suffer a supply-chain incident. Every component in the AI request path adds dependencies and trust assumptions. Self-hosting makes your organization responsible for operating and securing the deployment; a managed service means evaluating the provider's controls, policies and reliability instead.

Whichever model you choose, a few safeguards apply:

• Pin dependencies to verified versions, and prefer signed release artifacts where they exist.
• Check artifact provenance and scan the software supply chain.
• Limit which processes can read secrets, and keep development and production credentials separate.
• Keep a tested process for revoking exposed keys and investigating suspicious activity.

The question isn't which deployment model eliminates risk. It's whether your team can manage the risks it accepts.

Self-hosted, managed or OTel-only: which architecture fits your team?

Self-hosted proxies, managed gateways and telemetry-only setups are useful reference points, not exclusive choices. Many production systems combine a gateway with OpenTelemetry.

A self-hosted gateway makes sense when control over the deployment, customization and data path outweighs the cost of running it. It suits teams with platform engineers who can operate it reliably.

A managed gateway moves the gateway's deployment and maintenance to a service provider. It fits teams that need multi-provider access and central controls without running another service, in exchange for trusting the provider's availability, security practices, data handling and implementation.

An OTel-first approach is often the simplest starting point when the main need is visibility rather than enforcement. It also complements either type of gateway by giving the application and its dependencies a common telemetry pipeline.

In practice, the right choice depends on who you are:

• Solo developer on one subscription-based coding tool: start with the tool's native logs and telemetry. A proxy adds little, and subscription plans often can't be routed through an arbitrary gateway without unsupported workarounds, a limitation Siegel himself complained about earlier in October.
• Developer experimenting with several models: a lightweight self-hosted proxy is a good learning project, as long as the maintenance burden is acceptable.
• Platform team with strict deployment requirements: self-hosting makes sense when you need direct control over the infrastructure and have the capacity to secure and maintain it.
• Production team using several API providers without dedicated infrastructure capacity: a managed gateway, combined with application-level monitoring and OTel where useful, reduces operational work.
• Organization with complex observability requirements: a hybrid architecture provides gateway-level policies and routing alongside telemetry from applications, coding agents and supporting services.

One principle applies to all five: don't build a proxy because it sounds like the more serious engineering choice. Start from your requirements, identify the layer that can meet them, and don't operate components that don't earn their cost.

Where Eden AI fits

Eden AI is a managed option for teams that want one interface to many AI providers without operating their own gateway infrastructure.

It exposes an OpenAI-compatible API, so existing SDKs and agent frameworks can switch to it by changing the base URL. Behind that endpoint sit LLMs and expert models such as OCR, speech and translation from multiple providers. Gateway features include smart routing and fallbacks, cost-based routing, caching, budget caps per API key that reset daily, weekly or monthly, and the cost of each request returned in its response. Organization management attributes calls to a team, client or product, with roles and model access per key. With Bring Your Own Keys, teams can keep their negotiated provider pricing while Eden AI handles routing and monitoring. As with any gateway, confirm how each feature behaves for your workload before relying on it.

For European data-residency requirements, Eden AI offers a dedicated EU endpoint that routes requests only through EU-eligible providers and never silently falls back to the global endpoint. On data handling, its provider data policies state that Eden AI stores no request or response content by default and does not train on customer data. Retention by the underlying model provider depends on that provider, so end-to-end zero data retention means pinning models and fallbacks to providers that offer it. Its security page covers certifications and controls. Teams should assess all of this against their own configuration, contracts and data-processing requirements.

A managed gateway removes the work of running the gateway, not every operational responsibility. Applications still need sound access controls, monitoring, sensible retry behavior and a plan for service interruptions, and organizations must evaluate both the gateway provider and the model providers behind it.

A managed service is therefore not the universal answer to the proxy debate. It is the option for teams that would rather delegate part of their infrastructure in exchange for relying on a third party.

Visibility is the requirement. The architecture is a choice.

The Hacker News debate points to a real gap in how teams observe and control AI workloads. Provider dashboards alone rarely explain what happened inside an application, especially when several clients, agents and models are involved.

Running a proxy is only one way to close that gap. OpenTelemetry and application logs provide meaningful visibility; a gateway adds central policies, access control and routing; a managed service removes the burden of operating that gateway.

The real mistake isn't choosing the wrong proxy. It's deploying AI systems without knowing what evidence is available, which controls are enforced, and who is responsible when something fails. Answer those questions first, then choose the simplest architecture that answers them reliably.

FAQ