Summarize this article with:
AWS launched cross-Region inference routing for Amazon SageMaker AI on November 18, 2026. It moves traffic between SageMaker endpoints you already run in other Regions and accounts when platform faults, throttles or container 5xx errors occur. Client 4xx errors and slow responses do not trigger it. Cross-provider fallback covers a different failure: a model or vendor that is unavailable. This comparison explains what each layer protects, what stays your responsibility, and when to use both.
On November 18, 2026, AWS launched cross-Region inference routing for Amazon SageMaker AI. One inference router sends requests to SageMaker endpoints you already run in other AWS Regions and accounts when local endpoints fail or run short of capacity. AWS says teams previously had to build and maintain this routing, health checking and retry logic themselves.
The key detail is what triggers a move. According to AWS's documentation, platform faults, throttles and container 5xx errors shift traffic. Client 4xx errors and slow model responses do not, and a request that times out is not retried elsewhere.
This protects your own SageMaker deployments from regional failures. It does not cover a model or vendor outage, which is the job of cross-provider fallback. This article compares the two layers, what each leaves to you, and when to combine them.
| Launched | November 18, 2026, by AWS |
|---|---|
| Routes between | Existing SageMaker endpoints in other Regions and accounts |
| Routing order | In-Region endpoints first |
| Shifts traffic on | Platform faults, throttles, container 5xx errors |
| Does not shift on | Client 4xx errors, slow model responses |
| Availability | 32 commercial Regions (real-time), 25 (asynchronous) |
| Client change | New router runtime service and invoke APIs |
| Router charge | None; endpoints and CloudWatch metrics billed as usual |
Source: AWS, November 18, 2026.
Cross-Region routing and cross-provider fallback at a glance
SageMaker routing is for teams serving their own or custom models on SageMaker endpoints. Cross-provider fallback matters more for applications that call managed model APIs. The table compares both layers and what each leaves to you.
| Question | SageMaker routing | Provider fallback | Left to you |
|---|---|---|---|
| Failure covered | Regional endpoint fault or capacity shortage | Model or provider outage | Map each failure mode to a layer |
| Model served | The same deployed model | Possibly a different model | Test fallback outputs |
| Targets | Endpoints you already deployed | Models your policy allows | Approve every target first |
| Triggers | Platform faults, throttles, container 5xx | Set by the gateway or application | Handle 4xx errors and slow responses |
| Request modes | Real-time and asynchronous | Depends on the implementation | Check your request mode |
| Scope | Chosen Regions and accounts, same AWS partition | Approved providers and destinations | Enforce residency rules |
| Cost | No router fee; endpoints and metrics billed | Depends on provider pricing | Budget standby capacity |
Source for the SageMaker column: AWS, November 18, 2026. Checked November 19, 2026.
The router itself adds no fee, but the resilience is not free. Target endpoints in each Region are billed at standard SageMaker AI pricing, and router metrics at standard Amazon CloudWatch rates. The router never creates targets: they must already exist.
How SageMaker cross-Region inference routing works
You create an inference router, register the real-time endpoints, asynchronous endpoints or inference components you already run, and point your application at the router. It prefers endpoints in its own Region. As errors rise, it shifts new requests to healthy destinations in other Regions, then returns traffic gradually once the local endpoint recovers.
Endpoints, deployed models, containers and request bodies stay the same. So do ContentType, Accept, CustomAttributes and InferenceId. The client does change: it calls the sagemaker-inference-router-runtime service instead of sagemaker-runtime.
This pseudocode shows the change. Check the exact SDK method names in the SageMaker AI Developer Guide before you copy it.
Python
# Pseudocode: check method names in the AWS SDK reference
# Before: invoke one endpoint directly
runtime = aws_client("sagemaker-runtime")
runtime.invoke_endpoint(
EndpointName="my-endpoint",
Body=payload,
ContentType="application/json",
)
# After: invoke a routing target through the router
router = aws_client("sagemaker-inference-router-runtime")
router.invoke_inference_router(
InferenceRouterName="my-router",
RoutingTargetName="production",
Body=payload,
ContentType="application/json",
)
Synchronous clients call InvokeInferenceRouter, and asynchronous clients call InvokeInferenceRouterAsync.
Endpoint owners keep control of cross-account use. A resource-based policy with the sagemaker:SourceInferenceRouterArn condition decides which routers in other accounts can call an endpoint. An optional cross-Region traffic cap reserves capacity for the endpoint's local callers.
Neither control decides where your data goes: only the targets you configure do. Destinations must also sit in the same AWS partition as the router, so the router does not fail over from commercial AWS to AWS GovCloud.
Failures that do and do not move traffic
According to AWS's documentation, the router retries a failed attempt on another destination only for retriable errors. Examples are a throttle, or a container 5xx error before any response output begins. Platform faults also count against a destination's health.
Timeouts work differently. A request that times out is not retried, though repeated timeouts lower the destination's health and divert later requests.
Client 4xx errors and slow model responses never trigger failover. Routing also needs a healthy configured target with spare capacity. If every target is full or capped, the router has nowhere to send the request.
Each real-time response names the endpoint that served it and a RoutingReason: local-preferred, capacity-available, only-healthy or failover. Log these fields to see how often traffic leaves its home Region, alongside the router metrics in Amazon CloudWatch.
Cross-provider fallback at the application or gateway layer
Cross-provider fallback covers a different failure: the model or vendor itself. Your application or a gateway sends the request to another approved model when the preferred one is unavailable. That can mean an outage, a capacity limit, or even an account issue, as in this guide on what happens when an AI provider bans you.
The tradeoff is consistency. SageMaker routing serves the same deployment in another Region, while a fallback model may differ in output structure, tool support, safety behavior, latency and quality. Test each fallback against the same acceptance criteria as the primary: a matching request format does not mean matching behavior.
A gateway also joins the request path. It does not fix invalid requests, application bugs or unsafe outputs. Because data may reach a different provider or processing location, decide which destinations are allowed before you turn on automatic fallback.
The broader case for this layer is in our article on not relying on a single AI provider. For the integration work behind several providers, see how to handle complex AI integration.
Which failover strategy should you choose?
Choose the layer that matches the failure you need to survive. Many production systems need both.
Recommendations
Match the routing layer to the failure
SageMaker
Custom models on SageMaker
Use cross-Region routing when the main risk is an unhealthy endpoint or a Region short of capacity. Deploy, size and approve every target first.
Provider layer
Managed model APIs
Use provider fallback when your application must keep working after a model or vendor becomes unavailable. Test each fallback model on its own.
Both layers
High-availability applications
Combine them when you serve custom models on SageMaker and also call managed APIs. Each layer covers failures the other cannot see.
Governance
Residency-sensitive workloads
Register only acceptable AWS Regions and accounts as targets. Traffic caps and endpoint policies are not residency controls. Check each external provider's processing terms separately.
For chat or streaming workloads, check which request modes the router supports in the SageMaker AI Developer Guide before you move streamed traffic onto it.
A two-layer routing design
Application
Sends the request to its routing logic or gateway.
Provider layer
Picks a managed model API or your SageMaker model, with approved backups.
SageMaker router
For SageMaker traffic, picks a healthy configured endpoint.
Serving endpoint
Returns the response with the endpoint and routing reason.
Logs
Record the provider, endpoint and RoutingReason.
Where Eden AI fits
Eden AI sits at the provider layer, for the managed model APIs your application calls. With provider routing, you choose the order of providers, or sort them by price, speed or latency. Automatic fallback to backup models is a separate setting. Our post on rate limits and fallbacks in Eden AI API calls covers how the two work in practice.
This request sets an explicit provider order. Configure backup models as described in the fallback documentation.
Python
from openai import OpenAI
client = OpenAI(
api_key="YOUR_EDENAI_API_KEY",
base_url="https://api.edenai.run/v3",
)
response = client.chat.completions.create(
model="YOUR_PRIMARY_MODEL",
messages=[
{"role": "user", "content": "Summarize this incident."}
],
extra_body={
"routing": {
"order": ["YOUR_FIRST_PROVIDER", "YOUR_SECOND_PROVIDER"]
}
},
)
print(response.choices[0].message.content)
For European workloads, Eden AI offers an EU endpoint for European data residency, with zero data retention on the EU endpoint. You still need to check that each provider you allow meets your own processing requirements.
Eden AI does not deploy SageMaker endpoints or replace the SageMaker router. It is also one more service in the request path, so it does not remove every single point of failure. Regional resilience for your custom models stays with SageMaker.
What to keep in mind
- Provision targets first: SageMaker routing needs deployed, healthy destinations with spare capacity, and each one is billed.
- Latency is not a trigger: slow responses never cause failover, and a timed-out request is not retried.
- Check coverage: real-time routing covers 32 commercial Regions and asynchronous routing 25, all within one AWS partition.
- Validate every destination: review Regions, accounts, traffic caps, fallback-model behavior and data policies before production use.
Measure before you rely on it
AWS has not published the latency the router or an inter-Region hop adds. Compare direct and routed calls with your own payloads and target Regions before setting timeouts.
FAQ
What triggers SageMaker cross-Region routing?
Does slow inference trigger SageMaker failover?
Do clients need code changes to use the SageMaker inference router?
Do target endpoints need to exist already?
How does cross-account routing work in SageMaker?
Does the SageMaker inference router support streaming?
What does SageMaker cross-Region routing cost?
Can SageMaker routing and provider fallback be combined?
Further reading
- AWS announcement of cross-Region inference routing: availability, routing behavior and pricing, from AWS.
- Should you switch to SageMaker KV cache-aware routing?: a different SageMaker routing feature, aimed at inference efficiency.
- Eden AI is available on AWS Marketplace: how teams buying through AWS can access Eden AI.
- How to handle complex AI integration: what changes when an application uses several AI services.
- What to do when the OpenAI API goes down: a concrete case of a provider-level outage and how to prepare.



