AI Comparatives
Generative AI
8 min reading

SageMaker Cross-Region Routing vs Cross-Provider Fallback: Which Failures Each Covers

A railway switchyard where a first switch leads to different stations and a second switch leads to identical stations in several towns, showing provider fallback and regional routing as two layers.

Summarize this article with:

summary

AWS launched cross-Region inference routing for Amazon SageMaker AI on November 18, 2026. It moves traffic between SageMaker endpoints you already run in other Regions and accounts when platform faults, throttles or container 5xx errors occur. Client 4xx errors and slow responses do not trigger it. Cross-provider fallback covers a different failure: a model or vendor that is unavailable. This comparison explains what each layer protects, what stays your responsibility, and when to use both.

On November 18, 2026, AWS launched cross-Region inference routing for Amazon SageMaker AI. One inference router sends requests to SageMaker endpoints you already run in other AWS Regions and accounts when local endpoints fail or run short of capacity. AWS says teams previously had to build and maintain this routing, health checking and retry logic themselves.

The key detail is what triggers a move. According to AWS's documentation, platform faults, throttles and container 5xx errors shift traffic. Client 4xx errors and slow model responses do not, and a request that times out is not retried elsewhere.

This protects your own SageMaker deployments from regional failures. It does not cover a model or vendor outage, which is the job of cross-provider fallback. This article compares the two layers, what each leaves to you, and when to combine them.

LaunchedNovember 18, 2026, by AWS
Routes betweenExisting SageMaker endpoints in other Regions and accounts
Routing orderIn-Region endpoints first
Shifts traffic onPlatform faults, throttles, container 5xx errors
Does not shift onClient 4xx errors, slow model responses
Availability32 commercial Regions (real-time), 25 (asynchronous)
Client changeNew router runtime service and invoke APIs
Router chargeNone; endpoints and CloudWatch metrics billed as usual

Source: AWS, November 18, 2026.

Cross-Region routing and cross-provider fallback at a glance

SageMaker routing is for teams serving their own or custom models on SageMaker endpoints. Cross-provider fallback matters more for applications that call managed model APIs. The table compares both layers and what each leaves to you.

QuestionSageMaker routingProvider fallbackLeft to you
Failure coveredRegional endpoint fault or capacity shortageModel or provider outageMap each failure mode to a layer
Model servedThe same deployed modelPossibly a different modelTest fallback outputs
TargetsEndpoints you already deployedModels your policy allowsApprove every target first
TriggersPlatform faults, throttles, container 5xxSet by the gateway or applicationHandle 4xx errors and slow responses
Request modesReal-time and asynchronousDepends on the implementationCheck your request mode
ScopeChosen Regions and accounts, same AWS partitionApproved providers and destinationsEnforce residency rules
CostNo router fee; endpoints and metrics billedDepends on provider pricingBudget standby capacity

Source for the SageMaker column: AWS, November 18, 2026. Checked November 19, 2026.

The router itself adds no fee, but the resilience is not free. Target endpoints in each Region are billed at standard SageMaker AI pricing, and router metrics at standard Amazon CloudWatch rates. The router never creates targets: they must already exist.

How SageMaker cross-Region inference routing works

You create an inference router, register the real-time endpoints, asynchronous endpoints or inference components you already run, and point your application at the router. It prefers endpoints in its own Region. As errors rise, it shifts new requests to healthy destinations in other Regions, then returns traffic gradually once the local endpoint recovers.

Endpoints, deployed models, containers and request bodies stay the same. So do ContentType, Accept, CustomAttributes and InferenceId. The client does change: it calls the sagemaker-inference-router-runtime service instead of sagemaker-runtime.

This pseudocode shows the change. Check the exact SDK method names in the SageMaker AI Developer Guide before you copy it.

Python

# Pseudocode: check method names in the AWS SDK reference

# Before: invoke one endpoint directly
runtime = aws_client("sagemaker-runtime")
runtime.invoke_endpoint(
    EndpointName="my-endpoint",
    Body=payload,
    ContentType="application/json",
)

# After: invoke a routing target through the router
router = aws_client("sagemaker-inference-router-runtime")
router.invoke_inference_router(
    InferenceRouterName="my-router",
    RoutingTargetName="production",
    Body=payload,
    ContentType="application/json",
)

Synchronous clients call InvokeInferenceRouter, and asynchronous clients call InvokeInferenceRouterAsync.

Endpoint owners keep control of cross-account use. A resource-based policy with the sagemaker:SourceInferenceRouterArn condition decides which routers in other accounts can call an endpoint. An optional cross-Region traffic cap reserves capacity for the endpoint's local callers.

Neither control decides where your data goes: only the targets you configure do. Destinations must also sit in the same AWS partition as the router, so the router does not fail over from commercial AWS to AWS GovCloud.

Failures that do and do not move traffic

According to AWS's documentation, the router retries a failed attempt on another destination only for retriable errors. Examples are a throttle, or a container 5xx error before any response output begins. Platform faults also count against a destination's health.

Timeouts work differently. A request that times out is not retried, though repeated timeouts lower the destination's health and divert later requests.

Client 4xx errors and slow model responses never trigger failover. Routing also needs a healthy configured target with spare capacity. If every target is full or capped, the router has nowhere to send the request.

Each real-time response names the endpoint that served it and a RoutingReason: local-preferred, capacity-available, only-healthy or failover. Log these fields to see how often traffic leaves its home Region, alongside the router metrics in Amazon CloudWatch.

Cross-provider fallback at the application or gateway layer

Cross-provider fallback covers a different failure: the model or vendor itself. Your application or a gateway sends the request to another approved model when the preferred one is unavailable. That can mean an outage, a capacity limit, or even an account issue, as in this guide on what happens when an AI provider bans you.

The tradeoff is consistency. SageMaker routing serves the same deployment in another Region, while a fallback model may differ in output structure, tool support, safety behavior, latency and quality. Test each fallback against the same acceptance criteria as the primary: a matching request format does not mean matching behavior.

A gateway also joins the request path. It does not fix invalid requests, application bugs or unsafe outputs. Because data may reach a different provider or processing location, decide which destinations are allowed before you turn on automatic fallback.

The broader case for this layer is in our article on not relying on a single AI provider. For the integration work behind several providers, see how to handle complex AI integration.

Which failover strategy should you choose?

Choose the layer that matches the failure you need to survive. Many production systems need both.

Recommendations

Match the routing layer to the failure

SageMaker

Custom models on SageMaker

Use cross-Region routing when the main risk is an unhealthy endpoint or a Region short of capacity. Deploy, size and approve every target first.

Provider layer

Managed model APIs

Use provider fallback when your application must keep working after a model or vendor becomes unavailable. Test each fallback model on its own.

Both layers

High-availability applications

Combine them when you serve custom models on SageMaker and also call managed APIs. Each layer covers failures the other cannot see.

Governance

Residency-sensitive workloads

Register only acceptable AWS Regions and accounts as targets. Traffic caps and endpoint policies are not residency controls. Check each external provider's processing terms separately.

For chat or streaming workloads, check which request modes the router supports in the SageMaker AI Developer Guide before you move streamed traffic onto it.

A two-layer routing design

Application

Sends the request to its routing logic or gateway.

Provider layer

Picks a managed model API or your SageMaker model, with approved backups.

SageMaker router

For SageMaker traffic, picks a healthy configured endpoint.

Serving endpoint

Returns the response with the endpoint and routing reason.

Logs

Record the provider, endpoint and RoutingReason.

Where Eden AI fits

Eden AI sits at the provider layer, for the managed model APIs your application calls. With provider routing, you choose the order of providers, or sort them by price, speed or latency. Automatic fallback to backup models is a separate setting. Our post on rate limits and fallbacks in Eden AI API calls covers how the two work in practice.

This request sets an explicit provider order. Configure backup models as described in the fallback documentation.

Python

from openai import OpenAI

client = OpenAI(
    api_key="YOUR_EDENAI_API_KEY",
    base_url="https://api.edenai.run/v3",
)

response = client.chat.completions.create(
    model="YOUR_PRIMARY_MODEL",
    messages=[
        {"role": "user", "content": "Summarize this incident."}
    ],
    extra_body={
        "routing": {
            "order": ["YOUR_FIRST_PROVIDER", "YOUR_SECOND_PROVIDER"]
        }
    },
)

print(response.choices[0].message.content)

For European workloads, Eden AI offers an EU endpoint for European data residency, with zero data retention on the EU endpoint. You still need to check that each provider you allow meets your own processing requirements.

Eden AI does not deploy SageMaker endpoints or replace the SageMaker router. It is also one more service in the request path, so it does not remove every single point of failure. Regional resilience for your custom models stays with SageMaker.

What to keep in mind

  • Provision targets first: SageMaker routing needs deployed, healthy destinations with spare capacity, and each one is billed.
  • Latency is not a trigger: slow responses never cause failover, and a timed-out request is not retried.
  • Check coverage: real-time routing covers 32 commercial Regions and asynchronous routing 25, all within one AWS partition.
  • Validate every destination: review Regions, accounts, traffic caps, fallback-model behavior and data policies before production use.

Measure before you rely on it

AWS has not published the latency the router or an inter-Region hop adds. Compare direct and routed calls with your own payloads and target Regions before setting timeouts.

FAQ

What triggers SageMaker cross-Region routing?
Platform faults, throttles and container 5xx errors shift traffic to other configured endpoints. Client 4xx errors and slow model responses do not trigger failover.
Does slow inference trigger SageMaker failover?
No. Model response latency does not trigger failover, and a request that times out is not retried elsewhere, although repeated timeouts lower the destination's health for later requests.
Do clients need code changes to use the SageMaker inference router?
Yes. Models, containers, endpoints and payloads stay the same, but clients call the sagemaker-inference-router-runtime service with InvokeInferenceRouter or InvokeInferenceRouterAsync.
Do target endpoints need to exist already?
Yes. You register endpoints or inference components that are already running, and the router never creates capacity. It only sends requests to the targets you configure, so choosing acceptable Regions and accounts is up to you.
How does cross-account routing work in SageMaker?
Endpoint owners attach a resource-based policy that authorizes routers from other accounts. The sagemaker:SourceInferenceRouterArn condition restricts access to specific routers.
Does the SageMaker inference router support streaming?
The announcement covers real-time and asynchronous inference. Check the SageMaker AI Developer Guide for streaming support before moving streamed traffic to the router.
What does SageMaker cross-Region routing cost?
AWS charges nothing extra for the router. You pay standard SageMaker AI pricing for the endpoints that serve requests and standard Amazon CloudWatch rates for router metrics.
Can SageMaker routing and provider fallback be combined?
Yes. Provider fallback handles a model or vendor that is unavailable, while SageMaker routing keeps your own SageMaker models available across Regions.

Further reading

Similar articles

A coin rolls along three tracks of different lengths toward one finish line, showing how three Gemini tiers reach a completed task at different costs.
AI Comparatives
Generative AI
Gemini 4 Carbon vs Argon vs 3.8 Flash: Cost per Completed Task
11/16/2026
·
Written byClément Moreau
AI Comparatives
All
Time Series Foundation Models Explained: Timestral vs TimesFM-3 in Production
11/4/2026
·
Written byClément Moreau
Specialized AI models vs LLMs in 2027
AI Comparatives
Text Processing
Stop Using Your LLM for Everything: The Specialized AI Models Taking Over in 2027
11/2/2026
·
Written byClément Moreau
let’s start

Start building with Eden AI

A single interface to integrate the best AI technologies into your products.