routing
Smart Fallbacks & Retries
Always-on AI inference
Fallback routing tries your providers in priority order: on an error or a retryable status code, the gateway moves to the next target with exponential backoff. Pair it with per-target circuit breakers and a single failing provider never takes your AI features down.
Routing policy · chat-default
What you get
Everything in the box
- Priority-ordered fallback chains
- Per-target retry budgets (attempts)
- Retryable-status filtering (429, 502, 503, 504)
- Exponential backoff between attempts
- Circuit-breaker-aware target skipping
- Last upstream error surfaced if all targets fail
cfg := aigateway.Config{
Strategy: aigateway.StrategyConfig{
Mode: aigateway.ModeFallback,
},
Targets: []aigateway.Target{
{VirtualKey: "openai"}, // primary
{VirtualKey: "anthropic"}, // fallback 1
{VirtualKey: "google-gemini"}, // fallback 2
},
}
gw, _ := aigateway.New(cfg)
gw.RegisterProvider(openaiProvider)
gw.RegisterProvider(anthropicProvider)
gw.RegisterProvider(geminiProvider)
When to use it
When to reach for this
User-facing chat that can't go down
Put your fastest provider first and a second provider behind it, so a rate-limit spike or outage on the primary reroutes transparently and the chat keeps responding.
Absorb provider rate limits
List 429 in retry_on_status so throttled requests retry with backoff, then fail over to a second provider once the primary's budget is spent.
Cheapest-first with a safety net
Order targets cheapest to most reliable so most traffic lands on the low-cost provider, with a premium provider as the always-available backstop.
Skip dead providers during an incident
Add circuit breakers so a provider in the middle of an outage is dropped from the chain instead of retried on every request, keeping tail latency flat.
FAQ
Frequently asked questions
How does fallback routing decide when to fail over to the next provider?
On a transport error or a retryable HTTP status code (such as 429, 502, 503, or 504), the gateway retries within the current target's attempt budget using exponential backoff, then advances to the next target when that budget is exhausted.
Which status codes are retried?
Whichever you list in each target's retry_on_status — commonly 429, 502, 503, and 504. Status codes outside that list (like a 400 from a bad request) are not retried; the gateway fails over or returns the error immediately.
What happens if every provider in the chain fails?
The gateway returns the last upstream error to the client, so you see the real provider error rather than a generic gateway failure.
Do retries add latency when a provider is down?
Not if you add circuit breakers. A target whose breaker is open is skipped entirely, so the gateway fails over instantly instead of retrying a provider that is already known to be failing.
Does fallback require changes to my application code?
No. Fallback is configured in config.yaml. Your application keeps calling the same OpenAI-compatible endpoint, and the failover happens inside the gateway.
Related features
Feature deep dives
8 Routing Strategies
Choose from eight production-hardened routing strategies shipped out of the box. Switch between them with a single config key — no code changes required.
Circuit Breaker
Per-provider circuit breakers open after a configurable number of consecutive failures, removing a failing provider from rotation before it cascades. The breaker stays open for a cool-down, then half-opens and allows one probe request, closing again once it succeeds.
Real-time Observability
Four observability layers ship in the gateway: Prometheus metrics at /metrics, opt-in OpenTelemetry tracing, structured JSON logs, and a deep /health endpoint. The OTel trace ID, the log trace_id, and the X-Request-ID header are all the same value, so you can jump from a log line straight into your tracing backend.
Keep exploring
Solutions & docs
LLM Observability, Logs, and Tracing
Ferro Labs turns the gateway into the source of truth for LLM observability across request logs, traces, sessions, cost, latency, and provider errors.
Reliable LLM Routing, Fallbacks, and SLOs
Ferro Labs keeps production AI traffic moving with fallback routing, provider failover, retries, routing strategies, circuit breakers, SLO tracking, and incident workflows.
Read the docs
Full configuration reference and examples on docs.ferrolabs.ai