Skip to content
Fallback routing

Reliable LLM Routing, Fallbacks, and SLOs

Ferro Labs keeps production AI traffic moving with fallback routing, provider failover, retries, routing strategies, circuit breakers, SLO tracking, and incident workflows.

Use fallback routing, provider failover, retries, circuit breakers, SLOs, and incidents for production AI traffic.

Deploy OSSAll Solutions

Buyer Problem

Provider outages, rate limits, model incidents, and latency spikes can break user-facing AI features. Reliability needs to live in the gateway, not in every application.

Target Outcome

Teams can route around failures, standardize provider failover, monitor SLOs, manage incidents, and change model routing without redeploying application code.

Capabilities

Gateway controls for this solution

These are the gateway-level capabilities this solution depends on.

Priority fallback routing across approved providers and models.
Provider failover with retries, backoff, health checks, and circuit breakers.
Routing strategies for single target, load balance, least latency, cost optimized, conditional, content-based, and A/B traffic.
Config hot-reload and rollback patterns for model routing changes without app deploys.
SLO tracking for gateway reliability, latency, errors, and availability objectives.
Incident workflows that connect degraded providers, affected projects, and operational response.

Proof Points

Evaluation evidence to review

Fallback routing and other routing strategies are implemented.

Provider failover, retries, health checks, and circuit breaker controls are available at the gateway layer.

SLOs and incidents are implemented for reliability operations.

Multi-window SLO evaluation is a future roadmap item.

Status note

Live: fallback routing, routing strategies, SLOs, and incidents. Future: multi-window SLO evaluation.

Related Features

Feature deep dives

Smart Fallbacks & Retries

Fallback routing tries your providers in priority order: on an error or a retryable status code, the gateway moves to the next target with exponential backoff. Pair it with per-target circuit breakers and a single failing provider never takes your AI features down.

8 Routing Strategies

Choose from eight production-hardened routing strategies shipped out of the box. Switch between them with a single config key — no code changes required.

Circuit Breaker

Per-provider circuit breakers open after a configurable number of consecutive failures, removing a failing provider from rotation before it cascades. The breaker stays open for a cool-down, then half-opens and allows one probe request, closing again once it succeeds.

Real-time Observability

Four observability layers ship in the gateway: Prometheus metrics at /metrics, opt-in OpenTelemetry tracing, structured JSON logs, and a deep /health endpoint. The OTel trace ID, the log trace_id, and the X-Request-ID header are all the same value, so you can jump from a log line straight into your tracing backend.

FAQ

Common evaluation questions

Can Ferro Labs fail over between providers?

Yes. Fallback routing and provider failover are implemented so traffic can move through approved provider or model chains when a target fails.

Does the gateway support routing strategies beyond fallback?

Yes. Routing strategies include single target, load balance, least latency, cost optimized, conditional, content-based, and A/B traffic patterns.

Are SLOs supported?

Yes. SLOs and incidents are implemented. Multi-window SLO evaluation is future work.

Solution Review

Validate this solution against your deployment model

Start with the open-source gateway, review the docs, or scope a managed deployment with Ferro Labs.

Deploy OSSRead DocsBook Demo