Routing policy · chat-default
What you get
Everything in the box
- Single target — explicit provider pinning
- Fallback — priority-ordered chains with retries
- Weighted load balance — relative-weight traffic split
- Least latency — rolling P50 latency tracking
- Cost optimized — cheapest compatible model from the catalog
- Conditional — exact model or model-prefix matching
- Content based — substring or regex on message content
- A/B test — labelled traffic splits for live comparison
// Weighted load balance: 70% OpenAI, 30% Groq
cfg := aigateway.Config{
Strategy: aigateway.StrategyConfig{
Mode: aigateway.ModeLoadBalance,
},
Targets: []aigateway.Target{
{VirtualKey: "openai", Weight: 0.7},
{VirtualKey: "groq", Weight: 0.3},
},
}
// Or pick least-latency automatically:
cfg.Strategy.Mode = aigateway.ModeLeastLatency
// Or optimize purely by token cost:
cfg.Strategy.Mode = aigateway.ModeCostOptimized
When to use it
When to reach for this
Uptime above all
Use fallback routing when your chatbot must always respond — traffic rolls to the next provider, with retries and backoff, even if the primary is down.
Pin model families to providers
Use conditional routing to send all GPT models to OpenAI and all Claude models to Anthropic from a single endpoint, with no client-side branching.
Minimise spend without micromanaging
Use cost-optimized routing to let the catalog pick the cheapest compatible model per request, then layer retries to fall through to the next cheapest on failure.
Compare providers on live traffic
Use A/B test routing to split a fixed percentage of traffic between two providers, each tagged with a label you can aggregate in logs and analytics.
FAQ
Frequently asked questions
How many routing strategies does the Ferro Labs AI Gateway support?
Eight: single, fallback, weighted load balancing, conditional, least-latency, cost-optimized, content-based, and A/B test. You select one with the strategy.mode key in config.yaml.
Can I switch routing strategies without changing application code?
Yes. Routing is configured entirely in config.yaml. Change strategy.mode and your targets — your application keeps calling the same OpenAI-compatible endpoint with no code changes.
Does routing add latency to each request?
Minimal. Single and weighted routing are effectively free, least-latency adds a sub-microsecond mutex read, and content-based regex patterns are pre-compiled at startup so the hot path is a single match call.
How does cost-optimized routing decide which model is cheapest?
It estimates each eligible target's input-token cost using the built-in model catalog's pricing data and routes to the cheapest compatible provider. If cost data is missing it falls back to the first compatible target.
What happens when a provider fails partway through a strategy?
Every strategy respects per-target circuit breakers. A failing target's breaker opens and the target is excluded from selection, while fallback chains retry the next target with exponential backoff.
Related features
Feature deep dives
Smart Fallbacks & Retries
Fallback routing tries your providers in priority order: on an error or a retryable status code, the gateway moves to the next target with exponential backoff. Pair it with per-target circuit breakers and a single failing provider never takes your AI features down.
Circuit Breaker
Per-provider circuit breakers open after a configurable number of consecutive failures, removing a failing provider from rotation before it cascades. The breaker stays open for a cool-down, then half-opens and allows one probe request, closing again once it succeeds.
A/B Testing & Conditional Routing
A/B test routing splits traffic across weighted variants and tags every request with a variant label for downstream analysis, so you can compare quality, latency, and cost between providers on live traffic. Conditional routing complements it with exact-model and model-prefix rules — no client-side changes, no deploys.
Keep exploring
Solutions & docs
LLM Cost Control and Budget Enforcement
Ferro Labs gives finance, platform, and engineering teams shared LLM cost control across providers, models, projects, and API keys.
Reliable LLM Routing, Fallbacks, and SLOs
Ferro Labs keeps production AI traffic moving with fallback routing, provider failover, retries, routing strategies, circuit breakers, SLO tracking, and incident workflows.
Read the docs
Full configuration reference and examples on docs.ferrolabs.ai