Every feature you need to run AI in production
83 catalog providers. 8 routing strategies. 6 built-in plugins. One unified gateway — open source, Apache 2.0.
Featured providers from the 83 provider catalog
Overview
Everything included, nothing hidden
All capabilities ship in the open-source gateway. The FerroCloud managed tier adds multi-tenancy, analytics, and SLAs on top.
83catalog providers
OpenAI, Anthropic, Gemini, Bedrock, Groq, Mistral, Cohere, SambaNova, DeepInfra & more — one API.
2,505Catalog Models
Full catalog auto-discovered via OpenAI-compatible model endpoints.
8 Routing Strategies
Single, Fallback, Load Balance, Least Latency, Cost Optimised, Conditional, Content-Based, A/B Test.
Learn more
Semantic Caching
Vector-similarity cache reduces duplicate LLM calls with configurable thresholds.
Learn more
6 Built-in Plugins
Word filter, max-token limits, response cache, request logger, rate limiter & budget guard.
Learn more
Prometheus Metrics
Built-in /metrics endpoint. Plug into Grafana or any Prometheus-compatible stack.
Learn more
Circuit Breaker
Per-provider circuit breaker auto-opens on failures, self-heals on recovery.
Learn more
Admin API & Key Mgmt
REST API for key lifecycle, usage stats, config history & one-click rollback.
Learn more
MCP Server
Model Context Protocol server for IDE & agent tool-use integrations.
Learn more
OpenAI-Compatible API
Drop-in replacement — swap baseURL and you're done. Zero SDK changes.
A/B & Conditional Routing
Header or key-based traffic splits for safe model experiments in production.
Learn more
Hot-Reload Config
YAML/JSON config watched at runtime. Update routing without a restart.
Learn more
routing
Smart Fallbacks & Retries
Always-on AI inference
Fallback routing tries your providers in priority order: on an error or a retryable status code, the gateway moves to the next target with exponential backoff. Pair it with per-target circuit breakers and a single failing provider never takes your AI features down.
- Priority-ordered fallback chains
- Per-target retry budgets (attempts)
- Retryable-status filtering (429, 502, 503, 504)
- Exponential backoff between attempts
- Circuit-breaker-aware target skipping
- Last upstream error surfaced if all targets fail
cfg := aigateway.Config{
Strategy: aigateway.StrategyConfig{
Mode: aigateway.ModeFallback,
},
Targets: []aigateway.Target{
{VirtualKey: "openai"}, // primary
{VirtualKey: "anthropic"}, // fallback 1
{VirtualKey: "google-gemini"}, // fallback 2
},
}
gw, _ := aigateway.New(cfg)
gw.RegisterProvider(openaiProvider)
gw.RegisterProvider(anthropicProvider)
gw.RegisterProvider(geminiProvider)
routing
8 Routing Strategies
Right model, right time, right cost
Choose from eight production-hardened routing strategies shipped out of the box. Switch between them with a single config key — no code changes required.
- Single target — explicit provider pinning
- Fallback — priority-ordered chains with retries
- Weighted load balance — relative-weight traffic split
- Least latency — rolling P50 latency tracking
- Cost optimized — cheapest compatible model from the catalog
- Conditional — exact model or model-prefix matching
- Content based — substring or regex on message content
- A/B test — labelled traffic splits for live comparison
// Weighted load balance: 70% OpenAI, 30% Groq
cfg := aigateway.Config{
Strategy: aigateway.StrategyConfig{
Mode: aigateway.ModeLoadBalance,
},
Targets: []aigateway.Target{
{VirtualKey: "openai", Weight: 0.7},
{VirtualKey: "groq", Weight: 0.3},
},
}
// Or pick least-latency automatically:
cfg.Strategy.Mode = aigateway.ModeLeastLatency
// Or optimize purely by token cost:
cfg.Strategy.Mode = aigateway.ModeCostOptimized
throughput
Semantic Caching
Cache the meaning, not just the text
Semantic caching compares the meaning of a request, not its exact bytes, so near-duplicate prompts are served from cache. Queries are embedded and matched against an HNSW index in PostgreSQL with pgvector; a hit above the similarity threshold returns instantly without a billable provider call.
- Vector similarity matching (cosine)
- Configurable similarity threshold (0–1)
- TTL-based expiration for cached entries
- PostgreSQL pgvector + HNSW index
- Embeds and serves before a token is billed
# Semantic cache configuration
similarity_threshold: 0.92 # cosine similarity (0.0–1.0); higher = stricter
ttl_seconds: 3600 # entry time-to-live
vector_dimensions: 1536 # must match the embedding model output
embedding_model: text-embedding-3-small
guardrails
Built-in Plugins
Safety and control baked into every request
Plugins extend the request pipeline at three lifecycle stages — before_request, after_request, and on_error. Six ship built in, covering content filtering, token limits, response caching, request logging, rate limiting, and spend control. Each is declared in config.yaml and can be toggled independently.
- word-filter — block configured words or phrases
- max-token — cap tokens, messages, and input length
- response-cache — exact-match in-memory caching
- request-logger — structured logs, optional persistence
- rate-limit — token-bucket request throttling
- budget — cumulative USD spend caps per API key
plugins:
- name: word-filter
type: guardrail
stage: before_request
enabled: true
config:
blocked_words: ["confidential", "password", "secret"]
case_sensitive: false
- name: max-token
type: guardrail
stage: before_request
enabled: true
config:
max_tokens: 4096
max_messages: 50
max_input_length: 20000
observability
Real-time Observability
See everything, miss nothing
Four observability layers ship in the gateway: Prometheus metrics at /metrics, opt-in OpenTelemetry tracing, structured JSON logs, and a deep /health endpoint. The OTel trace ID, the log trace_id, and the X-Request-ID header are all the same value, so you can jump from a log line straight into your tracing backend.
- Prometheus /metrics endpoint (gateway_* metrics)
- Per-provider latency histograms
- Token & cost attribution per provider/model
- OpenTelemetry tracing over OTLP (opt-in)
- Structured JSON logs with a unified trace ID
- Deep /health endpoint with per-provider status
// Async event hook — emit to any backend
gw.AddHook(func(
ctx context.Context,
subject string,
data map[string]any,
) {
switch subject {
case aigateway.SubjectRequestCompleted:
metrics.Record(
data["latency_ms"],
data["cost_usd"],
data["tokens_out"],
)
case aigateway.SubjectRequestFailed:
alerts.Send(data["error"])
}
})
// Prometheus scrape: GET /metrics
// Deep health check: GET /health
routing
Circuit Breaker
Self-healing provider isolation
Per-provider circuit breakers open after a configurable number of consecutive failures, removing a failing provider from rotation before it cascades. The breaker stays open for a cool-down, then half-opens and allows one probe request, closing again once it succeeds.
- Per-provider independent breakers
- Opens on consecutive failures (failure_threshold)
- Half-open probe after a timeout cool-down
- Closes after success_threshold successes
- Open targets excluded from every routing strategy
- Self-heals with no manual intervention
targets:
- virtual_key: openai
circuit_breaker:
failure_threshold: 5 # consecutive failures before opening
success_threshold: 2 # successes in half-open before closing
timeout: "30s" # how long the breaker stays open
- virtual_key: anthropic
observability
A/B Testing & Conditional Routing
Data-driven model selection
A/B test routing splits traffic across weighted variants and tags every request with a variant label for downstream analysis, so you can compare quality, latency, and cost between providers on live traffic. Conditional routing complements it with exact-model and model-prefix rules — no client-side changes, no deploys.
- Weighted variant splits by relative weight
- Per-variant label emitted on every request event
- Conditional rules by model or model_prefix
- First-match-wins with a safe fall-through
- Gradual rollout and instant switchover
- Composes with fallback and load-balance strategies
strategy:
mode: ab-test
ab_variants:
- target_key: openai
weight: 70
label: control
- target_key: anthropic
weight: 30
label: challenger
targets:
- virtual_key: openai
- virtual_key: anthropic
guardrails
Rate Limiting & Access Control
Throttle traffic at every layer
Rate limiting works in three independent layers: per-IP HTTP middleware, a global token bucket applied before traffic reaches a provider, and per-API-key and per-user limits. Checks run in order — global, then per-key, then per-user — and the first exceeded limiter rejects the request with a distinct reason string.
- Per-IP HTTP middleware (RATE_LIMIT_RPS / BURST)
- Global token-bucket limit (requests_per_second + burst)
- Per-API-key limits (key_rpm)
- Per-user limits (user_rpm)
- Distinct rejection reason per layer for clean logs
- Requests without a key or user skip those checks
plugins:
- name: rate-limit
type: ratelimit
stage: before_request
enabled: true
config:
requests_per_second: 100 # global limit (optional)
burst: 200
key_rpm: 60 # max 60 req/min per API key
user_rpm: 30 # max 30 req/min per user ID
caching
MCP Server (Model Context Protocol)
Native tool-use for agents and IDEs
Add MCP tool servers to your config and the gateway injects their tools into chat-completion requests, then runs the full agentic loop — tool call, execution, result injection — inside the gateway. Your client sends a standard request and receives the final text answer, streaming included, without implementing tool-calling logic.
- Streamable HTTP MCP transport
- Full agentic loop runs inside the gateway
- allowed_tools whitelist per server
- Configurable max_call_depth per server
- Header auth with ${ENV_VAR} interpolation
- Streaming requests supported (stream: true)
mcp_servers:
- name: filesystem
url: "https://www.ferrolabs.ai/mcp"
allowed_tools: [read_file, list_directory, search_files]
max_call_depth: 4
timeout_seconds: 10
- name: database
url: "https://mcp-db.internal/mcp"
headers:
Authorization: "Bearer ${MCP_DB_TOKEN}"
allowed_tools: [query_readonly, list_tables]
max_call_depth: 5
routing
Model Aliases & Hot-Reload
Hot-swap models without code changes
Define semantic aliases like 'fast', 'smart', or 'cheap' and map them to concrete models in config. The gateway watches the config file and reloads on change, so you can migrate models at runtime — your application keeps calling the same alias while the target changes underneath it.
- Friendly semantic model names
- Hot-swap models in production
- Zero code changes in the app layer
- Config file watch & live reload
- Pairs with conditional and cost routing
# config.yaml — changed at runtime, no restart
aliases:
fast: groq/llama-3-8b-8192
smart: openai/gpt-4o
cheap: google/gemini-flash-1.5
vision: anthropic/claude-3-5-sonnet
# Application code never changes:
resp, _ := gw.Route(ctx, providers.Request{
Model: "smart", // resolves at the gateway
Messages: messages,
})
83
Catalog Providers
2,505
Catalog Models
6
Built-in Plugins
8
Routing Strategies
Ready to choose your path?
Deploy the OSS gateway, start the managed cloud path, or route enterprise review through the right owner.