observability
A/B Testing & Conditional Routing
Data-driven model selection
A/B test routing splits traffic across weighted variants and tags every request with a variant label for downstream analysis, so you can compare quality, latency, and cost between providers on live traffic. Conditional routing complements it with exact-model and model-prefix rules — no client-side changes, no deploys.
What you get
Everything in the box
- Weighted variant splits by relative weight
- Per-variant label emitted on every request event
- Conditional rules by model or model_prefix
- First-match-wins with a safe fall-through
- Gradual rollout and instant switchover
- Composes with fallback and load-balance strategies
strategy:
mode: ab-test
ab_variants:
- target_key: openai
weight: 70
label: control
- target_key: anthropic
weight: 30
label: challenger
targets:
- virtual_key: openai
- virtual_key: anthropic
When to use it
When to reach for this
Compare two providers on live traffic
Send a fixed share of requests to a challenger model, each tagged with a label, and compare quality, latency, and cost against the control with no client-side changes.
Roll out a new model gradually
Start the challenger at a small weight, watch the labelled results, and shift weight toward it as confidence grows — or revert instantly by editing config.
Pin model families to providers
Use conditional routing so claude* models go to Anthropic and gemini* models go to Gemini from a single endpoint, with a safe fall-through default.
Route a friendly alias to the right host
Alias a name like smart to a concrete model, then add a conditional rule on its prefix so the alias always reaches the provider that serves it.
FAQ
Frequently asked questions
How does A/B test routing split traffic?
By relative weight across ab_variants. Weights like 70 and 30 send roughly 70% and 30% of traffic; they need not sum to 100. Each request is tagged with its variant label.
What happens with a weight of 0 or a negative weight?
A weight of 0 is treated as 1, so the variant still gets an equal share. Negative weights are rejected at gateway startup, so a typo fails fast rather than skewing the experiment.
How do I analyse A/B results?
The variant label is emitted in every gateway.request.completed event. Persist those events (for example with the request-logger plugin to Postgres) and aggregate by label to compare cost, latency, and quality.
How does conditional routing choose a target?
It evaluates rules in order and the first match wins, matching on the requested model by exact name (model) or prefix (model_prefix). If nothing matches, the request falls through to the first target.
Do A/B and conditional routing require code changes?
No. Both are set in config.yaml via strategy.mode. Your application keeps calling the same endpoint while the gateway handles the split or the rule matching.
Related features
Feature deep dives
8 Routing Strategies
Choose from eight production-hardened routing strategies shipped out of the box. Switch between them with a single config key — no code changes required.
Real-time Observability
Four observability layers ship in the gateway: Prometheus metrics at /metrics, opt-in OpenTelemetry tracing, structured JSON logs, and a deep /health endpoint. The OTel trace ID, the log trace_id, and the X-Request-ID header are all the same value, so you can jump from a log line straight into your tracing backend.
Model Aliases & Hot-Reload
Define semantic aliases like 'fast', 'smart', or 'cheap' and map them to concrete models in config. The gateway watches the config file and reloads on change, so you can migrate models at runtime — your application keeps calling the same alias while the target changes underneath it.
Keep exploring