guardrails
Rate Limiting & Access Control
Throttle traffic at every layer
Rate limiting works in three independent layers: per-IP HTTP middleware, a global token bucket applied before traffic reaches a provider, and per-API-key and per-user limits. Checks run in order — global, then per-key, then per-user — and the first exceeded limiter rejects the request with a distinct reason string.
What you get
Everything in the box
- Per-IP HTTP middleware (RATE_LIMIT_RPS / BURST)
- Global token-bucket limit (requests_per_second + burst)
- Per-API-key limits (key_rpm)
- Per-user limits (user_rpm)
- Distinct rejection reason per layer for clean logs
- Requests without a key or user skip those checks
plugins:
- name: rate-limit
type: ratelimit
stage: before_request
enabled: true
config:
requests_per_second: 100 # global limit (optional)
burst: 200
key_rpm: 60 # max 60 req/min per API key
user_rpm: 30 # max 30 req/min per user ID
When to use it
When to reach for this
Protect spend from a runaway client
Set a global requests_per_second ceiling so a misbehaving script can't drive unbounded provider cost before anyone notices.
Fair-share across API keys
Add key_rpm so each API key gets its own allowance and one busy integration can't starve the others.
Throttle per end user
Set user_rpm to cap individual users in a multi-tenant app, independent of which API key the request arrives on.
Shed abuse at the edge
Use the per-IP middleware (RATE_LIMIT_RPS / BURST) to drop floods from a single source before they reach the pipeline.
FAQ
Frequently asked questions
What are the three rate-limiting layers?
Per-IP HTTP middleware (set with RATE_LIMIT_RPS and RATE_LIMIT_BURST), a global token bucket via the rate-limit plugin (requests_per_second + burst), and per-identity limits (key_rpm per API key, user_rpm per user).
In what order are the limits checked?
Global, then per-key, then per-user. The first limiter that is exceeded rejects the request with a distinct reason string, so your logs show which limit was hit.
What happens to a request with no API key or no user?
It skips that check. A request without an api_key isn't subject to the per-key limit, and a request without a user field isn't subject to the per-user limit.
What does a rate-limited request return?
HTTP 429 Too Many Requests with an OpenAI-style error body, so clients that already handle provider 429s handle the gateway's the same way.
Can I see which layer is rejecting traffic?
Yes. The Prometheus metric gateway_rate_limit_rejections_total is labelled by key_type (ip, api_key, or plugin), so you can attribute rejections to the layer that produced them.
Related features
Feature deep dives
Built-in Plugins
Plugins extend the request pipeline at three lifecycle stages — before_request, after_request, and on_error. Six ship built in, covering content filtering, token limits, response caching, request logging, rate limiting, and spend control. Each is declared in config.yaml and can be toggled independently.
Real-time Observability
Four observability layers ship in the gateway: Prometheus metrics at /metrics, opt-in OpenTelemetry tracing, structured JSON logs, and a deep /health endpoint. The OTel trace ID, the log trace_id, and the X-Request-ID header are all the same value, so you can jump from a log line straight into your tracing backend.
Smart Fallbacks & Retries
Fallback routing tries your providers in priority order: on an error or a retryable status code, the gateway moves to the next target with exponential backoff. Pair it with per-target circuit breakers and a single failing provider never takes your AI features down.
Keep exploring
Solutions & docs
Enterprise AI Governance and RBAC
Ferro Labs gives platform teams a single control plane for enterprise AI governance across providers, projects, teams, and API keys.
AI Guardrails for PII and Prompt Injection
Ferro Labs lets platform and security teams apply AI guardrails at the gateway before risky prompts, responses, or tool flows spread across applications.
MCP Gateway for Agentic AI
Ferro Labs gives agent platforms a governed MCP gateway that connects Model Context Protocol servers, tools, prompts, evals, logs, and access controls.
Read the docs
Full configuration reference and examples on docs.ferrolabs.ai