throughput
Semantic Caching
Cache the meaning, not just the text
Semantic caching compares the meaning of a request, not its exact bytes, so near-duplicate prompts are served from cache. Queries are embedded and matched against an HNSW index in PostgreSQL with pgvector; a hit above the similarity threshold returns instantly without a billable provider call.
What you get
Everything in the box
- Vector similarity matching (cosine)
- Configurable similarity threshold (0–1)
- TTL-based expiration for cached entries
- PostgreSQL pgvector + HNSW index
- Embeds and serves before a token is billed
# Semantic cache configuration
similarity_threshold: 0.92 # cosine similarity (0.0–1.0); higher = stricter
ttl_seconds: 3600 # entry time-to-live
vector_dimensions: 1536 # must match the embedding model output
embedding_model: text-embedding-3-small
When to use it
When to reach for this
FAQ and support assistants
Support bots field the same questions in countless phrasings. Semantic caching serves the repeats from cache so only genuinely novel questions reach the model.
High-traffic, repetitive prompts
When many users ask near-identical things — product lookups, definitions, boilerplate generation — caching the meaning collapses duplicate spend without changing your app.
Latency-sensitive surfaces
A cache hit returns before the provider would even start streaming, so repetitive queries on a latency-critical path feel instant.
Cost control on expensive models
Pair a high similarity threshold with an expensive model so only sufficiently-novel prompts incur the premium per-token cost.
FAQ
Frequently asked questions
How is semantic caching different from a normal response cache?
A normal cache keys on the exact request body, so reworded-but-equivalent prompts miss. Semantic caching embeds each query into a vector and matches on meaning, so near-duplicate prompts can share a cached response.
What backs the semantic cache?
PostgreSQL with the pgvector extension, using an HNSW index for fast approximate nearest-neighbour search over the stored embeddings. It is not a separate vector database.
How do I control how aggressively it caches?
With similarity_threshold (default 0.92). Raise it for stricter matching and fewer false hits; lower it to increase the hit rate at some cost to precision.
Will the cache return stale answers?
Entries expire after ttl_seconds (default 3600). Tune the TTL to balance freshness against hit rate for your content.
How much will it save me?
It depends on how semantically repetitive your traffic is, your threshold, and the models you use. Every hit removes a billable provider call, so the more repeated intent your traffic carries, the larger the saving.
Related features
Feature deep dives
8 Routing Strategies
Choose from eight production-hardened routing strategies shipped out of the box. Switch between them with a single config key — no code changes required.
Real-time Observability
Four observability layers ship in the gateway: Prometheus metrics at /metrics, opt-in OpenTelemetry tracing, structured JSON logs, and a deep /health endpoint. The OTel trace ID, the log trace_id, and the X-Request-ID header are all the same value, so you can jump from a log line straight into your tracing backend.
Built-in Plugins
Plugins extend the request pipeline at three lifecycle stages — before_request, after_request, and on_error. Six ship built in, covering content filtering, token limits, response caching, request logging, rate limiting, and spend control. Each is declared in config.yaml and can be toggled independently.
Keep exploring
Solutions & docs
LLM Observability, Logs, and Tracing
Ferro Labs turns the gateway into the source of truth for LLM observability across request logs, traces, sessions, cost, latency, and provider errors.
AI Guardrails for PII and Prompt Injection
Ferro Labs lets platform and security teams apply AI guardrails at the gateway before risky prompts, responses, or tool flows spread across applications.
LLM Cost Control and Budget Enforcement
Ferro Labs gives finance, platform, and engineering teams shared LLM cost control across providers, models, projects, and API keys.
Read the docs
Full configuration reference and examples on docs.ferrolabs.ai