Skip to content
LLM observability

LLM Observability, Logs, and Tracing

Ferro Labs turns the gateway into the source of truth for LLM observability across request logs, traces, sessions, cost, latency, and provider errors.

Trace every AI request with logs, analytics, sessions, cost attribution, and OTLP-ready telemetry.

Deploy OSSAll Solutions

Buyer Problem

When AI calls fail or spend spikes, teams often have only app logs and provider dashboards. That leaves gaps across retries, fallbacks, prompt flows, sessions, and user-level cost.

Target Outcome

Engineering, support, and platform teams can debug AI behavior from one timeline with trace context, request metadata, provider latency, token usage, errors, and audit-gated raw body access.

Capabilities

Gateway controls for this solution

These are the gateway-level capabilities this solution depends on.

AI request logs with model, provider, key, project, latency, tokens, cost, and error metadata.
LLM tracing for retries, fallbacks, provider attempts, tool events, and policy decisions.
Session views that connect multi-turn interactions across requests.
Analytics dashboards for usage, latency, error rate, cache behavior, and cost trends.
OTLP export support for sending gateway telemetry into existing observability stacks.
Sensitive raw body views protected by access checks and audit logging.

Proof Points

Evaluation evidence to review

Analytics, logs, traces, sessions, and OTLP telemetry are implemented.

Raw request and response body views are treated as sensitive and audit-logged.

Retention is tier-aware, so log availability can differ by customer plan.

Telemetry is captured at the gateway, including upstream provider outcomes and routing behavior.

Status note

Live: analytics, logs, traces, sessions, and OTLP. Sensitive raw body views are audit-logged. Retention varies by tier.

Related Features

Feature deep dives

Real-time Observability

Four observability layers ship in the gateway: Prometheus metrics at /metrics, opt-in OpenTelemetry tracing, structured JSON logs, and a deep /health endpoint. The OTel trace ID, the log trace_id, and the X-Request-ID header are all the same value, so you can jump from a log line straight into your tracing backend.

Smart Fallbacks & Retries

Fallback routing tries your providers in priority order: on an error or a retryable status code, the gateway moves to the next target with exponential backoff. Pair it with per-target circuit breakers and a single failing provider never takes your AI features down.

Circuit Breaker

Per-provider circuit breakers open after a configurable number of consecutive failures, removing a failing provider from rotation before it cascades. The breaker stays open for a cool-down, then half-opens and allows one probe request, closing again once it succeeds.

Semantic Caching

Semantic caching compares the meaning of a request, not its exact bytes, so near-duplicate prompts are served from cache. Queries are embedded and matched against an HNSW index in PostgreSQL with pgvector; a hit above the similarity threshold returns instantly without a billable provider call.

FAQ

Common evaluation questions

What does Ferro Labs capture for each AI request?

The gateway records request metadata such as project, key, model, provider, latency, token usage, cost, routing outcome, errors, and related trace or session context.

Can raw prompts and responses be viewed?

Yes, when enabled and permitted. Raw body views are sensitive, access-controlled, and audit-logged.

Can telemetry be exported?

Yes. OTLP support is implemented so teams can connect gateway telemetry to their existing observability tools.

Solution Review

Validate this solution against your deployment model

Start with the open-source gateway, review the docs, or scope a managed deployment with Ferro Labs.

Deploy OSSRead DocsBook Demo