LiteLLM Review

LiteLLM comes in two forms. The Python SDK calls 100+ LLMs through one interface with completion(), embedding(), and image_generation(), and includes a Router with retry, fallback, and load balancing and cost tracking through callbacks. The Proxy Server is a self-hosted, OpenAI-compatible gateway with virtual keys, budgets per key, team, and user, rate limits (RPM and TPM), spend tracking, centralized logging, guardrails, caching, an Admin UI, routing with automatic fallbacks, observability integrations with Langfuse, Arize Phoenix, LangSmith, and OpenTelemetry, pass-through endpoints, and S3 logging. It supports endpoints including /chat/completions, /responses, /embeddings, /images, /audio, /batches, /rerank, /a2a, and /messages, as well as MCP tools and A2A agents. Providers include OpenAI, Anthropic, Vertex AI, Bedrock, Ollama, Azure OpenAI, and Gemini.

The SDK installs with uv add litellm, and a leaner litellm-core distribution exists (its release is pending and it must be built from source; install only one SDK distribution per environment). The proxy installs with uv tool install 'litellm[proxy]' and starts with litellm --model gpt-4o, or runs as a Docker container. Virtual keys need a Postgres DATABASE_URL and a master key that starts with sk-. The production guide gives concrete numbers: Redis 7.0 or newer once you run more than one proxy instance (and enable the Redis transaction buffer at roughly 1,000 requests per second or 10+ instances), 1 vCPU and 4 Gi of memory per worker as both requests and limits (4 Gi is a floor), one Uvicorn worker per pod on Kubernetes or NUM_WORKERS equal to the vCPU count on a VM, a default database pool of 10 connections per worker, spend writes batched every 60 seconds, request_timeout of 600 seconds (default 6000), autoscaling on CPU at a 60% target, and a dedicated single-replica deployment for background jobs. Set LITELLM_MASTER_KEY and LITELLM_SALT_KEY (do not change the salt key after adding models or stored credentials become unreadable), set LITELLM_MODE to PRODUCTION, turn off debug logging, and run migrations from a pre-sync hook with DISABLE_SCHEMA_UPDATE on serving pods. The README claims 8 ms P95 latency at 1,000 requests per second and says images tagged -stable have undergone 12-hour load tests.

From the application's point of view the gateway looks like OpenAI: you change the base URL and key, and the proxy decides which provider and model actually serves the request. Administrators generate virtual keys through /key/generate with fields such as models, max_budget, budget_duration, tpm_limit, rpm_limit, and max_parallel_requests, group them under teams, set aliases that redirect a requested model name to another, give keys a duration, block or unblock them, and rotate them with a grace period so the old key keeps working during cutover. The routing documentation describes simple-shuffle as the default and recommended production strategy (it also supports weights), latency-based and least-busy routing, a usage-based strategy that uses Redis and is discouraged in production because of added latency, retries with exponential backoff for rate-limit errors, priority tiers with cross-group fallbacks (max_fallbacks defaults to 5), and cooldowns (allowed_fails 3 and cooldown_time 5 seconds by default, triggered by a 429 or more than 50% failures in a minute). Requests above max_parallel_requests get an immediate 429 with no queue, and without Redis each proxy instance tracks its own usage and cooldowns. This section reflects the documentation; we did not run the proxy.

• Unified OpenAI-compatible API across 100+ providers
• Virtual keys with budgets per key, team, and user, plus expiry, aliases, and rotation with a grace period
• RPM and TPM rate limits, load balancing, retries, cooldowns, and automatic fallbacks
• Observability with Langfuse, Arize Phoenix, LangSmith, and OpenTelemetry, plus S3 logging
• Guardrails, caching, an Admin UI, and pass-through endpoints
• MCP tools and A2A agents through the gateway

• Standardizes many providers behind one API, which makes switching and fallback easy
• Cost control through virtual keys and budgets, which hosted aggregators often lack
• Very popular and fast-moving, with about 61,000 stars and daily releases
• Unusually concrete production guidance, including sizing and secret-handling advice

• Enterprise features such as SSO, JWT auth, audit logs, and custom SLAs sit behind a commercial license
• The README refers to a commercial license without clearly describing the split, so verify the license for the pieces you use
• Another service to run and secure, and a single point of failure for your LLM traffic: it needs Postgres, Redis at scale, and about 4 Gi of memory per worker
• Misconfiguration traps are documented, such as changing the salt key after adding models, and the router returns an immediate 429 above max_parallel_requests with no queue
• Latency and throughput claims are the project's own, and the release pace (v1.105.0) means frequent upgrades

LiteLLM is free to self-host, and the open-source version covers the full feature set for most teams according to OpenAlternative. An enterprise license adds SSO, JWT authentication, audit logs, and custom SLAs, with no price published on the pages we reviewed.

OpenRouter, Cloudflare AI Gateway, and Vercel AI Gateway are hosted routing services that sit between your app and many models and bill you for usage. LiteLLM gives you the same single-API idea on your own infrastructure with your own provider keys, plus budgets, virtual keys, and logging you control. Choose a hosted gateway to avoid running anything, and LiteLLM when you want control over keys, data, and spend.

• Platform teams that need one governed entry point to many LLM providers
• Companies that want per-team budgets, rate limits, and spend tracking
• Developers who want provider fallback and load balancing

• Small apps that use a single provider
• Teams that do not want to operate a gateway service
• Organizations that need SSO and audit logs without an enterprise agreement