What Is an LLM Gateway?

An LLM gateway sits between your app and model providers. Your app talks to one endpoint; the gateway handles providers, routing, caching, limits and logging.

Quick answer

An LLM gateway is one API endpoint in front of many model providers, adding routing, failover, caching, budgets and cost tracking.

What it does

One API for many models and providers.
Routing and load balancing across providers and models.
Failover when a provider errors or slows down.
Caching of repeated prompts and, optionally, answers.
Budgets, rate limits and access control.
Logging and cost tracking.

LLM gateway vs API gateway

A traditional API gateway routes and secures HTTP traffic. An LLM gateway understands tokens, models, prompts and providers, so it can price, cache and route model calls.

LLM gateway vs model router

A model router picks the model for each prompt. A gateway carries the request and usually includes routing among other features.

Do you need one?

Once you use more than one provider, run production traffic, or spend enough that cost matters, usually yes. Read choosing an LLM gateway or build vs buy.

Frequently asked questions

What is an AI gateway?

Another name for an LLM gateway: a layer between your app and AI model providers that handles routing, failover, caching, limits and cost tracking.

Is an LLM gateway the same as a proxy?

A gateway is a proxy with model-aware features: routing, caching, budgets and cost tracking.

Want us to find your 20%?

Book a free 15-minute API cost teardown. Bring last month's provider usage and we'll show you your biggest waste and the first fix we'd ship.

Get a Free 15-Minute API Cost Teardown