Home › Glossary

LLM Cost and Gateway Glossary

Every term you need to understand and cut an LLM API bill, in one sentence each.

AI gateway

Another name for an LLM gateway: a layer between your app and AI model providers.

API base URL

The address your SDK sends requests to. Changing it is how you switch to a gateway such as apicuts.

Batch API

An asynchronous way to send many requests at once, at half the standard price on OpenAI, Anthropic and Google.

BYOK (bring your own key)

Using your own provider API keys through a gateway, so usage is billed to your provider accounts.

Cache hit rate

The share of input tokens served from a prompt cache. Higher means lower cost.

Cached input tokens

Input tokens read from a prompt cache and billed at a lower rate, such as $0.10 instead of $2.00 per 1M on GPT-6.1 Sol.

Cascading

Trying a cheaper model first and escalating to a stronger one only when a check fails.

Context caching

Google's name for caching prompt content on Gemini, with optional explicit caches billed for storage.

Context window

The maximum number of tokens a model can take in one request, including history and documents.

Cost per request

What one API call costs: input tokens times the input price plus output tokens times the output price.

Cost per successful task

Total cost, including retries and failures, divided by the number of tasks completed to standard. The number that matters.

Effort

A setting on some models that controls how much they reason before answering, and so how many output tokens they use.

Failover

Automatically moving a request to another provider or route when one errors or slows down.

Input tokens

Tokens you send: system prompt, tools, history, documents and the user's message.

LLM FinOps

Managing and optimizing spend on language model APIs: measuring, attributing, forecasting and reducing it.

LLM gateway

One API endpoint in front of many model providers, adding routing, failover, caching, budgets and cost tracking.

LLM router

Software that picks which model or provider handles each request.

Load balancing

Spreading requests across healthy providers, keys or regions so none becomes a bottleneck.

Model routing

Sending each request to the lowest-cost model that meets your quality bar.

OpenAI-compatible API

An API that accepts the same requests as OpenAI's, so OpenAI SDKs work by changing the base URL.

Output tokens

Tokens the model generates, including reasoning. Usually about five times the price of input or more.

Prompt caching

Reusing work for an identical prompt prefix and billing it at a lower cached rate.

Provider routing

Choosing among providers that serve the same model, by price, latency and reliability.

Rate limit

The maximum requests or tokens per minute a provider or gateway allows. Exceeding it returns a 429 error.

Reasoning tokens

Tokens a reasoning model generates while thinking. Billed as output.

Semantic caching

Serving a stored answer when a new request is close enough in meaning to an earlier one.

Thinking budget

A limit on how many tokens a model may spend reasoning, used on Gemini and other models.

Token

The unit LLMs read and write in, roughly a short word or part of a word. APIs bill per token.

Tokenizer

The scheme that splits text into tokens. Different models can count the same text as different numbers of tokens.

Zero data retention

A setup where prompts and outputs aren't stored after the request completes.

Next: what is an LLM gateway? and the LLM cost optimization guide.

Related