Home › Docs › Optimization techniques

How it works

Optimization techniques

Your 20%+ comes from many small, safe wins rather than one risky change. Here is every lever we use, and the guardrail on each.

Prompt caching

Repeated system prompts, tool definitions and documents are billed at the provider's cached rate. Guardrail: no change to what the model sees. See caching.

Semantic caching

Similar repeated requests on routes you approve are answered from cache. Guardrail: off by default, conservative matching, entries expire.

Batch processing

Work that doesn't need an instant answer moves to discounted batch APIs where the provider offers them. Guardrail: only for jobs you mark as non-urgent.

Prompt and context trimming

Redundant instructions, duplicate context and oversized retrieval results are removed. Guardrail: tested against your evals before shipping.

Output limits

Answers stay as long as they need to be, with sensible caps on output and reasoning tokens. Guardrail: quality checks on every capped route.

Model routing (optional)

Simple requests go to smaller equivalent models you approve. Guardrail: your evaluations and a canary rollout first. See routing.

Retry and agent-loop controls

Duplicate requests, runaway tool loops and wasted retries are caught and stopped. Guardrail: limits set with you per workflow.

Provider pricing

We compare official providers, negotiated pricing and commitment discounts for your volume.

Advanced optimization

Fine-tuning, distillation or self-hosting, only when your workload economics clearly support it.

What's automatic and what we set up

Automatic in the gateway
  • Provider routing and failover
  • Prompt caching and cache-aware routing
  • Semantic caching, on routes where you switch it on
  • Budgets, usage limits and alerts
  • The live savings dashboard
Set up with you by our engineer
  • Your routing rules and data region
  • Prompt and context trimming
  • Output limits
  • Model-routing tests against your evaluations
  • Which jobs move to batch processing
  • Retry and agent-loop limits

Once they're set up, these changes run in the gateway on every request.

How we decide what to switch on

  1. Baseline. We record your cost per equivalent successful request, plus quality, latency and error rate. See the savings methodology.
  2. Rank. Every opportunity is ranked by expected dollars saved, risk and effort.
  3. Test. Changes are tested against your evaluations, then rolled out to a small share of traffic.
  4. Measure. Savings, quality and speed are tracked live. A change that makes anything worse is rolled back.

Want us to find your 20%?

Book a free 15-minute API cost teardown. Bring last month's provider usage and we'll show you your biggest waste and the first fix we'd ship.

Get a Free 15-Minute API Cost Teardown