⚡ Your AI bill is eating your Q4 margin.Protect Your Margin. Get a FREE 360° Cost ReportFree 360° Cost Report

Home › Docs › Optimization techniques

How it works

Optimization techniques

Your 20%+ comes from many small, safe wins rather than one risky change. Here is every lever we use, and the guardrail on each.

Provider prompt & context caching

Repeated system prompts, tool definitions and long context prefixes are billed at the provider's cached rate. Guardrail: no change to what the model sees. See caching.

Batch processing

Work that doesn't need an instant answer moves to discounted batch APIs where the provider offers them. Guardrail: only for jobs you mark as non-urgent.

Prompt and context trimming

Redundant instructions, duplicate context and oversized retrieval results are removed. Guardrail: tested against your evals before shipping.

Output limits

Answers stay as long as they need to be, with sensible caps on output and reasoning tokens. Guardrail: quality checks on every capped route.

Model routing (optional)

Simple requests go to smaller equivalent models you approve. Guardrail: your evaluations and a canary rollout first. See routing.

Retry and agent-loop controls

Duplicate requests via Idempotency-Key, runaway concurrency loops and wasted retries are caught and stopped. Guardrail: key and workspace limits you control.

Provider pricing

We compare official providers, negotiated pricing and commitment discounts for your volume.

Fine-tuning and distillation

Task-specific fine-tuning or distillation, only when your workload economics clearly support it.

What's automatic and what we tune together

Automatic in the gateway
  • Provider routing and failover
  • Provider prompt, prefix and context caching with cache-aware routing
  • Prepaid wallet controls, concurrency caps and idempotency checks
  • The live savings dashboard (/app/overview)
Tuned with you during your audit
  • Allowed models and routing rules
  • Prompt and context trimming
  • Output limits
  • Model-routing tests against your evaluations
  • Which jobs move to batch processing
  • Retry and agent-loop limits

Once they're set up, these changes run in the gateway on every request.

How we decide what to switch on

  1. Baseline. We record your cost per equivalent successful request, plus quality, latency and error rate. See the savings methodology.
  2. Rank. Every opportunity is ranked by expected dollars saved, risk and effort.
  3. Test. Changes are tested against your evaluations, then rolled out to a small share of traffic.
  4. Measure. Savings, quality and speed are tracked live. A change that makes anything worse is rolled back.

Want us to find your 20%+?

Get a free 360° cost report. Bring last month's provider usage and we'll show you your biggest waste and the first fix we'd ship.

Get a Free 360° Cost Report