Home › Docs › Optimization techniques
How it worksOptimization techniques
Your 20%+ comes from many small, safe wins rather than one risky change. Here is every lever we use, and the guardrail on each.
Repeated system prompts, tool definitions and long context prefixes are billed at the provider's cached rate. Guardrail: no change to what the model sees. See caching.
Work that doesn't need an instant answer moves to discounted batch APIs where the provider offers them. Guardrail: only for jobs you mark as non-urgent.
Redundant instructions, duplicate context and oversized retrieval results are removed. Guardrail: tested against your evals before shipping.
Answers stay as long as they need to be, with sensible caps on output and reasoning tokens. Guardrail: quality checks on every capped route.
Simple requests go to smaller equivalent models you approve. Guardrail: your evaluations and a canary rollout first. See routing.
Duplicate requests via Idempotency-Key, runaway concurrency loops and wasted retries are caught and stopped. Guardrail: key and workspace limits you control.
We compare official providers, negotiated pricing and commitment discounts for your volume.
Task-specific fine-tuning or distillation, only when your workload economics clearly support it.
What's automatic and what we tune together
- Provider routing and failover
- Provider prompt, prefix and context caching with cache-aware routing
- Prepaid wallet controls, concurrency caps and idempotency checks
- The live savings dashboard (
/app/overview)
- Allowed models and routing rules
- Prompt and context trimming
- Output limits
- Model-routing tests against your evaluations
- Which jobs move to batch processing
- Retry and agent-loop limits
Once they're set up, these changes run in the gateway on every request.
How we decide what to switch on
- Baseline. We record your cost per equivalent successful request, plus quality, latency and error rate. See the savings methodology.
- Rank. Every opportunity is ranked by expected dollars saved, risk and effort.
- Test. Changes are tested against your evaluations, then rolled out to a small share of traffic.
- Measure. Savings, quality and speed are tracked live. A change that makes anything worse is rolled back.
Want us to find your 20%+?
Get a free 360° cost report. Bring last month's provider usage and we'll show you your biggest waste and the first fix we'd ship.
Get a Free 360° Cost Report