The fixed synthetic SproutRoute workload clears the 30% cost target without relying on cache hits.
Implemented platform capability / model access and economics
Control model cost, privacy, and fallback.
An OpenAI-compatible gateway moves provider choice, data protection, semantic reuse, fallback, quota, and cost attribution out of product code. SproutRoute has a tested, reversible integration path.
Local executionCommitted deterministic benchmark
- Gateway tests
- 47 passing
- Cold cost delta
- −66.41%
- Warm cost delta
- −82.81%
- Quality delta
- −1 point
- Decision
- Centralize routing policy while leaving product behavior and a direct-provider rollback with the application team.
- Why
- Duplicated provider logic obscures spend, weakens data controls, and makes fallback behavior impossible to govern consistently.
- Demo and benchmark
- Try the routing policy in your browser with a simulated provider. The figures below come from a 50-request synthetic benchmark.
Request pathFail closed
Policy sits between identity and inference.
Attribute
Authenticate
Hashed service credentials scope tenant, service, environment, role, model, and data class.
Protect
Classify
DLP blocks restricted data, redacts approved identifiers, and enforces regional routing before provider access.
Decide
Route
Versioned policy selects a model, samples quality, opens circuit breakers, and advances a bounded fallback chain.
Conserve
Reuse
Tenant-safe semantic caching and request, token, concurrency, and spend limits protect compute margin.
Measured evidence50-request workload
Lower cost on a fixed test workload.
A controlled 50% reuse pattern shows the economic upside of tenant-safe semantic caching.
The in-process load harness holds the shared envelope, denies 408 excess requests, and returns no cross-tenant cache result.
LLM_GATEWAY_URL and a scoped credential opt SproutRoute in; leaving either unset preserves the direct route.
Next decisionExternal validation
The next step: live-provider testing.
The next gate is a credentialed three-provider replay through LiteLLM and Redis, including one real provider failure, measured network p95, and a quality review using the same frozen workload.
Next capability: runtime tracesReturn to the operating modelTest reportsLocal execution
Explore the benchmark and routing decisions.
The lab shows how routing, privacy, and quota rules affect a request. The reports below contain the synthetic benchmark results and tenant-isolation tests.