Fast enough to forget it is there.
+1.52 ms median overhead when the request goes to the model; cache hits answered in 1.08 ms instead of a model call. A model call itself usually takes hundreds of milliseconds to seconds.
Latency
Milliseconds, concurrency 1, 2,000 requests per scenario after 200 warm-up requests. "Added" is the gateway percentile minus the same percentile for calling the mock provider directly (direct: p50 0.58, p95 1.07, p99 3.56).
| Scenario | p50 | p95 | p99 | Added p50 | Added p95 | Added p99 |
|---|---|---|---|---|---|---|
| Model call (cache miss) temperature 0.7, so the cache is bypassed: cache-control formatting, routing advice, the upstream round trip, usage parsing, the ledger write and receipt signing | 2.10 | 4.49 | 7.24 | +1.52 | +3.42 | +3.68 |
| Exact cache hit temperature 0, identical body: answered from the cache, no model call | 1.08 | 1.95 | 4.89 | +0.50 | +0.88 | +1.33 |
| Similarity cache hit temperature 0, whitespace-varied body: exact miss, then a SimHash match (opt-in feature) | 1.22 | 2.08 | 4.70 | +0.64 | +1.01 | +1.14 |
| Streaming, first byte stream: true; time until the first chunk reaches the client (direct: p50 1.14, p95 2.56, p99 4.36) | 3.12 | 6.14 | 8.63 | +1.98 | +3.58 | +4.27 |
| Streaming, complete mock emits the first chunk at once, then one every 15 ms; a buffering proxy would add ~75 ms to first byte; usage is read off the final chunk and the receipt signed after the stream ends | 80.0 | 83.4 | 94.8 | +2.46 | +3.01 | +12.7 |
Throughput, one process
Closed loop, 32 concurrent clients for 10 s against one gateway process. Production runs one process per core, and throughput grows with cores because ledger writes are batched; these single-process numbers are a floor, not a ceiling.
| Path | Requests / s | Errors |
|---|---|---|
| Mock provider, called directly (the load generator's ceiling) | 3,542 | 0 |
| Through TokenZero, model call | 641 | 0 |
| Through TokenZero, exact cache hit | 1,670 | 0 |
Method
A zero-latency mock provider and the gateway run as separate processes on the same machine, so the numbers isolate what TokenZero adds and leave out network distance to the provider. The request is ~1.4k input tokens, OpenAI chat, non-streaming, with the full engine path on: token counting, cache-control formatting, routing advice, usage parsing from the provider response, a ledger row and an Ed25519-signed receipt. Nothing is stubbed out of the gateway.
Machine: 8 vCPU Intel Core Processor (Haswell, no TSX), shared with other workloads (load average 6.0 / 5.5 / 6.1 during the run), so these are conservative. Reproduce it on your own hardware; it needs no keys and no network:
node --no-warnings scripts/bench/overhead.ts --n 2000 --seconds 10 --concurrency 32 --out results.json
Audit analyzer: golden corpus
13 committed files (7 with a known anti-pattern across Python and TypeScript, OpenAI, Anthropic, Gemini, LangChain and the Vercel AI SDK; 6 clean files that tempt a naive linter). Reproduce: node --no-warnings scripts/bench/audit-corpus.ts.
| Case | Kind | Expected | Found | Est. waste | Result |
|---|---|---|---|---|---|
app/triage.pypy-openai-classifier-uncached | anti-pattern | static-prompt, cascade, repeat-cache | cascade, static-prompt, repeat-cache | 70% | pass |
src/assistant.tsts-anthropic-system-no-breakpoint | anti-pattern | static-prompt | static-prompt | 70% | pass |
jobs/enrich.pypy-loop-per-item | anti-pattern | loop | loop | 15% | pass |
lib/extract.tsts-vercel-ai-json-extraction-frontier | anti-pattern | cascade | cascade | 70% | pass |
svc/sentiment.pypy-gemini-static-prefix-sentiment | anti-pattern | static-prompt, repeat-cache | static-prompt, repeat-cache | 12% | pass |
pipeline/translate.pypy-langchain-translate-frontier | anti-pattern | cascade | cascade | 70% | pass |
svc/digest.pypy-retiring-model | anti-pattern | lifecycle | lifecycle | 0% | pass |
src/agents/models/openai_chat.pylib-sdk-wrapper | clean | none | none | 0% | pass |
src/cached.tsts-anthropic-already-cached | clean | none | none | 0% | pass |
app/tag.pypy-already-cheap-model | clean | none | none | 0% | pass |
docs/models.mddocs-model-names-in-prose | clean | none | none | 0% | pass |
jobs/batch_enrich.pypy-batched-loop | clean | none | none | 0% | pass |
src/summarize.tsts-openai-cache-options-set | clean | none | none | 0% | pass |
Each case lists the finding kinds it should produce; a kind reported but not expected is a false positive, an expected kind not reported a false negative. The clean-code false-positive rate counts clean files that received any medium or high finding (info/low advice does not count).
Head to head
Only rows marked measured carry numbers from this script on this machine. The rest are product facts or the vendor's own claims, labelled as such; we do not publish competitor latency we did not measure.
| Product | Overhead per request | Signed receipts | Pricing model | Static waste analyzer | Runtime dependencies |
|---|---|---|---|---|---|
| TokenZero | +1.52 ms p50 / +3.42 ms p95 measured | Ed25519-signed per request, public key published | $0 on spend; 15% of verified savings above $500/mo routed (planned) | local static audit (CLI, MCP, GitHub Action), 12/12 golden corpus | single Node process, SQLite; no Redis/Postgres |
| LiteLLM Proxy | +16.3 ms p50 (17.0 vs 0.58 direct), p95 21.8 measured, litellm LiteLLM: Current Version = 1.106.4 | none | open source; enterprise tier | none | Python/FastAPI; Postgres and Redis for most features |
| Portkey | vendor claim: single-digit ms | none | per-request plans | none | hosted |
| Helicone | vendor claim: ~ms (edge) | none | per-request plans | none | hosted |
| OpenRouter | not measured | none | 5.5% on all spend | none | hosted |
| RouteLLM | library, not a proxy | none | open source | none | Python library, router models |