Grafferene
Response cache for LLM endpoints
Know what caching would save your LLM traffic, before serving a single cached answer.
Put Grafferene in front of your model endpoint in measure-only mode for a week. Every request still goes to your model. At the end you get a report of what caching would have saved, and the answers it would have served, side by side, for you to judge.
Would have been served
| Count | Share | |
|---|---|---|
| Requests | 34 | |
| Exact repeats | 8 | 23.5% |
| Similar questions | 5 | 14.7% |
| Prompt tokens saved | 14,458 | 39.4% |
Review sample
| Asked | Answer stored for |
|---|---|
| How fast does light travel in a vacuum? | What is the speed of light in a vacuum? |
| Which city is the capital of Australia? | What is the capital of Australia? |
From a 34-request test trace built to repeat. Your numbers will differ; that is the point of measuring.
Why measure first
How much a cache saves depends entirely on how much your traffic repeats, and a similar question is not always the same question. Both are worth knowing before any cached answer reaches a user.
Repetition varies
Some traffic barely repeats
In a public dataset of open-ended chat, only about 5% of prompts repeated across users, mostly greetings. Support desks, internal tools and templated workloads can look very different. The only reliable number is yours.
Similar ≠ same
Lookalikes are real
On a public set of human-labeled question pairs chosen to look alike, about one in five similar-question matches at a strict threshold was in fact a different question. That is why the report shows you what would have been served, not just how often.
Nothing changes
Your users see your model
In measure-only mode the cache answers nothing. It runs its lookups beside your traffic and records what it would have done, checked against your model server's own counters.
How it decides what to serve
When serving is switched on, an answer comes from the cache only if every check below passes. Anything else goes to your model, as it would without a cache.
Exact repeats
The same request, in the same tenant and namespace, gets the stored answer.
Similar questions, only in the same context
A reworded question can reuse an answer only when it was asked in the same context as the original. The same words asked under different instructions, or in another namespace, are treated as a different question and go to your model.
Optional verification before serving
A similar-question match can be sent to a separate verification service you run. It is served only if the verifier accepts it.
When in doubt, it refuses
If verification cannot complete, the match is not served.
Everything else goes to your model
A miss costs one lookup: about 20 ms over calling the model directly in our single-node test.
The pilot: one week, measure-only
It runs in your infrastructure, in front of your OpenAI-compatible endpoint. Your prompts and answers stay with you, and the report is generated where the cache runs.
1 · Deploy
Container and Helm chart
Point it at your endpoint with measure-only mode on. Tested in front of vLLM, on Kubernetes and on a bare host.
2 · Run
A week of real traffic
Every request reaches your model. The cache records would-be hits and the tokens those requests spent, as metrics and in a per-request decision trail.
3 · Decide
Read the report
Requests, would-be exact and similar-question hits, tokens and cost at your own prices, a day-by-day table, and a review sample of the least similar matches. Then decide whether, and how strictly, to serve.
What has been measured
Each result below comes from a benchmark with a stated setup. Savings were checked against the model server's own counters, not the cache's.
Measure-only mode reported exactly the hits a serving run of the same trace produced, row by row, while every request still went to the model.
One 32-request trace against vLLM, plus two scoping checks; laptop GPU.
With a strict threshold, the cache served 13 of 32 requests and cut prompt tokens by 41%, with no wrong answers on that trace. The same question asked under different instructions, or in another namespace, always reached the model.
A trace built to repeat, so the percentage describes the trace, not your traffic.
The approximate search index found every similar-question match the exhaustive search found, at 10 ms per search instead of 87 ms.
One million real embeddings, two held-out query sets of 1,000; laptop CPU, search only.
With 16 clients asking the same questions at once through an llm-d router, identical in-flight requests were combined: as many model calls as one client made, with no wrong answers.
One node, one GPU, clients in lockstep: the best case for combining requests.
Built for your infrastructure
Container image and Helm chart. Cache and decision trail survive restarts, encrypted at rest. Replicas can share one cache through Postgres. Tenant tokens keep one tenant from reading another's entries. One OpenTelemetry trace per request; Prometheus metrics.
What we do not claim
No savings percentage for your traffic until it is measured. No promise that similar-question matching is always right; the review sample is there so you can judge. Multi-replica sharing is tested on a single-node cluster, not yet across nodes.
Request a pilot
Send us a message on LinkedIn with what sits behind your endpoint (model server, rough daily volume, kind of traffic), and we will set up a one-week measure-only pilot with you. The pilot is free.
Who is behind it
Grafferene is built by Alexander Chernov, whose work is on governed AI systems: deciding, with evidence, what an automated system may do.
Every figure on this page comes from a documented benchmark run.
About the name
Grafferene comes from graphene: a plane one atom thick, where every atom sits in a lattice cell fixed by its neighbors. The cache works on a similar plane of meaning. Every stored question has its place, a new question lands next to the ones closest in meaning, and only a close enough neighbor in the same context can lend its answer.