Context caching needs reuse ledgers

Caching is a workflow decision, not a discount line

Context caching is easy to misunderstand because it arrives dressed as infrastructure optimization. A provider says repeated input tokens can be cached. A developer notices lower latency or lower cost on repeated prompts. A finance lead sees a cheaper run. Everyone relaxes too early.

The operating thesis is stricter: context caching needs reuse ledgers. If a recurring LLM workflow depends on the same policy pack, product documentation, rubric, catalogue, customer history or examples, the team should not treat caching as a hidden discount. It should record why that context is safe to reuse, when it becomes stale, how a cache hit is verified and what happens when reuse is no longer allowed.

Google’s Gemini API context caching documentation is a useful factual anchor because it describes repeated input tokens, implicit caching for Gemini 2.5 and newer models, prompt-prefix practices that improve the chance of a hit and usage metadata that reports cached tokens. Those facts are useful, but they are not an operating model. The missing artifact is a small ledger owned by the workflow team.

This is separate from deciding how much context an agent should carry at all. A context budget asks whether the agent is carrying too much state before release, as in Local agents need context budgets. A reuse ledger asks a different question: if this block appears again and again, should we cache it, refresh it, shorten it or externalize it?

What goes into Marco’s cache-reuse ledger?

The ledger is deliberately boring. It should fit in a spreadsheet, a YAML file, a product operations register or the workflow repository. If the team needs a complex tool before it can answer these questions, the workflow is already too opaque.

For each repeated context block, record seven fields.

First, the stable prefix. This is the part of the prompt that must remain structurally consistent for reuse to be plausible. In practice it might include the system instruction, the role frame and the large common content placed early in the request. If the prefix keeps changing because every team member edits the prompt differently, caching is not a governance strategy. It is luck.

Second, the reusable context block. Name the actual material, not a vague category. “Support refund policy v14” is inspectable. “Company knowledge” is not. A product or ops lead should be able to open the source, see the exact block and decide whether it belongs in the prompt at all.

Third, cache-hit evidence. The ledger should point to the run logs, usage metadata or dashboard field that shows cached tokens were actually counted. Gemini exposes cached-token reporting in usage metadata, but the broader rule is provider-neutral: do not call a workflow cached unless you can show evidence at the run level.

Fourth, the staleness owner. Every reusable block needs a human or team responsible for knowing when it has expired. Legal owns legal policy. Support owns macro language. Product owns feature entitlement rules. Data owns schema descriptions. If nobody owns staleness, the block should not be optimized for reuse.

Fifth, the invalidation trigger. Write the event that breaks reuse before the event happens. Examples include a policy update, pricing change, model migration, evaluation regression, incident review, documentation release, retrieval index change or customer segment expansion. A trigger that says “when it feels old” is not a trigger.

Sixth, fallback behavior. If the cache is missed, invalidated or suspected to be stale, does the workflow run uncached, retrieve fresh context, ask a human, downgrade the answer, or stop? This matters because cheap reuse can train teams to ignore the non-happy path.

Seventh, the decision threshold. The team should state the rule that justifies caching instead of shortening or externalizing the context. A simple rule works: cache only when the block is large, repeated, owned, measured and stable enough that the operational risk is lower than the savings. If one of those adjectives is not true, choose another path.

When should repeated context be cached?

Cache repeated context when reuse is frequent, the prefix is stable and the stale-context risk is actively owned. Do not cache merely because a block is long.

A product documentation pack used in every support triage run may be a good candidate. A compliance note that changes every week may not be, even if it is long. A customer-specific history block may belong outside the reusable prefix if it is volatile or privacy-sensitive. A set of examples for formatting an internal report may be safe to cache if it rarely changes and failures are easy to detect.

This is the decision rule I use with teams: repeated context has four possible destinations.

Cache it when the content is stable, large, reused often and measurable through cache-hit evidence. Refresh it when the content is valuable but has a known expiration rhythm. Shorten it when the block exists because nobody has edited the prompt since the prototype. Externalize it when the model should not carry the material in the prompt at all, for example because the answer should depend on retrieval, a database lookup, a policy service or a tool call.

This is where caching touches broader AI workflow governance. If a workflow already has escalation lanes, fallback states and owners, caching can make it more efficient. If the workflow is just a prompt in production, caching makes the mess cheaper. That is not progress. It is the same lesson behind AI workflows need escalation lanes: the model path is only one lane in the operating system.

Cache hits are not quality signals

A cache hit proves reuse, not correctness. It says the workflow benefited from repeated input tokens. It does not say the repeated context was current, sufficient, safe or well-scoped.

This is the adversarial risk. Caching can make bad context cheaper and faster. A stale refund rule becomes more scalable. A bloated prompt becomes more affordable. A brittle instruction becomes easier to keep. Optimization can delay the harder conversation: should this context still be here?

That is why the reuse ledger should sit beside evals, not replace them. Hamel Husain’s writing on AI product evals is useful here because it frames robust evaluation systems, trace inspection and recurring tests as central to improving LLM products. A cache ledger answers whether context reuse is controlled. Evals answer whether the workflow behavior is acceptable. You need both.

A practical review can be lightweight. Once per release cycle, sample the top recurring workflows by token spend or volume. For each one, inspect the ledger and ask five questions. Is the prefix still stable? Is the reusable block still the right block? Do logs show cache-hit evidence? Did any invalidation trigger fire? Did evals or incidents reveal failures tied to the repeated context?

If the answer to any question is unclear, do not simply disable caching. Decide whether to refresh, shorten or externalize. The goal is not to punish optimization. The goal is to make the optimization inspectable.

The operating artifact beats the provider feature

Provider features will keep changing. Minimum token thresholds, model families, usage fields and cache behavior will evolve. The stable practice is not a specific API setting. It is the discipline of recording reuse as a product decision.

A good cache-reuse ledger makes three conversations easier. Engineering can see whether the workflow is technically benefiting from reuse. Product can see whether the repeated context still represents the intended behavior. Operations can see who owns staleness and what happens when the safe path breaks.

The ledger does not need ceremony. It needs enough structure that an experienced reader can challenge the decision. If a row says “cache the policy pack” but has no owner, no invalidation trigger and no evidence of hits, the decision is not ready. If a row shows stable prefix, named reusable block, cached-token evidence, owner, trigger, fallback and threshold, the team can cache with eyes open.

That is the practical boundary: never let context caching be only a bill reduction mechanism. Treat it as a workflow-design decision. Cache the blocks that deserve reuse, refresh the blocks that age predictably, shorten the blocks that survived from the prototype and externalize the blocks that belong in retrieval or tools. The ledger is small, but it forces the right question before the discount hides the risk.