Local agents need context budgets

Local agents need context budgets. That is the thesis, and it becomes more important as local and hybrid AI workflows move from impressive demos to daily work.

A larger context window looks like a simple upgrade. The agent can see more documents, more chat history, more tool output, more code, and more instructions. In practice, context is not a drawer where everything can be thrown without consequence. It is an operating envelope. Every token you add competes with another token, consumes memory, can increase latency, and can make failure harder to explain.

Ollama’s own documentation defines context length as the maximum number of tokens available to the model in memory. It also ties default context length to VRAM, notes that web search, agents, and coding tools should use at least 64,000 tokens, and warns that larger context requires more memory. That is enough to stop treating context length as a preference slider and start treating it as a workflow budget. See the official Ollama context length documentation for the operational baseline.

Context is not free memory

The mistake is not wanting a larger window. Many agent workflows genuinely need one. A coding agent may need the task brief, several files, tool results, test output, and a final patch explanation. A research agent may need retrieved pages, search traces, source snippets, user constraints, and a structured answer. A local support assistant may need product policy, recent conversation turns, account-safe summaries, and escalation rules.

The mistake is assuming that a larger window removes the need to decide what belongs inside it.

A 64k or 128k window can make teams lazy in a very specific way. They stop asking whether the prompt, retrieval layer, tool trace, and answer contract are each doing their job. They add more retrieved chunks instead of fixing retrieval precision. They preserve full logs instead of summarizing tool state. They let the agent carry old ambiguity forward instead of forcing a checkpoint. The window becomes bigger, but the workflow becomes less inspectable.

This is why the argument in RAG needs retrieval contracts, not bigger windows applies beyond classic RAG. If the agent cannot explain why a document entered the context, more capacity only gives it more room to mix useful evidence with stale or irrelevant material. The model may still answer fluently, which is exactly the danger.

A context budget restores the question: what is this workflow allowed to spend attention on?

What belongs in a context budget?

A useful context budget fits on one page. It should not be a theoretical token spreadsheet for every possible run. It should describe one real workflow, with realistic inputs and an explicit failure path.

Start with the task type. “Answer customer policy questions” is too broad. “Draft a refund eligibility answer using the latest policy excerpt, order summary, and two prior messages” is closer. The narrower task tells you which inputs deserve budget.

Then list the context lanes. A practical local agent budget normally includes six lanes.

First, reserve space for system and developer instructions. These should include the role, boundaries, output format, tool rules, and refusal or escalation behavior. If these instructions are squeezed by documents and traces, the agent may still have facts but lose operating discipline.

Second, reserve space for user intent and conversation state. Do not blindly carry the whole conversation. Use a compact state summary when possible, especially when older turns contain superseded decisions.

Third, allocate retrieved inputs. This is where teams often overspend. Instead of “top 20 chunks,” define document classes, maximum snippets, freshness rules, and exclusion rules. If you need more than the budget allows, that is a retrieval design problem, not only a context problem.

Fourth, allocate tool traces. Agents that call search, shell commands, databases, or internal APIs can flood the window with logs. Keep the evidence the final answer needs, not every line the tool produced. For governance, the full trace can live outside the prompt in observability storage, while the model receives a compressed working trace.

Fifth, reserve output space. This is the lane many teams forget. If the model fills the window with input, the answer has too little room to reason, cite, structure, or produce a patch. A context budget should explicitly reserve answer tokens for the expected final artifact.

Sixth, define the fallback. When the budget is exceeded, the agent should not silently stuff more material into the prompt. It should choose a known behavior: ask for a narrower scope, summarize intermediate state, run a second pass, escalate to a human, or refuse to proceed without a required source. That aligns with the operating mindset in AI workflows need escalation lanes.

How should builders test the budget before release?

Do not test only the happy path. A context budget is useful because it reveals where the workflow breaks.

Create three fixtures. The first is a normal case that should fit comfortably. The second is a boundary case that nearly fills the budget. The third is an overflow case that must trigger fallback. If all three cases simply pass because the agent answers something plausible, the test is too weak.

For each fixture, inspect four things.

Check inclusion. Did the required source material enter the context? If the final answer depends on a policy, code file, or customer fact, that material must be visible in the working context or explicitly summarized there.

Check exclusion. Did irrelevant material stay out? This matters as much as inclusion. Long-context failures often come from plausible noise, not from missing data.

Check trace shape. Did tool output arrive as usable evidence, or as a raw dump? Local agents are especially vulnerable here because teams can feel that local compute makes waste acceptable. Waste is still waste if it hides the reason for the answer.

Check output reserve. Did the model have enough room to produce the expected artifact? If the desired answer is a structured decision memo, a diff explanation, or a customer-safe response, reserve space for that shape.

This testing discipline belongs with runtime governance. Production agents need runtime guardrails because model capability alone does not make an agent production-ready. The same is true for context. A large setting is not a guardrail. It is a capacity limit that must be governed.

When should you shorten instead of enlarge?

Sometimes the right move is to increase the local context setting. If the workflow genuinely requires many pieces of evidence, the hardware can support it, and the agent still passes boundary tests, a larger window may be justified.

But many teams should shorten before they enlarge.

Shorten when retrieval is noisy. A larger window will not fix a weak source contract. It will only let weak retrieval bring more clutter.

Shorten when the agent repeats old assumptions. Carrying more chat history can preserve context, but it can also preserve confusion. Summarize decisions, discard superseded turns, and make current intent explicit.

Shorten when tool traces are verbose. The model does not need a complete database response if the task only needs three fields. The full response can be logged elsewhere.

Shorten when the answer format is unstable. If the model cannot reliably produce the required output, do not spend the whole budget on inputs. Reserve output room and tighten the contract.

Shorten when latency or memory pressure makes the workflow feel fragile. Ollama’s documentation is direct that larger context requires more memory. On local hardware, that constraint is not abstract. It affects what else can run, whether the model stays on GPU, and whether the workflow feels dependable.

Write the budget before moving the slider

The practical call to action is simple: before increasing a local agent context setting, write a one-page context budget for one workflow.

Name the task. List the lanes. Put rough token limits beside instructions, user state, retrieved inputs, tool traces, and reserved output. Define the fallback when the budget is exceeded. Add three test fixtures: normal, boundary, overflow. Then run the agent and inspect what actually enters the context.

This will feel slower than dragging a slider. It is also how a local agent becomes operable.

The point is not to keep context small for its own sake. The point is to make context accountable. Local agents need context budgets because context length is not capability by itself. It is a scarce operating surface where evidence, instructions, memory, traces, and answers compete. If you do not budget it, the agent will spend it for you.