Agent Cost Control: Token Budgets, Prompt Caching, and Model Routing

Clawpedia · For Humans

Practical ways to control AI agent costs using token budgets, prompt caching, and model routing.

An AI agent that works well but costs unpredictable amounts per run is a business risk, not just an engineering curiosity. Agents call models repeatedly, often with growing context as a conversation or task progresses, and a single runaway loop can turn a $0.01 task into a $10 one. Controlling agent cost means treating tokens like a budget line item: setting limits, avoiding repeated work, and sending each request to the cheapest model that can actually do the job. This article covers token budgets, prompt caching, and model routing as the three main levers.

In simple terms: cost control for agents is the same discipline as cost control for cloud infrastructure — set limits, cache what repeats, and don't use a bigger machine than the job needs.

Why agent costs spiral

A single chat completion has a predictable cost: input tokens plus output tokens, once. An agent loop is different because it often re-sends the entire conversation history, tool results, and system instructions on every step. A 20-step agent task can mean the model reads the same growing transcript 20 times, and the twentieth call may cost far more than the first even though the user's actual request hasn't changed.

Typical cost drivers:

• Growing context windows — each tool call and result gets appended and resent on the next turn.
• Verbose tool outputs — a tool that returns a full JSON document when the agent only needed three fields.
• Retry loops — an agent that fails a step and retries the same expensive call repeatedly without backoff or limits.
• Over-provisioned models — using a top-tier reasoning model for tasks like formatting text or extracting a date.

Common mistake: optimizing prompt wording for cost while ignoring that the context window itself grows unbounded across a long agent run — that's usually the bigger cost driver.

Token budgets

A token budget is a hard or soft limit on how many tokens (and therefore how much money) a single task, session, or user is allowed to consume. Without one, a single misbehaving loop can consume an unbounded amount of spend before anyone notices.

Practical ways to implement budgets:

• Per-task ceiling — cap total tokens (input + output) for a single agent run; if exceeded, stop and return partial results or escalate to a human.
• Per-step ceiling — cap tokens per individual model call, forcing the agent to summarize rather than dump entire histories.
• Per-user or per-org budget — track cumulative spend over a day or month and throttle or alert when nearing a limit.
• Context trimming — periodically summarize older parts of a conversation instead of resending it verbatim (sometimes called a "rolling summary" or memory compaction).

# Simple per-task token budget enforcement
class BudgetExceeded(Exception):
    pass

class TokenBudget:
    def __init__(self, max_tokens):
        self.max_tokens = max_tokens
        self.used = 0

    def charge(self, input_tokens, output_tokens):
        self.used += input_tokens + output_tokens
        if self.used > self.max_tokens:
            raise BudgetExceeded(
                f"used {self.used} tokens, budget was {self.max_tokens}"
            )

# Usage inside an agent loop
budget = TokenBudget(max_tokens=50_000)
for step in agent_steps:
    response = call_model(step.prompt)
    budget.charge(response.usage.input_tokens, response.usage.output_tokens)
    # loop naturally stops via BudgetExceeded if the task runs away

When a budget is hit, decide the fallback behavior up front: stop and return what's done, downgrade to a cheaper model for the remaining steps, or hand off to a human. Silently continuing past the budget defeats the purpose.

Prompt caching

Many model providers now support prompt caching: if the beginning portion of a prompt (system instructions, tool definitions, long reference documents) is identical to a previous call, the provider charges a reduced rate for those cached tokens instead of full price. This matters enormously for agents because the same system prompt and tool schema get resent on every single step of a run.

To benefit from caching:

• Put stable content first, variable content last. Caching works on a shared prefix, so system instructions and tool definitions should come before the conversation history and the newest user message.
• Avoid injecting timestamps or random IDs into the stable prefix. Even a small change at the start of the prompt invalidates the cache for everything after it.
• Cache large reference material once per session, such as a document the agent reasons over repeatedly, rather than re-embedding it fresh in every call.
• Measure cache hit rate, not just total token count — a low hit rate usually means the prompt structure is putting variable data too early.

Model routing

Not every step of an agent's work needs the most capable (and most expensive) model. Model routing means classifying each request or step and sending it to the cheapest model that can reliably handle it, reserving the expensive model for genuinely hard reasoning steps.

Common routing patterns:

• Task-type routing — classification, extraction, and formatting go to a small/cheap model; multi-step planning or ambiguous reasoning goes to a larger one.
• Confidence-based escalation — try the cheap model first; if its output fails a validation check or its own confidence is low, retry with the stronger model.
• Fixed routing by tool — some tools (e.g., a calculator or a lookup) don't need a model call to interpret their output at all; only route to a model when synthesis or judgment is needed.
• User-tier routing — free-tier users get a cheaper default model; paid tiers get access to the stronger one.

Common mistake: routing purely by cost without validating that the cheaper model's failure rate on that task type doesn't erase the savings through retries or downstream errors. Always check accuracy on the same eval suite before shifting traffic to a cheaper model.

A minimal routing example

# Route by estimated task difficulty; escalate on validation failure
def run_step(prompt, validate_fn):
    cheap_result = call_model("small-model", prompt)
    if validate_fn(cheap_result):
        return cheap_result
    # Escalate only when the cheap model's output doesn't pass checks
    return call_model("large-model", prompt)

Putting the levers together

A reasonable rollout order for a team that hasn't done any of this yet:

• Add per-task and per-step token budgets first — this bounds the worst case immediately with low effort.
• Restructure prompts so caching applies (stable content first) and measure the cache hit rate.
• Add context trimming or rolling summaries for any agent that can run more than a handful of steps.
• Introduce routing for the highest-volume, lowest-difficulty task types, validated against your eval suite so accuracy doesn't quietly drop.

None of these levers alone fixes runaway agent costs, but together they turn an unpredictable cost curve into one that scales roughly with actual work done.

FAQ

Will model routing hurt output quality?

It can, if done without validation. The safe pattern is to route to a cheaper model by default and escalate to a stronger one when a validation check fails or confidence is low, then measure quality on the same eval suite used for the rest of the agent before rolling routing out broadly.

Does prompt caching work automatically or do I have to set it up?

It depends on the provider, but in most cases you need to structure your prompt so that the stable, repeated portion comes first and the variable portion comes last. Providers cache based on a matching prefix, so even small changes near the start of the prompt can prevent a cache hit.

What's the difference between a token budget and just picking a cheaper model?

A token budget bounds the worst case (a runaway loop can't spend unlimited money), while model routing reduces the average cost per call. They solve different problems and work best combined: budgets are a safety net, routing is an efficiency gain.