Post cover image

July 5, 2026

How I cut my AI Agent’s Token Bill - A 3 Layers Model

Your AI agent re-sends the whole conversation every turn. Here is the 3-layer system I use to cut the token bill, and how each layer works.

By Amit Karni

8 min read

Your AI agent re-sends the whole conversation every turn. Here is the 3-layer system I use to cut the token bill, and how each layer works.

By Amit Karni · Last updated: July 2026

Your AI agent has the memory of a goldfish and the spending habits of a taxi with the meter running. Every turn, it forgets the whole conversation, re-sends it to remind itself, and hands you the bill for the reminder.

That is the cost model nobody puts on the pricing page. You think you are paying for answers. You are really paying to re-transmit the entire transcript, again and again, one turn at a time. Cost tracks the length of the session, not the length of your question. The reply is the cheap part. The context is where your money quietly goes to die.

Once that clicks, the goal stops being "find a cheaper model" and becomes "stop shipping tokens you do not need to ship." That does not happen in one place. Tokens sit in three pools, so I run one layer per pool. They compose cleanly, because no two of them touch the same tokens.

  • Transport: shrink the context on the wire. → Headroom
  • Output: trim what the model says back. → Caveman
  • Design: generate less code in the first place. → Ponytail

The savings are nice. The mechanisms are the fun part, and yes, I said fun about a token bill. Stay with me.

Layer 1 · Headroom: treat context as a transport problem

Headroom is a reverse proxy that speaks the model provider's own API. You repoint one base URL at it and nothing else in your stack changes. What earns it a place in the request path is the engineering underneath.

Prefix stability for cache reuse. The highest-leverage move is not compression at all. It is keeping the front of each request byte-stable across turns so the provider's prompt cache keeps hitting instead of re-reading context it has already seen. Most setups quietly break their own cache by letting the prefix drift. Headroom's job is to stop that from happening.

Content-aware routing. It classifies each block and compresses it with the right codec: structural crushing for JSON, syntax-aware compression for code so your symbols and identifiers survive, and a learned model for prose. A single generic "summarize the prompt" pass cannot do this without shredding fidelity. Different content wants different compression.

Reversibility as a first-class property. This is what makes it safe to run inline. Compression is lossless-in-effect: it keeps the originals and injects a retrieval tool, so if the model needs the full text of anything, it asks for it and gets it back. You are not betting that a summary was good enough. The full context is one tool call away, on demand.

An honest tradeoff, exposed. Aggressively rewriting old turns maximizes shrink but works against cache reuse. Freezing them maximizes cache hits but shrinks less. Headroom exposes both modes instead of pretending the tension does not exist, and you pick per workload.

The reason a senior engineer trusts this layer is that it behaves like infrastructure: transparent, model-agnostic, reversible, observable, and toggleable. You reason about it the way you reason about a caching proxy, because that is what it is.

Layer 2 · Caveman: steer the output, do not post-process it

Caveman is a skill for the agent, and the elegant part is what it is not. It is not a filter that rewrites responses after they arrive. It is instruction injection: a hook loads the ruleset when the session starts, and another reinforces it on every turn so the model never drifts back to its verbose default. You are shaping the output distribution live, not cleaning up afterward.

The properties that matter if you care about correctness:

  • It strips only scaffolding. Pleasantries, hedging, and restating the question go. Code, symbols, and error strings stay verbatim.
  • It touches output tokens only, never the reasoning. It is orthogonal to whether the answer is right.
  • It degrades gracefully. If the model ignores it on a hard turn, you lose nothing. It is just context, so it composes with everything.

And there is a quality-of-life win that has nothing to do with the invoice. Ask a normal model for a one-line change and it hands you back a small Harry Potter book: a chapter of preamble, a recap of what it is about to do, three paragraphs of context, and a heartfelt epilogue about next steps. For a one-character fix. Caveman skips the saga. You get the fix and get on with your day, and honestly that alone is worth it.

The shape of the change: instead of a page of preamble wrapped around a one-line fix, you get the one-line fix. Same content, far less packaging. Smaller mouth, not smaller brain.

Layer 3 · Ponytail: attack the tokens before they are generated

Same delivery mechanism as Caveman, injected instructions rather than a post-processor, but aimed at the most upstream pool of all: the code itself. Think of it as a lazy senior developer, the good kind, who deletes more than they add and calls it a productive afternoon. The other two layers make an existing payload smaller. Ponytail keeps the payload from being large to begin with.

It encodes the taste a good staff engineer already has, as a decision ladder the agent walks, stopping at the first rung that holds:

  1. Does this need to exist at all? Skip speculative work.
  2. Does the standard library already do it?
  3. Does the platform already do it? Reach for the native primitive before a dependency.
  4. Does something you already depend on solve it?
  5. Can it be one line?
  6. Only then, the least code that works.

What makes it disciplined rather than reckless: it never trades away validation, error handling, security, or accessibility, and it marks every deliberate shortcut with a comment, so the debt is auditable instead of hidden. And it is the highest-leverage layer of the three, because code you do not generate today is context you do not re-send on every turn after. It cuts output now and input forever.

Why running all three works

They are orthogonal by construction, so they stack without interference. In a single turn: Ponytail shapes a smaller solution, Caveman trims the prose around it, and Headroom compresses whatever is left while keeping the prefix cacheable. Then the loop closes on itself. Because Ponytail wrote less, the next turn's re-sent context is already smaller, which means less for Headroom to compress and less for you to pay to transmit. Each layer lowers the baseline the next one runs against.

None

My setup, in a few words

Headroom runs as two always-on proxies. One sits in front of the Anthropic API, the other in front of Ollama Cloud, where I run models like GLM. Same compression either way. That is the whole point of doing it at the transport layer: it rides below the model, so whatever I route to still gets compressed. One global base-URL variable puts every agent profile through it with nothing configured per project, and a dashboard shows what happened. Set once, always on.

Taking it globally to production: compression in front of the model gateway

On one laptop this is a personal optimization. For a team it becomes an architecture decision, and this is where it gets genuinely interesting from an AI standpoint.

We route the whole team through LiteLLM, a model gateway that presents one API and fans out to many providers behind it. One endpoint, and a request can land on Claude, GPT, Gemini, GLM, or whatever else we have wired in. That is already the right shape for an AI platform: callers ask for a capability, not a vendor.

The move that pays off is putting Headroom in front of the gateway, so compression happens once, centrally, for every model the router can reach:

client → compression → gateway (routing across models) → provider

Three things make this click as an AI design, not just an ops one:

  • Compression becomes provider-agnostic. Because Headroom sits above the router and the router speaks to every backend, one compression tier covers the entire model fleet. Add a new model to the gateway and it inherits compression for free. Nobody re-implements anything per provider.
  • Reversibility survives routing. Headroom's compression is lossless-in-effect and protocol-faithful, so the gateway can still route or fail over to a different model and the request stays intact. The model on the other end can still pull back any original it needs. Compression and routing do not fight.
  • Governance rides along. Each agent authenticates with a scoped key that carries its own budget and model allow-list, so you control which agent can reach which model and cap what it spends, while the real provider credentials stay behind the gateway and never touch a client. For a fleet of autonomous agents, that containment matters as much as the savings.

We run the compression tier as its own service with a switch to bypass it instantly if it ever misbehaves. The deployment mechanics are ordinary platform work and not the interesting part. The interesting part is that "compress the context" stopped being a per-developer trick and became a property of the platform, applied uniformly to every model, every agent, every request.

The takeaway

The deliverable is the mental model, not a shopping list. Tokens live in three pools, so you intervene at three layers: compress the wire, steer the reply, generate less code. They stack because they are orthogonal, and the biggest wins live on the way in, where your agent keeps re-sending everything it already sent. Do it once per developer with a local proxy, or once for everyone by putting compression in front of your model gateway. Either way, the cheapest token is still the one you never send again.

FAQ

What does "saving LLM tokens" mean? It means reducing the tokens sent to and returned from a language model, which lowers cost, speeds up responses, and frees space in the context window. You can intervene at three points: the context you send in, the words the model writes back, and the code it produces.

What is Headroom? Headroom is a local proxy that sits between an AI client and the model API and compresses the context before it is sent. It keeps request prefixes stable so the provider's cache keeps hitting, routes content by type for smarter compression, and stores originals so the model can retrieve anything it needs. You route to it by changing one base URL, with no code change.

What are Caveman and Ponytail? They are skills for coding agents. Caveman makes the agent write shorter answers to cut output tokens. Ponytail makes the agent write the smallest working solution to cut generated code. Both work by injecting instructions into the model, so they steer its behavior rather than post-processing its output.

Do these tools reduce answer quality? No. Headroom keeps originals and lets the model fetch them back, so accuracy holds. Ponytail never removes validation, error handling, security, or accessibility. Caveman only shortens wording, not reasoning.

Why put compression in front of a model gateway? Because a gateway routes one API to many providers, placing compression in front of it means every model the gateway can reach inherits the same compression, with no per-provider work. The compression is reversible, so routing and failover still work, and provider keys stay behind the gateway.

Where do the biggest savings come from? The context you send in, not the reply. Agents re-send the whole conversation every turn, so keeping that context small and cacheable is the highest-leverage move, which is why the transport layer tends to matter most.

References and further reading

The tools in this article:

Background on the mechanisms:

  • Anthropic, Prompt caching documentation, on why a stable request prefix keeps the provider cache warm: docs.anthropic.com

Disclosure: these are tools I actually run in my own setup. I am not affiliated with any of them, and this is not sponsored. Internal infrastructure names have been left out for publication.