Token Compression

I came across Token Compression recently, with AI Model costs increasing and latency becoming important especially with voice AI Agents. This topic struck a chord. Lets dive into Token Prompt Compression

Token / Prompt compression is not about shorter prompts.

It is about higher-signal context.

As AI agents move from single-turn chats to long-running workflows, the prompt starts filling up with everything:

chat history
retrieved chunks
tool outputs
API responses
memory
user preferences
system instructions
intermediate state

Most teams keep sending all of it back to the model.

That creates three problems:

more tokens  → higher cost
more context → higher latency
more noise   → worse answers

The surprising part:

More context does not always mean better output.

The model does not need everything.

It needs the right things.

That is where prompt compression helps.

Prompt compression converts large, noisy context into smaller, task-relevant context.

A simple example:

Instead of passing five pages of conversation history, convert it into state:

{
  "user_goal": "renew home insurance",
  "objection": "does not want auto bundle",
  "status": "ready to buy",
  "next_action": "prepare final quote"
}

This is not just shorter.

It is easier for the model to reason over.

There are a few useful types of compression:

1. Lexical compression
Remove repeated text, boilerplate, empty fields, metadata, HTML, formatting noise.

2. Semantic compression
Use a smaller model to summarize long conversations, documents, or tool traces.

3. Structural compression
Convert messy context into JSON, state, entities, tasks, and constraints.

4. Retrieval compression
Retrieve less, rerank better, dedupe chunks, and pass only what matters.

5. Memory compression
Store durable facts instead of replaying the entire conversation history.

In production, I think of prompt compression as a pipeline, not one function:

clean
→ structure
→ preserve exact facts
→ compress
→ budget
→ generate

For example, start with cheap deterministic compression first:

import re

def lexical_compress(text: str) -> str:
    text = re.sub(r"<[^>]+>", "", text)          # remove HTML
    text = re.sub(r"\s+", " ", text)             # normalize whitespace
    text = re.sub(r"(?i)disclaimer:.*", "", text)
    return text.strip()

Then convert verbose tool output into compact state:

def compress_customer_record(api_response: dict) -> dict:
    return {
        "customer_id": api_response.get("id"),
        "plan": api_response.get("plan"),
        "status": api_response.get("status"),
        "open_invoice": api_response.get("billing", {}).get("open_invoice"),
        "renewal_date": api_response.get("billing", {}).get("renewal_date"),
        "support_tier": api_response.get("support", {}).get("tier"),
    }

Then apply a context budget before calling the final model:

def fit_context(chunks, max_tokens):
    selected = []
    used = 0

    chunks = sorted(chunks, key=lambda x: x["priority"], reverse=True)

    for chunk in chunks:
        tokens = chunk["tokens"]

        if used + tokens <= max_tokens:
            selected.append(chunk)
            used += tokens

    return selected

But compression is not free.

If you need another LLM call to compress the prompt, the math has to work.

Bad tradeoff:
2,000 tokens → 1,200 tokens
extra LLM call required
probably not worth it

Good tradeoff:
40,000 tokens → 5,000 tokens
before calling a larger reasoning model
often worth it

The real question is not:

Can we compress the prompt?

The real question is:

Does compression improve cost, latency, or accuracy?

There is also risk.

Compression can remove the wrong thing.

Bad compression:

Customer wants refund or replacement.

Better compression:

{
  "customer_preference": "replacement if delivered by Friday",
  "fallback": "refund",
  "condition": "refund only if replacement takes more than 3 days"
}

Good compression preserves decision logic.

Bad compression preserves only topic.

The best agent systems do not compress everything.

They use context budgeting.

Before every model call, they decide:

What must be included verbatim?
What can be summarized?
What can be retrieved?
What can be dropped?
What should be stored as state?
What should be cached?

This is the real architecture:

retrieval
+ memory
+ state
+ reranking
+ caching
+ compression
+ context budgeting

Prompt compression is not the architecture.

It is one layer inside the architecture.

The goal is not to send fewer tokens.

The goal is to send better context.

Learn more at Twig.so