Token Compression for LLMs: How to Reduce API Costs by 60-95% Without Losing Context

Token compression is a set of techniques that reduce the number of tokens sent to an LLM API by summarizing, extracting, filtering, or chunking input content before it reaches the model. Tools like Headroom achieve 60% to 95% token reduction while preserving answer quality. The most impactful optimization for agent pipelines is compressing tool outputs between calls, which can cut per-task API costs from $0.05 to under $0.01.

Why Token Compression Matters in 2026

LLM API costs scale linearly with input tokens. A typical agent task that reads a file, queries a database, and searches the web can easily consume 10,000 to 50,000 input tokens per step. At GPT-4 class pricing ($10 to $15 per million input tokens), a single complex agent run can cost $0.50 to $2.00.

The problem is context bloat. MCP tools, RAG retrievers, and logging systems return far more data than the LLM actually needs. A search result might return 2,000 tokens when only 200 are relevant. A file read might pull in 5,000 lines when the answer lives in 10 lines. A database query returns full rows when only three columns matter.

Token compression sits between your data sources and the LLM, removing the noise while keeping the signal. The result is the same answer quality at a fraction of the cost.

The Five Core Compression Strategies

1. Summarization

Summarization replaces verbose content with a shorter version that preserves the key information. This is the most common and effective strategy for long documents, conversation histories, and tool outputs.

For example, a 3,000-token tool output describing a web search result can be summarized to 300 tokens that capture only the relevant facts. The LLM then receives a compact context that still answers the user's question.

You can implement summarization using a smaller, cheaper model as the compressor. A common pattern is to use a fast model like GPT-4o-mini or Claude Haiku to summarize tool outputs before passing them to a more expensive model for the final answer.

2. Extraction

Extraction pulls specific data points from unstructured content. Instead of sending an entire invoice PDF to the LLM, you extract only the line items, totals, and vendor name. Instead of sending a full log file, you extract only error messages and stack traces.

Extraction is more precise than summarization. While summarization creates a shorter narrative, extraction produces structured data that the LLM can reason over directly. This is particularly valuable for financial documents, legal contracts, and technical logs.

3. Semantic Filtering

Semantic filtering scores each chunk of content by relevance to the current query and drops chunks below a threshold. This is the standard approach in RAG pipelines, where you retrieve the top-K most relevant document chunks rather than sending the entire corpus.

The key insight is that most retrieved content is not relevant to the specific question being asked. A RAG system might retrieve 20 chunks, but only 5 contain information that helps answer the query. Semantic filtering drops the other 15, saving thousands of tokens.

4. Structured Output Constraints

Constraining the output format reduces output tokens, which are typically more expensive than input tokens. Instead of asking the model to "analyze this data and share your thoughts," you ask it to return a JSON object with specific fields.

This approach saves 50% to 80% of output tokens. A free-form analysis might generate 800 output tokens, while a structured JSON response with the same information uses only 150 to 200 tokens.

5. Context Window Chunking

When content exceeds the context window, chunking breaks it into smaller pieces and processes each chunk separately. Results from each chunk are then combined in a final synthesis step.

This is essential for processing long documents (100K+ tokens) that exceed even the largest context windows. The tradeoff is additional API calls, but each call uses far fewer tokens than attempting to fit everything into one massive request.

Headroom: The Open-Source Token Compression Tool

Headroom is the most popular open-source tool for token compression, with over 51,000 GitHub stars as of June 2026. It provides three deployment modes: a Python library, an HTTP proxy, and an MCP server.

The library mode lets you compress content programmatically before sending it to any LLM API. The proxy mode sits between your application and the LLM, automatically compressing requests in transit. The MCP server mode integrates directly with agent frameworks that support the Model Context Protocol.

Headroom claims 60% to 95% token reduction depending on the content type. Tool outputs and logs tend to compress the most (80% to 95%) because they contain significant redundancy. RAG chunks compress less (60% to 80%) because they have already been filtered for relevance.

Implementing Token Compression with Eden AI

Eden AI provides summarization and extraction APIs that function as a compression layer without requiring you to deploy additional infrastructure. You can use these to compress content before sending it to any LLM through the Eden AI unified endpoint.

Summarization as Compression

Use Eden AI's summarization feature to compress verbose tool outputs:

import urllib.request
import json
import os

url = "https://api.edenai" + ".run/v3/universal-ai"
headers = {
    "Authorization": "Bearer " + os.environ["EDENAI_API_KEY"],
    "Content-Type": "application/json",
}

# Step 1: Summarize verbose tool output
compress_payload = json.dumps({
    "model": "text/summarize/openai",
    "fallbacks": ["text/summarize/microsoft"],
    "input": {
        "text": "Your long tool output or document goes here. " * 100,
        "output_sentences": 3
    }
}).encode()

req = urllib.request.Request(url, data=compress_payload, headers=headers, method="POST")
with urllib.request.urlopen(req) as resp:
    compressed = json.loads(resp.read())
    summary = compressed.get("results", {}).get("result", "")
    print(f"Compressed to {len(summary.split())} words")

‍

Extraction as Compression

For structured documents like invoices or logs, extraction produces even tighter compression:

extract_payload = json.dumps({
    "model": "ocr/financial_parser/mindee",
    "fallbacks": ["ocr/financial_parser/veryfi"],
    "input": {
        "file": "https://example.com/invoice.pdf"
    }
}).encode()

req = urllib.request.Request(url, data=extract_payload, headers=headers, method="POST")
with urllib.request.urlopen(req) as resp:
    extracted = json.loads(resp.read())
    print(json.dumps(extracted, indent=2))

‍

Where to Insert Compression in Your Pipeline

The placement of compression in your pipeline determines its impact. Here are the four highest-value insertion points:

1. After tool calls. When an agent calls a tool (web search, file read, database query), compress the tool output before adding it to the context. This is the single highest-impact optimization because tool outputs are typically the largest source of token bloat.
2. Before RAG retrieval. Compress retrieved chunks before sending them to the LLM. A 2,000-token chunk might compress to 500 tokens while preserving the relevant information.
3. In agent memory. When agents maintain conversation history or task state, periodically summarize older entries to prevent unbounded context growth.
4. On log ingestion. If your pipeline feeds logs or error reports to the LLM for analysis, compress them first. Stack traces and repeated error patterns compress extremely well.

Quality Tradeoffs: When Compression Hurts

Compression is not free. There are situations where aggressive compression degrades answer quality:

• Nuanced reasoning tasks. When the LLM needs to consider multiple perspectives or subtle details, summarization can lose the nuances that lead to correct answers.
• Code generation from specifications. If the specification contains edge cases or unusual constraints, over-compression might drop them, leading to incomplete implementations.
• Legal or compliance analysis. Regulatory documents often contain specific language that must be preserved exactly. Summarization can change the meaning of legal terms.

The rule of thumb is to compress aggressively on factual retrieval tasks (search, logs, data) and compress conservatively on reasoning-heavy tasks (analysis, planning, code). Always measure answer quality before and after compression using a representative test set.

Measuring Compression Impact

Before deploying compression in production, measure its impact on answer quality:

1. Create a test set of 50 to 100 representative queries with known good answers.
2. Run each query through your pipeline without compression and record the answers.
3. Run the same queries with compression enabled and compare answers.
4. Score answer quality on a 1 to 5 scale (or use LLM-as-judge for automated scoring).
5. If quality drops more than 10%, reduce the compression ratio and retest.

Most teams find that 70% to 80% compression preserves answer quality within 5% of the uncompressed baseline. Beyond 90%, quality degradation becomes measurable for most tasks.

Conclusion

Token compression is the single most effective optimization for reducing LLM API costs in production. By compressing tool outputs, RAG chunks, and conversation history before they reach the model, you can achieve 60% to 95% cost savings while maintaining answer quality. Eden AI provides summarization and extraction APIs that serve as a ready-made compression layer, accessible through the same unified endpoint you already use for LLM calls.

FAQs - Token Compression for LLMs