Token Optimization Techniques: Reducing LLM Costs and Maximizing Context Window Efficiency

Every token sent to a large language model costs money, consumes context window space, and increases latency. GPT-4o charges $2.50 per million input tokens. A context transfer of 30,000 tokens costs $0.075. Across 10,000 daily active users averaging 3 transfers, the annual input cost exceeds $82,000.

Token optimization is also about quality. Research demonstrates that models lose track of information in the middle of long contexts — the "lost in the middle" effect. By reducing context size, you improve the model's ability to use the context you provide.

Modern LLMs use subword tokenization (Byte Pair Encoding or SentencePiece). Key facts:

• Average English word ≈ 1.3 tokens
• Technical terms get split: "PostgreSQL" → ["Post", "gre", "SQL"] (3 tokens)
• Code is token-dense: variable names, operators, syntax each consume tokens
• JSON is highly token-inefficient due to structural characters

Format	Tokens (3 records)	Reduction
JSON	42	Baseline
YAML	31	26%
TSV	18	57%
Compressed prose	14	67%

Technique 1: Instruction Compression

// Before: 47 tokens
You are a helpful AI assistant. When the user asks you a question,
you should provide a detailed, accurate, and helpful response.

// After: 18 tokens  
Role: Technical assistant. Provide precise, example-rich answers.

The compressed version conveys the same instructions in 62% fewer tokens.

Technique 2: Few-Shot Example Pruning

Often 1-2 examples suffice. Going from 5 to 2 examples can save thousands of tokens. Use minimal examples that demonstrate the pattern, not comprehensive ones.

Technique 3: Request Structured Output

Requesting tables or bullet points instead of prose produces responses with the same information density in 60% fewer tokens.

Strategy 1: Sliding Window

Include only the N most recent messages. Simple but loses foundational context.

Strategy 2: Pinned + Recent

Pin the first 2-3 messages (project setup) and include the last N messages:

function selectMessages(messages: Message[], maxTokens: number): Message[] {
  const pinned = messages.slice(0, 3);
  const recent = messages.slice(-10);
  let selected = [...pinned, ...recent];
  let totalTokens = countTokens(selected);
  while (totalTokens > maxTokens && selected.length > 3) {
    selected.splice(3, 1);
    totalTokens = countTokens(selected);
  }
  return selected;
}

Strategy 3: Semantic Pruning

Score each message by information value. Decisions and entity definitions score high. Conversational filler ("thanks!", "got it") scores zero. This achieves the best compression ratios by eliminating low-value content regardless of position.

When transferring context to a different model, adapt the injected context to fit the target's capacity:

const MODEL_CONSTRAINTS: Record<string, { maxInput: number; reserveOutput: number }> = {
  'gpt-4o':            { maxInput: 120000, reserveOutput: 8000 },
  'claude-3.5-sonnet': { maxInput: 190000, reserveOutput: 10000 },
  'gemini-1.5-pro':    { maxInput: 990000, reserveOutput: 10000 },
};

function budgetAwareInjection(bundle: ToffeeBundle, targetModel: string): string {
  const constraints = MODEL_CONSTRAINTS[targetModel];
  const available = constraints.maxInput - constraints.reserveOutput - 500;
  if (bundle.metrics.compressed_tokens <= available) return bundle.compressed_context;
  return truncateByPriority(bundle, available);
}

The truncateByPriority function removes lowest-priority information first until the bundle fits. Core decisions, entities, and preferences are the last to be removed.

Selective Minification

Reduce indentation, remove blank lines, collapse single-statement blocks:

// Before: 45 tokens
async function fetchUser(id: string): Promise<User> {
    const response = await fetch('/api/users/' + id);
    if (!response.ok) {
        throw new Error('User not found');
    }
    const data = await response.json();
    return data as User;
}

// After: 31 tokens (31% reduction)
async function fetchUser(id: string): Promise<User> {
  const res = await fetch('/api/users/' + id);
  if (!res.ok) throw new Error('User not found');
  return await res.json() as User;
}

Comment Stripping

Preserve comments that explain why. Strip comments that describe what (when the code is self-explanatory).

Track these metrics:

• Compression ratio: (original - compressed) / original
• Reconstruction accuracy: Can a fresh session answer questions using only the compressed context?
• Task completion rate: Can the model continue work without asking for clarification?
• Cost per transfer: Actual API cost of compression + injection

The sweet spot is 60-70% compression. Below 60%, savings are insufficient. Above 75%, you risk losing important context. The optimal point varies by conversation type and target model.