The wrong million tokens

Source: CIO.com

The Challenge

Leaders often react to surging AI bills by slamming on arbitrary budget brakes—like per-seat spending caps or model downgrades. However, blanket spending ceilings do not fix the underlying measurement deficit. Most organizations have no idea what an AI-assisted task actually costs relative to the value it returns. When an agentic workflow re-sends unoptimized, noisy context on every step, billed tokens compound quadratically. Capping spending without fixing the underlying retrieval pipeline merely starves productive teams while leaving wasteful context architecture intact, blinding leadership to true operational return.

Core Findings

Using Uber's rapid AI budget burn as a case study, author Pete Johnson illustrates that spending caps are emergency measures masking an inability to price what tokens actually deliver. In agentic workflows, conversational history reserializes each turn, causing token bills to escalate quadratically rather than linearly. While prompt caching cuts costs by up to 79%, poor prompt construction—such as placing dynamic retrieval chunks before stable instructions—breaks cache keys. Crucially, research shows excessive context degrades model performance; token bloat actively damages accuracy due to dilution effects demonstrated in recent empirical studies. The solution lies in engineering cleaner context via semantic filtering, reranking, smart chunking, context compaction, and cross-turn deduplication to track cost per resolved task rather than cost per token.

Strategic Takeaway

Shift your FinOps metrics from 'cost per seat' or 'cost per token' to 'cost per task'—such as closed tickets or shipped features. Before enacting blanket limits, audit your engineering prompts to ensure static system definitions precede dynamic context, maximizing prompt cache hit rates. Restructure retrieval pipelines to eliminate redundant context, because irrelevant tokens dilute reasoning quality while driving compounding quadratic expenses. Measuring task-level unit economics gives leadership the clarity needed to defend and expand high-performing AI workflows to finance.

Deep Dive Q&A

Why do agentic AI token costs grow quadratically instead of linearly?

In multi-turn agent loops, the entire interaction history is typically re-injected at each new step. As message history expands turn by turn, every subsequent call repays for the accumulation of all previous turns, causing total billed input tokens to compound quadratically.

How does prompt caching help, and what causes it to fail?

Providers discount cached prefix tokens significantly (often by 75% to 90%). However, caches rely on exact prefix matching; inserting dynamic chunks or timestamps at the front of a prompt invalidates the cache for all downstream tokens, erasing cost savings.

Does trimming context risk losing valuable model intelligence?

No. Peer-reviewed research shows that extraneous context actively degrades LLM accuracy and reasoning, even within advertised context window limits. Refining context to only high-relevance tokens lowers costs while simultaneously improving output quality.

Related Research & Implementation Playbooks

Cross-disciplinary frameworks and practical blueprints connected to this insight.