The Challenge
Leaders often react to surging AI bills by slamming on arbitrary budget brakes—like per-seat spending caps or model downgrades. However, blanket spending ceilings do not fix the underlying measurement deficit. Most organizations have no idea what an AI-assisted task actually costs relative to the value it returns. When an agentic workflow re-sends unoptimized, noisy context on every step, billed tokens compound quadratically. Capping spending without fixing the underlying retrieval pipeline merely starves productive teams while leaving wasteful context architecture intact, blinding leadership to true operational return.
Core Findings
Using Uber's rapid AI budget burn as a case study, author Pete Johnson illustrates that spending caps are emergency measures masking an inability to price what tokens actually deliver. In agentic workflows, conversational history reserializes each turn, causing token bills to escalate quadratically rather than linearly. While prompt caching cuts costs by up to 79%, poor prompt construction—such as placing dynamic retrieval chunks before stable instructions—breaks cache keys. Crucially, research shows excessive context degrades model performance; token bloat actively damages accuracy due to dilution effects demonstrated in recent empirical studies. The solution lies in engineering cleaner context via semantic filtering, reranking, smart chunking, context compaction, and cross-turn deduplication to track cost per resolved task rather than cost per token.
Strategic Takeaway
Shift your FinOps metrics from 'cost per seat' or 'cost per token' to 'cost per task'—such as closed tickets or shipped features. Before enacting blanket limits, audit your engineering prompts to ensure static system definitions precede dynamic context, maximizing prompt cache hit rates. Restructure retrieval pipelines to eliminate redundant context, because irrelevant tokens dilute reasoning quality while driving compounding quadratic expenses. Measuring task-level unit economics gives leadership the clarity needed to defend and expand high-performing AI workflows to finance.