I Was Burning Money on Tavern Small Talk

I was burning money on narration

So here's the situation. I'm running an AI Game Master that uses Claude for everything. Combat resolution, NPC dialogue, skill checks, scene descriptions, the works. Every player action hits the API. And I was sending every single request to the expensive model.

"I walk into the tavern." Expensive model. "I order a drink." Expensive model. "I ask the barkeep about the weather." Expensive model. Full price for a guy ordering ale. (Much like me swiping the family credit card on Pokemon cards. The bill always comes.)

The cheaper model costs roughly 3x less on both input and output. For a product where every player action is an API call, that 3x adds up real fast. But I couldn't just switch everything to the small model. Combat needs to track initiative, spell slots, action economy, conditions, death saves. The small model fumbles the complex stuff.

I needed the big brain for the hard parts and the fast brain for the easy parts.

Which actions actually need the big model?

I went through every tool the AI can call and asked: does this actually need heavyweight reasoning?

Turns out most of them don't. Scene descriptions, storing NPC data, presenting choices to the player. That's narration. The small model handles narration fine.

The tools that actually need the big model: starting combat, modifying creatures mid-fight, advancing turns with action economy, skill checks with DC adjudication, spell slot tracking, tactical movement. Anything with real game mechanics math.

Everything else? Small model. You don't need a PhD to describe a sunset.

The routing is dynamic. Start on the small model for most actions. If the AI calls a tool that requires complex reasoning, upgrade the model for the next round. If it's just narration tools, stay cheap. First session of a new campaign always gets the big model because you want quality world-building on first impressions.

Stop re-reading the rulebook every turn

Here's the other half of the cost problem. Every API call sends the full system prompt. Game rules, character sheets, campaign memories, combat state, module context. Easily 8,000-14,000 tokens. We were paying full price for those tokens every. single. time.

Think of it like a DM who has to re-read the entire Player's Handbook before answering every question. "Can I use my bonus action to dash?" (DM reads 300 pages.) "Yes." That's what we were doing.

Anthropic's prompt caching fixes this. You mark certain message blocks as cacheable, and subsequent calls that send the same prefix get a 90% discount on those tokens. First call pays a small surcharge to write the cache. Every call after that reads for a fraction of the normal price.

We applied this to the system prompt (biggest win), the tool definitions (~2,100 tokens of schemas), and a couple of background services that were making completely uncached LLM calls. Episode summarization, session recaps. Both were just sending raw messages at full price. (The kind of thing you don't notice until you actually look at the bill.)

Stop guessing, start counting

Before this change, I was estimating costs by dividing string length by a magic number and calling it tokens. That's... fine? For rough math? But I was making billing decisions based on vibes.

The API returns actual token counts in every response. Input tokens, output tokens, cache reads, cache writes. I was ignoring all of it. Now every AI call logs actual usage to our observability pipeline. Model used, all token counts, cache hit rate, tools invoked, estimated cost in dollars.

The cache hit rate is the metric I watch closest. First call in a session writes the cache (0% hit rate). Every subsequent call should read it (80-95%). If I see low hit rates, something is invalidating the cache between calls, which means I'm paying MORE than before. That metric is my early warning.

The before and after

For a typical narration action ("I search the room for traps"):

Roughly 80% reduction for narration-heavy sessions. Combat sessions still use the big model so savings are smaller there (40-50% from caching alone). Blended across all session types, I'm targeting 50% overall.

What could go wrong (and probably will)

The small model might miss things. If a player does something that should trigger combat but doesn't use obvious combat words, the small model might narrate through it instead of starting the encounter. I'm watching the tool usage logs. So far it hasn't been an issue, but I've only been running this for a few days. (Haven't tested this one tis but a claim.)

Cache invalidation. If the system prompt changes between calls in the same session (shouldn't happen, but bugs exist), the cache misses and we pay full price plus the write surcharge. That would actually make it more expensive than no caching.

The upgrade is mid-request. If the small model starts a response and then calls a tool that needs the big model, we upgrade for the next round. The first round's output is still small-model quality. In practice this is fine because round 1 is usually narration and the real work happens in round 2.

The numbers so far

All from about 100 lines of changes. No user-facing differences. Same AI quality for combat. Faster responses for narration (the small model is quicker). And I can finally see what things actually cost instead of guessing.

Your AI bill might be your best optimization target. Start counting the real tokens.