An LLM feature that delights ten test users can quietly become slow and ruinously expensive at ten thousand. The economics of AI in production are an engineering problem, and mostly a solved one — if you design for them.
- Cost and latency scale with tokens and model choice. Both are under your control, and both are usually ignored until the bill or the complaints arrive.
- Cache aggressively, route easy work to small cheap models, and only reach for the big model when the task actually needs it.
- Stream responses so the feature feels fast even when the total generation time is unchanged. Perceived latency is the one users judge.
In the demo, the LLM feature is a delight. Answers appear, they're good, everyone's impressed. What the demo doesn't show is the invoice, because at ten test users the invoice is a rounding error and every request is fast because nothing's under load.
Then it ships to real traffic, and two things happen at once. The finance team starts asking pointed questions about a fast-growing line item, and users start complaining that it's slow. Both problems were entirely predictable, and both are solvable — but only if you treat cost and latency as engineering constraints from the start rather than surprises to discover in production.
The mental model: you're paying by the token
Almost everything about LLM economics follows from one fact: you pay per token, in and out, and you wait per token generated. Tokens are roughly word-fragments; both your prompt (input) and the model's reply (output) are counted.
That single fact has direct consequences:
- A prompt that stuffs in 4,000 tokens of context costs more and starts slower than one that sends the 500 tokens that actually matter.
- Output tokens are generated one at a time, so a long answer is a slow answer — latency scales with how much the model says.
- Bigger, smarter models cost more per token and generate more slowly than small ones.
input tokens ──┐
├─▶ model (size) ─▶ output tokens (generated one by one)
context size ──┘ │
▲ ▼
trim what you send pick the smallest model that's good enoughOnce you see it this way, the optimisation levers are obvious: send fewer input tokens, generate fewer output tokens, use a smaller model where you can, and avoid making the call at all when you don't have to. Let's take them in order of impact.
Cache: the cheapest call is the one you never make
A huge amount of LLM traffic is repetitive. The same questions, the same documents, the same prompts, over and over. Every one of those is an identical call you can serve from a cache for free and instantly.
async function answer(question: string): Promise<string> {
const key = `llm:${hash(question)}`;
const cached = await cache.get(key);
if (cached) return cached; // free, ~1ms instead of ~2s and a token bill
const result = await llm.complete(question);
await cache.set(key, result, { ttl: 60 * 60 * 24 }); // a day
return result;
}Exact-match caching is trivial and catches genuinely identical requests. Semantic caching goes further: embed the incoming question and, if it's very close to one you've already answered, reuse that answer — so "what's your refund window?" and "how long do I have to return something?" share a single generation. On support-style workloads where the same handful of questions dominate, caching alone can remove a large fraction of calls.
There's also prompt caching offered by the providers themselves: if a big chunk of your prompt is identical across calls — a long system prompt, a fixed document — they'll cache that prefix and charge a fraction for it on subsequent calls. When you're sending the same 2,000-token instruction block every time, turning this on is close to free money.
Route: don't send easy work to the expensive model
Teams reach for the biggest, smartest model and use it for everything, because it gives the best answers. But most workloads are a mix: a few genuinely hard requests that need the big model, and a majority of easy ones a small, cheap, fast model handles perfectly.
Model routing sends each request to the cheapest model that can do that job well:
async function route(task: Task): Promise<string> {
// Simple, mechanical tasks -> small, fast, cheap model.
if (task.type === "classify" || task.type === "extract") {
return smallModel.complete(task.prompt);
}
// Open-ended reasoning or drafting -> the capable model.
return largeModel.complete(task.prompt);
}The price gap between model tiers is large — often an order of magnitude per token. Classification, extraction, short factual answers, and routing rarely need the flagship model; reserve it for open-ended reasoning and nuanced drafting. Getting the routing right frequently cuts cost dramatically with no visible drop in quality, because the small model was never going to be worse at the easy tasks — it just costs a fraction as much to do them.
| Task | Model tier | Why |
|---|---|---|
| Classify a ticket | small | narrow, well-defined, cheap wins |
| Extract fields from text | small | structured, verifiable output |
| Draft a nuanced reply | large | open-ended, quality matters |
| Multi-step reasoning | large | small models stumble here |
Trim: send only the context that matters
Because you pay for input tokens and they slow the first response, padding the prompt is pure waste. This is where good retrieval pays off twice: sending the model the five genuinely relevant passages instead of twenty mediocre ones is cheaper, faster, and produces better answers, because the signal isn't buried. Rerank, keep the best few, and cut the rest. Long conversations need the same discipline — summarise old turns rather than resending the entire history on every message.
Stream: make it feel fast even when it isn't
Latency has two versions — how long the response actually takes, and how long it feels. Users only experience the second one, and streaming lets you win it almost for free.
Instead of waiting for the whole answer and showing it at once, stream tokens as they generate, so text starts appearing within a few hundred milliseconds:
const stream = await llm.completeStream(prompt);
for await (const token of stream) {
send(token); // words appear as they're produced
}The total generation time is unchanged, but the experience is transformed. "Time to first token" is what people register as speed, and a response that starts immediately and flows feels fast even if it takes the same three seconds overall. A feature that sits on a spinner for three seconds and then dumps a paragraph feels broken. Same latency, opposite verdict.
Put a ceiling on it
However well you tune the averages, protect yourself from the tail. Set a max output length so a single request can't run away and generate — and bill for — thousands of tokens. Set timeouts so a slow upstream call fails fast instead of hanging your whole request. And track cost per request as a first-class metric, so a change that quietly triples token usage shows up on a dashboard the day it ships, not on the invoice a month later.
cache ─▶ route to right-sized model ─▶ trim context ─▶ stream out
│
monitor cost/request + p95 latencyThe takeaway
None of this is exotic. Cost and latency in LLM features come down to a handful of familiar engineering moves — cache what repeats, right-size the model, send less, stream the output, and cap the extremes. What makes them matter is that they're almost always skipped, because the demo hid the problem they solve.
The feature that delights ten users and the one that survives ten thousand can be the exact same idea. The difference is whether someone engineered the economics — or just shipped the demo and waited for the invoice.