How to cut your LLM API bill: caching, model routing, prompt trimming and batching

LLM bills rarely explode in one go. They creep. A feature launches with a short prompt, then gains a longer system prompt, more examples, a retrieval step that adds ten documents, an agent loop that calls the model six times, and a retry on failure. Usage grows, and one month the invoice becomes a line item that finance asks about.

The good news is that most LLM applications have a lot of room to get cheaper without getting worse. The techniques are well known. What teams usually lack is measurement, so they cannot tell which technique will matter for them. This guide starts there.

Measure cost per task, not cost per month

Your provider dashboard shows total spend. That is the wrong unit. You need the cost of one task: one ticket answered, one document processed, one lead enriched. Only then can you compare it to the value of the task, spot the expensive outliers and know whether a change helped.

Every model response includes token usage. Log it for every call, tagged with the feature, the model and a task ID, and multiply by your current price table. An agent run that makes several calls becomes one task with a summed cost.

-- llm_calls: one row per model call, with token usage and price per million tokens
SELECT
  feature,
  model,
  COUNT(DISTINCT task_id)                                        AS tasks,
  SUM(input_tokens + cache_read_tokens + output_tokens)
    / COUNT(DISTINCT task_id)                                    AS tokens_per_task,
  SUM(
      input_tokens      * input_price
    + cache_read_tokens * cache_read_price
    + output_tokens     * output_price
  ) / 1e6 / COUNT(DISTINCT task_id)                              AS cost_per_task
FROM llm_calls
WHERE created_at >= now() - interval '7 days'
GROUP BY feature, model
ORDER BY cost_per_task DESC;

Look at the distribution as well as the average. Often a small share of tasks, the ones where an agent loops or retrieval pulls in huge documents, accounts for a large share of the spend. Fixing those outliers can be the cheapest win available.

Prompt caching for repeated prefixes

Many applications send the same large block on every call: a long system prompt, tool definitions, few-shot examples, a policy document. Prompt caching lets the provider reuse its processing of that identical prefix, and cached reads are billed at a substantial discount compared to fresh input. Latency usually improves too.

Caching works on prefixes, so ordering matters. Put stable content first (tools, system prompt, reference documents) and variable content last (the user's question, retrieved results). A timestamp or user name at the top of the system prompt silently breaks the cache on every request. Some providers cache automatically; others, like the Anthropic API, let you mark the breakpoint explicitly:

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

export async function answer(question: string, policyDoc: string) {
  const response = await client.messages.create({
    model: "claude-sonnet-5-5",
    max_tokens: 600,
    system: [
      { type: "text", text: "You answer questions about our refund policy. Be concise." },
      // Everything up to and including this block is cached and reused across calls.
      { type: "text", text: policyDoc, cache_control: { type: "ephemeral" } },
    ],
    messages: [{ role: "user", content: question }],
  });

  // Log cache effectiveness alongside cost.
  const { input_tokens, cache_read_input_tokens, cache_creation_input_tokens, output_tokens } =
    response.usage;
  console.log({ input_tokens, cache_read_input_tokens, cache_creation_input_tokens, output_tokens });
  return response;
}

Check the usage fields after deploying. If cache reads stay at zero, something in your prefix is changing between calls.

Caches also expire after a period of inactivity, so caching pays off most on high-traffic paths where the same prefix is reused within minutes. Writing to the cache can cost slightly more than a normal input token, which means a prefix that is only ever used once is better left uncached. Check your provider's documentation for the exact rules and minimum cacheable length.

Route easy tasks to smaller models

Not every call needs your most capable model. Classification, extraction from clean text, short summaries, routing decisions and formatting are often handled well by smaller, much cheaper and faster models. Reserve the largest model for multi-step reasoning, ambiguous inputs and high-stakes outputs.

Routing can be static (this feature always uses the small model) or dynamic (try the small model, escalate to the large one if a validation check fails or confidence is low). Static routing is simpler and covers most of the benefit. Whichever you choose, run your eval set on the smaller model first; the point is to save money on tasks where quality is unchanged, not to quietly degrade the product.

Keep the model choice in configuration rather than scattered through code, so you can move a feature between models with a one-line change and an eval run. New, cheaper models are released regularly, and a setup that makes switching easy lets you take advantage of them without a rewrite.

Trim prompts and cap outputs

Prompts accumulate. Audit them like code:

• Remove dead instructions for edge cases that no longer exist, and duplicated rules phrased three ways.
• Retrieve less, better. Ten mediocre chunks cost more and often answer worse than three good ones after reranking.
• Compact conversation history. Summarise older turns or drop stale tool results in long agent runs instead of resending everything.
• Send only the fields you need. A tool that returns a full customer record when the model needs two fields wastes tokens on every call.
• Cap output. Ask for concise or structured answers and set max_tokens to a realistic ceiling. Output tokens are typically the most expensive ones.

Batch work that does not need an instant answer

Nightly enrichment, bulk document processing, backfills and offline evaluations do not need real-time responses. Major providers offer batch APIs that process large sets of requests asynchronously, usually within hours, at a discount compared to standard calls. Move anything that can wait into a batch job.

Also check for work you should not be doing at all: re-processing documents that have not changed, re-generating the same summary for every viewer, calling the model to answer questions a simple lookup or rule could handle. A response cache keyed on normalised input, for deterministic tasks like classification, can remove a surprising number of calls.

Key takeaways

• Log token usage per call and compute cost per task; optimise the outliers first.
• Order prompts stable-first and use prompt caching for long repeated prefixes.
• Route simple tasks to smaller models, verified against an eval set.
• Trim prompts, retrieve less but better, compact history and cap outputs.
• Move non-urgent work to batch APIs and stop calling the model for work a lookup could do.

If your AI features cost more than they should, our AI consulting team can audit usage and implement these changes with you. Contact us to get started.