Prompt Caching: Calculate the Savings Before Rewriting Your AI Workflow

An AI application sends the same instructions, reference material, and tool definitions again and again. The resulting input bill can make those repeated words look wasteful. Prompt caching may help, but a cheaper cached-input rate is only one part of the calculation. A write can have a different price, a changed prefix can miss the cache, and the model still generates output.

Current OpenAI documentation describes reuse of the model's stored key-value states for a matching prompt prefix. Supported models have different controls and prices; newer documentation also distinguishes cache writes from cache reads. Claude's documentation describes its own cache controls and lifetime options. Do not transfer one provider's settings to another by analogy.

The practical question is therefore measurable: which tokens can our requests genuinely reuse, and what does the observed mix of writes and reads cost? This guide uses a synthetic rate schedule and a small Python calculator. It makes no API calls and does not quote a provider's dollar prices.

Distinguish prefix reuse from answer reuse

Imagine an extraction service with a stable field specification and a changing document. The field specification may be reusable context; the document is fresh input. Reusing work on the specification does not mean the previous answer can be returned for the new document.

This distinction matters to both cost and correctness. If you build an ordinary response cache keyed only by the instructions, you could return one customer's extracted values for a different customer's document. A provider's prompt cache and your application's response cache have different keys, purposes, and invalidation rules.

Choosing a mechanism starts with your workload. A frequently repeated lookup might suit response caching. A changing document with shared extraction instructions is a different case. Keep the success test explicit rather than describing every optimization as “caching.”

Put stability where it can be reused

Write down the parts of a request and how often each changes. A document identifier, current timestamp, or customer-specific instruction near the beginning may change the context before the large shared section. Whether that prevents reuse depends on the provider's actual rendered-prefix rules and cache controls.

Do not move a security instruction merely to improve a hit rate. If order affects behavior, the optimization has changed the application and needs evaluation. A shorter bill with weaker task compliance is not a successful optimization.

Define a synthetic batch before doing the arithmetic

Our example processes 20 independent documents. Every request contains 6,000 reusable prefix tokens and 400 changing input tokens, and produces 200 output tokens. For this exercise, every eligible cold prefix is charged as a cache write; a full hit is charged as a cache read. Real providers may classify requests differently.

These inputs are chosen to expose the tradeoff, not to recommend a model. Use your model's current rate schedule and reported usage when assessing a real application. In particular, do not add a write fee to ordinary-input pricing when the schedule instead assigns the token to one charging category.

Compare the whole bill, not one discount

Without caching, input costs 20 × 6,400 × $2 / 1,000,000 = $0.256. Output costs another $0.032, giving $0.288 for the batch.

With one prefix write and 19 full reads, prefix processing costs $0.015 + $0.0228 = $0.0378. Changing input still costs $0.016, and output still costs $0.032. The total is $0.0858, about 70.2% below the uncached total.

The last row is the useful warning. A discounted read price contributes nothing when there are no reads. The example's higher write price then dominates. This is a property of the stated synthetic charging model, not a claim that every provider charges every cache miss this way.

Find the break-even read share

Let h be the share of requests with a full prefix read. The remaining share, 1 − h, incurs the example's write rate. The average prefix rate is therefore 2.50 × (1 − h) + 0.20 × h. Caching beats the $2.00 ordinary-input rate when this expression is below 2.00.

Rearranging gives h > (2.50 − 2.00) / (2.50 − 0.20), or a full-read share above approximately 21.74%. For a 20-request batch, at least five reads are needed. Sixteen writes and four reads cost $0.2928 overall; fifteen writes and five reads cost $0.2790, compared with the $0.2880 baseline.

This threshold works because prefix length, changing input, and output are fixed across the comparison. If request sizes differ, token-weighted reuse matters more than the fraction of requests labelled hits. Different prices or additional costs require a new threshold.

Make the calculation repeatable

The following Python 3.12 program uses decimal arithmetic. It accepts counts of full-prefix writes and reads rather than pretending to know whether a live service will achieve them.

from decimal import Decimal

MILLION = Decimal("1000000")
PREFIX, VARIABLE, OUTPUT, REQUESTS = 6000, 400, 200, 20
BASE, WRITE, READ, OUTPUT_RATE = map(
    Decimal, ("2.00", "2.50", "0.20", "8.00")
)

def batch_cost(writes, reads):
    if writes < 0 or reads < 0 or writes + reads != REQUESTS:
        raise ValueError("Every request needs one prefix outcome")
    prefix_cost = PREFIX * (writes * WRITE + reads * READ)
    other_cost = REQUESTS * (VARIABLE * BASE + OUTPUT * OUTPUT_RATE)
    return (prefix_cost + other_cost) / MILLION

baseline = REQUESTS * (
    (PREFIX + VARIABLE) * BASE + OUTPUT * OUTPUT_RATE
) / MILLION

assert baseline == Decimal("0.288")
assert batch_cost(1, 19) == Decimal("0.0858")
assert batch_cost(11, 9) == Decimal("0.2238")
assert batch_cost(20, 0) == Decimal("0.348")
assert batch_cost(16, 4) == Decimal("0.2928")
assert batch_cost(15, 5) == Decimal("0.2790")
for writes, reads in [(1, 19), (11, 9), (20, 0)]:
    print(writes, reads, batch_cost(writes, reads))

Partial prefix matches, multiple writes, retries, additional reasoning tokens, and tool charges need additional accounting in a real workflow. The simple model deliberately holds output constant so the contribution of prefix reuse is visible.

Interpret expiry and usage as operational evidence

Cache lifetime, minimum prefix size, and mode depend on the model and API. As checked on October 14, OpenAI documents a 30m lifetime control for GPT-5.6 and later; earlier models use different retention options. Claude documents five-minute and one-hour cache lifetimes. Eligibility and retention do not guarantee that a particular request will find a reusable entry.

Record the model and request configuration with each experiment. Then inspect the provider's usage fields to distinguish ordinary input, cache creation, cache reads, and output. For Claude these include cache_creation_input_tokens and cache_read_input_tokens; other APIs use different names and totals. Verify the meaning of each field before adding counts together.

Avoid an unweighted average of per-request percentages when request sizes differ. If you want an aggregate read share, divide the relevant total read tokens by the corresponding total input tokens under that API's accounting. Also inspect results per workflow version so a small successful cohort does not conceal a large group of cold requests.

Decide whether the optimization earns its complexity

Run a limited comparison using the same task, evaluation fixtures, and output expectations. Measure total cost and latency separately: a lower input bill does not establish a faster end-to-end workflow. Changes to output length, retries, or model choice can overwhelm the saving you attributed to caching.

Accept the design when it preserves quality and reduces an observed constraint. Keep a rollback path and an explanation of what happens after the prefix, schema, or model changes. A cache-friendly request is a useful engineering choice when it serves the task; it is not a reason to add irrelevant tokens or retain sensitive context longer than your requirements allow.

Sources and scope

• OpenAI: prompt caching — current prefix-reuse behavior, model-dependent controls, charging distinctions, and usage guidance.
• Claude: prompt caching — provider-specific lifetime and usage fields.
• Python 3.12: decimal — the local calculator's arithmetic.

Sources checked October 14, 2026. All workload sizes, dollar rates, hit outcomes, and cost results are original illustrative scenarios. No live API benchmark or saving is claimed.