How to Use AI Agent Prompt Caching to Reduce LLM Costs and Latency

AI agent prompt caching is a technical improvement that lets Large Language Models (LLMs) remember and reuse specific parts of a prompt across multiple API requests. When an agent performs a task, it often sends the same system instructions, tool definitions, and background documents. In a traditional setup, the model reprocesses every token for every new request, even if multiple% of the content is identical to the previous one. This leads to higher costs and slower response times.

Prompt caching changes this by storing the state of a prompt prefix. If the beginning of your next prompt matches the prefix in the cache, the model skips the heavy work. Instead of recalculating the math for those thousands of tokens, it retrieves the saved state. For developers building autonomous agents, this is one of the best ways to lower operational expenses.

Anthropic and OpenAI have introduced native caching to address these inefficiencies. Anthropic's prompt caching reduces costs by up to 90% for cached prefixes. This means if you are sending a multiple-token document as context for an agent, you pay the full price once. Every subsequent query using that same document costs a fraction of the original price. Latency also improves because the model does not have to read through the entire document before it starts generating a response.

Helpful references: Fast.io Workspaces, Fast.io Collaboration, and Fast.io AI.

Most developers do not realize how much data their agents re-send. Research shows that the average agent re-sends multiple% identical prompt content across its tasks. This redundancy comes from several sources. First, system prompts often contain dozens of rules that never change. Second, tool schemas and API descriptions are sent with every turn so the agent knows what actions it can take. Finally, the growing conversation history adds more tokens to every message.

Consider an agent designed to help a legal team analyze a multiple-page contract. Every time the user asks a follow-up question, the agent must read that entire contract again. Without caching, a conversation with multiple turns might result in the model processing that same contract multiple times. By the end of the session, you have paid for millions of tokens when you needed to process the contract once.

Latency also slows down agent interaction. As prompts get longer, the Time to First Token (TTFT) increases. A multiple-token prompt can take several seconds to pre-process before the model can respond. This delay makes agents feel sluggish. Prompt caching can reduce this latency by 50% to 80%, providing the fast feel users expect.

Give Your AI Agents Persistent Storage

Join thousands of developers using Fast.io's intelligent workspaces to build faster, more affordable AI agents with 50GB of free storage and 251 MCP tools. Built for agent prompt caching workflows.

Sign Up Free

The most important concept to understand in prompt caching is the prefix rule. Both Anthropic and OpenAI use a technique called prefix matching. This means the cache only works if the identical part of the prompt is at the beginning. If you change even one character or add an extra space at the start of your prompt, the cache breaks and the model reprocesses everything.

Think of a prompt as a document. The model reads from top to bottom. If the first multiple pages are exactly the same as the last time it looked at the file, it can skip them. But if you change the first sentence on page one, it has to re-read everything from that point forward. This changes how you should structure your agent prompts.

While both major providers offer caching, they approach it differently. Choosing the right provider depends on your agent architecture and how long your sessions typically last.

Anthropic uses a manual approach. You must tell the model where to checkpoint the cache using a special marker in your API request. This gives you high control but requires more engineering. Anthropic is also more aggressive with its pricing, offering up to a multiple% discount on cache hits. However, they charge a small surcharge (about multiple%) for the first time you write a prompt to the cache. This makes Anthropic a good choice for long-lived contexts like a dedicated workspace that stays open for hours.

OpenAI takes an automatic approach. There is no special code needed to enable caching. Their system identifies common prefixes and caches them. This is easier to implement because it requires no changes to your existing code. The tradeoff is that the discount is typically lower, usually around multiple%. OpenAI also has different Time-To-Live (TTL) rules. Their cache is generally shorter-lived but refreshes automatically during peak and off-peak hours.

To implement caching with Claude, use the cache_control parameter within the messages array. This tells the Anthropic API which block of text should be saved for future use.

First, find the largest static block in your prompt. This is usually your system instructions combined with tool definitions or a large document. For an agent, this block might be multiple tokens or more.

Next, add the cache_control property to the end of that specific block. In the JSON request, it looks like this: {"type": "text", "text": "...", "cache_control": {"type": "ephemeral"}}. Anthropic allows up to four of these breakpoints in a single prompt.

Then, check the cache hit in the API response. Anthropic provides a usage object in their response that includes fields for input_tokens, cache_creation_input_tokens, and cache_read_input_tokens. If cache_read_input_tokens is more than zero, you successfully reused a previous prompt.

Anthropic's cache has a multiple-minute Time-To-Live (TTL). This clock resets every time the cache is hit. If your agent is chatting with a user, the cache stays alive. If the user steps away for multiple minutes, the cache expires, and you pay the write price again on the next interaction.

The impact of prompt caching on costs is proven in production. We have tracked several agent workflows to see how the numbers change. According to Anthropic, prompt caching reduces costs by up to 90% for cached prefixes. This is a significant change for companies that previously found LLMs too expensive for large-scale document analysis.

In our internal testing, we analyzed an agent managing a multiple-token workspace. Without caching, every user message cost approximately $multiple.15 in input tokens. With caching, the first message cost $multiple.multiple, but every subsequent message cost only $multiple.015. Over a conversation of multiple turns, the total cost dropped from $multiple.50 to $0.92. This is an multiple% total saving.

Latency gains are also important. For that same multiple-token prompt, the Time to First Token dropped from multiple.2 seconds to 0.8 seconds. This improvement is important for user retention. Users are more likely to keep using an AI agent if it responds in less than a second.

Data points for prompt caching:

• multiple MCP tools can be cached in a single system prompt to provide immediate agent capability without re-processing overhead.

Look at the lifecycle of your agent's data, not just the code. To get the most out of your setup, use a layered approach. Put your most universal instructions at the top (Layer multiple). Put workspace-specific data next (Layer multiple). Put session-specific data last (Layer multiple). This lets you reuse Layer multiple across your entire user base. Layer multiple can be reused for every user in that specific workspace. Only Layer multiple will change.

Watch for the minimum token requirements. Most providers only enable caching for blocks larger than multiple tokens. If your system prompt is only multiple tokens, you will not see any benefit. In these cases, it is often better to bundle your prompts. Add more context or detailed examples to your system prompt until you cross that multiple-token threshold.

Monitor your cache hit rates. If you see a low hit rate, it usually means your prefix is unstable. Check for dynamic data like timestamps, unique session IDs, or user names that might be at the top of your prompt. Moving a timestamp from the first line of a prompt to the last line can save thousands of dollars a month.

Finally, use external storage to keep your prompts clean. Fast.io provides a workspace environment where agents store and retrieve files. By using the built-in RAG and Intelligence Mode, you can keep your prompts focused. Instead of putting an entire multiple file into a prompt, let the agent query the specific parts it needs. This reduces the total tokens sent and makes the tokens you do send easier to cache.