FollowAgents AI AGENTS DIRECTORY
Home / AI Agent Glossary / Prompt Caching
Ecosystem & Emerging Terms

Prompt Caching

Also called: context caching · prefix caching

Prompt caching is a provider feature that stores the processed form of a repeated prompt prefix so later requests starting with the same content are served faster and at lower input cost.

An agent re-sends a lot of the same text on every turn: the system prompt, tool definitions, project instructions, and the growing conversation history. Processing that text from scratch each time wastes compute. Prompt caching lets the provider keep the intermediate state computed for a prefix and reuse it when a later request begins with exactly the same tokens.

Major model providers offer some form of it, under names such as prompt caching or context caching. Cached input is typically billed at a discount and is faster to process, though the exact discount, cache lifetime, and minimum size vary by provider and change over time, so check current documentation rather than assuming numbers.

For long-running agents this is one of the most effective cost and latency levers, and it changes how you should structure prompts: put stable content first, volatile content last.

How it works

Caching works on prefixes: the request is compared token-by-token from the start against previously processed content, and everything up to the first difference can be reused. Some providers cache automatically; others require you to mark cache breakpoints. Cache entries expire after a period without use. Any edit early in the prompt, such as inserting a timestamp near the top or reordering tools, invalidates everything after it.

Example

A coding agent's prompt is laid out as: system instructions, tool definitions, repository guide, then the conversation so far. Each turn appends new messages at the end, so the first several thousand tokens are identical and served from cache. If someone later adds the current time to the top of the system prompt, every request becomes a cache miss.

Common misconceptions

Often assumed: Prompt caching stores and reuses the model's previous answers.
Actually: It caches the processing of the input prefix, not outputs. The model still generates a fresh response each time.
Often assumed: Any repeated text anywhere in the prompt will be cached.
Actually: It generally matches only a shared leading prefix. Identical text placed after a differing section is not reused.

FAQ

What is prompt caching?
A provider feature that reuses the processed state of an unchanged prompt prefix across requests, reducing latency and input cost.
How do I get a higher cache hit rate in an agent?
Keep the beginning of the prompt stable: system prompt and tool definitions first, append new turns at the end, and avoid inserting changing values like timestamps early on.
Does prompt caching change the model's output?
It is intended to be an optimization only; the model computes the response as usual, with the input prefix processing reused.

Last checked: 2026-09-20

Related terms

Context Engineering

Context engineering is the practice of deciding what information — instructions, tools, retrieved documents, history, memory — actually goes into a model's context window for a given step, and how it's structured and kept up to date.

Context Rot

Context rot is the observed pattern that a language model's reliability degrades as the amount of text in its input grows — not only when the context window is nearly full, but incrementally as tokens accumulate, even on tasks the model handles easily at short lengths.

Agent Loop

The agent loop is the repeating perceive-plan-act-observe cycle by which an AI agent turns a goal into a sequence of actions, checking each result before deciding what to do next.

Mixture of Experts (MoE)

A mixture-of-experts (MoE) model contains many specialist sub-networks and a router that sends each token to only a few of them, so only part of the model's parameters run for any given token.