Context engineering: what it is and how to do it for AI agents

Context engineering is the practice of deciding what information sits in a model's context window on every inference call: the system prompt, the tool definitions, the documents you retrieved, the running history of the task, and whatever the agent wrote to memory in an earlier session. Prompt engineering optimises how you ask. Context engineering optimises what the model knows when it answers.

It matters for anyone running agents in production, which means multi-step work, tools attached, sessions that last hours rather than seconds. A single-turn classifier does not need it. An agent that opens 40 files, calls six tools and runs for 90 minutes lives or dies on it. Anthropic describes the goal as finding the smallest set of high-signal tokens that produce the outcome you want.

The 30-second version

Models do not read their context window evenly. Bury the one fact that matters in the middle of 100,000 tokens of tool output and retrieval accuracy falls. Let a session run long enough and the agent keeps answering the version of the task it assumed 20 turns ago. Context engineering is the set of moves that keep the window small, ordered and current: retrieve just in time, compact when it fills, write durable state outside the window, and hand separable work to agents with clean windows of their own.

Why did prompt engineering stop being enough?

Prompts kept improving and agents kept failing in the same places. The reason is measurable. Researchers at Microsoft Research and Salesforce Research ran more than 200,000 simulated conversations across six generation tasks and found that models lose an average of 39% of their performance when a fully specified instruction arrives in pieces across turns instead of all at once (Laban et al., 2025). The loss came from unreliability, up 112%, rather than from lost capability, down 15%. Bigger models did not resist it: GPT-4.1, Claude 3.7 Sonnet and Gemini 2.5 Pro all degraded. The paper's summary of the behaviour is blunt. When a model takes a wrong turn early, it does not recover.

The second finding is about length rather than turns. Chroma tested 18 models on retrieval and text replication, tasks that are close to trivial at short input lengths, and found reliability dropping as the input grew (Context Rot, 2025). Position matters too: material at the start and the end of a window is handled better than material in the middle. The industry settled on context rot as the name for it, and the replication toolkit is public.

Neither problem has a prompt fix. Rewording an instruction does not repair a retrieval step that returned the wrong document, and no phrasing recovers an agent that committed to a bad assumption 20 turns back. The fix sits upstream, in what enters the window and when it leaves.

What actually sits in an agent's context window

• System prompt. Role, constraints, output format, house rules. Usually static, usually the cheapest thing to cache.
• Tool definitions. Every schema you attach costs tokens on every call, whether the agent uses the tool or not. Twenty tools is a tax paid continuously.
• Retrieved data. Documents, database rows, API responses, file contents. The largest and most volatile block.
• Task history. The running transcript of steps taken, tool calls made and outputs returned. Grows monotonically unless you intervene.
• Memory. Anything the agent wrote down in an earlier session and reads back now.

Prompt engineering touches the first item. Context engineering touches all five, plus the order they arrive in and the moment each one gets dropped.

How context engineering works in practice

Just-in-time retrieval

Load identifiers, not payloads. The agent holds file paths, saved queries and links, then calls a tool to pull the content at the moment it needs it. Claude Code works this way: the CLAUDE.md file goes in up front, and glob and grep fetch source files during the run. The cost is a round trip per lookup, which adds latency. The gain is a window that stays small on a task touching 200 files. We wrote about the file itself in this guide to CLAUDE.md.

Compaction

When the window nears its limit, summarise the session and restart with the summary. The hard part is what to keep. Anthropic's guidance is to preserve architectural decisions, unresolved bugs and implementation details, and to drop tool outputs the agent has already consumed. Compact too aggressively and you throw away the constraint the agent needed at step 40. Compact too late and it has been degrading for an hour.

Structured note-taking

Write state to a file outside the window and read it back. A to-do list, a findings log, a decisions record. The window can then be reset without losing the thread, because the thread lives on disk. A side effect worth having: the run becomes auditable, which is the difference between explaining an agent's behaviour to a client and guessing at it.

Sub-agents with clean windows

Give a narrow, separable task to a fresh agent, let it burn its own context, and return a summary to the caller. The main agent stays lean. This is the architecture behind most production research agents. It costs more tokens in total and adds a coordination surface to debug, so it earns its place on work with genuinely independent scope rather than as a default. The patterns that survive contact with production are in our piece on multi-agent design.

What it costs, and what it saves

Two numbers make the economics concrete. The team behind Manus, a production agent platform, reported an average input-to-output token ratio of roughly 100:1, which means almost the entire bill is context rather than generation. They also reported cached input at $0.30 per million tokens against $3.00 uncached on Claude Sonnet, a 10x gap (Manus, 2025). List prices move, so check current rates before you model anything, but the shape of the gap has held.

That turns cache hit rate into a first-class metric rather than an optimisation you get to later. It also makes small design choices expensive. Putting a timestamp at the top of a system prompt invalidates the cache from that token onward, on every call, for the life of the agent. Stable prefixes, append-only history and deterministic JSON serialisation are cheap habits with a direct line to the invoice. The same arithmetic from the billing side is in our post on AI feature token cost, and the caching mechanics in the Anthropic API in Next.js.

When you do not need this

The discipline earns its cost when runs are long, tools are many, or the same agent gets called thousands of times a day. Below that line, a well-written prompt and one retrieval step will do, and the engineering hours are better spent elsewhere. A support chat capped at ten messages does not need compaction. A classifier does not need a memory file.

Two honest limits. Context engineering does not repair a bad index: if the vector store returns the wrong three documents, careful window management delivers the wrong three documents more efficiently. And it does not remove the need to evaluate. Most of the moves above are trade-offs with a tuning parameter, and the only way to know where to set it is to measure task success on long runs rather than on the demo.

Is it a real discipline or a rebrand?

Some of both. The DataHub 2026 State of Context Management report found 82% of IT and data leaders saying prompt engineering alone no longer powers AI at scale, and 95% calling context engineering important for running agents at scale (DataHub, 2026). Vendor surveys about a category tend to find that the category matters, so discount the numbers. The shift underneath them is real, and it shows up in the research rather than the marketing: the failure modes moved from how you word a request to what the model was holding when it answered. The job title is new. The work is retrieval design, caching and state management, which backend engineers have been doing for twenty years under other names.

Adjacent concepts

• Context window as a token budget, the day-to-day version of the same discipline inside a coding agent.
• Skills, where progressive disclosure keeps capability out of the window until it is called.
• Multi-agent design patterns, for when one clean window is not enough.

The practical takeaway is smaller than the discourse around it. Treat the context window as a scarce, ordered, expensive resource with an owner, measure what fills it, and remove things on a schedule rather than when the model starts failing. Most agent reliability problems we see in production are a supply problem, and the supply is fixable.

Sources

• Effective context engineering for AI agents, Anthropic
• LLMs Get Lost In Multi-Turn Conversation, Laban et al., arXiv 2505.06120
• Context Rot: How Increasing Input Tokens Impacts LLM Performance, Chroma Research
• chroma-core/context-rot, replication toolkit
• Context Engineering for AI Agents: Lessons from Building Manus
• Context Engineering vs Prompt Engineering, DataHub