好文集ChatGPT Plus¥119

a beginners guide to context engineering from first principles

文章目录

before we begin

context window is the amount of text measured in tokens that a LLM can process in one time. why does it have a limit? it has a limit because models have a maximum amount of info they can process in a single interaction.

agent is an LLM working autonomously in loops using tools.

context engineering

context engineering refers to the set of strategies used for curating and maintaining a set of tokens used during LLM inference. it is the art and science of deciding what goes into the limited context window of the LLM.

managing context is the key to building successful agents.

the attention budget

as the number of tokens in a context window increases, the ability of an LLM to recall information from the context decreases. LLMs have an attention budget that they draw when parsing large amounts of data. every new token added decreases this budget. models are generally better at understanding shorter sequences of data due to their training distribution and struggle of longer sequences. techniques like positional encoding interpolation allows models to handle larger sequences by adapting them to originally trained smaller context.

good context engineering is finding the best and smallest set of high signal tokens that produce the desirable outcome.

building good context

system prompt should be clear, direct language and simple. it should not be very strict and not too vague as well. finding the correct balance between the two is the key. you should decide the minimal set of information that perfectly describes your task.

tools allow agents to pull in new information while staying in their environment. tools should promote efficiency by returning info that is token efficient and correct. their functionality should not overlap with each other and they should be clear of their purpose. creating a minimal set of tools for the agent can lead to better context management and it can also result in failure if the tool set is too bloated.

how agents use context

agents are just LLMs autonomously using tools in loop. agents these days dynamically load data into their context using tools. they use bash commands like head and tail to analyze large volumes of data without directly ever loading them into the context.

the main idea here is that before giving all the information to the agent beforehand, let it find and retrieve the information when it needs it.

but there is a catch here: this approach takes time and requires good tools. to overcome this, we need to take a hybrid approach that is provide the agent with initial information and let it retrieve more information with tools. it balances upfront context with autonomous exploration.

giving the agent the ability to use, find and store information at the right time is the key to agentic design.

how context fails

people think that with a large enough context window they can put anything and everything in a prompt and the model will take care of it but longer contexts donot generate better responses. context can become poisoned, distracting or even confusing and conflicting.

  • context poisoning: when a hallucination or error makes it into the context

  • context distraction: when the context grows too large and the model starts focusing on the context rather than the data it was trained on. for smaller models, the distracting ceiling is much lower

  • context confusion: happens when irrelevant info, superfluous content or excess tools degrade a model’s judgement and reduce response quality

  • context clash: when you put new info or tools in the context that conflicts with other info in the context. this is more problematic than context confusion

handling long-horizon tasks

then how do we handle tasks that require continuous work over a long period, sometimes hours because increasing the context window is not the solution and can lead to context pollution.

  1. compaction: compaction is summarizing the info when the context window is almost full and then starting a new context window with this summarized information. the agent preserves the architectural decisions, bugs and other important info and discards the useless info such as unnecessary tool outputs. the challenge is which info to retain and which to discard.

  2. note taking (agentic memory): instead of relying on the context window the agent stores important info in external files or memory such as completed tasks, pending tasks, unsolved bugs etc. after context is reset, it reads these notes and continues from where it left.

  3. sub agent architecture: instead of one agent handling a long complex task, it is divided into multiple subagents which are independent and have their own context window. these subagents report their findings to the main agent when finished.

how to fix our context

everything in context engineering is about information management. everything in the context influences response.

  • RAG: First of all, RAG is not dead as people claim it. We cannot treat context like a junk drawer and put everything into it. The more junk you put into the context, the worse your response will be.

  • tool loadout: tool loadout is selecting only relevant tools to add to your context. The simplest way to do it is to use RAG for your tool descriptions. By storing the tool descriptions in a vector database, we can select the most relevant tools given an input prompt. Selecting the right tools is critical when you have more than 30 tools. Beyond 30 or above 100, the tools begin to overlap and create confusion.

  • context quarantine: we achieve better results when the context is not too big. We can achieve this if we break our task into smaller isolated jobs, each with its own context.

  • context pruning: this is removing unwanted or irrelevant information from our context. When an agent uses a tool and assembles a document, it acquires a lot of context so we can task an LLM or do it ourselves to clear off or check if it is acquiring any irrelevant or non-essential documents.

  • context summarization: it is the process of summarizing the context we have acquired. Knowing what information should be preserved and detailing it to an LLM is the hard part.

  • context offloading: context offloading is the act of storing information outside the LM context, usually via a tool that stores and manages the data.

the key insight

The key insight is that context is not free. Every token in the context influences your model’s response, be it in a better way or in a worse way, and higher context does not always mean better results.

as models become more capable over time, the key is not just prompt engineering but how we effectively utilize the attention budget during long, complex tasks. the key is always finding the best and smallest set of high signal tasks that can execute the task in the right manner.


references

https://www.anthropic.com/research/effective-context-engineering-for-ai-agents

https://www.dbreunig.com/2025/06/22/how-contexts-fail-and-how-to-fix-them.html

https://www.dbreunig.com/2025/06/26/how-to-fix-your-context.html

全文完
查看原文