Memory, retrieval and context engineering

A language model can only use the information in its context window, the text it receives on each call. Everything an agent knows about its task, its past steps, the user's preferences and the relevant documents must either fit in that window or be retrieved into it at the right moment. As tasks grow longer, spanning hundreds of tool calls, many files or days of interaction, managing what goes into the context becomes one of the central problems of agent design, often called context engineering. This lesson covers the kinds of memory agents need, the limits of long contexts, retrieval-augmented generation and vector search, strategies for summarization, note-taking and context compaction, and the privacy and correctness issues that memory raises.

Kinds of memory

Borrowing loosely from cognitive science, agent memory is often divided into:

• Working memory: the current context window, holding instructions, the task, recent steps and tool results.
• Episodic memory: records of past interactions and experiences, such as previous conversations with a user or earlier attempts at similar tasks.
• Semantic memory: facts and knowledge, such as company policies, product documentation and user preferences.
• Procedural memory: knowledge of how to do things, such as reusable instructions, scripts and skills that the agent can load when relevant.

In practice, these are implemented as the prompt itself, conversation logs, databases and document stores, files written by the agent, and packaged instructions or skills that are loaded on demand. The model's own weights also contain vast general knowledge, but it is fixed at training time, may be outdated, and cannot be inspected or corrected easily, which is why external memory matters.

The limits of long context

Context windows have grown from a few thousand tokens to hundreds of thousands or more, enough to hold a novel or a sizeable codebase. Larger windows help, but they do not solve memory. Cost and latency grow with context length. Models attend less reliably to information buried in the middle of very long contexts, a finding known as "lost in the middle", and performance on tasks requiring the combination of many scattered facts degrades as contexts grow. Irrelevant material distracts the model and can lead it astray. And some tasks generate far more information than any window can hold, such as days of agent activity or millions of documents. Effective agents therefore curate their context, keeping it focused on what matters for the current step, an approach Anthropic and others describe as treating context as a scarce resource to be engineered.

Retrieval-augmented generation

Retrieval-augmented generation (RAG) gives a model access to large external knowledge by retrieving relevant passages at query time and inserting them into the context. A typical pipeline splits documents into chunks of a few hundred words, converts each chunk into an embedding vector using an embedding model, and stores the vectors in a vector index. At query time, the query is embedded, the most similar chunks are found by cosine similarity, and they are inserted into the prompt with instructions to answer from them and cite sources. Keyword search methods such as BM25 complement vector search for exact terms, names and codes, and hybrid search combining both, followed by a reranking model, is common in production.

import numpy as np
from collections import Counter
# Agent memory: store notes as vectors and retrieve the most relevant ones for a query
notes = [
    "customer Dana prefers email contact and morning meetings",
    "the staging server password rotates every 30 days",
    "Dana asked for the Q3 invoice to be resent with the purchase order number",
    "the deploy script fails if the cache directory is missing",
    "team lunch is moved to Thursday this week",
    "invoice numbering restarted in July after the accounting system change",
    "the purchase order number for Dana is PO-7731",
]
stop = {"the", "is", "to", "and", "be", "if", "a", "for", "with", "this", "after", "every", "in", "of", "s"}
def tokens(text):
    words = text.lower().replace(",", " ").replace(".", " ").replace("-", " ").split()
    return [w for w in words if w not in stop]
vocab = sorted({w for n in notes for w in tokens(n)})
doc_freq = Counter(w for n in notes for w in set(tokens(n)))
def embed(text):
    v = np.zeros(len(vocab)); counts = Counter(tokens(text))
    for w, c in counts.items():
        if w in vocab:
            v[vocab.index(w)] = c * np.log((1 + len(notes)) / (1 + doc_freq[w])) + c
    return v / (np.linalg.norm(v) + 1e-9)
M = np.array([embed(n) for n in notes])
for query in ("resend Dana the invoice with her PO number", "why did the deploy fail"):
    scores = M @ embed(query)
    top = np.argsort(scores)[::-1][:3]
    print(f"query: {query}")
    for i in top:
        print(f"  {scores[i]:.2f}  {notes[i]}")

query: resend Dana the invoice with her PO number
  0.60  the purchase order number for Dana is PO-7731
  0.44  Dana asked for the Q3 invoice to be resent with the purchase order number
  0.16  invoice numbering restarted in July after the accounting system change
query: why did the deploy fail
  0.41  the deploy script fails if the cache directory is missing
  0.00  invoice numbering restarted in July after the accounting system change
  0.00  the purchase order number for Dana is PO-7731

The example uses simple TF-IDF word vectors, which weight words by how distinctive they are, instead of a neural embedding model, so the mechanics are transparent. For the request to resend Dana the invoice with her purchase order number, the two most relevant notes, the order number and Dana's request, score highest, and an unrelated note about invoice numbering comes third with a much lower score. For the question about the deploy failure, only the note about the deploy script matches; the others score zero. Neural embeddings capture meaning beyond shared words, so they would also match "release" with "deploy" or "PO" with "purchase order", but the principle is the same: represent text as vectors and retrieve by similarity. RAG quality depends heavily on chunking, metadata filters (such as only this customer's notes), freshness, and evaluating whether the retrieved passages actually contain the answer.

Managing context in long tasks

Agents working for hours or across many steps need strategies to keep context useful:

• Summarization and compaction: when the history grows long, replace older steps with a summary of decisions, findings and open issues, keeping recent steps in full. Good summaries preserve facts the agent will need, such as file names, errors and decisions, while dropping verbose tool output.
• Structured note-taking: the agent maintains files such as a task list, progress notes or a plan, updating them as it works and rereading them after compaction or when resuming. Writing state to files makes it persistent, inspectable by humans and independent of the context window.
• Just-in-time retrieval: instead of loading everything up front, the agent keeps lightweight references, such as file paths, search queries or document IDs, and loads content when needed, as a person uses a file system rather than memorizing it.
• Tool result trimming: return only relevant parts of large outputs, and remove old tool results once they have been processed.
• Sub-agents: delegate a self-contained subtask, such as searching a large codebase, to a separate agent with its own context, which returns only a concise result (Lesson 11).

Long-term memory and personalization

Assistants that remember users across sessions can be more helpful: recalling preferences, ongoing projects and previous decisions. Long-term memory systems typically extract salient facts from conversations, store them with timestamps and sources, retrieve relevant ones in later sessions, and update or delete them when they change. Design questions include what to remember (useful, durable facts rather than everything), how to resolve conflicts between old and new information, how to show users what is remembered, and how to let them correct or delete memories. Poor memory systems recall outdated or wrong facts confidently, mix up users, or surface sensitive information in the wrong context.

Privacy, security and correctness

Memory raises significant responsibilities. Stored conversations and extracted facts may contain personal and sensitive information subject to privacy laws, so systems need clear purposes, retention limits, access controls and user rights to access and delete data. Retrieval must respect permissions: an enterprise agent must not retrieve documents the current user is not allowed to see, which requires carrying access control information into the retrieval index. Memory can also be poisoned: if an attacker can insert text into documents or past conversations that the agent later retrieves, they can plant false facts or prompt injection instructions that persist across sessions. Treating retrieved content as untrusted data, recording provenance, and limiting who can write to shared memory reduce these risks.

Evaluating retrieval and memory

Retrieval quality should be measured, not assumed. Useful metrics include recall at k (whether the passage containing the answer appears among the top k results), precision (how many retrieved passages are relevant), and end-to-end answer accuracy with citations checked against sources. Building a test set of realistic questions with known supporting passages lets teams compare chunk sizes, embedding models, hybrid search settings and rerankers objectively. For long-term memory, tests should check that the agent recalls relevant facts, does not recall facts from other users, updates outdated information, and forgets what it has been asked to delete. Failures in retrieval often look like model failures, so measuring the retrieval step separately saves considerable debugging time.

Instructions as memory

Persistent instruction files, such as project guidelines, style guides and lists of known pitfalls, are a simple but powerful form of procedural memory. Loaded at the start of each task, they encode lessons learned so the agent does not repeat past mistakes. Keeping them short, current and specific matters: long, outdated instruction files consume context and can mislead the agent as easily as they help it.

Context engineering is an ongoing practice rather than a one-time design. As tasks, tools and models change, teams revisit what goes into the context, measure the effect of changes on success rates, and prune instructions, examples and tool outputs that no longer help. Small improvements in context quality often produce larger gains than switching to a bigger model.

Retrieval and memory also interact with cost. Every retrieved passage and remembered fact consumes tokens on every call that includes it, so retrieving fewer, better passages saves money and often improves accuracy at the same time.