Why Memory Is the Hardest Problem

Here's a scenario that exposes 90% of agent implementations: Your agent researches a topic across 15 web pages, collects findings, and starts writing a report. Halfway through the report, it contradicts something it found in page 3 — because page 3's content fell out of the context window 8 steps ago.

The agent didn't make an error. It forgot.

Context windows are getting larger — Claude offers 200K tokens, Gemini 2M — but bigger windows don't solve the problem. They delay it. And they're expensive: processing 200K tokens 15 times across an agent loop costs real money.

Production agents need a memory architecture that's smarter than "stuff everything into the prompt." This lesson shows you how to build one.

The Three Memory Types (Implementation Guide)

1. Conversation Memory (Working Memory)

The simplest form: keep the conversation history in the LLM's context. LangChain provides several strategies for managing this:

from langchain_core.chat_history import InMemoryChatMessageHistory
from langchain_core.messages import HumanMessage, AIMessage

# Basic: keep all messages (works until context fills up)
history = InMemoryChatMessageHistory()
history.add_message(HumanMessage(content="Research AI agent frameworks"))
history.add_message(AIMessage(content="I'll search for the latest information..."))

# With a window: keep only the last N messages
from langchain_core.runnables.history import RunnableWithMessageHistory

def trim_messages(messages, max_messages=20):
    """Keep the system prompt + last N messages."""
    if len(messages) <= max_messages:
        return messages
    # Always keep the first message (system prompt) + last N
    return [messages[0]] + messages[-max_messages:]

The window approach is the most common in production. But it has a fundamental flaw: important information from early in the conversation gets dropped. Step 2's critical finding disappears when step 22 pushes it out of the window.

2. Summary Memory (Compressed Working Memory)

A smarter approach: periodically summarize older messages and replace them with a condensed version:

from langchain_openai import ChatOpenAI
from langchain_core.messages import SystemMessage

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)  # Cheap model for summaries

async def summarize_and_trim(messages: list, threshold: int = 30) -> list:
    """When messages exceed threshold, summarize the oldest half."""
    if len(messages) <= threshold:
        return messages

    # Split: keep system prompt + summarize old + keep recent
    system = messages[0]
    old_messages = messages[1:threshold // 2]
    recent_messages = messages[threshold // 2:]

    # Generate summary of old messages
    summary_prompt = f"""Summarize the key findings, decisions, and context
    from this conversation. Preserve specific facts, numbers, and tool results.
    Be concise but don't lose important details:\n\n
    {chr(10).join(str(m.content)[:200] for m in old_messages)}"""

    summary = await llm.ainvoke([HumanMessage(content=summary_prompt)])

    # Replace old messages with summary
    summary_msg = SystemMessage(
        content=f"Summary of earlier conversation:\n{summary.content}"
    )
    return [system, summary_msg] + recent_messages

This gives you the best of both worlds: older context is preserved (compressed), and recent context is exact. The tradeoff is an extra LLM call for summarization — use a cheap, fast model like GPT-4o-mini.

3. Vector Store Memory (Long-term Retrieval)

For information that needs to persist across sessions and be retrieved semantically, you need a vector store. This is the foundation of RAG (Retrieval-Augmented Generation):

from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
from langchain_core.documents import Document
import chromadb

# Initialize the vector store
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma(
    collection_name="agent_memory",
    embedding_function=embeddings,
    persist_directory="./agent_memory_db"  # Persists to disk
)

# Store a finding from an agent's research
def store_memory(content: str, metadata: dict = None):
    """Store a piece of information in long-term memory."""
    doc = Document(
        page_content=content,
        metadata=metadata or {}
    )
    vectorstore.add_documents([doc])

# Retrieve relevant memories
def recall(query: str, k: int = 5) -> list[str]:
    """Retrieve the k most relevant memories for a query."""
    results = vectorstore.similarity_search(query, k=k)
    return [doc.page_content for doc in results]

# Example usage in an agent
store_memory(
    "LangGraph has 47M+ PyPI downloads and is the most popular agent framework",
    metadata={"source": "alphamatch.ai", "date": "2026-03", "topic": "frameworks"}
)

store_memory(
    "CrewAI has 20K+ GitHub stars and is the fastest-growing for multi-agent use cases",
    metadata={"source": "arsum.com", "date": "2026-03", "topic": "frameworks"}
)

# Later, when the agent needs this information:
relevant = recall("Which agent framework is most popular?")
# Returns: ["LangGraph has 47M+ PyPI downloads...", "CrewAI has 20K+ GitHub stars..."]

Putting It All Together: The Memory-Augmented Agent

Here's how to integrate all three memory types into a LangGraph agent:

from typing import TypedDict, Annotated, Sequence
from langchain_core.messages import BaseMessage, SystemMessage, HumanMessage
from langgraph.graph import StateGraph, END
from langgraph.graph.message import add_messages

class MemoryAgentState(TypedDict):
    messages: Annotated[Sequence[BaseMessage], add_messages]
    long_term_context: str  # Retrieved from vector store
    summary: str            # Compressed conversation history

def inject_memory(state: MemoryAgentState) -> dict:
    """Before the agent thinks, inject relevant long-term memories."""
    # Get the latest user message
    last_user_msg = None
    for msg in reversed(state["messages"]):
        if isinstance(msg, HumanMessage):
            last_user_msg = msg.content
            break

    if last_user_msg:
        # Retrieve relevant memories
        memories = recall(last_user_msg, k=3)
        context = "\n".join(f"- {m}" for m in memories)
        return {"long_term_context": context}
    return {}

def agent_with_memory(state: MemoryAgentState) -> dict:
    """Agent node that uses all three memory types."""
    system = f"""You are a research agent with access to past findings.

    Relevant information from previous sessions:
    {state.get('long_term_context', 'None')}

    Summary of earlier in this conversation:
    {state.get('summary', 'None')}"""

    messages = [SystemMessage(content=system)] + list(state["messages"])
    response = llm_with_tools.invoke(messages)

    # Store any new findings in long-term memory
    if not response.tool_calls:
        store_memory(response.content[:500], {"type": "agent_output"})

    return {"messages": [response]}

# Build graph with memory injection
graph = StateGraph(MemoryAgentState)
graph.add_node("inject_memory", inject_memory)
graph.add_node("agent", agent_with_memory)
graph.add_node("tools", ToolNode(tools))

graph.set_entry_point("inject_memory")
graph.add_edge("inject_memory", "agent")
graph.add_conditional_edges("agent", router, {"tools": "tools", "end": END})
graph.add_edge("tools", "agent")

Choosing the Right Memory Strategy

Use Case	Memory Type	Implementation
Simple chatbot (< 20 turns)	Full history	Keep all messages in context
Long conversations (20-100 turns)	Window + summary	Recent messages + periodic summarization
Multi-session agent	Vector store + summary	ChromaDB/Pinecone for persistence
Team of agents (CrewAI)	Shared + entity	CrewAI's built-in memory system
Production at scale	Hybrid (all types)	Redis short-term + vector long-term + SQL structured

The Production Memory Stack

For serious production deployments, here's the architecture that works:

Redis for short-term/session memory (fast reads, auto-expiry with TTL)
ChromaDB or Pinecone for vector/semantic memory (similarity search across all past interactions)
PostgreSQL for structured memory (user preferences, entity facts, explicit knowledge)

The key insight: different memory types serve different retrieval patterns. Session memory needs fast key-value lookup. Semantic memory needs similarity search. Structured memory needs exact queries. No single system handles all three well.

Explore vector database tools and memory solutions on tools.skila.ai.

Agent Memory — Short-term, Long-term, and Vector Stores

Related Resources