Memory-Constrained AI Agents: Designing Efficient Reasoning Under Limited Context Windows
Most AI systems assume infinite memory. You send a conversation history. You include all context. You let the model process everything at once.
This works in labs. In production, it breaks. Context windows have limits. Tokens cost money. Memory fills up. Real agents—robots, embedded devices, agent swarms—can’t store everything forever.
Memory-constrained agents are different. They work like human brains: short-term recall for active tasks, mid-term memory for recent events, long-term storage for important patterns. They decide what to keep, what to compress, and what to forget.
This article shows how to build memory-constrained agents. We’ll design layered memory architectures. We’ll implement eviction strategies. We’ll write code that compresses context without losing critical information. We’ll also measure whether these constraints actually improve efficiency and interpretability.
The Hidden Cost of Infinite Context
Every LLM has a context window. GPT-4 handles 128K tokens. Claude handles 200K. But these limits create problems even before you hit them.
First, cost scales with context. More tokens mean more compute. A 100K token context costs roughly 50x more than a 2K token context per request. At scale, this adds up fast.
Second, latency increases. Processing 100K tokens takes time. Real-time systems can’t wait. Chatbots need instant responses. Embedded devices have strict timing requirements.
Third, context quality degrades. Early experiments show that models perform worse when context gets very long. Important information gets buried. The model struggles to find relevant details in the noise.
Fourth, memory fills up. Agents that run continuously accumulate context. A week of conversations can exceed any context window. Something has to go.
These aren’t edge cases. Production agents face them daily. On-device AI has strict memory budgets. Multi-agent systems share compute resources. Chatbots need to remember conversations without hitting API limits.
The solution isn’t bigger context windows. It’s smarter memory management.
Architectural Concept: Layered Memory Design
Human memory works in layers. Sensory memory lasts milliseconds. Working memory holds a few items for seconds. Episodic memory stores events for days. Semantic memory keeps facts for years.
Memory-constrained agents use the same pattern.
Short-Term Memory
Short-term memory holds the active context window. This is the raw token buffer the LLM processes directly. Typical size: 2K to 8K tokens for current conversation turns.
This layer stores:
- The current user message
- Recent agent responses (last 3-5 turns)
- Immediate context needed for the current task
- Active tool calls and their results
It’s fast but limited. When full, data must move to mid-term memory or get evicted.
Mid-Term Episodic Memory
Mid-term memory stores recent interactions and events. Think of it as episodic memory: specific conversations, tool executions, and decision points from recent sessions.
Typical capacity: 50 to 200 conversation turns, or about 2-4 hours of interaction.
This layer stores:
- Summarized conversation segments
- Key decisions and their outcomes
- User preferences discovered during interaction
- Contextual patterns (e.g., “user asked about X, then Y, then Z”)
Mid-term memory uses compression. Conversations get summarized. Redundant information gets removed. Only the essentials stay.
Long-Term Vector Store Memory
Long-term memory is a vector database. It stores semantic patterns, learned behaviors, and cross-session knowledge.
This layer stores:
- Embeddings of important conversation segments
- Learned user preferences and patterns
- Successful action sequences
- Domain knowledge and facts
Unlike short-term and mid-term memory, this layer isn’t bound by token limits. It grows over time. Retrieval uses semantic search: the agent queries for relevant past experiences when needed.
Memory Hierarchy Flow
Here’s how memory moves through layers:
- New interaction enters short-term memory
- After a few turns, short-term memory gets summarized and moved to mid-term
- Important patterns in mid-term memory get embedded and stored in long-term
- When needed, the agent retrieves from long-term memory back into short-term for current reasoning
This creates a memory lifecycle. Hot data stays in short-term. Warm data in mid-term. Cold data in long-term. The agent pulls from long-term when context suggests it’s relevant.
Designing a Memory-Constrained Agent
A memory-constrained agent needs three components: a memory controller, a summarization system, and a retrieval mechanism.
Memory Controller Logic
The memory controller decides what to keep, what to compress, and what to evict. It scores each memory item for relevance, recency, and importance.
from typing import Dict, List, Tuple
from dataclasses import dataclass
from datetime import datetime, timedelta
import math
@dataclass
class MemoryItem:
content: str
timestamp: datetime
access_count: int
relevance_score: float
importance_score: float
tokens: int
class MemoryController:
def __init__(self,
max_short_term_tokens: int = 4000,
max_mid_term_items: int = 100,
recency_decay: float = 0.95):
self.max_short_term_tokens = max_short_term_tokens
self.max_mid_term_items = max_mid_term_items
self.recency_decay = recency_decay
self.short_term_memory: List[MemoryItem] = []
self.mid_term_memory: List[MemoryItem] = []
self.total_tokens = 0
def add_to_short_term(self, content: str, tokens: int,
importance: float = 0.5):
"""Add new content to short-term memory."""
item = MemoryItem(
content=content,
timestamp=datetime.now(),
access_count=1,
relevance_score=1.0, # New items are maximally relevant
importance_score=importance,
tokens=tokens
)
self.short_term_memory.append(item)
self.total_tokens += tokens
# Evict if over limit
while self.total_tokens > self.max_short_term_tokens:
self._evict_from_short_term()
def _evict_from_short_term(self):
"""Evict lowest-scoring item from short-term memory."""
if not self.short_term_memory:
return
# Score each item for retention
scored_items = []
for item in self.short_term_memory:
retention_score = self._calculate_retention_score(item)
scored_items.append((retention_score, item))
# Sort by score (lowest first)
scored_items.sort(key=lambda x: x[0])
# Evict lowest-scoring item
lowest_item = scored_items[0][1]
self.short_term_memory.remove(lowest_item)
self.total_tokens -= lowest_item.tokens
# Move to mid-term if important enough
if lowest_item.importance_score > 0.3:
self._promote_to_mid_term(lowest_item)
def _calculate_retention_score(self, item: MemoryItem) -> float:
"""Calculate how valuable it is to keep this item."""
# Time decay: older items score lower
age_hours = (datetime.now() - item.timestamp).total_seconds() / 3600
time_factor = math.exp(-age_hours * (1 - self.recency_decay))
# Access frequency: frequently accessed items score higher
access_factor = math.log(1 + item.access_count) / 10
# Importance: user-marked important items score higher
importance_factor = item.importance_score
# Relevance: currently relevant items score higher
relevance_factor = item.relevance_score
# Combined score (lower = more likely to evict)
retention_score = (
0.3 * time_factor +
0.2 * access_factor +
0.3 * importance_factor +
0.2 * relevance_factor
)
return retention_score
def _promote_to_mid_term(self, item: MemoryItem):
"""Move item from short-term to mid-term memory."""
# Compress before storing
compressed = self._compress_content(item.content)
mid_term_item = MemoryItem(
content=compressed,
timestamp=item.timestamp,
access_count=item.access_count,
relevance_score=item.relevance_score * 0.8, # Slightly less relevant
importance_score=item.importance_score,
tokens=len(compressed.split()) # Approximate
)
self.mid_term_memory.append(mid_term_item)
# Evict from mid-term if over limit
if len(self.mid_term_memory) > self.max_mid_term_items:
self._evict_from_mid_term()
def _evict_from_mid_term(self):
"""Evict lowest-scoring item from mid-term memory."""
if not self.mid_term_memory:
return
scored_items = [
(self._calculate_retention_score(item), item)
for item in self.mid_term_memory
]
scored_items.sort(key=lambda x: x[0])
# Remove lowest-scoring item (could promote to long-term here)
lowest_item = scored_items[0][1]
self.mid_term_memory.remove(lowest_item)
def _compress_content(self, content: str) -> str:
"""Compress content by removing redundancy."""
# Simple compression: remove filler words, condense
# In production, use LLM-based summarization
sentences = content.split('. ')
if len(sentences) > 3:
# Keep first and last sentences, summarize middle
compressed = f"{sentences[0]}. [Summarized {len(sentences)-2} sentences]. {sentences[-1]}"
return compressed
return content
def update_relevance(self, query: str):
"""Update relevance scores based on current query."""
# Simple keyword matching (in production, use embeddings)
query_words = set(query.lower().split())
for item in self.short_term_memory + self.mid_term_memory:
content_words = set(item.content.lower().split())
overlap = len(query_words & content_words)
relevance = min(1.0, overlap / max(1, len(query_words)))
item.relevance_score = relevance * 0.7 + item.relevance_score * 0.3 # Smoothing
def get_active_context(self, max_tokens: int = None) -> str:
"""Get the current active context for the LLM."""
if max_tokens is None:
max_tokens = self.max_short_term_tokens
# Sort by relevance and importance
active_items = sorted(
self.short_term_memory,
key=lambda x: (x.relevance_score, x.importance_score),
reverse=True
)
context_parts = []
token_count = 0
for item in active_items:
if token_count + item.tokens <= max_tokens:
context_parts.append(item.content)
token_count += item.tokens
item.access_count += 1
else:
break
return "\n\n".join(context_parts)This memory controller implements basic retention scoring. It considers recency, access frequency, importance, and relevance. Items with low scores get evicted first. Important items get promoted to mid-term memory.
Adaptive Summarization
Summarization compresses context without losing critical information. The challenge is deciding what matters.
from langchain.llms import OpenAI
from langchain.chains.summarize import load_summarize_chain
from langchain.text_splitter import RecursiveCharacterTextSplitter
import tiktoken
class AdaptiveSummarizer:
def __init__(self, llm, target_compression_ratio: float = 0.3):
self.llm = llm
self.target_compression = target_compression_ratio
self.encoder = tiktoken.encoding_for_model("gpt-3.5-turbo")
def summarize_conversation(self, messages: List[Dict],
max_output_tokens: int) -> str:
"""Summarize a conversation segment to fit token budget."""
# Combine messages into text
conversation_text = self._messages_to_text(messages)
# Count input tokens
input_tokens = len(self.encoder.encode(conversation_text))
# If already under budget, return as-is
if input_tokens <= max_output_tokens:
return conversation_text
# Calculate compression needed
compression_ratio = max_output_tokens / input_tokens
# Split into chunks if very long
if input_tokens > 4000:
summary = self._hierarchical_summarize(conversation_text,
max_output_tokens)
else:
summary = self._single_pass_summarize(conversation_text,
max_output_tokens)
return summary
def _messages_to_text(self, messages: List[Dict]) -> str:
"""Convert message list to plain text."""
text_parts = []
for msg in messages:
role = msg.get('role', 'unknown')
content = msg.get('content', '')
text_parts.append(f"{role.upper()}: {content}")
return "\n\n".join(text_parts)
def _single_pass_summarize(self, text: str, max_tokens: int) -> str:
"""Single-pass summarization."""
prompt = f"""Summarize the following conversation, preserving:
1. Key decisions and their reasoning
2. User preferences and constraints
3. Important facts or data points
4. Action items or next steps
Target length: approximately {max_tokens} tokens.
Conversation:
{text}
Summary:"""
chain = load_summarize_chain(self.llm, chain_type="stuff")
# Split text if needed
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=3000,
chunk_overlap=200
)
docs = text_splitter.create_documents([text])
summary = chain.run(docs)
# Verify token count
summary_tokens = len(self.encoder.encode(summary))
if summary_tokens > max_tokens:
# Recursively summarize further
return self._single_pass_summarize(summary, max_tokens)
return summary
def _hierarchical_summarize(self, text: str, max_tokens: int) -> str:
"""Hierarchical summarization for very long texts."""
# Split into chunks
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=3000,
chunk_overlap=200
)
chunks = text_splitter.split_text(text)
# Summarize each chunk
chunk_summaries = []
tokens_per_chunk = max_tokens // len(chunks)
for chunk in chunks:
chunk_summary = self._single_pass_summarize(chunk, tokens_per_chunk)
chunk_summaries.append(chunk_summary)
# Combine and summarize again
combined = "\n\n".join(chunk_summaries)
final_summary = self._single_pass_summarize(combined, max_tokens)
return final_summary
def extract_key_points(self, text: str, max_points: int = 5) -> List[str]:
"""Extract key points from text for ultra-compressed storage."""
prompt = f"""Extract the {max_points} most important points from this text.
Return each point as a single sentence.
Text:
{text}
Key points:
1."""
response = self.llm(prompt)
points = [line.strip() for line in response.split('\n')
if line.strip() and not line.strip().startswith('#')]
return points[:max_points]This summarizer compresses conversations adaptively. Short conversations get single-pass summarization. Long ones get hierarchical compression. The system targets a token budget and recursively summarizes until it fits.
Integration with Retrieval-Augmented Models
Memory-constrained agents need retrieval. When short-term memory lacks context, they query long-term storage.
from langchain.vectorstores import FAISS
from langchain.embeddings import OpenAIEmbeddings
from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import LLMChainExtractor
from langchain.document_loaders import TextLoader
class RAGLiteMemory:
"""Lightweight RAG system for memory-constrained agents."""
def __init__(self, embedding_model: str = "text-embedding-3-small"):
self.embeddings = OpenAIEmbeddings(model=embedding_model)
self.vector_store = None
self.memory_items = []
def initialize_vector_store(self):
"""Initialize the vector database."""
# Start with empty store
self.vector_store = FAISS.from_texts(
[""], # Dummy text
self.embeddings
)
def add_memory(self, content: str, metadata: Dict = None):
"""Add a memory item to long-term storage."""
if self.vector_store is None:
self.initialize_vector_store()
# Create document
doc = {
"content": content,
"metadata": metadata or {},
"timestamp": datetime.now().isoformat()
}
self.memory_items.append(doc)
# Add to vector store
self.vector_store.add_texts(
[content],
metadatas=[doc["metadata"]]
)
def retrieve_relevant(self, query: str, top_k: int = 3,
min_score: float = 0.7) -> List[Dict]:
"""Retrieve relevant memories for a query."""
if self.vector_store is None or len(self.memory_items) == 0:
return []
# Search vector store
results = self.vector_store.similarity_search_with_score(
query, k=top_k * 2 # Get more, filter by score
)
# Filter by minimum score
relevant = []
for doc, score in results:
# FAISS returns distance, lower is better
# Convert to similarity score (assuming cosine distance)
similarity = 1 - min(1.0, score)
if similarity >= min_score:
relevant.append({
"content": doc.page_content,
"metadata": doc.metadata,
"similarity": similarity
})
# Return top K
return sorted(relevant, key=lambda x: x["similarity"],
reverse=True)[:top_k]
def update_memory_importance(self, memory_id: str, importance: float):
"""Update importance score for a memory item."""
# In production, store importance in metadata and re-embed
pass
def get_conversation_summary(self, conversation_id: str) -> str:
"""Get a summary of a past conversation."""
relevant = self.retrieve_relevant(
f"conversation {conversation_id}",
top_k=10
)
if not relevant:
return ""
# Combine relevant memories
combined = "\n\n".join([r["content"] for r in relevant])
return combinedThis RAG-Lite system stores memories as embeddings. When the agent needs context, it queries the vector store. Relevant memories get pulled back into short-term memory for reasoning.
Implementation Guide
Let’s build a complete memory-constrained agent that uses all these components.
import os
from typing import Dict, List, Optional
from langchain.chat_models import ChatOpenAI
from langchain.schema import HumanMessage, AIMessage, SystemMessage
class MemoryConstrainedAgent:
def __init__(self,
max_context_tokens: int = 4000,
llm_model: str = "gpt-3.5-turbo"):
self.llm = ChatOpenAI(model=llm_model, temperature=0.7)
self.memory_controller = MemoryController(
max_short_term_tokens=max_context_tokens
)
self.summarizer = AdaptiveSummarizer(
llm=self.llm,
target_compression_ratio=0.3
)
self.rag_memory = RAGLiteMemory()
self.conversation_history = []
def process_message(self, user_message: str) -> str:
"""Process a user message with memory constraints."""
# Update relevance scores based on new query
self.memory_controller.update_relevance(user_message)
# Retrieve relevant long-term memories
relevant_memories = self.rag_memory.retrieve_relevant(
user_message, top_k=3
)
# Add relevant memories to short-term if space allows
for memory in relevant_memories:
content = f"[Retrieved memory] {memory['content']}"
tokens = len(self.summarizer.encoder.encode(content))
self.memory_controller.add_to_short_term(
content, tokens, importance=memory['similarity']
)
# Add user message to short-term
user_tokens = len(self.summarizer.encoder.encode(user_message))
self.memory_controller.add_to_short_term(
user_message, user_tokens, importance=0.7
)
# Get active context for LLM
active_context = self.memory_controller.get_active_context(
max_tokens=self.memory_controller.max_short_term_tokens
)
# Build messages for LLM
messages = [
SystemMessage(content="You are a helpful assistant with limited memory. "
"Use the provided context efficiently."),
]
# Add context
if active_context:
messages.append(SystemMessage(
content=f"Context from memory:\n{active_context}"
))
messages.append(HumanMessage(content=user_message))
# Get response from LLM
response = self.llm(messages)
response_text = response.content
# Add response to short-term memory
response_tokens = len(self.summarizer.encoder.encode(response_text))
self.memory_controller.add_to_short_term(
response_text, response_tokens, importance=0.6
)
# Store important interactions in long-term
if self._is_important_interaction(user_message, response_text):
summary = self.summarizer.summarize_conversation(
[
{"role": "user", "content": user_message},
{"role": "assistant", "content": response_text}
],
max_output_tokens=200
)
self.rag_memory.add_memory(
summary,
metadata={
"type": "conversation_turn",
"timestamp": datetime.now().isoformat()
}
)
# Periodically summarize and compress short-term memory
if len(self.memory_controller.short_term_memory) > 20:
self._compress_short_term_memory()
return response_text
def _is_important_interaction(self, user_msg: str, response: str) -> bool:
"""Determine if an interaction should be stored long-term."""
# Simple heuristic: store if contains key phrases
important_phrases = [
"remember", "important", "preference", "always", "never",
"decision", "agree", "disagree"
]
combined = (user_msg + " " + response).lower()
return any(phrase in combined for phrase in important_phrases)
def _compress_short_term_memory(self):
"""Compress older items in short-term memory."""
# Get older items
old_items = sorted(
self.memory_controller.short_term_memory,
key=lambda x: x.timestamp
)[:10] # Oldest 10 items
if not old_items:
return
# Summarize them together
conversation_text = "\n\n".join([item.content for item in old_items])
summary = self.summarizer.summarize_conversation(
[{"role": "system", "content": conversation_text}],
max_output_tokens=500
)
# Replace old items with summary
total_tokens = sum(item.tokens for item in old_items)
for item in old_items:
self.memory_controller.short_term_memory.remove(item)
self.memory_controller.total_tokens -= item.tokens
# Add summary as single item
summary_tokens = len(self.summarizer.encoder.encode(summary))
summary_item = MemoryItem(
content=summary,
timestamp=datetime.now(),
access_count=1,
relevance_score=0.5,
importance_score=0.5,
tokens=summary_tokens
)
self.memory_controller.short_term_memory.append(summary_item)
self.memory_controller.total_tokens += summary_tokens
def get_memory_stats(self) -> Dict:
"""Get statistics about memory usage."""
return {
"short_term_items": len(self.memory_controller.short_term_memory),
"short_term_tokens": self.memory_controller.total_tokens,
"mid_term_items": len(self.memory_controller.mid_term_memory),
"long_term_items": len(self.rag_memory.memory_items),
"max_short_term_tokens": self.memory_controller.max_short_term_tokens
}This agent integrates all three components. It manages short-term memory, retrieves from long-term when needed, and compresses context to stay within token limits.
Benchmarking Context Utilization
Measuring memory efficiency requires metrics beyond token counts.
import matplotlib.pyplot as plt
from typing import List, Dict
import numpy as np
class MemoryEfficiencyMetrics:
def __init__(self):
self.history: List[Dict] = []
def record_interaction(self,
tokens_used: int,
tokens_available: int,
retrieval_count: int,
compression_ratio: float):
"""Record metrics for an interaction."""
self.history.append({
"timestamp": datetime.now(),
"tokens_used": tokens_used,
"tokens_available": tokens_available,
"utilization": tokens_used / tokens_available if tokens_available > 0 else 0,
"retrieval_count": retrieval_count,
"compression_ratio": compression_ratio
})
def calculate_efficiency(self) -> Dict:
"""Calculate overall efficiency metrics."""
if not self.history:
return {}
utilizations = [h["utilization"] for h in self.history]
compression_ratios = [h["compression_ratio"] for h in self.history]
retrieval_counts = [h["retrieval_count"] for h in self.history]
return {
"avg_utilization": np.mean(utilizations),
"utilization_std": np.std(utilizations),
"avg_compression": np.mean(compression_ratios),
"total_retrievals": sum(retrieval_counts),
"retrieval_rate": np.mean(retrieval_counts)
}
def plot_token_efficiency(self, save_path: str = None):
"""Plot token utilization over time."""
timestamps = [h["timestamp"] for h in self.history]
utilizations = [h["utilization"] * 100 for h in self.history] # As percentage
plt.figure(figsize=(12, 6))
plt.plot(timestamps, utilizations, marker='o', markersize=4)
plt.xlabel("Time")
plt.ylabel("Token Utilization (%)")
plt.title("Memory Token Utilization Over Time")
plt.grid(True, alpha=0.3)
plt.ylim(0, 100)
if save_path:
plt.savefig(save_path)
else:
plt.show()
def plot_retention_accuracy(self, ground_truth: Dict[str, float],
save_path: str = None):
"""Plot how well the agent retains important information."""
# ground_truth: {memory_id: importance_score}
# Compare with what agent actually retained
# This would require tracking what was retained vs evicted
passThese metrics track how well the agent uses available memory. High utilization with good retrieval means efficient memory management.
Practical Use Cases
Memory-constrained agents fit several real-world scenarios.
Edge AI Devices
Edge devices have strict memory and compute limits. A smart home assistant can’t store all conversation history. It needs to remember user preferences while staying within hardware constraints.
The layered memory design works here. Short-term holds the current interaction. Mid-term remembers recent preferences. Long-term stores learned patterns like “user prefers temperature at 72°F.”
Agent Swarms
In multi-agent systems, each agent has limited context. Agents need to remember inter-agent communications and shared state without exceeding memory budgets.
Memory-constrained agents can share a long-term vector store. Each agent maintains its own short-term memory. When agents need context, they query the shared store.
Chatbots in Restricted Environments
Some chatbots run in environments with strict API limits. Free tiers might allow only 4K tokens per request. Paid tiers still benefit from lower costs with efficient memory use.
The memory controller ensures the agent stays within limits. Summarization compresses history. Retrieval pulls relevant context only when needed.
Persisting Context Across Sessions
Agents that run across multiple sessions need persistence. But they can’t load everything into memory each time.
The solution: store compressed summaries in long-term memory. On session start, retrieve relevant summaries. Load only what’s needed for the current session. This reduces startup time and memory usage.
Conclusion
Memory-constrained agents aren’t about limitations. They’re about efficiency. By managing memory intentionally, agents become more scalable, interpretable, and cost-effective.
The key insight: not all context is equal. Recent, relevant, important information deserves premium storage. Older, redundant information can be compressed or discarded. The agent decides based on retention scores and relevance.
This creates systems that think more like humans. They remember what matters. They forget what doesn’t. They retrieve relevant past experiences when needed. They don’t try to hold everything at once.
Future Trends
Memory-constrained agents are still evolving. Several trends are emerging:
Multi-modal memory compression: Beyond text, agents need to compress images, audio, and structured data. Compression techniques will adapt to different modalities.
Neurosymbolic recall: Combining neural embeddings with symbolic knowledge graphs. Agents can retrieve both semantic patterns (neural) and explicit facts (symbolic).
Distributed memory: Agents sharing memory across networks. Federated memory systems where agents contribute to and query from shared knowledge pools.
Adaptive compression: Compression strategies that learn what to preserve based on downstream task performance. The agent improves its memory management through experience.
Memory specialization: Different memory architectures for different agent types. Real-time agents prioritize speed. Analytical agents prioritize retention. The architecture adapts to the agent’s purpose.
The future of AI agents isn’t infinite context. It’s intelligent memory management. Agents that know what to remember, what to compress, and what to forget will scale better and cost less.
Start with layered memory. Add retention scoring. Implement summarization. Integrate retrieval. Measure efficiency. Iterate based on metrics.
Memory constraints aren’t obstacles. They’re design constraints that lead to better systems.
Discussion
Loading comments...