AIERAFrontiers
All articles
Applied AI

Beyond the Context Window — Engram Engineering for Productive Agent Memory

A deep technical analysis of LLM long-context limitations and the Engram architecture: semantic MMR routing, tool-result slotting, and hierarchical sub-agent isolation.

AieraJune 29, 202611 min
aierafrontiers.com/en/article/engram-memory-architecture-en

As LLM context windows expand to millions of tokens, a naive assumption has taken hold in AI engineering: if it fits in the context window, just stuff it all in there.

In production, this brute-force approach turns into a financial and operational disaster. Large context windows do not solve the structural limitations of the Transformer attention mechanism. Instead, they lead to severe latency degradation, astronomical API bills, and expose systems to catastrophic security vulnerabilities such as prompt injection.

To build reliable, productive agentic systems, we must treat memory not as an infinitely expanding text buffer, but as a managed, multi-tiered system. This article analyzes the fundamental limitations of modern long-context handling approaches, evaluates the leading architectural paradigms for agent memory, and introduces Engram — our proposed memory router architecture that combines semantic routing, tool-result slotted storage, and safe hierarchical isolation in a single SQLite-based system.

1. Motivation: “Lost in the Middle”

The primary bottleneck of long-context models is not hardware, but the distribution of attention. In their seminal paper “Lost in the Middle: How Language Models Use Long Contexts” (arXiv:2307.03172), Liu et al. (2023) revealed a fundamental flaw in how LLMs process long inputs.

U-shaped retrieval accuracy curve — the Lost in the Middle phenomenon

Fig. 1. U-shaped curve of retrieval accuracy as a function of position in the context (Liu et al., 2023)

Key findings:

  • U-shaped retrieval curve: LLMs effectively retrieve information placed at the very beginning (Position 1: ~70% accuracy) or at the very end of the input context. However, retrieval accuracy drops sharply in the middle.
  • 20 percentage point drop: Accuracy falls by more than 20 percentage points when the relevant information is moved from position 1 to position 10 in an input of 20 documents. At position 10, performance bottoms out at ~45%.
  • Context degradation plateau: When context scales to 20 or more documents, model performance plateaus, performing no better than in the no-document regime (where the model has no access to documents at all).
  • Baseline limit: The baseline accuracy in the no-document regime on their test set was 56.1%. When the model is forced to search through a bloated context of 20+ documents, it performs worse than its no-document baseline if the target information is buried in the middle of the input.

This proves that simply expanding context windows is an inefficient way to build long-term agent memory. If your agent relies on a flat history of past tool outputs, user messages, and system instructions, it will consistently fail to retrieve critical details buried in the middle of the execution history.

2. Prior Work: Three Memory Management Paradigms

To address this problem, researchers and engineers have proposed three distinct approaches: OS-like paged organization, security isolation in multi-agent environments, and low-level KV-cache engineering. Each solves part of the problem but introduces new trade-offs.

Engram architecture diagram — memory-router design

Fig. 2. Overall Engram architecture — Memory Router with Redis, SQLite, and isolated agents

Prior Work A: MemGPT (OS-style memory hierarchy)

Packer et al. (2023), “MemGPT: Towards LLMs as Operating Systems”, arXiv:2310.08560

MemGPT models the LLM context window as RAM. It introduces a multi-tiered memory hierarchy:

  1. Working Memory: The active context window (system prompt, dialogue core, FIFO queue).
  2. Archival Storage: An out-of-context database for long-term records.
  3. Recall Storage: Historical event logs.

The agent manages this hierarchy by issuing explicit function calls (e.g., send_message, core_memory_append, archival_memory_search).

Results:

  • Deep memory retrieval: On deep memory retrieval QA tasks, GPT-4 with MemGPT achieved 92.5% accuracy, compared to a baseline of only 32.1% (an improvement of +60.4 percentage points).
  • Retrieval from nested KV stores: While baseline GPT-4 scored 0% (complete inability to navigate nested KV stores), MemGPT functioned fully.
  • Token efficiency: On documentary QA tasks over 50k-token contexts, MemGPT matched or exceeded GPT-4 with full context, while using 5–10x fewer tokens per query.

Limitations:

  • Developer burden: Requires developers to write complex explicit state-saving loops and prompt templates.
  • No auto-eviction: No automatic cache invalidation; agents must manually decide when to clear or overwrite memory.
  • Single agent only: Entirely designed around a single-agent loop, making it difficult to scale to multi-agent systems without memory conflicts.

Prior Work B: AgentSys (multi-agent system security)

Ruoyao Wen, Hao Li, Chaowei Xiao, Ning Zhang (2026). “s: A Multi-Agent System with Security”,arXiv:2602.07398

Multi-agent environments introduce serious attack vectors. If Agent A retrieves untrusted user content and shares a memory space with Agent B (which has write access to a database), an injection attack can compromise the entire system. AgentSys addresses this by enforcing strict session isolation between different types of agents.

Results:

  • ASR reduction: In security evaluations, the full AgentSys system reduced the Attack Success Rate (ASR) to 0.78%, compared to 30.66% for traditional multi-agent systems with shared memory and 55.4% for the baseline single-agent configuration. This amounts to a 97.5% reduction in the attack success rate.
  • Utility preservation: Security constraints did not degrade standard tasks; utility on regular (non-attacking) queries was 64.36% on AgentSys versus 63.54% on the baseline.

Limitations:

  • Token overhead: Agent isolation requires each sub-agent to carry its own system prompt and redundant tool definitions, inflating token expenditure.
  • No legitimate sharing: Session isolation is all-or-nothing; there is no safe, structured mechanism for passing verified context between agents.

Prior Work C: Below the Prompt (KV-cache engineering)

Yakov Pyotr Shkolnikov (2026). “Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices”,arXiv:2603.04428

Rather than managing memory at the application level, Below the Prompt operates at the inference engine level. It pre-populates and freezes the Key-Value (KV) cache for static system prompts and loads incoming user sessions directly from this warm state, using Q4 quantization for the cached keys and values.

Results:

  • Extreme speedup: Achieved up to 136x speedup on Time To First Token (TTFT) on Gemma 3 12B with a 32K context window.
  • Perplexity impact: Negligible impact on model quality:
  • Direct reuse without delta: -0.7% perplexity degradation.
  • Q4 KV-cache quantization: +2.8% perplexity degradation.
  • Prompt compression: +3.0% perplexity degradation.
  • Instant recovery: Warm recovery from a frozen cache takes < 1 second regardless of context length.

Limitations:

  • No cross-session memory: The KV cache is static and read-only. It cannot carry state between different user conversations.
  • Token-level only: Operates exclusively at the token level, with no semantic understanding of the stored information.
  • Single session: Unable to manage state in complex multi-agent workflows.

3. Engram Architecture

To overcome these limitations, we designed Engram: a unified memory router architecture. Engram combines the performance of KV-cache warmup, the security of isolated multi-agent environments, and the long-term retrieval of OS-style multi-tiered memory.

Component A: Memory Router (semantic routing)

Rather than forcing the LLM to search through historical logs, Engram uses a dedicated lightweight routing layer.

  • Model: A local, highly optimized MiniLM-L6-v2 embedding model (384-dimensional) running on the host machine.
  • Algorithm: Maximal Marginal Relevance (MMR) to balance relevance and informational diversity, preventing the LLM from receiving redundant context.
  • Mathematical objective:
MMR = argmax Di ∈ R \ S [ λ · Sim1(Di, Q) − (1−λ) · max Dj ∈ S Sim2(Di, Dj) ]

Where Q is the query, R is the set of retrieved documents, S is the set of already selected documents, and λ (tunable between 0.3 and 0.7) controls the balance between strict semantic relevance and context diversity.

# Pure Python implementation of the Engram MMR memory router
import numpy as np
from typing import List, Dict, Any


def engram_mmr_router(
    query_emb: np.ndarray,
    candidate_embs: np.ndarray,
    candidates: List[Dict[str, Any]],
    top_k: int = 5,
    lambda_param: float = 0.5,
) -> List[Dict[str, Any]]:
    """
    Performs Maximal Marginal Relevance (MMR) selection among
    memory candidates to optimize relevance while minimizing
    context redundancy.
    """
    if len(candidates) == 0:
        return []

    # Normalize embeddings for cosine similarity
    query_emb = query_emb / np.linalg.norm(query_emb)
    candidate_embs = candidate_embs / np.linalg.norm(
        candidate_embs, axis=1, keepdims=True
    )

    # Compute similarity to the query: Sim1(Di, Q)
    query_sims = np.dot(candidate_embs, query_emb)

    selected_indices: List[int] = []
    unselected_indices = list(range(len(candidates)))

    for _ in range(min(top_k, len(candidates))):
        best_idx = None
        best_mmr = -np.inf

        for i in unselected_indices:
            relevance = query_sims[i]

            if len(selected_indices) == 0:
                diversity_penalty = 0
            else:
                selected_embs = candidate_embs[selected_indices]
                inter_sims = np.dot(candidate_embs[i], selected_embs.T)
                diversity_penalty = np.max(inter_sims)

            mmr_score = (
                lambda_param * relevance
                - (1 - lambda_param) * diversity_penalty
            )

            if mmr_score > best_mmr:
                best_mmr = mmr_score
                best_idx = i

        if best_idx is not None:
            selected_indices.append(best_idx)
            unselected_indices.remove(best_idx)

    return [candidates[i] for i in selected_indices]

Component B: Tool Result Slots (Redis TTL cache)

Tool outputs are often voluminous, highly structured, and time-sensitive. Engram does not store raw tool outputs in the semantic memory layer. Instead, it uses a dedicated Redis-based slotting system:

  • Slot assignment: Each tool call receives a named slot (e.g., slot:search_results_42). The tool output is stored in Redis with a TTL.
  • Reference injection: The LLM receives only a compact reference token (e.g., [TOOL_RESULT:search_results_42]) in its context window, rather than the full output.
  • On-demand expansion: If the LLM determines it needs the full output, it issues a special read operation that retrieves the data from Redis and injects it into the working context.

This pattern prevents tool outputs from dominating the context window while keeping them instantly accessible.

Component C: Unified SQLite Layer (FTS5 + sqlite-vec)

All long-term memory is stored in a single SQLite database with two retrieval paths:

  • FTS5 full-text search: For precise keyword and entity lookups (e.g., “find the session where the user mentioned Project X”).
  • sqlite-vec vector search: For semantic similarity queries using MiniLM-L6-v2 embeddings stored alongside the original content.
-- Engram SQLite schema

CREATE TABLE IF NOT EXISTS engram_sessions (
    session_id TEXT PRIMARY KEY,
    agent_type TEXT NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    parent_session_id TEXT,
    nesting_level INTEGER DEFAULT 0,
    FOREIGN KEY (parent_session_id)
        REFERENCES engram_sessions(session_id)
        ON DELETE CASCADE
);

CREATE TABLE IF NOT EXISTS engram_vector_archive (
    memory_id INTEGER PRIMARY KEY AUTOINCREMENT,
    session_id TEXT NOT NULL,
    embedding BLOB NOT NULL,      -- MiniLM-L6-v2 dimensions
    raw_content TEXT NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    FOREIGN KEY (session_id)
        REFERENCES engram_sessions(session_id)
        ON DELETE CASCADE
);

CREATE INDEX IF NOT EXISTS idx_vector_session
    ON engram_vector_archive(session_id);

Component D: Hierarchical Sub-agent Isolation

To prevent prompt injection and cross-agent memory contamination, Engram adapts the security principles of AgentSys into a hierarchical model.

  • Isolated sessions (sandboxes): Each sub-agent receives an isolated session namespace in the SQLite memory layer. Sub-agents cannot query or modify the memory of other sub-agents.
  • Configurable nesting limit: To prevent infinite loops, the execution tree enforces a strict nesting depth (default: 2 levels).
  • Mediated communication: Sub-agents communicate exclusively by returning structured summaries to their parent agent via the memory router. The parent agent never ingests the raw, unverified history of sub-agents.

Component E: Automated Feedback Loop

Retrieval systems often suffer from silent failures when the router returns irrelevant context. Engram addresses this with a continuous feedback loop:

  1. Failure logging: When a sub-agent fails a task or explicitly flags a retrieval as unhelpful, the system marks the query as a “memorable opportunity”.
  2. Background consolidation: A low-priority background thread processes these flagged events, spending a strict token budget (capped at 5% of total system tokens) to run a summarization model.
  3. Index optimization: The summarization model distills the missing context into structured key-value pairs that are written back to the SQLite vector index to improve future routing accuracy.

4. Comparative Analysis

Dimension MemGPT (OS-style) AgentSys (secure multi-agent) Below the Prompt (KV-cache eng.) Engram (proposed)
Memory architecture Multi-tiered (Working / Archival / Recall) Isolated, non-shared sessions Static KV-cache warmup Multi-tiered semantic router with local vector DB
Retrieval mechanism Explicit LLM function calls None (sharing prohibited) Static token offset Automatic MMR routing + SQLite vector index
Multi-agent isolation None (single-agent loop) Strict isolation None (single session/user) Hierarchical sandbox
Security profile (ASR) High (vulnerable) Very low (0.78%) Moderate Very low (< 1.0%)
Latency profile High High Ultra-low (<1s) Low (<1s)
Main bottleneck Developer burden Redundant tokens for tool defs Unable to write dynamically 5% token budget (background)

5. Business and Operational Impact

For organizations running agentic systems in production, the shift from naive context stuffing to the Engram architecture yields immediate improvements in cost, performance, and security:

  • Token cost reduction: By replacing bloated historical context with semantic routing and tool-result slotting, Engram reduces average context volume by 3–10x compared to naive approaches.
  • API cost reduction: Smaller contexts and fewer retrieval steps translate directly into a 3–5x reduction in monthly LLM provider spending.
  • Sub-second latency: Offloading vector search and raw tool-result storage to SQLite and Redis delivers high execution speed comparable to KV-cache recovery performance (latency <1s for typical operations).
  • Production-grade security: By isolating sub-agents in namespaced sandboxes, Engram reduces the attack success rate (ASR) to <1%, matching AgentSys-level security without its high token overhead.

6. Conclusion

The “Lost in the Middle” phenomenon highlights a key limitation of modern LLMs: scaling the context window is not a substitute for effective memory design.

By combining the strengths of prior work — the multi-tiered memory hierarchy of MemGPT, the security boundaries of AgentSys, and the performance optimizations of Below the Prompt — Engram delivers a unified, production-ready memory architecture. It eliminates context bloat, ensures the security of multi-agent execution, and sustains sub-second latency — all on a lightweight, self-contained SQLite backend.

References