When retrieving information and remembering its meaning take different routes.
Imagine an AI assistant with access to thousands of documents. Ask a question, and it searches for relevant information to answer.
Sounds familiar? That’s what Retrieval-Augmented Generation (RAG) does.
But what if an AI could use information from thousands of documents without repeatedly reading their text?
A new approach called Memorilla offers an interesting possibility.
Except there’s one uncomfortable question: What if compression loses the very detail that matters?
Meet Memorilla. But Wait… Isn’t This RAG?
In MEMORILLA: Latent Semantic Memory for LLMs (v3, November 2026), researchers introduce a trained memory module that supplies a frozen language model with just 16 query-dependent soft memory tokens.
These aren’t 16 words. They’re numerical vectors that the model can process.
But RAG already uses numerical representations called embeddings, doesn’t it?
It does!
Both approaches encode documents into embeddings and store them for later use. The difference lies in what happens when a question arrives.
RAG compares the question with stored embeddings, retrieves selected original passages, and gives the language model their text.
Memorilla uses the question to condition learned memory slots, which attend across the stored document embeddings. It then constructs compact numerical representations and passes them directly into the frozen LLM’s input-embedding space.
Same starting point. Different delivery mechanism.
Three Reports, Two Routes
Imagine three aircraft maintenance reports:
- A: Overheating increased during summer flights.
- B: Cooling efficiency decreased at high ambient temperatures.
- C: A later investigation identified a faulty temperature sensor.
Question: What caused the reported overheating?
A top-two RAG retriever might select A and B, leaving out C.
Memorilla instead processes all three document embeddings through learned attention.
A simplified illustration of one weighted combination is:
m = 0.6zA + 0.3zB + 0.1zC
Here, zA, zB, and zC are the embeddings of the reports A, B and C respectively. A contributes 60%, B 30%, and C 10%.
These are hypothetical weights, not actual Memorilla outputs. In reality, its trained attention mechanism computes question-dependent weights across multiple slots and attention heads, then refines and projects the representations into 16 soft memory tokens.
The underlying document encoder and LLM remain frozen. The memory module is trained beforehand and can handle new document collections without being retrained for each one, though in the reported tests, it was also trained on practice data from all eight benchmarks, with extra tuning for two personalisation tasks.
Oh! Wait… What If C Changes Everything?
Suppose C establishes that the apparent overheating was actually a sensor problem.
Now we have a dilemma.
RAG’s risk: It might never retrieve C.
Memorilla’s risk: Its learned compression might fail to preserve C’s decisive significance.
Memorilla could also successfully prioritise C. The example in this blog illustrates a possible risk, not a demonstrated failure.
The distinction is important:
RAG may lose evidence through selection. Memorilla may lose details through compression.
Neither guarantees that conflicting evidence will be interpreted correctly.
What Did the Research Actually Show?
The researchers tested Memorilla on eight different tasks. Overall, it had the best average score among the methods compared.
A few results stood out.
On a fact-checking task (FactKG), Memorilla scored 84%. That’s higher than an AI that read every document in full, which scored 80%. So giving an AI everything isn’t always better. This suggests that a short, focused summary built for the question can sometimes beat giving the AI all the original text.
On a task about remembering personal details from past chats (PersonaMem-v2), Memorilla scored 45%, while the next best method scored 31%. But Memorilla got some extra training for this particular task.
It also kept things small. Instead of passing the AI about 6,300 pieces of text from the documents, it passed just 16 memory tokens, about 394 times less. That’s less for the AI to read. It doesn’t mean 394 times less computing power or energy overall.
It didn’t win everywhere. On medical research questions with very short documents (PubMedQA), Memorilla scored 59%, while simply reading the documents scored 67%. When the text is already short, squeezing it can lose useful detail.
And the speed? With 512 documents, Memorilla answered in under a second, while reading everything took about 20 seconds. But that’s compared with reading all the documents, not with RAG.
The researchers also reported improved performance in a text-based agent game, TextWorld, where trained memory helped reduce repeated navigation.
But the benefits aren’t free. Memorilla’s main configuration has approximately 194 million trainable memory-module parameters and required upfront training, using four H100 GPUs in the reported supervised setup.
And as PubMedQA illustrates, compressing already-short evidence may offer little benefit and can hurt accuracy.
RAG, Memorilla, or Both?
Exact facts, quotations and citations: Source-grounded RAG
Broad relationships across lengthy histories: Memorilla, subject to testing
Both contextual understanding and traceable evidence: Hybrid retrieval and learned memory
Small documents that fit comfortably: Direct context
These are starting points, not fixed rules.
The researchers themselves acknowledge potential losses of fine-grained details and suggest that combining latent memory with traditional retrieval deserves further investigation.
The Question Worth Remembering
Can compressed memory preserve rare but decisive facts? Will its advantages carry over to unfamiliar domains? When does the training investment justify the inference savings?
These remain important questions.
Sometimes finding the right page is enough. Sometimes understanding connections across hundreds of pages matters more.
The real question isn’t how much an AI can remember. It’s whether it preserves what matters, accurately enough for the task at hand.
Research: Fallahpour et al., MEMORILLA: Latent Semantic Memory for LLMs, arXiv:2610.21375v3 (November 3, 2026).
Project: https://memory.stanford.edu/