Memorilla: Latent Semantic Memory for LLMs

Memorilla is a memory system that gathers information from an entire document collection into a fixed number of query-dependent 'memory tokens' for a frozen LLM. Using only 16 memory tokens, Memorilla achieved 84.33% accuracy on FactKG versus 80.43% with full-context inference. Memorilla improved an agent's average success rate from 18.6% to 24.2% in a TextWorld environment.