KVCacheMemory: Key-Value Cache for Activation Memory

KV-cache Memory Use Cases

In MemOS, KV-cache memory is best suited for storing semantically stable and frequently reused background content such as:

• Frequently asked questions (FAQs) or domain-specific knowledge
• Prior conversation history

These stable plaintext memory items are automatically identified and managed by the MemScheduler module. Once selected, they are converted into KV-format representations (KVCacheItem) ahead of time. This precomputation step stores the activation states (Key/Value tensors) of the memory in a reusable format, allowing them to be injected into the model’s attention cache during inference.

Once converted, these KV memories can be reused across queries without requiring re-encoding of the original content. This reduces the computational overhead of processing and storing large amounts of text, making it ideal for applications that require rapid response times and high throughput.

Why KV-cache Memory

Integrating MemScheduler with KV-cache memory enables significant performance optimization, particularly in the prefill phase of LLM inference.

Without KVCacheMemory

• Each new query is appended to the full prompt, including the background memory.
• The model must recompute token embeddings and attention over the full sequence — even for unchanged memory.

With KVCacheMemory

• The background content is cached once as Key/Value tensors.
• For each query, only the new user input (query tokens) is encoded.
• The previously cached KV is injected directly into the attention mechanism.

Benefits

This separation reduces redundant computation in the prefill phase and leads to:

• Skipping repeated encoding of background content
• Faster attention computation between query tokens and cached memory
• Lower Time To First Token (TTFT) latency during generation

This optimization is especially valuable in:

• Multi-turn chatbot interactions
• Retrieval-augmented or context-augmented generation (RAG, CAG)
• Assistants operating over fixed documentation or FAQ-style memory

KVCacheMemory Acceleration Evaluation

To validate the performance impact of KV-based memory injection, we conducted a set of controlled experiments simulating real memory reuse in MemOS.

Experiment Setup

During typical usage, the MemScheduler module continuously tracks interaction patterns and promotes high-frequency, stable plaintext memory into KV format. These KV memories are loaded into GPU memory as activation caches and reused during inference.

The evaluation compares two memory injection strategies:

1. Prompt-based injection: background memory is prepended as raw text.
2. KV-cache injection: memory is injected directly into the model’s attention cache.

We test these strategies across:

• Three context sizes: short, medium, and long
• Three query types: short-form, medium-form, and long-form

The primary metric is Time To First Token (TTFT), a key latency indicator for responsive generation.

Results

The following table shows results across three models (Qwen3-8B, Qwen3-32B, Qwen2.5-72B). TTFT under KV-cache injection is consistently lower than prompt-based injection, while the output tokens remain identical across both strategies.

KV-cache Memory Structure

KV-based memory reuse via KVCacheMemory offers substantial latency reduction across model sizes and query types, while maintaining identical output. By shifting reusable memory from plaintext prompts into precomputed KV caches, MemOS eliminates redundant context encoding and achieves faster response times—especially beneficial in real-time, memory-augmented LLM applications.

Each cache is stored as a KVCacheItem:

API Summary (KVCacheMemory)

Initialization

KVCacheMemory(config: KVCacheMemoryConfig)

Core Methods

When calling dump(dir), the system writes to:

<dir>/<config.memory_filename>

This file contains a pickled dictionary of all KV caches, which can be reloaded using load(dir).

How to Use

from memos.configs.memory import KVCacheMemoryConfig
from memos.memories.activation.kv import KVCacheMemory

config = KVCacheMemoryConfig(
    extractor_llm={
        "backend": "huggingface",
        "config": {"model_name_or_path": "Qwen/Qwen3-1.7B"}
    }
)
mem = KVCacheMemory(config)

# Extract and add a cache
cache_item = mem.extract("The capital of France is Paris.")
mem.add([cache_item])

# Retrieve and merge caches
merged_cache = mem.get_cache([cache_item.kv_cache_id])

# Save/load
mem.dump("tmp/act_mem")
mem.load("tmp/act_mem")

Developer Notes

• Uses HuggingFace DynamicCache for efficient key-value storage
• Pickle-based serialization for fast load/save
• All methods are covered by integration tests in /tests