To Keep or Not to Keep: Learning KV Cache Retention in Disaggregated LLM Serving Systems

Researchers present KVLearn, a learning-based framework for cost-aware KV-cache retention in disaggregated LLM serving systems. KVLearn uses a Prefix Reuse Predictor, a Cost-Aware Retention Score, and an Adaptive Threshold Controller to improve keep/admit decisions by accounting for recomputation, transfer, and storage costs.