#KVCache
LightNews LightNews
paper.bsky.social
(1/1) 170 Upvotes, 0 Comments, 28 Oct 2026, alphaXiv
FlatCake: A Flat KVCache Storage Architecture for LLM Serving
KV cache reuse accelerates LLM serving, but existing systems manage host DRAM and NVMe SSD as a tiered hierarchy where DRAM acts as an intermediate cache. Production LLM workloads break both...
www.alphaxiv.org
November 7, 2026 at 12:03 AM
paper.bsky.social
Links: abs, pdf
Search: Bluesky, Twitter, Reddit, Hacker News, Hugging Face, alphaXiv
FlatCake: A Flat KVCache Storage Architecture for LLM Serving
KV cache reuse accelerates LLM serving, but existing systems manage host DRAM and NVMe SSD as a tiered hierarchy where DRAM acts as an intermediate cache. Production LLM workloads break both premises ...
arxiv.org
November 7, 2026 at 12:03 AM
labmemo.bsky.social
https://labmemo.com/llama-cpp-v0-9-0-fp8-kvcache-mistral-large4-2026-11/
November 4, 2026 at 11:32 PM
pytorch.kr
KVCache Meetup Korea의 후기가 @pytorch.org Foundation 공식 블로그에 게재되었습니다! 🎉

한국어 후기는 @pytorch.kr 블로그에서, 공개된 행사 영상은 PyTorchKR YouTube에서 보실 수 있습니다:

* 한국어 블로그: pytorch.kr/blog/2026/kv...

* KVCache Meetup 영상: www.youtube.com/playlist?lis...

#PyTorch #PyTorchKR #KVCache #LLMInference #Meetup
November 3, 2026 at 11:36 AM
paper.bsky.social
[29/30] 170 Upvotes, 0 Comments, 1 Posts, arXiv:2610.38355

🆕FlatCake: A Flat KVCache Storage Architecture for LLM Serving

Xin Huang, Teng Ma, Zhuoran Li, Yibo Zhao, Zhaoshi Li, Qianchao Zhu
KV cache reuse accelerates LLM serving, but existing systems manage host DRAM and NVMe SSD as a tiered hierarchy where DRAM acts as an intermediate cache. Production LLM workloads break both premises of this hierarchy. First, bimodal reuse defeats DRAM temporal filtering, since workloads are dominated either by one-shot blocks with long reuse distances or by multi-gigabyte agentic sessions that exceed DRAM capacity, putting SSD on the critical read path. Second, DRAM staging and eviction-ordered layout bottleneck SSD restoration. Holding host staging buffers for in-flight prefixes causes contention under load that aborts prefetches, while incremental eviction scatters a session's blocks across the SSD and prevents large sequential reads. We present FlatCake, a flat KV storage architecture that exposes host DRAM and NVMe SSD as peer media in a unified 64-bit address space. Exploiting the fact that KV blocks are write-once, immutable, and reused at session scope, FlatCake places each block directly onto its target medium in session-contiguous, layer-major order at creation time and never migrates it. FlatCake streams coalesced SSD reads directly into a bounded GPU window via GPUDirect Storage and decouples DRAM and SSD request queues to prevent head-of-line blocking. On a production Claude Code trace with DeepSeek-V4-Flash, FlatCake sustains a 94% cache hit rate as load increases (versus 64% for Mooncake), eliminates Mooncake's $6.5\times$--$6.7\times$ SSD read amplification, and reduces peak-load mean TTFT by $1.6\times$ while improving total token throughput by $1.6\times$. Across 8K--128K contexts on Llama-3.1-70B, FlatCake further reduces mean TTFT by up to $2.4\times$ over Mooncake and $1.8\times$ over Mooncake-GDS.
November 7, 2026 at 12:03 AM
pytorch.org
Catch the recap from KVCache Meetup Korea, co-hosted by PyTorchKR and XCENA. Speakers explored LLM serving, CXL memory pooling, and inference optimizations across hardware and software.

Read the full event recap by PyTorch Foundation Ambassador Junghwan Park: https://bit.ly/4sLT8bG

November 3, 2026 at 10:00 AM
realfeedapp.com
The team built KVGuard, a sanitizer for LLM serving systems that detects a bug where the server reads data from a shared KV cache buffer belonging to other users. Testing on SGLang 0.5.19 revealed 4095 such uninitialized reads.

#KVCache #LLM #ServingEngine
KVGuard detected the KV cache leak bug in SGLang
Source: HackerNews AI
realfeedapp.com
November 5, 2026 at 11:22 AM
osrepos.bsky.social
tair-kvcache: Manage and Simulate LLM Inference KV Caches
https://osrepos.com/repo/alibaba-tair-kvcache?utm_source=bluesky

Alibaba Cloud Tair KVCache combines global KV-cache management with HiSim, a CPU-based simulator for LLM inference performance.

#opensource
tair-kvcache: Manage and Simulate LLM Inference KV Caches
Alibaba Cloud Tair KVCache combines global KV-cache management with HiSim, a CPU-based simulator for LLM inference performance. It is aimed at teams integrating cache storage with inference engines or evaluating workloads before allocating GPU resources.
osrepos.com
October 29, 2026 at 8:05 PM
csdc-bot.bsky.social
Xin Huang, Teng Ma, Zhuoran Li, Yibo Zhao, Zhaoshi Li, Qianchao Zhu, Wei Yang, Chenxi Wang, Tao Xie, Youwei Zhuo: FlatCake: A Flat KVCache Storage Architecture for LLM Serving https://arxiv.org/abs/2610.38355 https://arxiv.org/pdf/2610.38355 https://arxiv.org/html/2610.38355
October 30, 2026 at 6:40 AM
csai-bot.bsky.social
Suhas Kumar Morisetty, Vaibhavi Lokegaonkar, Sachin Mathew Varghese, Wesley Harris: WAKMan: Workload Aware KVCache Manager https://arxiv.org/abs/2610.36355 https://arxiv.org/pdf/2610.36355 https://arxiv.org/html/2610.36355
October 29, 2026 at 6:39 AM
sipirtu.com
WAKMan is a workload-aware KV cache manager that picks its eviction policy from live traffic without knowing request types or reading prompts.

Source: arXiv
WAKMan: Workload Aware KVCache Manager
Abstract page for arXiv paper 2610.36355: WAKMan: Workload Aware KVCache Manager
arxiv.org
October 28, 2026 at 8:58 AM
csai-bot.bsky.social
Suhas Kumar Morisetty, Vaibhavi Lokegaonkar, Sachin Mathew Varghese, Wesley Harris: WAKMan: Workload Aware KVCache Manager https://arxiv.org/abs/2610.36355 https://arxiv.org/pdf/2610.36355 https://arxiv.org/html/2610.36355
October 28, 2026 at 6:39 AM
realfeedapp.com
A vulnerability was found allowing access to KV cache from one request by another when using SGLang PD. Developers implemented a deferred memory release mechanism and checksum verification to fix the issue.

#Sglang #KVCache #Security
SGLang fixed KV cache exposure in GLM-5.3
Source: HackerNews AI
realfeedapp.com
October 27, 2026 at 6:00 AM
ayourtch.bsky.social
Rougher quants make for some interesting failure modes - so ensure you test the function as well! (same about KVCache quant, anything below 8 bit is risky - the model works but has “hazy brain”).
October 26, 2026 at 7:35 AM
aidailypost.com
Stanford & UC Berkeley just turbo‑charged long‑context prefill with TokenGate—up to 11.3× faster! 🚀 See how token pruning, KV‑cache tricks, and a Qwen3 LoRA adapter reshape LLM inference. #TokenGate #LongContext #KVCache

🔗 aidailypost.com/news/stanfor...
October 25, 2026 at 1:33 PM
pytorch.kr
KVCache Meetup Korea 후기: 메모리 병목과 KV 캐시 최적화를 함께 살펴본 시간 | 파이토치 한국 사용자 모임

발표를 듣기 위해 모인 KVCache Meetup Korea 참석자들.

KVCache Meetup Korea 개요

지난 2026년 9월 30일 저녁, 서울 강남 드림플러스에서 KVCache Meetup Korea 행사가 진행되었습니다. PyTorchKR과 CXL 전문 스타트업 XCENA가 공동으로 주최하고, LMCache의 개발사 Tensormesh가 후원했습니다. 이번 행사는 대규모 언어 모델(LL…
KVCache Meetup Korea 후기: 메모리 병목과 KV 캐시 최적화를 함께 살펴본 시간 | 파이토치 한국 사용자 모임
발표를 듣기 위해 모인 KVCache Meetup Korea 참석자들. KVCache Meetup Korea 개요 지난 2026년 9월 30일 저녁, 서울 강남 드림플러스에서 KVCache Meetup Korea 행사가 진행되었습니다. PyTorchKR과 CXL 전문 스타트업 XCENA가 공동으로 주최하고, LMCache의 개발사 Tensormesh가 후원했습니다. 이번 행사는 대규모 언어 모델(LLM, Large Language Model)의 추론 과정에서 메모리 병목을 줄이는 방법과 경험을 나누는 자리였습니다. 이번 밋업의 주제는 KV 캐시(KV Cache)였습니다. KV 캐시는 모델이 앞서 처리한 토큰의 어텐션 키(Key)와 값(Value)을 저장해, 다음 토...
discuss.pytorch.kr
October 23, 2026 at 11:10 AM
donwebmedia.bsky.social
MiniMax M3.1 Flash: los pesos ya están abiertos

¿Ya podés self-hostear MiniMax M3.1 Flash? Los pesos salieron el 23 de octubre con soporte día cero de vLLM: mirá el checklist antes de migrar

#minimaxm31flash #vllm #kvcache #openweights #nvfp4
MiniMax M3.1 Flash: los pesos ya están abiertos
Pesos liberados el 23 de octubre, soporte día cero de vLLM en tres plataformas de hardware, y el caveat de memoria que nadie puede ignorar.
blog.donweb.com
October 23, 2026 at 4:27 PM
discuss.huggingface.co.web.brid.gy
Live Transformer Graphs: Running and Editing LLM Architectures on a Custom CUDA Engine
Has anyone ever wondered what it would be like to edit the model while its actually running live? well, this is the result… also note every part of the Kvcache persisted as state carries forever forward. This project is built into Leviathan which I have previously mentioned in Show/tell it’s all about manipulation of when a model is running live for the most part.
discuss.huggingface.co
October 22, 2026 at 8:05 PM
tokenburn.co.uk
DeepSeek open-sourced 3FS, a file system built for AI training and inference. The KVCache part is the hook: SSDs instead of DRAM for caching, with much bigger capacity. You can git clone it, though their test cluster was 180 storage nodes, so maybe not this weekend.
GitHub - deepseek-ai/3FS: A high-performance distributed file system designed to address the challenges of AI training and inference workloads.
A high-performance distributed file system designed to address the challenges of AI training and inference workloads. - deepseek-ai/3FS
github.com
October 21, 2026 at 7:00 PM
csdb-bot.bsky.social
Jun Kong, Qi Lin, Zhichao Cao: SARP: An On-SSD Index for KVCache with Fast Prefix Matching and Recovery https://arxiv.org/abs/2610.26849 https://arxiv.org/pdf/2610.26849 https://arxiv.org/html/2610.26849
October 20, 2026 at 6:40 AM
ai-bloom.warp-studio.com
vLLMの双方向KV転送によるマルチターンエージェントのTTFT最適化

vLLMにおける双方向KVキャッシュ転送を用いたマルチターンエージェントのTTFT改善手法。

#LLM #vLLM #KVCache #PerformanceOptimization #AIAgents
vLLMの双方向KV転送によるマルチターンエージェントのTTFT最適化
vLLMにおける双方向KVキャッシュ転送を用いたマルチターンエージェントのTTFT改善手法。
ai.warp-studio.com
October 20, 2026 at 12:09 AM
iskamag.com
FP16 kvcache, used only 25k output tokens, and this is even better...
October 16, 2026 at 4:30 PM
siliconsignalai.bsky.social
New arXiv work finds remote KV cache hits can beat GPU recompute latency in theory but cause queue instability in practice, with p99 TTFB ballooning on GLM-5.3 FP8 MoE serving. Synchronous loading left GPUs only 7% busy while…

#KVCache #LLMServing #AIInfra #GPUs
https://arxiv.org/abs/2610.21500
October 17, 2026 at 12:01 AM
futuregearai.bsky.social
SparPR is a runtime mechanism that recovers the key-value cache during LLM inference on consumer GPUs by selecting, budgeting, and recomputing blocks on demand, without changing model weights or requiring offline…

#LLMInference #KVCache #ConsumerGPU #EfficientAI
https://arxiv.org/abs/2610.18140
October 14, 2026 at 8:01 PM
ai-firehose.column.social
Mooncake enhances LLM serving via a KVCache-centric architecture, achieving a 525% throughput increase and enabling workloads to manage 75% more requests. By separating prefill and decoding phases, it alleviates overload while optimizing for strict latency demands. https://arxiv.org/abs/2407.00079
Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving
ArXiv link for Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving
arxiv.org
October 10, 2026 at 4:10 AM