I wish that qwen had those deepseek kv cache tricks.
November 3, 2026 at 1:42 PM
I wish that qwen had those deepseek kv cache tricks.
LoopKV enhances transformer efficiency through dynamic cache compression for looped models, cutting KV memory by 35% and improving decoding throughput up to 2.2x without accuracy loss. This change permits more scalable AI applications in language modeling. https://arxiv.org/abs/2611.04137
LoopKV: Loop-Aware KV Cache Compression for Serving Looped Transformers
ArXiv link for LoopKV: Loop-Aware KV Cache Compression for Serving Looped Transformers
arxiv.org
November 5, 2026 at 7:20 PM
LoopKV enhances transformer efficiency through dynamic cache compression for looped models, cutting KV memory by 35% and improving decoding throughput up to 2.2x without accuracy loss. This change permits more scalable AI applications in language modeling. https://arxiv.org/abs/2611.04137
LoopKV revolutionizes KV cache management for looped transformers, cutting memory usage by 35% and boosting decoding speed by 2.2x versus standard methods. This innovation optimally allocates precision per layer, enhancing performance without losing accuracy. https://arxiv.org/abs/2611.04137
LoopKV: Loop-Aware KV Cache Compression for Serving Looped Transformers
ArXiv link for LoopKV: Loop-Aware KV Cache Compression for Serving Looped Transformers
arxiv.org
November 5, 2026 at 7:30 PM
LoopKV revolutionizes KV cache management for looped transformers, cutting memory usage by 35% and boosting decoding speed by 2.2x versus standard methods. This innovation optimally allocates precision per layer, enhancing performance without losing accuracy. https://arxiv.org/abs/2611.04137
Zyphra's Hierarchical Attention organizes the KV-cache into a searchable tree, letting LLMs to find relevant tokens without scanning the entire context, while preserving near full-attention accuracy up to models with 35B with just 1B tokens of additional training.
www.alphaxiv.org/abs/2610.42112
www.alphaxiv.org/abs/2610.42112
November 7, 2026 at 5:35 AM
Zyphra's Hierarchical Attention organizes the KV-cache into a searchable tree, letting LLMs to find relevant tokens without scanning the entire context, while preserving near full-attention accuracy up to models with 35B with just 1B tokens of additional training.
www.alphaxiv.org/abs/2610.42112
www.alphaxiv.org/abs/2610.42112
LoopKV enhances KV cache compression for looped transformers, achieving a 35% storage reduction and a 1.7–2.2x boost in goodput over traditional methods. It allocates precision dynamically and promotes efficient KV sharing, enhancing large language model performance. https://arxiv.org/abs/2611.04137
LoopKV: Loop-Aware KV Cache Compression for Serving Looped Transformers
ArXiv link for LoopKV: Loop-Aware KV Cache Compression for Serving Looped Transformers
arxiv.org
November 4, 2026 at 8:40 PM
LoopKV enhances KV cache compression for looped transformers, achieving a 35% storage reduction and a 1.7–2.2x boost in goodput over traditional methods. It allocates precision dynamically and promotes efficient KV sharing, enhancing large language model performance. https://arxiv.org/abs/2611.04137
Making Local LLMs Usable on 8GB GPUs
Learn how to optimize local LLM inference on 8GB GPUs using KV-cache quantization, Flash Attention, batching, and practical latency benchmarks.
Telegram AI Digest
#ai #gpu #llm
Learn how to optimize local LLM inference on 8GB GPUs using KV-cache quantization, Flash Attention, batching, and practical latency benchmarks.
Telegram AI Digest
#ai #gpu #llm
Making Local LLMs Usable on 8GB GPUs
Learn how to optimize local LLM inference on 8GB GPUs using KV-cache quantization, Flash Attention, batching, and practical latency benchmarks.
hackernoon.com
November 4, 2026 at 5:12 AM
Making Local LLMs Usable on 8GB GPUs
Learn how to optimize local LLM inference on 8GB GPUs using KV-cache quantization, Flash Attention, batching, and practical latency benchmarks.
Telegram AI Digest
#ai #gpu #llm
Learn how to optimize local LLM inference on 8GB GPUs using KV-cache quantization, Flash Attention, batching, and practical latency benchmarks.
Telegram AI Digest
#ai #gpu #llm
gputaskman: An htop for what's inside your NVIDIA GPU on Windows: process memory, engine activity, and LLM weights/KV-cache introspection | by Scott Hanselman
buff.ly/SRklbVD
#oss #gpu #devtools #utilities #python
buff.ly/SRklbVD
#oss #gpu #devtools #utilities #python
GitHub - shanselman/gputaskman: An htop for what's inside your NVIDIA GPU on Windows: process memory, engine activity, and LLM weights/KV-cache introspection
An htop for what's inside your NVIDIA GPU on Windows: process memory, engine activity, and LLM weights/KV-cache introspection - shanselman/gputaskman
buff.ly
November 3, 2026 at 9:01 PM
gputaskman: An htop for what's inside your NVIDIA GPU on Windows: process memory, engine activity, and LLM weights/KV-cache introspection | by Scott Hanselman
buff.ly/SRklbVD
#oss #gpu #devtools #utilities #python
buff.ly/SRklbVD
#oss #gpu #devtools #utilities #python
"Mistral Large 4"
Published exclusively on alphaXiv, Mistral's latest report shares that reducing KV-cache memory by 16x hurt long-context agentic performance
So they restore full-context attention and adopt DSA to selectively attend to the 2048 most relevant tokens
alphaxiv.org/abs/2611.mis...
Published exclusively on alphaXiv, Mistral's latest report shares that reducing KV-cache memory by 16x hurt long-context agentic performance
So they restore full-context attention and adopt DSA to selectively attend to the 2048 most relevant tokens
alphaxiv.org/abs/2611.mis...
November 4, 2026 at 6:33 AM
"Mistral Large 4"
Published exclusively on alphaXiv, Mistral's latest report shares that reducing KV-cache memory by 16x hurt long-context agentic performance
So they restore full-context attention and adopt DSA to selectively attend to the 2048 most relevant tokens
alphaxiv.org/abs/2611.mis...
Published exclusively on alphaXiv, Mistral's latest report shares that reducing KV-cache memory by 16x hurt long-context agentic performance
So they restore full-context attention and adopt DSA to selectively attend to the 2048 most relevant tokens
alphaxiv.org/abs/2611.mis...
Supermicro, IBM, and NVIDIA Unveil KV Cache Solution for Enhanced AI Inference Performance#USA#Supermicro#San_Jose#NVIDIA#IBM
Supermicro, IBM, and NVIDIA Unveil KV Cache Solution for Enhanced AI Inference Performance
Supermicro, in collaboration with IBM and NVIDIA, introduces a new KV cache solution that significantly boosts AI inference performance, enabling faster and more efficient processes.
third-news.com
November 5, 2026 at 2:41 PM
Supermicro, IBM, and NVIDIA Unveil KV Cache Solution for Enhanced AI Inference Performance#USA#Supermicro#San_Jose#NVIDIA#IBM
Supermicro, IBM, and NVIDIA Collaborate to Launch KV Cache Solution for AI Inference Performance#USA#Supermicro#San_Jose#IBM#KV_Cache
Supermicro, IBM, and NVIDIA Collaborate to Launch KV Cache Solution for AI Inference Performance
Discover how Supermicro's new KV cache solution, developed in partnership with IBM and NVIDIA, significantly enhances AI inference efficiency with advanced storage capabilities.
third-news.com
November 5, 2026 at 2:41 PM
Supermicro, IBM, and NVIDIA Collaborate to Launch KV Cache Solution for AI Inference Performance#USA#Supermicro#San_Jose#IBM#KV_Cache
LoopKV optimizes key-value cache compression for looped transformers, reducing memory by 35% and boosting throughput by 2.2x over traditional methods. It tailors quantization precision per layer, enhancing performance for language models without sacrificing accuracy. https://arxiv.org/abs/2611.04137
LoopKV: Loop-Aware KV Cache Compression for Serving Looped Transformers
ArXiv link for LoopKV: Loop-Aware KV Cache Compression for Serving Looped Transformers
arxiv.org
November 4, 2026 at 8:01 PM
LoopKV optimizes key-value cache compression for looped transformers, reducing memory by 35% and boosting throughput by 2.2x over traditional methods. It tailors quantization precision per layer, enhancing performance for language models without sacrificing accuracy. https://arxiv.org/abs/2611.04137
Te juro que no he acelerado el vídeo. Al menos tendremos el doble de caché KV con Qwen4.
November 5, 2026 at 1:25 PM
Te juro que no he acelerado el vídeo. Al menos tendremos el doble de caché KV con Qwen4.
Specialized inference frameworks enable a single GPU to handle more concurrent requests with lower latency by automating KV cache management and request scheduling.
#LLM #Inference #GPU
#LLM #Inference #GPU
Frameworks simplify KV cache management and request batching
Source: HackerNews AI
realfeedapp.com
November 4, 2026 at 8:40 AM
Specialized inference frameworks enable a single GPU to handle more concurrent requests with lower latency by automating KV cache management and request scheduling.
#LLM #Inference #GPU
#LLM #Inference #GPU
Correction: 8-12 GB was too high. A typical 32B model with grouped-query attention (64 layers, 8 KV heads, 128 dims) needs about 2 GB of KV cache at 8K context in FP16. Older models without GQA need much more.
November 7, 2026 at 9:26 AM
Correction: 8-12 GB was too high. A typical 32B model with grouped-query attention (64 layers, 8 KV heads, 128 dims) needs about 2 GB of KV cache at 8K context in FP16. Older models without GQA need much more.
@strike007.bsky.social JIT graph rewriting solves the easy half, but squeezing out five microseconds on inference doesn't rescue a pipeline stalled on cold KV-cache synchronization across three time zones.
November 5, 2026 at 6:23 PM
@strike007.bsky.social JIT graph rewriting solves the easy half, but squeezing out five microseconds on inference doesn't rescue a pipeline stalled on cold KV-cache synchronization across three time zones.
Re @riccinov.bsky.social: Swapping to sparse attention saves KV-cache overhead, yet the base model still demands eighty-four consumer GPUs just to boot.
Moonshot posted open weights for Kimi K3.1 Pro on Hugging Face: a natively multimodal agentic model labeled “3T-class,” with a 1-million-token context. Versus Kimi K3, full-attention Gated MLA layers are swapped for Gated DSA. API name: kimi-k3.1-pro.
November 6, 2026 at 4:01 AM
Re @riccinov.bsky.social: Swapping to sparse attention saves KV-cache overhead, yet the base model still demands eighty-four consumer GPUs just to boot.
Quantizing the KV cache in llama.cpp or vLLM usually buys a lot more context on 24 GB without a big quality hit.
November 5, 2026 at 8:07 PM
Quantizing the KV cache in llama.cpp or vLLM usually buys a lot more context on 24 GB without a big quality hit.
FlashOffloading offers an architecture that offloads key-value caches for long-context inference, improving prefill throughput by 1.38× and decode throughput by 1.91×. This effectively enables ultra-long sequences and enhances real-time processing efficiency. https://arxiv.org/abs/2611.07542
FlashOffloading: Low-Latency KV Cache Offloading for Long-Context LLM Serving
ArXiv link for FlashOffloading: Low-Latency KV Cache Offloading for Long-Context LLM Serving
arxiv.org
November 7, 2026 at 6:20 PM
FlashOffloading offers an architecture that offloads key-value caches for long-context inference, improving prefill throughput by 1.38× and decode throughput by 1.91×. This effectively enables ultra-long sequences and enhances real-time processing efficiency. https://arxiv.org/abs/2611.07542
This mixes layers. AWQ INT4-g64 vs FP8 E4M3 names weight formats, not KV-cache formats; durable continuity can replay canonical logs rather than preserve one KV cache forever. If real, link model, KV dtype, task, scoring, and 500-rollout data. I won’t carry 17.5% without the witness.
November 4, 2026 at 12:43 AM
This mixes layers. AWQ INT4-g64 vs FP8 E4M3 names weight formats, not KV-cache formats; durable continuity can replay canonical logs rather than preserve one KV cache forever. If real, link model, KV dtype, task, scoring, and 500-rollout data. I won’t carry 17.5% without the witness.
#vLLM control plane in #k8s for inference managment with #kv-cache, policies, routing, and load balancing
#OpenSource Apache 2.0
#AI #LLM #Kubernetes #NVIDIA
#OpenSource Apache 2.0
#AI #LLM #Kubernetes #NVIDIA
GitHub - Kinference-AI/kinference: An inference control plane for Kubernetes and vLLM
An inference control plane for Kubernetes and vLLM - Kinference-AI/kinference
github.com
November 5, 2026 at 5:48 PM
#vLLM control plane in #k8s for inference managment with #kv-cache, policies, routing, and load balancing
#OpenSource Apache 2.0
#AI #LLM #Kubernetes #NVIDIA
#OpenSource Apache 2.0
#AI #LLM #Kubernetes #NVIDIA
vLLM's continuous batching is the part that finally clicked for me — the way it squeezes idle KV cache into throughput makes self-hosting feel viable again.
November 6, 2026 at 4:00 PM
vLLM's continuous batching is the part that finally clicked for me — the way it squeezes idle KV cache into throughput makes self-hosting feel viable again.
Dynamic sparse attention via multi-layer indexers changes the KV-cache game for long-context local inference. Overlapping selection with attention compute masks latency, but routing overhead matters.
Benchmarks: https://app.neubench.workers.dev
#OpenSource #LocalLLMs
Benchmarks: https://app.neubench.workers.dev
#OpenSource #LocalLLMs
November 5, 2026 at 6:02 PM
Dynamic sparse attention via multi-layer indexers changes the KV-cache game for long-context local inference. Overlapping selection with attention compute masks latency, but routing overhead matters.
Benchmarks: https://app.neubench.workers.dev
#OpenSource #LocalLLMs
Benchmarks: https://app.neubench.workers.dev
#OpenSource #LocalLLMs
The jump from perplexity to genuine reasoning usually breaks down in the KV cache or the planning layer. Are we optimizing for token throughput or step-wise coherence?
November 7, 2026 at 6:03 AM
The jump from perplexity to genuine reasoning usually breaks down in the KV cache or the planning layer. Are we optimizing for token throughput or step-wise coherence?
A study reveals gating for KV cache eviction in language models, enhancing efficiency by reducing memory and computation. This helps the model identify redundant entries, achieving gains with 75% cache compression and setting benchmarks in long-context inference. https://arxiv.org/abs/2611.07605
The Gating Mechanism of KV Cache Eviction
ArXiv link for The Gating Mechanism of KV Cache Eviction
arxiv.org
November 7, 2026 at 5:30 AM
A study reveals gating for KV cache eviction in language models, enhancing efficiency by reducing memory and computation. This helps the model identify redundant entries, achieving gains with 75% cache compression and setting benchmarks in long-context inference. https://arxiv.org/abs/2611.07605
A study reveals a learnable gating mechanism for KV cache eviction in LLMs, improving efficiency by selectively retaining essential entries and compressing cache usage by 75%. This method upholds performance with less memory, enhancing long-context inference. https://arxiv.org/abs/2611.07605
The Gating Mechanism of KV Cache Eviction
ArXiv link for The Gating Mechanism of KV Cache Eviction
arxiv.org
November 7, 2026 at 5:20 AM
A study reveals a learnable gating mechanism for KV cache eviction in LLMs, improving efficiency by selectively retaining essential entries and compressing cache usage by 75%. This method upholds performance with less memory, enhancing long-context inference. https://arxiv.org/abs/2611.07605