The rapid progress of software engineering agents drives interest in applying them to open-ended problems such as GPU kernel optimization. However, we find that existing coding agents converge to local optima on real-world, expert-written kernels (e.g., FlashInfer). We introduce Evolving Agent Team…

The rapid progress of software engineering agents drives interest in applying them to open-ended problems such as GPU kernel optimization. However, we find that existing coding agents converge to local optima on real-world, expert-written kernels (e.g., FlashInfer). We introduce Evolving Agent Team…

Source: HackerNews AI

Winner 🏆 (Agent-only) MLSys 2026 - FlashInfer AI Kernel Generation Contest for the DeepSeek Sparse Attention (DSA) track with an average speedup of 34.93x - Dogacel/auto-gpu-kernel

What Happened Across the open-source LLM ecosystem this week there were two coordinated trends: (1) infrastructure-level releases and fixes in the ggml/llama.cpp ecosystem that broaden platform and backend support, add…

We study whether general-purpose code agents can produce state-of-the-art GPU kernels without any manually written CUDA code. We investigate this question using representative workloads from FlashInfer-Bench, focusing on the Fused MoE, DSA TopK Indexer, and DSA Sparse Attention, and evaluate all gen

Notable changes Prefill chunking for VLMs. What's Changed Fixing Qwen 2.5 VL (32B). by @Narsil in #3157 Fixing tokenization like https://github.com/huggingface/text-embeddin… by @Narsil in #3156...

**TL;DR:** In collaboration with the open-source community, vLLM + NVIDIA has achieved significant performance milestones on the `gpt-oss-120b` model running on NVIDIA’s Blackwell GPUs. Through deep integration with FlashInfer, novel kernel fusions via `torch.compile`, and various inference runtime features, we have set a new record for the model’s performance Pareto frontier —simultaneously optimizing for maximum throughput (+38%) and best interactivity (+13%). This post details the engineering journey, technical breakthroughs, and instructions to reproduce the results. Continuous benchmarks are also available on **SemiAnalysis Inference MAX and vLLM Recipes**. ## Table of Contents * Introduction * FlashInfer + torch.compile * Runtime Improvements * Deployment Recipes * Results * Next Steps * Acknowledgements * * * ## Introduction Optimizing for a single metric—like maximum throughput or single-batch latency—is often insufficient for real-world deployments. Different use cases require different latency constraints and request concurrency. As a result, the real challenge lies in optimizing the **Pareto frontier** : the curve that represents the best possible trade-off between **Tokens Per Second (TPS) per GPU** (TCO, total cost of ownership) and **TPS per User** (interactivity). Pushing this curve upwards and to the right means delivering faster generation for individual users while allowing more users to share the hardware. SemiAnalysis InferenceMAX has identified this critical need to measure, report and improve performance data for such LLM inference workloads on modern GPUs. One of the key use-cases is serving OpenAI’s `gpt-oss-120b` model, a natively 4-bit quantized (MXFP4) Mixture-of-Experts (MoE) LLM. It has achieved SoTA model accuracy for its size along with strong agentic capabilities. At the recent SemiAnalysis InferenceMAX showcase, vLLM demonstrated its capability to handle this workload efficiently on NVIDIA’s latest Blackwell (B200/GB200) architecture. The heart of the optimizations is hardware-software co-design. The NVIDIA B200/GB200 GPUs introduce powerful features like native FP4 TensorCores and 192GB HBM per GPU, which are critical for serving large MoE models like `gpt-oss`. To leverage this hardware fully, vLLM and NVIDIA teams have integrated with **FlashInfer** and adopted a rigorous optimization strategy focusing on kernel fusion, communication overhead reduction, and host-device overlapping. ## FlashInfer Integration and torch.compile based fusion To maximize the utilization of Blackwell’s tensor cores, vLLM leverages **FlashInfer** as its primary kernel backend for attention, MoE, and other compute-intensive and fused operations. **1. Key Compute Kernel Integration** : * **MoE Backends:** We enabled both `trtllm-gen` (PR23819) and `cutlass` (PR23696) backends for MoE operations with FlashInfer. This allows vLLM to select the most performant kernel for expert routing and computation. In addition to providing the best-performing kernels for LLMs, FlashInfer also includes jit-in-time compilation, auto-tuning, and kernel caching, which greatly improves the user experience for any developer with high-performance kernel needs. * **FP8 KV-Cache:** Storing kv-cache in FP8 precision allows the engine to serve more concurrent requests with the same kv-cache budget. Moreover, carrying out some of the attention operations in FP8 precision also reduces the compute/memory complexity of the attention operation. To achieve the best performance for this use case, vLLM has integrated FlashInfer’s optimized attention kernels in PR25674. **2. Graph Fusions via torch.compile** A significant portion of our optimization effort focused on kernel fusion to reduce memory access and kernel launch overhead. Instead of hard-coded fusion optimizations, vLLM has built an extensive infrastructure based on `torch.compile` to conduct kernel fusion automatically. This approach not only improves performance, but significantly reduces the effort to enable, generalize, and maintain such improvements. * **AR + Norm Fusion:** We implemented the fusion of AllReduce (AR) and RMSNorm operations. This is particularly important for tensor-parallel (TP) deployments, where communication overhead can become a bottleneck, details please see PR20691. * **Pad + Quant & Finalize + Slice:** We are actively rolling out the fusion passes, PR30647 for padding/quantization and finalize/slice operations to further streamline the MoE execution path, with an expected 6% performance gain. As we identify and develop new fused operations, the team will continue to deliver automatic performance gains via this infrastructure. ## Runtime Improvements On next-generation hardware like Blackwell, the GPU is so fast that the CPU (host) often becomes the bottleneck, struggling to dispatch kernels quickly enough to keep the GPU busy. In addition, `prepare\_batch`, request scheduling and sampling logic also require heavy CPU side logic. This “host overhead” manifests as gaps between kernel executions, degrading performance and overall GPU utilization. To address this, we implemented both **Async Scheduling** and **Stream Interval** to vLLM that effectively eliminate host-side overhead. Async Scheduling: * **Mechanism:** This scheduler decouples the CPU’s request scheduling from the GPU’s execution. By allowing the CPU to prepare the next batch of requests while the GPU is still processing the current batch, we effectively hide the host overhead. * **Impact:** This optimization is crucial for the `gpt-oss` model, particularly in both high-throughput and min-latency scenarios. On more capable GPUs (H200s, B200s, GB200s), you can expect around a 10% performance gain. * **Configuration:** This has been turned on by default in recent vLLM releases. Stream Interval: * **Mechanism:** This feature reduces the granularity of network responses by buffering generated tokens before sending them to the client. Instead of triggering a network call for every single token, the engine waits until a specified buffer size (the “interval”) is reached. Crucially, the implementation preserves responsiveness by ensuring the **first token is always sent immediately** (keeping Time-To-First-Token low), while subsequent tokens are batched. * **Impact:** By reducing the frequency of HTTP/gRPC response dispatching, this significantly lowers the CPU overhead associated with network I/O and serialization. In high-concurrency benchmarks (e.g., `gpt-oss-20b` with 1024 concurrent requests), this optimization relieved output queue bottlenecks, resulting in a **57% end-to-end performance gain** and improved Time Per Output Token (TPOT). * **Configuration:** Users can configure this behavior using the `--stream-interval <num_tokens>` argument. The default value is `1` (standard streaming), but increasing this value (e.g., to `10`) is highly effective for reducing host overhead in high-throughput deployments. ## Deployment Recipes Most of the optimizations are already applied by default on the latest vLLM release. In addition, to reproduce the optimized performance for `gpt-oss` on Blackwell GPUs (B200/GB200), we recommend the following configurations in your vLLM deployment recipes. They can also be found under vLLM Recipes page. **Recommended Configuration Flags:** * **Graph Capture:** * `--cuda-graph-capture-size 2048` * **Scheduling:** * `--api-server-count 20` or `--stream-interval 20`: This helps decouple the HTTP API server overhead from the inference engine, stabilizing performance at high concurrency. * **MoE Backend:** * Explicitly enable the optimized Cutlass backend for FP8/FP4 MoE to ensure maximum throughput: `VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8=1`. ## Results The combined effect of the optimizations has resulted in a significant uptick of performance since the launch of InferenceMax. Notably, a 38% performance increase at max-throughput, and 13% performance increase at min-latency Such improvements are not just for a single use-case, but rather **across the entire Pareto curve to benefit the vLLM community at large**. ## Next steps Our work on `gpt-oss` is ongoing. Here is a look at the active engineering tracks to further push the Pareto frontier. The list can also be found in Issue 30758. ### Disaggregation By separating the Prefill stage and the Decode stage on to different GPUs, we can potentially achieve better throughput per GPU. We are currently experimenting with this setup and find the correct configs that achieve better performance. ### Data+Expert parallel performance Our projection shows that using DEP2 (Attention DP + MoE EP on 2 GPUs) can potentially achieve higher throughput per GPU compared to TP1 and TP2 at the same latency (TPS/user). However, currently the DEP2 performance is worse than TP1/TP2 mainly due to the MoE kernel selection issue. We are actively working on this to resolve it. ### Minimum latency performance We have identified a few performance optimization opportunities for min-latency scenario, or TP8 concurrency 8 more specifically: * RoPE+Q+Cache fusion: Kernel is available in FlashInfer. Integration in vLLM is in progress. * The router gemm and fc_qkv/fc_o_proj gemms: we can use specialized tiny gemm kernels with better performance and PDL support. ## Acknowledgements We would like to give thanks to the many talented people in the vLLM community who worked together as a part of this effort: * Red Hat: Michael Goin, Alexander Matveev, Lucas Wilkinson, Luka Govedič, Wentao Ye, Ilia Markov, Matt Bonanni, Varun Sundar Rabindranath, Bill Nell, Tyler Michael Smith, Robert Shaw * NVIDIA: Po-Han Huang, Pavani Majety, Shu Wang, Elvis Chen, Zihao Ye, Duncan Moss, Kaixi Hou, Siyuan Fu, Benjamin Chislett, Xin Li, Vadim Gimpelson, Minseok Lee, Amir Samani, Elfie Guo, Lee Nau, Kushan Ahmadian, Grace Ho, Pen Chun Li * vLLM: Chen Zhang, Yongye Zhu, Bowen Wang, Kaichao You, Simon Mo, Woosuk Kwon, Zhuohan Li * Meta: Yang Chen, Xiaozhu Meng, Boyuan Feng, Lu Fang

This section highlights vAttention's ability to add dynamic memory allocation support to unmodified FlashAttention and FlashInfer prefill kernels, simplifying development while boosting performance.

Introduction: 传统三维重建长期依赖LIDAR硬件或迭代优化算法，前者成本高昂，后者计算繁重、难以实时。Robbyant团队发布的LingBot‑Map，作为一个纯前馈式3D基础模型，首次在单一框架内统一了坐标 grounding、密集几何线索与长程漂移校正，实现了无需LIDAR、无需迭代优化、无需后处理的流式3D重建。该模型以约20 FPS的吞吐量处理518×378分辨率输入，稳定支持超过10,000帧的长序列，性能超越现有流式方法和多种离线方法。 Learning Objectives: Objective 1: 理解LingBot‑Map的Geometric Context Transformer（GCT）架构原理及其三大上下文机制（anchor context、pose‑reference window、trajectory memory） Objective 2: 掌握LingBot‑Map在Linux环境下的完整部署流程，包括conda环境配置、PyTorch与FlashInfer安装、模型权重下载 Objective 3: 学会使用LingBot‑Map进行实时流式推理，涵盖图像文件夹输入、视频输入、关键帧间隔调优、窗口推理及天空遮罩等高级功能 You Should Know: 1. 核心架构：Geometric Context Transformer（GCT）与三大记忆机制 流式3D重建的本质问题是“记忆”——保留什么、以什么形式保留。LingBot‑Map通过Geometric Context Attention（GCA）给出了一个端到端学习的答案。GCA维护三种互补的上下文： Anchor Context（锚定上下文）：负责坐标与尺度 grounding，为整个场景提供稳定的空间参考系。 Pose‑Reference Window（位姿参考窗口）：维护局部密集几何信息，确保短程几何一致性。 Trajectory Memory（轨迹记忆）：将完整历史压缩为紧凑的每帧token，使每帧内存和计算量在10,000+帧序列上近乎恒定。 这一设计的巧妙之处在于：它将SLAM（同步定位与地图构建）的经典理念融入纯前馈Transformer架构，通过分页KV缓存注意力（paged KV cache attention）实现稳定高效的流式推理。与依赖束调整（bundle adjustment）等迭代优化的传统方法不同，LingBot‑Map一次性前馈完成全部预测，无需任何后处理清理步骤。 2. 环境准备与安装（Linux / Windows WSL2） LingBot‑Map官方推荐在Linux环境（或Windows WSL2）下使用CUDA 12.8 + PyTorch 2.9.1部署。 Step‑by‑step 安装指南： 1. 创建conda环境（Python 3.10） conda create -1 lingbot-map python=3.10 -y&hellip;

A team of University of Washington and NVIDIA researchers developed a system that can help make large language models (LLMs) faster and more adaptable. The researchers introduced FlashInfer, a versatile LLM inference kernel library that is open source as well as highly optimized for new techniques including key-value cache reuse…