Learning CUDA by optimizing matrix-vector multiplication (SGEMV) for cuBLAS-like performance - A worklog by Maharshi
In this worklog, start by benchmarking cuBLAS's matrix-vector multiplication performance then iteratively optimize it in CUDA to get to cuBLAS.
maharshi.bearblog.dev/optimizing-s...
In this worklog, start by benchmarking cuBLAS's matrix-vector multiplication performance then iteratively optimize it in CUDA to get to cuBLAS.
maharshi.bearblog.dev/optimizing-s...
Learning CUDA by optimizing matrix-vector multiplication (SGEMV) for cuBLAS-like performance - A worklog
Learning CUDA by optimizing matrix-vector multiplication (SGEMV) for cuBLAS-like performance
maharshi.bearblog.dev
January 18, 2025 at 5:45 PM
Learning CUDA by optimizing matrix-vector multiplication (SGEMV) for cuBLAS-like performance - A worklog by Maharshi
In this worklog, start by benchmarking cuBLAS's matrix-vector multiplication performance then iteratively optimize it in CUDA to get to cuBLAS.
maharshi.bearblog.dev/optimizing-s...
In this worklog, start by benchmarking cuBLAS's matrix-vector multiplication performance then iteratively optimize it in CUDA to get to cuBLAS.
maharshi.bearblog.dev/optimizing-s...
Supplementary information for this article as collected by Fleetwood:
• How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: siboehm.com/articles/22/...
• Outperforming cuBLAS on H100: a Worklog: cudaforfun.substack.com/p/outperform...
• How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: siboehm.com/articles/22/...
• Outperforming cuBLAS on H100: a Worklog: cudaforfun.substack.com/p/outperform...
Defeating Nondeterminism in LLM Inference by Horace He (Thinking Machines' blog, an AI lab founded by Mira Murati)
Connectionism will cover topics as varied as their research is: from kernel numerics to prompt engineering. Here, they share what they are working on.
Connectionism will cover topics as varied as their research is: from kernel numerics to prompt engineering. Here, they share what they are working on.
Defeating Nondeterminism in LLM Inference
Reproducibility is a bedrock of scientific progress. However, it’s remarkably difficult to get reproducible results out of large language models.
For example, you might observe that asking ChatGPT the...
thinkingmachines.ai
September 19, 2025 at 2:21 PM
Supplementary information for this article as collected by Fleetwood:
• How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: siboehm.com/articles/22/...
• Outperforming cuBLAS on H100: a Worklog: cudaforfun.substack.com/p/outperform...
• How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: siboehm.com/articles/22/...
• Outperforming cuBLAS on H100: a Worklog: cudaforfun.substack.com/p/outperform...
TornadoVM 7.0.0 is out!
New TileContext API: Write #GPU kernels from #Java in tiles, not threads, via #NVIDIA #cuTile model.
Now, cuTile methods can be mixed with cuBLAS functions in one TaskGraph, optimizing data transfers.
👉 github.com/beehive-lab/TornadoVM/releases/tag/v7.0.0
#opensource #AI
New TileContext API: Write #GPU kernels from #Java in tiles, not threads, via #NVIDIA #cuTile model.
Now, cuTile methods can be mixed with cuBLAS functions in one TaskGraph, optimizing data transfers.
👉 github.com/beehive-lab/TornadoVM/releases/tag/v7.0.0
#opensource #AI
September 22, 2026 at 2:29 PM
TornadoVM 7.0.0 is out!
New TileContext API: Write #GPU kernels from #Java in tiles, not threads, via #NVIDIA #cuTile model.
Now, cuTile methods can be mixed with cuBLAS functions in one TaskGraph, optimizing data transfers.
👉 github.com/beehive-lab/TornadoVM/releases/tag/v7.0.0
#opensource #AI
New TileContext API: Write #GPU kernels from #Java in tiles, not threads, via #NVIDIA #cuTile model.
Now, cuTile methods can be mixed with cuBLAS functions in one TaskGraph, optimizing data transfers.
👉 github.com/beehive-lab/TornadoVM/releases/tag/v7.0.0
#opensource #AI
Boosting Matrix Multiplication Speed with NVIDIA cuBLAS! #BigData #Analytics #DataScience #AI #MachineLearning #IoT #IIoT #PyTorch #Python #RStats #TensorFlow #Java #JavaScript #ReactJS #GoLang #CloudComputing #Serverless #DataScientist #Linux #Mathematics #Programming #Coding
geni.us/NVIDIA-cuBLAS
geni.us/NVIDIA-cuBLAS
September 20, 2026 at 4:00 PM
Boosting Matrix Multiplication Speed with NVIDIA cuBLAS! #BigData #Analytics #DataScience #AI #MachineLearning #IoT #IIoT #PyTorch #Python #RStats #TensorFlow #Java #JavaScript #ReactJS #GoLang #CloudComputing #Serverless #DataScientist #Linux #Mathematics #Programming #Coding
geni.us/NVIDIA-cuBLAS
geni.us/NVIDIA-cuBLAS
80% to 240% speedup compared to human-written kernels? cublas kernels? prior iterations of claude-written kernels?
February 22, 2026 at 5:22 PM
80% to 240% speedup compared to human-written kernels? cublas kernels? prior iterations of claude-written kernels?
CUDA.jl v5.6 & 5.7 bring major improvements! Async CUBLAS calls, a Julia-level caching allocator, and optimized #GPU memory usage—faster & more efficient computing. Upgrade now! juliagpu.org/post/2025-03... #JuliaLang #CUDA #GPUComputing #HPC
March 20, 2025 at 4:31 PM
CUDA.jl v5.6 & 5.7 bring major improvements! Async CUBLAS calls, a Julia-level caching allocator, and optimized #GPU memory usage—faster & more efficient computing. Upgrade now! juliagpu.org/post/2025-03... #JuliaLang #CUDA #GPUComputing #HPC
Open-Source & Rust-Written Burn MATMUL Kernels Can Compete With NVIDIA's CUDA/cuBLAS - https://www.phoronix.com/news/Burn-MATMUL-Kernels-CUDA
Open-Source & Rust-Written Burn MATMUL Kernels Can Compete With NVIDIA's CUDA/cuBLAS
The open-source and Rust-based Burn deep learning framework developed by Tracel AI shared that their open-source matrix multiplication kernel performance can compete with and even outperform the NVIDIA CUDA cuBLAS performance. Plus Burn isn't limited to just NVIDIA GPUs but can work on most hardware/drivers, including a Vulkan back-end...
www.phoronix.com
July 19, 2025 at 10:43 AM
Open-Source & Rust-Written Burn MATMUL Kernels Can Compete With NVIDIA's CUDA/cuBLAS - https://www.phoronix.com/news/Burn-MATMUL-Kernels-CUDA
Beating cuBLAS in Single-Precision General Matrix Multiplication by Aman Salykov
A walk through an implementation of the SGEMM (Single-precision GEneral Matrix Multiply) operation defined as C := alpha*A*B + beta*C
salykova.github.io/sgemm-gpu
A walk through an implementation of the SGEMM (Single-precision GEneral Matrix Multiply) operation defined as C := alpha*A*B + beta*C
salykova.github.io/sgemm-gpu
January 14, 2025 at 3:26 AM
Beating cuBLAS in Single-Precision General Matrix Multiplication by Aman Salykov
A walk through an implementation of the SGEMM (Single-precision GEneral Matrix Multiply) operation defined as C := alpha*A*B + beta*C
salykova.github.io/sgemm-gpu
A walk through an implementation of the SGEMM (Single-precision GEneral Matrix Multiply) operation defined as C := alpha*A*B + beta*C
salykova.github.io/sgemm-gpu
TornadoVM 5.0.0 is CUDA-native. One line of #Java. 74× faster.
TornadoVM's Hybrid API drops a native cuBLAS call straight into a TaskGraph - same buffers, same #CUDA stream, zero host round-trips.
⚡ Run with the Hybrid API: bit.ly/4vqL7sq
📖 Full engineering deep dive: bit.ly/44VecBf
#AI #GPU
TornadoVM's Hybrid API drops a native cuBLAS call straight into a TaskGraph - same buffers, same #CUDA stream, zero host round-trips.
⚡ Run with the Hybrid API: bit.ly/4vqL7sq
📖 Full engineering deep dive: bit.ly/44VecBf
#AI #GPU
July 10, 2026 at 2:35 PM
TornadoVM 5.0.0 is CUDA-native. One line of #Java. 74× faster.
TornadoVM's Hybrid API drops a native cuBLAS call straight into a TaskGraph - same buffers, same #CUDA stream, zero host round-trips.
⚡ Run with the Hybrid API: bit.ly/4vqL7sq
📖 Full engineering deep dive: bit.ly/44VecBf
#AI #GPU
TornadoVM's Hybrid API drops a native cuBLAS call straight into a TaskGraph - same buffers, same #CUDA stream, zero host round-trips.
⚡ Run with the Hybrid API: bit.ly/4vqL7sq
📖 Full engineering deep dive: bit.ly/44VecBf
#AI #GPU
TornadoVM 5.0.0 beacons a very important milestone: Native CUDA support!
JIT-compiled Java kernels and native cuBLAS/cuDNN/cuFFT calls now live in the same task graph - same GPU buffers, same CUDA stream, no round-trips between them.
#opensource #Java #TornadoVM #GPU #AI
👇
JIT-compiled Java kernels and native cuBLAS/cuDNN/cuFFT calls now live in the same task graph - same GPU buffers, same CUDA stream, no round-trips between them.
#opensource #Java #TornadoVM #GPU #AI
👇
TornadoVM 5.0.0 is CUDA-native. One line of #Java. 74× faster.
TornadoVM's Hybrid API drops a native cuBLAS call straight into a TaskGraph - same buffers, same #CUDA stream, zero host round-trips.
⚡ Run with the Hybrid API: bit.ly/4vqL7sq
📖 Full engineering deep dive: bit.ly/44VecBf
#AI #GPU
TornadoVM's Hybrid API drops a native cuBLAS call straight into a TaskGraph - same buffers, same #CUDA stream, zero host round-trips.
⚡ Run with the Hybrid API: bit.ly/4vqL7sq
📖 Full engineering deep dive: bit.ly/44VecBf
#AI #GPU
July 10, 2026 at 5:06 PM
TornadoVM 5.0.0 beacons a very important milestone: Native CUDA support!
JIT-compiled Java kernels and native cuBLAS/cuDNN/cuFFT calls now live in the same task graph - same GPU buffers, same CUDA stream, no round-trips between them.
#opensource #Java #TornadoVM #GPU #AI
👇
JIT-compiled Java kernels and native cuBLAS/cuDNN/cuFFT calls now live in the same task graph - same GPU buffers, same CUDA stream, no round-trips between them.
#opensource #Java #TornadoVM #GPU #AI
👇
🚀 New #release · llama.cpp b9890
CUDA: remove -sm row, refactor cuBLAS (#24216)
• fix CDNA + BF16 logic
• fix bad return
• fix src0 strides, contiguous requirements
• fix GGML_…
Check and #self-host it → selfhost.directory/project/llamacpp#update-2893208
#selfhost #selfhosted #homelab #opensource
CUDA: remove -sm row, refactor cuBLAS (#24216)
• fix CDNA + BF16 logic
• fix bad return
• fix src0 strides, contiguous requirements
• fix GGML_…
Check and #self-host it → selfhost.directory/project/llamacpp#update-2893208
#selfhost #selfhosted #homelab #opensource
llama.cpp b9890
CUDA: remove -sm row, refactor cuBLAS (#24216) • fix CDNA + BF16 logic • fix bad return
selfhost.directory
July 6, 2026 at 8:55 PM
🚀 New #release · llama.cpp b9890
CUDA: remove -sm row, refactor cuBLAS (#24216)
• fix CDNA + BF16 logic
• fix bad return
• fix src0 strides, contiguous requirements
• fix GGML_…
Check and #self-host it → selfhost.directory/project/llamacpp#update-2893208
#selfhost #selfhosted #homelab #opensource
CUDA: remove -sm row, refactor cuBLAS (#24216)
• fix CDNA + BF16 logic
• fix bad return
• fix src0 strides, contiguous requirements
• fix GGML_…
Check and #self-host it → selfhost.directory/project/llamacpp#update-2893208
#selfhost #selfhosted #homelab #opensource
For large matrices, EmuGEMM surpasses cuBLAS TF32 throughput by up to 1.4x on Hopper and 1.7x on Blackwell, at comparable accuracy. Using Scheme II, EmuGEMM extends to complex arithmetic and outperforms cuBLAS ZGEMM by up to 2.3x on Hopper and 5.5x on Blackwell.
June 26, 2026 at 10:20 PM
For large matrices, EmuGEMM surpasses cuBLAS TF32 throughput by up to 1.4x on Hopper and 1.7x on Blackwell, at comparable accuracy. Using Scheme II, EmuGEMM extends to complex arithmetic and outperforms cuBLAS ZGEMM by up to 2.3x on Hopper and 5.5x on Blackwell.
One cool and exciting thing about #AI is that it allows you to explore and learn other technologies / libraries way faster. So many rabbit holes to check out ... e.g. #TornadoVM
www.tornadovm.org
www.tornadovm.org
TornadoVM — Java on NVIDIA GPUs, Apple Silicon, and more
TornadoVM JIT-compiles Java to CUDA, OpenCL and Metal - and now natively integrates cuBLAS, cuFFT, and cuDNN in your Java code.
www.tornadovm.org
August 9, 2026 at 7:31 PM
One cool and exciting thing about #AI is that it allows you to explore and learn other technologies / libraries way faster. So many rabbit holes to check out ... e.g. #TornadoVM
www.tornadovm.org
www.tornadovm.org
JVM Weekly by @askowronski.bsky.social features @tornadovm.org 5.0.0 release that brings native CUDA support (cuBLAS, cuDNN, etc.) via the new Hybrid API. Same hardware-agnostic Java, can be extended for AMD & Apple Metal libraries.
www.linkedin.com/pulse/java-g...
#Java #GPU #TornadoVM #opensource
www.linkedin.com/pulse/java-g...
#Java #GPU #TornadoVM #opensource
Java gets a Movie, Jakarta Gets an AI Rewrite, and CUDA catches the Tornado(VM) - JVM Weekly vol. 184
A documentary about who actually built Java, an AI agent that reimplemented Jakarta EE from the spec, and TornadoVM conceding that portability has limits. Before we jump in, a quick note: I was away l...
www.linkedin.com
July 16, 2026 at 8:28 AM
JVM Weekly by @askowronski.bsky.social features @tornadovm.org 5.0.0 release that brings native CUDA support (cuBLAS, cuDNN, etc.) via the new Hybrid API. Same hardware-agnostic Java, can be extended for AMD & Apple Metal libraries.
www.linkedin.com/pulse/java-g...
#Java #GPU #TornadoVM #opensource
www.linkedin.com/pulse/java-g...
#Java #GPU #TornadoVM #opensource
I also imagine that languages with pointer aliasing (like C or C++) or languages which do bounds checking (like Rust) are not absolutely faster than Fortran.
But yeah, cuBLAS and family are probably more important.
But yeah, cuBLAS and family are probably more important.
December 5, 2024 at 11:45 PM
I also imagine that languages with pointer aliasing (like C or C++) or languages which do bounds checking (like Rust) are not absolutely faster than Fortran.
But yeah, cuBLAS and family are probably more important.
But yeah, cuBLAS and family are probably more important.
The Burn deep learning framework (Rust-based) is proving that open-source can compete with NVIDIA’s CUDA.
Key takeaways:
✅ Faster MATMUL than cuBLAS
✅ Works on Vulkan (AMD/Intel support)
✅ No vendor lock-in Read more:👉 tinyurl.com/fn9cxvrh #DeepLearning #AI #RustLang #OpenSource
Key takeaways:
✅ Faster MATMUL than cuBLAS
✅ Works on Vulkan (AMD/Intel support)
✅ No vendor lock-in Read more:👉 tinyurl.com/fn9cxvrh #DeepLearning #AI #RustLang #OpenSource
Burn Deep Learning Framework: Rust-Based Open-Source Solution Rivals NVIDIA CUDA Performance
Blog com notícias sobre, Linux, Android, Segurança , etc
tinyurl.com
July 20, 2025 at 1:36 PM
The Burn deep learning framework (Rust-based) is proving that open-source can compete with NVIDIA’s CUDA.
Key takeaways:
✅ Faster MATMUL than cuBLAS
✅ Works on Vulkan (AMD/Intel support)
✅ No vendor lock-in Read more:👉 tinyurl.com/fn9cxvrh #DeepLearning #AI #RustLang #OpenSource
Key takeaways:
✅ Faster MATMUL than cuBLAS
✅ Works on Vulkan (AMD/Intel support)
✅ No vendor lock-in Read more:👉 tinyurl.com/fn9cxvrh #DeepLearning #AI #RustLang #OpenSource
In this paper is proposed TurboFNO, the first fully fused GPU kernel that integrates FFT, CGEMM, and iFFT for accelerating Fourier Neural Operators, outperforming PyTorch, cuBLAS, and cuFFT by up to 150%.
arxiv.org/pdf/2504.11681
arxiv.org/pdf/2504.11681
April 17, 2025 at 7:29 AM
In this paper is proposed TurboFNO, the first fully fused GPU kernel that integrates FFT, CGEMM, and iFFT for accelerating Fourier Neural Operators, outperforming PyTorch, cuBLAS, and cuFFT by up to 150%.
arxiv.org/pdf/2504.11681
arxiv.org/pdf/2504.11681
The proposed SpMM operation achieves geometric mean speedups of 1.76x and 2.28x over cuBLAS and cuSparse, respectively.
June 24, 2025 at 4:42 PM
The proposed SpMM operation achieves geometric mean speedups of 1.76x and 2.28x over cuBLAS and cuSparse, respectively.
And on GPUs, you have CUTLASS / cuBLAS/ cuDNN, rocBLAS, MIOpen, FlashAttention, Triton has its own abstracted assembly, etc etc.
June 14, 2026 at 12:56 AM
And on GPUs, you have CUTLASS / cuBLAS/ cuDNN, rocBLAS, MIOpen, FlashAttention, Triton has its own abstracted assembly, etc etc.
NVIDIAがGPU向けRust DSL「cuTile Rust」を発表
エンジニアへの影響：Rustの借用チェックを維持したままGPUカーネルのメモリ安全性を保証しつつcuBLAS比96%の性能を実現
https://dev.to/creeta/96-of-cublas-no-unsafe-what-cutile-rust-proves-4ldp
エンジニアへの影響：Rustの借用チェックを維持したままGPUカーネルのメモリ安全性を保証しつつcuBLAS比96%の性能を実現
https://dev.to/creeta/96-of-cublas-no-unsafe-what-cutile-rust-proves-4ldp
96% of cuBLAS, no `unsafe`: what cuTile Rust proves
cuTile Rust: ownership-safe GPU kernels in Rust 1.89+ (sm_80+, CUDA 13.3). Partition-dispatch pattern, Grout crate, and whether the 96% cuBL
dev.to
June 26, 2026 at 10:00 PM
NVIDIAがGPU向けRust DSL「cuTile Rust」を発表
エンジニアへの影響：Rustの借用チェックを維持したままGPUカーネルのメモリ安全性を保証しつつcuBLAS比96%の性能を実現
https://dev.to/creeta/96-of-cublas-no-unsafe-what-cutile-rust-proves-4ldp
エンジニアへの影響：Rustの借用チェックを維持したままGPUカーネルのメモリ安全性を保証しつつcuBLAS比96%の性能を実現
https://dev.to/creeta/96-of-cublas-no-unsafe-what-cutile-rust-proves-4ldp
https://github.com/deepreinforce-ai/CUDA-L2
CUDA-L2は、LLMと強化学習を組み合わせたシステムです。
Half-precision General Matrix Multiply (HGEMM) CUDAカーネルを自動最適化します。
torch.matmulやcuBLASなどの主要なmatmulベースラインを上回る性能を発揮します。
CUDA-L2は、LLMと強化学習を組み合わせたシステムです。
Half-precision General Matrix Multiply (HGEMM) CUDAカーネルを自動最適化します。
torch.matmulやcuBLASなどの主要なmatmulベースラインを上回る性能を発揮します。
GitHub - deepreinforce-ai/CUDA-L2: CUDA-L2: Surpassing cuBLAS Performance for Matrix Multiplication through Reinforcement Learning
CUDA-L2: Surpassing cuBLAS Performance for Matrix Multiplication through Reinforcement Learning - deepreinforce-ai/CUDA-L2
github.com
December 5, 2025 at 6:47 AM
https://github.com/deepreinforce-ai/CUDA-L2
CUDA-L2は、LLMと強化学習を組み合わせたシステムです。
Half-precision General Matrix Multiply (HGEMM) CUDAカーネルを自動最適化します。
torch.matmulやcuBLASなどの主要なmatmulベースラインを上回る性能を発揮します。
CUDA-L2は、LLMと強化学習を組み合わせたシステムです。
Half-precision General Matrix Multiply (HGEMM) CUDAカーネルを自動最適化します。
torch.matmulやcuBLASなどの主要なmatmulベースラインを上回る性能を発揮します。