I'm fascinated by the creative use of SIMD instructions. When you first learn about SIMD, it is clear that doing more multiplications in a single instruction is useful for speeding up matrix multi ...

On the JVM, optimizing a hot kernel is not only about writing faster code: it is also about understanding how much the result depends on the machine code HotSpot derives from the scalar loop. Using…

Wow, thanks a lot. Your code is now on par with the C++ example. In my case the mean time is `0.07355s ± 0.00054s`.

Bruecki: > The numba asm partially unrolls the loop. Yeah, that looks like the core of the difference. If you isolate just the summing loop, the optimized LLVM IR is one-`double`-at-a-time on the default target: https://rust.godbolt.org/z/E6j64acnz You can get much shorter ASM if you target AVX2, but it's still one-at-a-time: https://rust.godbolt.org/z/5Yzbazo8n. I don't know if that's materially faster. Of course, on nightly you can also manually vectorize it, if you're ok with getting a different answer: https://rust.godbolt.org/z/7oWMeMPeG

xoredarray: > `println!("{:8.4},{}", sum, t2.as_secs_f64());` Try changing this to println!("{:8.4},{}", {sum}, t2.as_secs_f64()); Or extract the loop out into a separate function. Taking the address of something -- like println does -- makes it harder for LLVM to optimize. (I don't know if that's the problem here, but we've seen it be a problem before. Benchmarking properly is _hard_ , and I'm always skeptical of anyone doing it by hand with `Instant` instead of something using statistical methods.)

Hi! I tried both your suggestions but it didn't improve anything at all. Anyway, thank you for your help. I agree that `-ffast-math` is dangerous to use and without it the C++ version loses its competitive power to Rust.

The numba asm partially unrolls the loop. I am unsure why LLVM does that. Maybe numba emmits some IR for loops that does something like `chunks_exact`? .LBB0_5: vmovsd (%rdx), %xmm3 vcmpltsd %xmm3, %xmm1, %xmm4 vblendvpd %xmm4, %xmm3, %xmm2, %xmm3 vaddsd %xmm3, %xmm0, %xmm0 vmovsd (%r8,%rdx), %xmm3 vcmpltsd %xmm3, %xmm1, %xmm4 vblendvpd %xmm4, %xmm3, %xmm2, %xmm3 vaddsd %xmm3, %xmm0, %xmm0 vmovsd (%rdx,%r8,2), %xmm3 vcmpltsd %xmm3, %xmm1, %xmm4 vblendvpd %xmm4, %xmm3, %xmm2, %xmm3 vaddsd %xmm3, %xmm0, %xmm0 vmovsd (%rbx,%rdx), %xmm3 addq $4, %rsi vcmpltsd %xmm3, %xmm1, %xmm4 vblendvpd %xmm4, %xmm3, %xmm2, %xmm3 vaddsd %xmm3, %xmm0, %xmm0 addq %r14, %rdx cmpq %rsi, %r11 jne .LBB0_5 while the rust versions does not. Here is the LLVM IR of numba Summary (click for more details) I do not properly understand LLVM IR but `unroll_iter` smells a bit like manual unrolling.

thanks for the suggestion. TIL that -Ofast enables `-ffast-math` flag. I removed `-ffast-math` and set -O3: g++ -Wall -Wextra -O3 -march=native -mtune=native -std=c++23 select_elements.cpp the elapsed time is `0.58802 s ± 0.01281 s`, so it is slower than Rust version now. Interesting. But I still wonder why is numba then produces quite fast code (even if I explicitly turn off fast math)? In [3]: @jit(fastmath=False) ...: def sum_selected(v: np.ndarray): ...: res_sum = np.float64(0.0) ...: for i in v: ...: if i > 0.5: ...: res_sum += i ...: return res_sum ...: In [4]: v = np.random.rand(100000000) In [5]: %timeit sum_selected(v) 163 ms ± 888 μs per loop (mean ± std. dev. of 7 runs, 1 loop each)

Hi, everyone! Before you close the tab I can tell you that I built Rust code with `--release` option What I am trying here is to achieve at least the same order of magnitude with C++ in this simple benchmark. Here are the details: the idea of this benchmark is to sum all elements in a vector that are greater than 0.5. Here is the reference code using NumPy: In [1]: import numpy as np In [2]: v = np.random.rand(100000000) In [3]: %timeit np.sum(v[v>0.5]) 907 ms ± 1.39 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) If we use jit from numba, the time can be significantly reduced: In [4]: from numba import jit In [5]: @jit ...: def sum_selected(v: np.ndarray): ...: res_sum = np.float64(0.0) ...: for i in v: ...: if i > 0.5: ...: res_sum += i ...: return res_sum ...: In [6]: %timeit sum_selected(v) 161 ms ± 2.53 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) Now, here is the Rust version using iterators: use rand::prelude::*; use std::time::Instant; fn main() { let mut rng = rand::rng(); let arr: Vec<f64> = Vec::from_iter((0..100000000).map(|_| rng.random::<f64>())); let t1 = Instant::now(); let sum: f64 = arr .iter() .filter(|&&x| x > 0.5f64) .sum(); let t2 = t1.elapsed(); println!("{:8.4},{}", sum, t2.as_secs_f64()); } The mean elapsed time is `0.22727 s ± 0.00361 s`. As you can see it is even slower than Python with jit. I also tried looping over vector but got almost no difference compared to the iterator version - `0.22353 s ± 0.00308 s`. use rand::prelude::*; use std::time::Instant; fn main() { let mut rng = rand::rng(); let arr: Vec<f64> = Vec::from_iter((0..100000000).map(|_| rng.random::<f64>())); let mut sum: f64 = 0.0; let t1 = Instant::now(); for el in arr.iter() { if *el > 0.5f64 { sum += *el; } } let t2 = t1.elapsed(); println!("{:8.4},{}", sum, t2.as_secs_f64()); } Both examples were built with the following options: RUSTFLAGS="-C target-cpu=native -C target-feature=+avx2" cargo run --release In emitted assembly I see that the code was vectorized but still it's slower than python with jit. This C++ code is better by one order of magnitude than Rust version - `0.07285 s ± 0.00139 s` (I also benchmarked Fortran with similar to C++ results, but I will omit the Fortran code). #include <chrono> #include <climits> #include <cstdlib> #include <ctime> #include <iostream> #include <stdfloat> #include <vector> #define N 100000000 int main() { std::srand(std::time(NULL)); std::vector<std::float64_t> arr; for (size_t i = 0; i < N; ++i) { arr.push_back(static_cast<std::float64_t>(std::rand())/RAND_MAX); } std::chrono::steady_clock::time_point t1 = std::chrono::steady_clock::now(); std::float64_t sum_res = 0.0f64; for (auto el: arr) { if (el > 0.5f64) sum_res += el; } std::chrono::steady_clock::time_point t2 = std::chrono::steady_clock::now(); auto ellapsed = std::chrono::duration_cast<std::chrono::microseconds>(t2-t1).count(); std::cout << sum_res << "," << static_cast<std::float64_t>(ellapsed)/1000000 << std::endl; } Here is how I compiled the C++ code: g++ -Wall -Wextra -Ofast -march=native -mtune=native -ffast-math -std=c++23 select_elements.cpp Why is Rust variant slower than C++? Do you have any ideas how can I speed up the code? I am not an expert in Rust so maybe I am doing something wrong or miss important compiler options. I am aware there is `simd` and `intrinsics` modules but they require rust nightly and frankly speaking offer me to do vectorization manually in contrast to C++ where the work is delegated to the compiler. Here is the versions of the software I used: gcc version 14.2.1 20250207 rust 1.85.0 python-numba 0.61.0 python-numpy 2.2.3

Anything that short-circuits. If we know that closures are pure than we can call them more often than promised by the API because we'll know that the distinction is not observable. Then we can for example fetch them in chunks of 4, let autovectorization do its job and recover the result as-if it short-circuited by examining the lanes more closely after dropping out of the vectorized loop. Among the simplest cases is `iter.all(f)` could put items in `[T; 4]`, call `f` 4 times, put the results in a `[bool as u32; 4]` and check for `== [1u32; 4]` or whatever pleases the optimizer. This could also be extended to other final methods like `cmp`, `eq` and also adapters `map`, `filter`, `take_while`, .... Pure map and filter closures aren't uncommon.

In part 1 of the "SIMD in zlib-rs" series, we've seen that, with a bit of nudging, autovectorization can produce optimal code for some problems. But that does not always work: with SIMD clever pr ...

In part 1 of the "SIMD in zlib-rs" series, we've seen that, with a bit of nudging, autovectorization can produce optimal code for some problems.

Indexing by other integer types would be quite convenient, and eliminate a lot of `as usize` boilerplate. But that doesn't cover cases that actually need truncation. I rarely want wrapping truncation specifically, but I often need truncation without additional overhead of `try_from().unwrap()`. That branch can be prohibitively expensive due to preventing autovectorization and other code motion. `u8::try_from(u16 >> 8)` may be free, but `T::try_from(sum / len)` may not. Checking when it optimises out is an overhead, both cognitive and taking actual time. Having said that, having truncation with a debug-only overflow check would be ideal, similar to arithmetic overflow checks.

the8472: > how much of a perf difference are you seeing Lack of inlining in the wrong loops typically results in the complete loss of some major loop optimizations. These include... * Autovectorization * For inner loops that fit well in the L1d cache, the slowdown ranges from 2x to 64x depending on the target CPU and data type. For my own work, I'm typically manipulating f32s on x86 without AVX-512, so we're in the 4-8x range. * Unrolling * This typically results in under-utilization of hardware ILP and extra slowdown caused by latency-bound dependency chains. Depending on how much the hardware tolerates the later (x86 CPUs tend to be quite tolerant with respect to other arches like GPUs) we're typically talking about an extra 2-6x slowdown in compute-bound loops. * It should be noted that compiler optimizers are not super-skilled at this optimization and I often need to give them a hand anyway. But my favorite manual unrolling hack, which avoids duplicating compute code or tweaking loop bounds (which is bad for maintainability), also relies on aggressive inlining to work... * Bound check elision * This optimization is even more brittle than unrolling, so whenever I can use iterators instead, I do so. But some loops are just not easy to rewrite using available iterators, and then the choice becomes rolling my own iterator (yuck, lots of unsafe) vs relying on the compiler to elide the bound checks, so in that case I appreciate when the bound check elision works. * Perf impact is a little nuanced: bound checks themselves are not that expensive (though a typical CPU can only do one every 1-2cy, so if you need to access many different arrays it can easily cause a 2-6x slowdown wrt ALU-bound code which can do multiple ops per cycle). But loss of bound checks elision typically also results into loss of vectorization and unrolling, and that's way more costly. * And we can also lose some other less critical but nice-to-have optimizations, like this one whose name I don't know which does the ASM equivalent of turning `for (int i = 0; i < N; ++i) { /* use arr[i] */ }` into `for (T* elem = &arr[0]; elem < &arr[arr.len()]; ++elem) { /* use elem */ }`, and thus reduces the register pressure and AGU workload linked to array indexing a little. * I would also put in this "less bad, still bad" bucket the overhead of function calls themselves, or more precisely the overhead of spilling and reloading registers on every function calls, losing SRoA for iterator structs, etc (I have yet to see code that is bottlenecked by `call`/`ret` themselves). That will still cost a ~factors slowdown, but it's not in the same badness range as losing more important optimizations like vectorization and unrolling. It is, however, highly correlated with losing these other optimizations. Because these losses multiply each other, the total loss of compute throughput in the affected loop can easily go in the ~100x range. But depending on how performance-critical this particular compute loop is to the overall application, the application-wide impact will vary. HPC computations are typically completely bottlenecked by a few tight compute loops (if you managed to avoid being IO-bound), so the impact on these will be huge. But for less compute-bound activities like O(N) data crunching or web servers, we would probably go into more reasonable territory, like maybe a 1.1-3x slowdown or something. the8472: > how critical is it to your application I get these issues all the time, in many different codebases, so maybe the easiest answer is to rephrase this as "how critical are my applications to the broader ecosystem" Right now, I have not convinced #work to embrace Rust for Serious Big Projects, so I mostly use it for smaller utilities where wasting a bit of compute time is not a huge deal. What I did manage, however, is to integrate a Rust chapter in our high-performance numerical computing course, which attract a large audience (500 people registered to the last "multilingual" course in July, 200 to a smaller-audience Rust and online-only spinoff next week). My goal with this Rust compute teaching activity is to convince people that Rust can be a solid contender to the big players of that ecosystem (C++, Fortran, Python + native backend...) in scenarios where long-term human scalability and maintainability matters more than having the best established library or GPU ecosystem. Because attracting more people to the Rust dark side is the only way we are ever going to get a more established library or GPU ecosystem anyhow But in this performance-focused crowd, it will be much harder to convince people if I cannot show that given a reasonable amount of work, Rust can achieve similar performance to C++ and Fortran, which to me means either "I look at the ASM from the compiler and I have no idea how I could have done it better" or "the computation is simple enough that I can compute the theoretical hardware throughput limit and we are right there as we should be". In this context, any language or compiler feature that makes it easier to get good codegen out of Rust helps. I will leave you to decide how critical that is. the8472: > as mentioned earlier, have you tried PGO? I guess I should give it a try at some point, but I really hope we won't end up with the official statement that PGO is necessary to get good performance out of even relatively simple numerical computations in Rust. Because PGO is a huge pain from the user side, and on the C++/Fortran side it is only rarely needed, so that's not a good look from a language comparison PoV... the8472: > have you tried extracting the hot part to a crate and then tuning inlining thresholds and the like Could that affect the compiler's ability to inline code from third-party crates? Because that's most often where the issue lies. I have regularly tried to call LTO for help there, but never got any significant performance gain out of it, as even when LTO allows the compiler to inline the incriminated function it stubbornly refuses to do so. Perhaps a combination of LTO and aggressive compiler backend flags could work? the8472: > could your yak-shaving be solved by attacking the problem elsewhere (you mention allocations and inlining decisions around target_features) Well, that's the specific issue that led me to throw the towel and open this particular forum thread, but ndarray and iterator adapter inlining issues are an everyday annoyance. Resolving that for good (not iteratively in a fine-grained way, little by little) would, on its own, be a major quality-of-life improvement for numerical Rust practicioners. It's just that since I got inlining issues in other contexts, I figured that maybe attacking the problem from the general inlining angle was most useful.