14 ms·
Rust SIMD on the GPU
- westurner 1mo agoRust SIMD in pgrust, on the GPU? "Pgrust v0.2: Now faster than Postgres" (2026-06) https://news.ycombinator.com/item?id=49111925 https://news.ycombinator.com/item?id=49111925
- efnx 1mo agoCongrats to the Rust-GPU folks! Nice to see the good work flowing.
- queisoy 1mo ago[flagged]
- LegNeato 1mo agoAuthor here, AMA.
- guess__who 1mo ago[flagged]
- jcranmer 1mo agoThe post is kind of vague on the IR you're targeting. Can you give some examples of what the SIMD-ized IR looks like, and how it maps to the target PTX?
- the__alchemist 1mo agoI'm confused too. How does this fit between these approaches for paraellization: - CUDA kernels and Tiles (e.g. Cudarc, cuda-oxide, rust-gpu etc) - SIMD on the GPU. (E.g. as in the title...) - CPU SIMD using avx or SSE instructions (And probably thin wrappers for vectors so you can have sane syntax). Or the maybe-upcoming core simd which should abstract over architecture-specific instructions. Magic floats etc which do 4-16 computations at once, but are a bit clumsy to work with - Rayon thread pools - arbitrary parallel computations, including SIMD, one per CPU core. It looks like from the code samples like maybe a cleaner syntax for writing code on the GPU than CUDA kernels? E.g. without mucking with serialization, host and device by abstracting over it? And inspired by core::simd. (Good choice if so, in the interest of standardizing on syntax; I did this for my x86 SIMD vector/quaternion lib as well)
- LegNeato 1mo agoDidn't want to go into crazy detail in the post. Each family of operations is a trait parameterized by the operation itself: pub trait EvaluateReduction<Operation, T>: LaneEvaluator { /// Reduce one distributed definition to an ordinary uniform scalar. fn evaluate_reduction(&self, value: LaneValue<Self, role::Distributed, T>) -> T; } Call sites name the operation: let one = evaluator.splat::<Splat, _>(1_u32); let two = evaluator.splat::<Splat, _>(2_u32); let three = evaluator.binary::<Add, _>(one, two); let total = evaluator.reduce::<Sum, u32>(three); // a uniform u32 let running = <Executor as EvaluateScan<Scan<Sum, Exclusive>, u32>>::scan(&evaluator, three); Operations like Sum, Max, ReduceXor, Inclusive, and Exclusive are all distinct types. As mentioned in the post, execution shape is typed too. A static shuffle takes its control as a type-level constant, and the shuffle mode constrains which controls are expressible: // Shift down one lane, keeping our own value where the source is inactive. let down = <Executor as EvaluateShuffle<Shuffle<Down>, DownOrSelf<1>, u32>>::shuffle(&ev, v); // Broadcast from lane zero. let bcast = <Executor as EvaluateShuffle<Shuffle<Broadcast>, WarpLane<0>, u32>>::shuffle(&ev, down); // Butterfly exchange with the neighbor one bit away. let bfly = <Executor as EvaluateShuffle<Shuffle<Xor>, Butterfly<1>, u32>>::shuffle(&ev, bcast); For an example of errors caught, a warp-scoped executor for a device-scoped barrier is a compile error: <ScopedWarpExecutor<'_, WarpUniform> as EvaluateBarrier<Barrier<Device>>>::barrier(evaluator) // error[E0277]: the trait bound `Device: NvptxBarrierScope` is not satisfied // help: the trait `NvptxBarrierScope` is implemented for `Warp` Strip mining is typed on the amount of work and the lane capacity, and it hands back one chunk at a time along with the predicate saying which lanes live in that chunk: // Six work items across four active lanes: two chunks, based at 0 and 4. <Executor as EvaluateStripMine<StripMine, (WorkItems, ActiveLanes<StripMined<4>>), i32>>:: for_each_strip_mined( &evaluator, (WorkItems::new(6)?, ActiveLanes::new(4)?), |index, active| { // ... }, ); Hopefully that gives the flavor of it.
- lbhdc 1mo agoThis is really cool! It sounds like y'all have a compiler fork that you are using to make this work. I wanna tinker with this, is your compiler available?
- LegNeato 1mo agoIt is not currently available but we intend to make it available after we launch our products.
- lbhdc 1mo agoWhat is vectorware's business model? Are you planning to sell support/consulting to companies using your stack? Or are you looking to sell licenses to your tool? Or something else?
- Eridrus 1mo agoGiven the massive demand for GPUs for LLMs, what sorts of work do you expect to economically benefit from utilizing GPUs more?
- LegNeato 1mo agoPart of our thesis is that decent GPUs are in every shipping device and most software doesn't use them and should.
- Eridrus 1mo agoI guess you're looking at consumer hardware then since servers have exactly what you pay for. Can you say more about the application space you're targeting?
- PoignardAzur 1mo agoAny thoughts about SIMD-related crates?
- adityazero 1mo agoReally like that the barrier scopes and shuffle controls are type-level, that is a lot of footguns turned into compile errors. Once you lower to PTX, does any of that structure survive for ptxas to optimize on, or is it fully erased by then? Also curious whether the strip mining abstraction fights the register allocator at higher lane counts, or if it stays friendly.
- shay_ker 1mo agoHm is the intent to one day replace the CPU?
- LegNeato 1mo agoThe goal is to use similar abstractions and code across both the CPU and GPU where it makes sense.
- max-m 1mo agoHow was your day?
- bbminner 1mo agoIf you have to express your computation using an "array programming DSL" with things like scan and gather anyways - why not opt to use torch/tensorflow/jax or anything else that targets MLIR? An example of writing a relu using an embedded array DSL is really not helping your case either - that's exactly the problem that these other solutions mentioned above are successfully solving for the past ~15y (starting with theano etc). Not sure what this brings to the table - doing that AoT instead of at runtime?
- LegNeato 1mo agoThe goal of this work is to run existing unmodified CPU libraries (which may use core::simd) on the GPU. If you are manually writing ML-shaped workloads, it doesn't add any value over writing with tech like torch/tensorflow/jax which are custom built for those use-cases (except maybe familiarity if you are a CPU programmer).
- peterbower 1mo agoAll well and good but where can we install it now?
- guess__who 1mo ago[flagged]
- the__alchemist 1mo agoI care because it means I can use this in a Rust program without a FFI barrier. Regrettably, we have built computing infrastructure as a society with many barriers; programming language is one.
- LegNeato 1mo agoWe never mention anything about superiority nor compare with other languages or programming models. This post is about making existing Rust CPU code work on the GPU.
- guess__who 1mo ago[flagged]
- fluffybucktsnek 1mo agoThose aren't lines you are reading. Those are your hallucinations. At worst, the post reads like a propaganda for VectorWare, but, overall, it reads more like their insights on the matter.
- kooi 1mo ago[flagged]
- fire_wheel 1mo ago[flagged]
- throwaway894345 1mo agoI'm not a Rust user apart from an occasional toy program here and there, but you seem really triggered about a language that other people use. What's the issue?
- 6r17 1mo agoMy heard hurts - i was stupid enough to think that SIMD was a CPU only thing - I don't understand why it would be ported to GPU - huge kudos to managing to surprise me
- hingler36 1mo agoWelcome to the lucky 10,000! SIMD is actually a pretty integral part of how GPUs are able to work efficiently, it's part of why there's such a strong focus on branchless programming in the field.
- chlorion 1mo agoGPUs work on vectors and matrices very often, that's what they are good at, so it makes a lot of sense that they can operate with SIMD I think!
- ismailmaj 1mo agoThere is something very SIMD-coded in GPU programming which is coalesced stores/loads, if a warp (32 threads) handles contiguous memory, it will create ~4 transactions instead of 32.
- monocasa 1mo agoGPU "cores" are basically what a CPU would call SIMD lanes. So a GPU with 1024 'CUDA cores' might be structured as 16 relatively independent pieces that a CPU might call a core, each with a 64 wide SIMD unit.
- mathisfun123 1mo ago32 wide - only AMD has a 64 wide mode
- y1n0 1mo ago64 what? Bits/bytes/something bigger?
- corysama 1mo ago
- the__alchemist 1mo agoHey - this is probably off-topic/meta, but what is going on with the comments here? Is it bots?
- dev_l1x_be 1mo agoNo idea, but it seems HN needs POW challenges.
- lukan 1mo agoCould also just be trolls attracted by the Rust topic.
- dev_l1x_be 1mo agoI bet some kid is bored out of his mind and wrote a bot.
- deleted 1mo ago[deleted]
- lx-user 1mo ago[flagged]
- deleted 1mo ago[deleted]
- rust-lang 1mo ago[flagged]
- O3marchnative 1mo agoThe author mentions Rust's portable SIMD library [0]. The only issue with portable SIMD is it's only available on nightly. I used it in my FFT crate, but we had to switch to the fearless_simd crate in order to get a portable SIMD solution that works on stable [1]. [0] https://doc.rust-lang.org/std/simd/index.html https://doc.rust-lang.org/std/simd/index.html [1] https://github.com/linebender/fearless_simd https://github.com/linebender/fearless_simd
- jonkoops 1mo agoPretty common for Rust to cook things in nightly for a very long time; I wouldn't consider it a bad thing, tbh.
- LoganDark 1mo agoIt's been annoying to me as an end user that so many basic things require nightly. I use nightly as my main toolchain, but enabling unstable features makes a project nightly-only, which is undesired for crates that don't already revolve around the unstable feature. I most often encounter unstable features when I reach for a basic common-sense utility method and discover that it's not stable. Like just earlier today I would have reached for bool::toggle which not only is unstable, but is also newly added as of like a month ago! but some unstable methods have been sitting around for years. And now that IntelliJ-Rust is proprietary, I can't even make a feature request anymore for the ability to exclude unstable features from the autocomplete. So they will taunt me forever, perfect little helpers just locked away.
- nirvdrum 1mo ago[dead]
- stymaar 1mo ago> It's been annoying to me as an end user that so many basic things require nightly It used to be the case a decade ago, but now I wouldn't agree that any "basic" things require nightly (I wouldn't call portable SIMD "basic" at all for instance). > Like just earlier today I would have reached for bool::toggle which not only is unstable, but is also newly added as of like a month ago! This is very likely not the kind of feature that will stay on nightly for a long time, but is instead one of the many convenience feature that land on stable every release. The 6-weeks release cadence with beta in between means there's always at least 6 weeks and up to 3 months between the time a feature land on nightly and the day it reaches stable, even if the feature is as consensual as this one. > And now that IntelliJ-Rust is proprietary, I can't even make a feature request anymore for the ability to exclude unstable features from the autocomplete. Can't you tell it to use stable as the default target, and use nightly manually in cargo?
- nynx 1mo agoDo you have examples of complex algorithms running on the gpu with rust with competative performance? Radix sort might be a good one to start with
- camel-cdr 1mo agoI love how ever example of portable SIMD isn't portable. They specifies a constant SIMD width so it's non-portable. Well, not performance portable, but why are we using SIMD again?
- tyho 1mo agoGo's implementation is vector size independant https://pkg.go.dev/simd@master https://pkg.go.dev/simd@master
- IshKebab 1mo agoSure but there's no real way to use that in a portable way, at least not a way that maximises performance on every CPU you run it on. That's pretty much impossible at the moment.
- krapht 1mo agoWhich is why I've never quite understood the appeal of portable SIMD libraries for performance-critical code. If I'm explicitly writing SIMD rather than relying on the auto-vectorizer, it's usually because I want access to the particular capabilities of the target ISA. For many problems, choosing the right instruction or instruction sequence makes a large difference. Portable SIMD abstractions necessarily expose some common semantic layer, but SIMD ISAs don't actually have equivalent capabilities. Instructions like pshufb, for example, enable algorithmic tricks that don't necessarily have an equally efficient analogue on another architecture. If maximum performance matters, I generally want intrinsics and architecture-specific implementations; if portability matters more, I'd rather move further up the abstraction stack and use something designed to target multiple architectures, such as ISPC. There are certainly cases where portable SIMD gets close enough to optimal, but I don't think there's a compiler or abstraction that can express every useful SIMD idiom and lower it equally efficiently across fundamentally different ISAs.
- pjmlp 1mo agoBecause usually they achieve a very good middle ground, they are useful for when autovectorization isn't good enough, and it is possible to give a little help to the compiler. There are many ways that performance matters without trying to win a F1 race. Go isn't alone, .NET, Java have similar portable libraries, and C++ is in the process of getting one.
- donald-trump 1mo ago[flagged]
- donald-trump 1mo ago[flagged]
- nperez19 1mo agoLove the pendantic mode setting on the website
- deleted 1mo ago[deleted]
- minraws 1mo agoI don't get what's the value of it not being enabled by default what does the toggle get us, really? Maybe I don't understand web design and it makes it harder to read for some, I am dyslexic and never had any issues.
- LegNeato 1mo agoIt's just a way for us to add minutia and details that most don't care / need to know about. There are three audiences we try to make the posts accessible for: Rust people who don't know about GPUs, GPU people who don't know about Rust, and non-Rust non-GPU people. The toggle lets knowledgable readers go "wait, what about..." and hopefully the toggle answers it.
- grokcodec 1mo agoI would love to have an open source Rust SIMD library with the scope and maturity that https://github.com/google/highway https://github.com/google/highway brings to C++.
- raphlinus 1mo agoThis is basically the goal of fearless_simd, but of course achieving the same level of maturity will take time.
- dev_dan_2 1mo agoReally exciting work and great write up, thanks a lot and all the best to your startup! `core` instead of `std` is great too! This will become useful in one of my sideproject where I use bitmaps to speed up pathfinding, exited to try it out!
- melodyogonna 1mo agoVery interesting. But GPU programming gets complicated when you start doing 3d computation on very large data, will be interesting to see how tensor abstraction is built on top of this. Another point is that this is using fixed-width SIMD vectors; unless there is a way to compute this statically based on available GPU info, performance will always be left on the table.
- frollogaston 1mo agoI've noticed a lot of articles about SIMD on the HN front page. That's cool, but just wondering, is there some reason this is more in focus lately?
- LoganDark 1mo agoMaybe things being on the front page reminds others? After seeing something, sometimes you can have ideas relating to it for a while.
- samuell 1mo agoYes, this kind of thing seems to happen quite often. Popular posts spurring further posts on a theme.
- skitsofrandom 1mo agoI’m wondering if AI has made SIMD intrinsics much more approachable for many and so there are just more people working on abstractions for their workflow of choice right now. There’s probably a lot of code out there that could benefit from SIMD but the effort to actually use it was too high for the return.
- vatsachak 1mo agoSIMD is actually underrated still. Programmers should always be thinking about it. It's a free 4x in a lot of cases
- frollogaston 1mo agoI did see the article about that too. Don't know about "always" since there are applications like web backends where you're never going to add arrays of floats or something. Even if it's data science stuff, if that's in Python, Numpy is doing the SIMD for you.
- vatsachak 1mo agoFair I work in very mathematical code
- neonsunset 1mo ago[dead]
- reindeer2 1mo ago[dead]