10 ms·
Wild – A fast linker for Linux
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- throwaway106382 2y ago> Mold is already very fast, however it doesn't do incremental linking and the author has stated that they don't intend to. Wild doesn't do incremental linking yet, but that is the end-goal. By writing Wild in Rust, it's hoped that the complexity of incremental linking will be achievable. Can someone explain what is so special about Rust for this?
- ComputerGuru 2y ago[flagged]
- nicce 2y agoI guess they meant that why write in Rust.
- ComputerGuru 2y agoYou caught me as I was trying to delete/move the comment because I missed that.
- dralley 2y agoI assume they're referring to thread-safety and the ability to more aggressively parallelize.
- compiler-guy 2y agoMold and lld are already very heavily parallelized. It’s one of the things that makes them very fast already.
- senkora 2y agoI assume that he is referring to "fearless concurrency", the idea that Rust makes it possible to write more complex concurrent programs than other languages because of the safety guarantees: https://doc.rust-lang.org/book/ch16-00-concurrency.html https://doc.rust-lang.org/book/ch16-00-concurrency.html So the logic would go: 1. mold doesn't do incremental linking because it is too complex to do it while still being fast (concurrent). 2. Rust makes it possible to write very complex fast (concurrent) programs. 3. A new linker written in Rust can do incremental linking while still being fast (concurrent). EDIT: I meant this originally, but comments were posted before I added it so I want to be clear that this part is new: (Any of those three could be false; I take no strong position on that. But I believe that this is the motivating logic.)
- compiler-guy 2y agoBoth mold and lld are already very heavily concurrent. There is no fear at all there.
- ComputerGuru 2y agoActually a lot of the hacks that mold uses to be the fastest linker would be, ironically, harder to reproduce with rust because they’re antithetical to its approach. Eg Mold intentionally eschews used resource collection to speed up execution (it’ll be cleaned up by the os when the process exits) while rust has a strong RAII approach here that would introduce slowdowns.
- Philpax 2y agoI mean, that's pretty easy to do in Rust: https://doc.rust-lang.org/std/mem/struct.ManuallyDrop.html https://doc.rust-lang.org/std/mem/struct.ManuallyDrop.html Also see various arena allocator crates, etc.
- ComputerGuru 2y agoNot really. You would have to either wrap any standard library types in newtypes with ManuallyDrop implemented or (for some) use a custom allocator. And if you want to free some things in one go but not others that gets much harder, especially when you look at how easy a language like zig makes it. And if you intentionally leak everything it is onerous to get the borrow checker to realize that unless you use a leaked box for all declaration/allocations, which introduces both friction and performance regressions (due to memory access patterns) because the use of custom allocators doesn’t factor into lifetime analysis. (Spoken as a die-hard rust dev that still thinks it’s the better language than zig for most everything.)
- devit 2y agoIt's feasible to write complex correct programs with optimal performance in Rust, unlike any other programming language (complex+correct is not feasible in C/C++/assembly/Zig/etc., optimal performance not possible in any other language).
- compiler-guy 2y agoThat’s puzzling to me too. Rust is a great language, and probably makes developing Wild faster. But the complexity of incremental linking doesn’t stem from the linker’s implementation language. It stems from all the tracking, reserved spacing, and other issues required to link a previously linked binary (or at least parts of it) a second time.
- tialaramex 2y agoI would guess the idea is that in Rust the complexity is cheaper on a "per unit" basis so you can afford more complexity. So yes, it is a more complicated problem than the previous linkers, but, in Rust maybe you can get that done anyway.
- IshKebab 2y agoRust allows your to enforce more invariants at compile time, so implementing a complex system where you are likely to make a mistake and violate those invariants is easier.
- IshKebab 2y agoThere are two main factors: 1. Rust's well designed type system and borrow checker makes writing code that works just easier. It has the "if it compiles it works" property (not unique to Rust; people say this about e.g. Haskell too). 2. Rust's type system - especially its trait system can be used to enforce safety constraints statically. The obvious one is the Send and Sync traits for thread safety, but there are others, e.g. the Fuchsia network code statically guarantees deadlocks are impossible. Mold is written in C++ which is extremely error prone in comparison.
- manoweb 2y agoThat is baffling. Maybe the author assumes that a language with many safeguards will lead to keeping complexity under control for a difficult task. By the way I had to lookup what incremental linking is, in practice I think it means that code from libraries and modules that have not changed won’t need to be re-packed each time which ch will save time for frequent development builds, it’s actually ingenious
- deleted 2y ago[deleted]
- the_duke 2y agoRust has a pretty good incremental caching compiler that makes debug builds relatively fast. Linking is often a very notable bottleneck for debug binaries and mold can make a big difference. So interest in speeding up linking for Rust is expected.
- panstromek 2y agoApart from what others said, maybe he plans to use Salsa or something like that. Rust has a few popular libraries for doing this.
- wffurr 2y agoI went looking for some writing by the author about how he made wild fast, but couldn't find much: https://davidlattimore.github.io/ https://davidlattimore.github.io/
- devit 2y agoI think the optimal approach for development would be to not produce a traditional linked executable at all, but instead just place the object files in memory, and then produce a loader executable that hooks page faults in those memory areas and on-demand mmaps the relevant object elsewhere, applies relocations to it, and then moves it in place with mremap. Symbols would be resolved based on an index where only updated object files are reindexed. It could also eagerly relocate in the background, in order depending on previous usage data. This would basically make a copyless lazy incremental linker.
- IshKebab 2y agoSounds like dynamic linking, sort of.
- fsfod 2y agoYou can sort of do that with some of LLVM's JIT systems https://llvm.org/docs/JITLink.html https://llvm.org/docs/JITLink.html, I'm surprised that no one has yet made a edit and continue system using it.
- all2 2y agoMy parens sense is tingling. This sounds like a lisp-machine, or just standard lisp development environment.
- klibertp 2y agoMaybe of interest: https://github.com/clasp-developers/clasp/ https://github.com/clasp-developers/clasp/ (Lisp env. that uses LLVM for compilation; new-ish, actively developed.) However, my impression (I didn't measure it) is that the compilation speed is an order of magnitude slower than in SBCL, never mind CCL.
- samatman 2y agoThey have! It's called Julia and it's great.
- 2y ago
- satvikpendem 2y agoI looked at this before, is it ready for production? I thought not based on the readme, so I'm still using mold. For those on macOS, Apple released a new linker about a year or two ago (which is why the mold author stopped working on their macOS version), and if you're using it with Rust, put this in your config.toml: [target.aarch64-apple-darwin] rustflags = [ "-C", "link-arg=-fuse-ld=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld", "-C", "link-arg=-ld_new", ]
- brink 2y agoI don't even use mold for production. It's for development.
- dralley 2y agoNo, the author is pretty clear that it shouldn't be used for production yet
- satvikpendem 2y agoGreat, I'll keep a look out but will hold off on using it for now.
- newman314 2y agoCan you confirm that's still the right location for Sequioa? I have the command line tools installed and I only have /usr/bin/ld and /usr/bin/ld-classic
- satvikpendem 2y agoThen it'd be the /usr/bin/ld as I believe my solution was for before they moved the linker it seems.
- saagarjha 2y ago/usr/bin/ld will correctly invoke the right linker, it's a stub to look at your developer dir and reexec.
- ComputerGuru 2y agoThere’s been a lot of interest in faster linkers spurred by the adoption and popularity of rust. Even modest statically linked rust binaries can take a couple of minutes in the link stage of compilation in release mode (using mold). It’s not a rust-specific issue but an amalgam of (usually) strictly static linking, advanced link-time optimizations enabled by llvm like LTO and bolt, and a general dissatisfaction with compile times in the rust community. Rust’s (clinically) strong relationship with(read: dependency on) LLVM makes it the most popular language where LLVM link-time magic has been most heavily universally adopted; you could face these issues with C++ but it wouldn’t be chalked up to the language rather than your toolchain. I’ve been eyeing wild for some time as I’m excited by the promise of an optimizing incremental linker, but to be frank, see zero incentive to even fiddle with it until it can actually, you know, link incrementally.
- sitkack 2y agoI solved this by using Wasm. Your outer application shell calls into Wasm business logic, only the inner logic needs to get recompiled, the outer app shell doesn't even need to restart.
- ComputerGuru 2y agoI don’t think I can use wasm with simd or syscalls, which is the bulk of my work.
- sitkack 2y agoI haven't used SIMD in Rust (or Wasm). Syscalls can be passed into the Wasm env. https://doc.rust-lang.org/core/arch/wasm32/index.html#simd https://doc.rust-lang.org/core/arch/wasm32/index.html#simd https://nickb.dev/blog/authoring-a-simd-enhanced-wasm-library-with-rust/ https://nickb.dev/blog/authoring-a-simd-enhanced-wasm-librar... Could definitely be more effort than it is worth just to speed up compilation.
- SkiFire13 2y agoHow is this different than dynamically linking the business logic library?
- pzmarzly 2y agoEver since mold relicensed from AGPL to MIT (as part of mold 2.0 release), the worldwide need for making another fast linker has been greatly reduced, so I wasn't expecting a project like this to appear. And definitely wasn't expecting it to already be 2x faster than mold in some cases. Will keep an eye on this project to see how it evolves, best of luck to the author.
- secondcoming 2y agoMaybe I'm holding it wrong, but mold isn't faster at all if you're using LTO, which you probably should be.
- account42 2y agoYou should be using LTO where incremental build times are a concern, i.e. for development builds. And for realease builds link time is hardly a concern.
- 0x457 2y agoI think we're talking about non-release builds here. In those, you don't want to use LTO, you just want to get that binary as fast as possible.
- deleted 2y ago[deleted]
- compiler-guy 2y agoMold will be faster than LLD even using LTO, but all of its benefits will be absolutely swamped by the LTO process, which is, more or less, recompiling the entire program from high-level LLVM-IR. That's extremely expensive and dwarfs any linking advantages. So the benefit will be barely noticable. As another comment points out, LTO should only be used when you need a binary optimized to within an inch of its life, such as a release copy, or a copy for performance testing.
- paulddraper 2y ago
- deleted 2y ago[deleted]
- iceenforcer 2y ago[dead]
- kryptiskt 2y agoWhat would be refreshing would be a C/C++ compiler that did away with the intermediate step of linking and built the whole program as a unit. LTO doesn't even have to be a thing if the compiler can see the entire program in the first place. It would still have to save some build products so that incremental builds are possible, but not as object files, the compiler would need metadata to know of the origin and dependencies of all the generated code so it would be able to replace the right things. External libs are most often linked dynamically these days, so they don't need to be built from source, so eliminating the linker doesn't pose a problem for non-open source dependencies. And if that's not enough letting the compiler also consume object files could provide for legacy use cases or edge cases where you must statically link to a binary.
- almostgotcaught 2y ago[flagged]
- nn3 2y ago>Secondly, if you think any compiler is meaningfully doing anything optimal >>("whole program analysis") on a TU scale greater than say ~50kloc (ie ~10 files) >relative to compiling individually you're dreaming. That's wrong. gcc generates summaries of function properties and propagate those up and down the call tree, which for LTO is then build in a distributed way. It does much more than mere inlining, but even advanced analysis like points to analysis. https://gcc.gnu.org/onlinedocs/gccint/IPA.html https://gcc.gnu.org/onlinedocs/gccint/IPA.html https://gcc.gnu.org/onlinedocs/gccint/IPA-passes.html https://gcc.gnu.org/onlinedocs/gccint/IPA-passes.html It scales to millions of lines of code because it's partioned.
- jcalvinowens 2y ago> if you think any compiler is meaningfully doing anything optimal ("whole program analysis") on a TU scale greater than say ~50kloc (ie ~10 files) relative to compiling individually you're dreaming. You can build the Linux kernel with LTO: simply diff the LTO vs non-LTO outputs and it will be obvious you're wrong.
- 2y ago
- deleted 2y ago[deleted]
- shmerl 2y agoThat looks promising. In Rust to begin with and with the goal of being fast and support incremental linking. To use it with Rust, this can probbaly also work using gcc as linker driver. In project's .cargo/config.toml: [target.x86_64-unknown-linux-gnu] rustflags = ["-C", "link-arg=-fuse-ld=wild"] Side note, but why does Rust need to plug into gcc or clang for that? Some missing functionality?
- sedatk 2y agoBecause Rust compiler generates IR bytecode, not machine code.
- davidlattimore 2y agoUnfortunately gcc doesn't accept arbitrary linkers via the `-fuse-ld=` flag. The only linkers it accepts are bfd, gold lld and mold. It is possible to use gcc to invoke wild as the linker, but currently to do that, you need to create a directory containing the wild linker and rename the binary (or a symlink) to "ld", then pass `-B/path/to/directory/containing/wild` to gcc. As for why Rust uses gcc or clang to invoke the linker rather than invoking the linker directly - it's because the C compiler knows what linker flags are needed on the current platform in order to link against libc and the C runtime. Things like `Scrt1.o`, `crti.o`, `crtbeginS.o`, `crtendS.o` and `crtn.o`.
- shmerl 2y agoAh, good to know, thanks! May be it's worth filing a feature request for gcc to have parity with clang for arbitrary linkers?
- sylware 2y agoThe real issue is actually runtime ELF (and PE) which are obsolete on modern hardware architecture.
- bmacho 2y agoWhat do you mean by this?
- sylware 2y agoELF(COFF) should now be only an assembler output format on modern large hardware architecture. On modern large hardware architecture, for executable files/dynamic libraries, ELF(PE[+]) has overkill complexity. I am personnally using a executable file format of my own I do wrap into an "ELF capsule" on linux kernel. With position independent code, you kind of only need memory mapped segments (which dynamic libraries are in this very format). I have two very simple partial linkers I wrote in plain and simple C, one for risc-v assembly, one for x86_64 assembly, which allow me to link into such executable file some simple ELF object files (from binutils GAS). There is no more centralized "ELF loader". Of course, there are tradeoffs, 1 billion times worth it in regards of the accute simplicity of the format. (I even have a little vm which allows me to interpret simple risc-v binaries on x86_64).
- o11c 2y agoYou're giving up a lot if you stop using a format that supports multiple mapping, relro, dynamic relocations, ...
- sylware 2y agoThis is where the "scam" from those excessively complex formats is: I, pertinently, do not give up a lot since I get the job done... but on the other side, I gain the removal of tons and tons of complexity, and nullify significant developer/vendor lock-in at the same time. A good analogy to "feel" that, it is a bit like "json vs xml" but for executable binary formats. But, I keep in mind, those formats (excrutiatingly simple) can work only on modern hardware architectures.
- ajb 2y ago2008: Gold, a new linker, intended to be faster than Gnu LD 2015(?): Lld a drop in replacement linker, at least 2x as fast as Gold 2021: mold, a new linker, several times faster than lld 2025: wild, a new linker...
- dundarious 2y agoFor windows, there is also [The RAD Linker](https://github.com/EpicGamesExt/raddebugger?tab=readme-ov-file#the-rad-linker https://github.com/EpicGamesExt/raddebugger?tab=readme-ov-fi...) though quite early days.
- wolfd 2y agoI’m not sure if you’re intending to leave a negative or positive remark, or just a brief history, but the fact that people are still managing to squeeze better performance into linkers is very encouraging to me.
- ajb 2y agoCertainly no intention to be negative. Not having run the numbers, I don't know if the older ones got slower over time due to more features, or the new ones are squeezing out new performance gains. I guess it's also partly that the bigger codebases scaled up so much over this period, so that there are gains to be had that weren't interesting before.
- wolfd 2y agoGood question, I always wonder the same thing. https://www.phoronix.com/news/Mold-Linker-2024-Performance https://www.phoronix.com/news/Mold-Linker-2024-Performance seems to show that that the newer linkers still outperform their predecessors, even after maturing. But of course this doesn’t show the full picture.
- cbmuser 2y agoGold is slated for removal from binutils for version 2.44.0, so it's officially dead.
- bjourne 2y agoWhat a coincidence. :) Just an hour ago I compared the performance of wild, mold, and (plain-old) ld on a C project I'm working on. 23 kloc and 172 files. Takes about 23.4 s of user time to compile with gcc+ld, 22.5 s with gcc+mold, and 21.8 s with gcc+wild. Which leads me to believe that link time shouldn't be that much of a problem for well-structured projects.
- ndesaulniers 2y agoHow about ld.lld?
- searealist 2y agoFast linkers are mostly useful in incremental compilation scenarios to cut down on the edit cycle.
- davidlattimore 2y agoIt sounds like you're building from scratch. In that case, the majority of the time will be spent compiling code, not linking. The case for fast linkers is strongest when doing iterative development. i.e. when making small changes to your code then rebuilding and running the result. With a small change, there's generally very little work for the compiler to do, but linking is still done from scratch, so tends to dominate.
- commandersaki 2y agoYep in my case I have 11 * 450MB executables that take about 8 minutes to compile and link. But for small iterative programming cycles using the standard linker with g++, it takes about 30 seconds to link (If I remember correctly). I tried mold and shaved 25% of that time, which didn't seem worth the change overall; attempted wild a year ago but ran into issues, but will revisit at some point.
- menaerus 2y agoExactly. But also even in build-from-scratch use-case when there's a multitude of binaries to be built - think 10s or 100s of (unit, integration, performance) test binaries or utilities that come along with the main release binary etc. Faster linkers giving even a modest 10% speedup per binary will quickly accumulate and will obviously scale much better.
- ndesaulniers 2y agoCan it link the Linux kernel yet? Was a useful milestone for LLD.
- davidlattimore 2y agoNot yet. The Linux kernel uses linker scripts, which Wild doesn't yet support. I'd like to add support for linker scripts at some point, but it's some way down the priority list.
- oguz-ismail 2y agoDoes it at least support -Ttext, -Tdata, etc.?
- KerrAvon 2y agoI'm curious: what's the theory behind why this would be faster than mold in the non-incremental case? "Because Rust" is a fine explanation for a bunch of things, but doesn't explain expected performance benefits. "Because there's low hanging concurrent fruit that Rust can help us get?" would be interesting but that's not explicitly stated or even implied.
- davidlattimore 2y agoI'm not actually sure, mostly because I'm not really familiar with the Mold codebase. One clue is that I've heard that Mold gets about a 10% speedup by using a faster allocator (mimalloc). I've tried using mimalloc with Wild and didn't get any measurable speedup. This suggests to me that Mold is probably making heavier use of the allocator than Wild is. With Wild, I've certainly tried to optimise the number of heap allocations. But in general, I'd guess just different design decisions. As for how this might be related to Rust - I'm certain that were Wild ported from Rust to C or C++, that it would perform very similarly. However, code patterns that are fine in Rust due to the borrow checker, would be footguns in languages like C or C++, so maintaining that code could be tricky. Certainly when I've coded in C++ in the past, I've found myself coding more defensively, even at a small performance cost, whereas with Rust, I'm able to be a lot bolder because I know the compiler has got my back.
- menaerus 2y ago> Mold gets about a 10% speedup by using a faster allocator (mimalloc). I've tried using mimalloc with Wild and didn't get any measurable speedup Perhaps it is worth repeating the experiment with heavy MLoC codebases. jmalloc or mimalloc.
- einpoklum 2y agoRust is a perfectly fine language, and there's no reason you should not be able to implement fast incremental linking using Rust, so - I wish you success in doing that. ... however... > code patterns that are fine in Rust due to the borrow checker, would be footguns in languages like C or C++, That "dig" is probably not true. Or rather, your very conflation of C and C++ suggests that you are talking about the kind of code which would not be used in modern C++ of the past decade-or-more. While one _can_ write footguns in C++ easily, one can also very easily choose not to do so - especially when writing a new project.
- fuzztester 2y agoRelated, and a good one, though old: The book Linkers and Loaders by John Levine. Last book in the list here: https://www.johnlevine.com/books.phtml https://www.johnlevine.com/books.phtml I had read it some years ago, and found it quite interesting. It's a standard one in the field. He has also written some other popular computer books (see link above - pun not intended, but noticed).
- 1vuio0pswjnm7 2y ago"These benchmark were run on David Lattimore's laptop (2020 model System76 Lemur pro), which has 4 cores (8 threads) and 42 GB of RAM." https://news.ycombinator.com/item?id=33330499 https://news.ycombinator.com/item?id=33330499 NB. This is not to suggest wild is bloated. The issue if any is the software being developed with it and the computers of those who might use such software.
- 1vuio0pswjnm7 2y agohttps://news.ycombinator.com/item?id=42896619 https://news.ycombinator.com/item?id=42896619 "... I have 16 GB of ram, I can't upgrade it..."
- klibertp 2y agoHalf in jest, but I'd think anybody coding in Rust already has 32GB of RAM... (Personally, upgrading my laptop to 64GB at the expense of literally everything else was almost a great decision. Almost, because I really should have splurged on RAM and display instead of going all-in on RAM. The only downside is that cleaning up open tabs once a week became a chore, taking up the whole evening.)
- unit149 2y ago[dead]
- juujian 2y agoIs it too late to ask what a linker is?
- nappy-doo 2y agoI'll ELI5: Compilers take the code the programmer writes, and turns it into things called object files. Object files are close to executable by the target processor, but not completely. There are little places where the code needs to be rewritten to handle access to subroutines, access operating system functionality, and other things. A linker combines all these object files, does the necessary rewriting, and generates something that the operating system can use. It's the final step in building an executable. -- More complicatedly: a linker is a little Turing machine that runs over the object files. Some can do complicated things like rewriting code, or optimizing across function calls. But, fundamentally, they plop all the object files together and follow little scripts (or rewrites) that clean up the places the compiler couldn't properly insert instructions because the compiler doesn't know the final layout of the program.
- Mikhail_K 2y agoI just knew it's going to be Rust as soon as I've read the title.
- deleted 2y ago[deleted]