9 ms·

The state of binary compatibility on Linux and how to address it

201 points by generichuman 1y ago

sylware 1y ago
This article missed a critical point which is "the right way" to select a glibc ABI version: see binutils ld documentation, second part of the page related to VERSION support. This must include glibc internal symbols. This will allow to craft ELF binaries on a modern distro which will run on "older" distros. This is critical for games and game engines. There is an significant upfront only-once work in order to select an "old" glibc ABI. The quick and dirty alternative being having a toolchain configured to link with an "old" glibc on the side. This article missed the -static-libstdc++ critical option for c++ applications (the c++ ABI is hell on earth), but did not miss the -static-libgcc and the dynamic loading of system interface shared libs.
- api 1y ago
  Is there a reason glibc can't just do a better job at keeping some legacy symbols around? It's not like it's big stuff. They're things like legacy string functions. We're talking a few kilobytes of code in most cases. The Linux kernel goes to a lot of effort to not break user space, at least for non-exotic core features and syscalls. It seems like a lot of user-space in Linux-land does not make the same effort. It's particularly bad when it's the C library doing this, since that's at the center of the dependency graph for almost everything.
  - Conan_Kudo 1y ago
    The legacy symbols are still around. It's just that the linker won't use them by default.
  - FitCodIa 1y ago
    > a lot of user-space in Linux-land does not make the same effort I believe that, what the article misses is that glibc is maintained and extended with an entirely different community and development model. Windows remains compatible over decades because Microsoft (a) is the sole distributor, and (b) puts an immense effort towards backwards compat. In Linux userspace, it's simply a non-goal across distributions. If you want to ship a binary for a particular distro, you need to build the binary on / for that distro; even within a distro, a major release bump (or especially a major release downgrade) may break a binary. Ultimately, it's a consequence of Conway’s Law. Microsoft is the sole distributor of Windows, so they can enforce compatibilty with an iron fist, and there are people working for Microsoft whose pay depends on said compatibility. With "Linux" in general, there is no common authority to appeal to, and (again) most vendors don't even promise a seamless userspace upgrade path from one major release to another. This is unfixable; it will never change -- as long as independent parties are permitted to distribute different operating systems yet call them all "Linux". Ship multiple binaries, or distribute the source code (and let users build it). EDIT: you'll notice that "ship multiple binaries" is what distros (especially commercial distros) do. They maintain separate branches, backport fixes to old branches, and employ software maintenance engineers to focus on this kind of work. If you want to target multiple major releases, this is what you have to do, too. If you (as a commercial ISV) target a commercial distro with long-term support, and can convince your users to use / license the same distro, you'll have a good, stable development experience. You only need to port like once every decade, when you jump major releases. The Linux user base / the Linux market is fragmented; that's the whole goal. The technical proliferation / inconsistency is just a consequence. Unless you take away the freedom of users to run their own flavors of "Linux", there won't be a uniform Linux target. In a way, it's funny to even expect otherwise. Why do you expect to ship the same binaries when the foundations are diverse, with no standardization authority that all Linux distributors recognize as such? And even POSIX is an API spec, not an ABI spec. And, any authority that controls binary aspects will immediately accrue political capital. This is exactly what shouldn't happen in Linux. The fact that anyone can fork (or start) a distro, and contribute to the chaos, is good for freedom.
    - cozzyd 1y ago
      The big Linux distros (EL, Ubuntu) in fact have the opposite incentive, to get proprietary vendors to rather their distribution specifically.
      - myk9001 1y ago
        This theory would check out if a proprietary vendor could easily get away with shipping a single binary package for all supported versions of, say, Ubuntu. Having to build and maintain a binary packege separately for each version of the same distro probably isn't that appealing to them.
        cozzyd 1y ago
        For Ubuntu they would only target LTS releases, most likely. On EL it's easier, now you would just support 2 or 3 of EL7, EL8, and EL9. As an example of something I use, Xfdtd only officially supports one version of Ubuntu and 2 versions of EL https://www.remcom.com/system-requirements#xfdtd-system-requirements https://www.remcom.com/system-requirements#xfdtd-system-requ... In practice, it wasn't too hard to get it running on EL9 or Fedora either...
    - yjftsjthsd-h 1y ago
      > If you (as a commercial ISV) target a commercial distro with long-term support, and can convince your users to use / license the same distro, you'll have a good, stable development experience. You only need to port like once every decade, when you jump major releases. If things go well, it's even better than that: If you target ex. RHEL 8, there's a very good chance that your binaries will work on RHEL 9 and a decent shot at RHEL 10 with zero changes (though of course you should test all versions you want to work). And the same for Ubuntu 20.04/22.04/24.04/... and Debian/SUSE/whatever. Backwards incompatibilities can happen, but within a single stable distro they're not super common so the lazy ISV can probably only really port forward after more than a decade if they really want. (Incidentally, this isn't a hypothetical: I once had the joy of working on software that targeted RHEL 5, and those binaries ran on RHEL/CentOS 7 without any problems.)
    - 1y ago
  - AshamedCaptain 1y ago
    The problem is the opposite: they are trying to run executables built using a newer glibc in a system that has an older glibc. glibc keeps all the old function definitions since practically forever. Frankly, I do not understand who would think glibc symbols themselves would be the challenge in this case. Even if you statically link glibc there's zero guarantee the syscalls will be present in the older Linux (cue .ABI-tag failures). Or even damn ELF format changes (e.g. gnu-style hashes). The simple solution is to build in the older Linux (&glibc). In my long experience with ancient binaries, glibc has almost never been the problem, and its ability to _run_ ancient binaries is all but excellent; even Linux is more of a problem than glibc is (for starters paths to everywhere in /proc, /sys change every other half-decade).
    - forrestthewoods 1y ago
      > executables built and using a newer glibc It’s an abomination that Linux uses system libraries when building. Catastrophically terrible and stupid decision. It should be trivial for any program to compile and specify any arbitrary previous version of glibc as the target. Linux got this so incredibly wildly wrong. It’s a shame.
      - AshamedCaptain 1y ago
        Ehm... you _can_ use the older toolchain even in the newer "Linux", in the same way you have to use a older Visual Studio to target Windows 8 from Windows 10.
        myk9001 1y ago
        So maybe the difference here is between can and actually doing it? I tend to stay on the oldest supported version of Windows until they drop support and haven't ever seen an application that wouldn't run because it's built on a newer version of Windows.
        forrestthewoods 1y ago
        I am saying that compiler toolchains on Linux should never ever under any circumstances ever rely on anything on the system for compiling. Compiling based on the system global version of glibc is stupid, bad, wrong, and Linus should be ashamed for letting it happen. It should be trivial for Windows to cross-compile for Linux for any distro and for any ancient version of glibc. It is not trivial. Here is a post describing the mountain range of bullshit that Zig had to move to enable trivial cross-compile and backwards targeting. https://andrewkelley.me/post/zig-cc-powerful-drop-in-replacement-gcc-clang.html https://andrewkelley.me/post/zig-cc-powerful-drop-in-replace... Linux is far and away the worst offender out of Linux, Mac, and Windows. By leaps and bounds.
- AndyKelley 1y ago
  One of the features Zig provides is ability to target any glibc version. See https://github.com/ziglang/glibc-abi-tool/ https://github.com/ziglang/glibc-abi-tool/ for more details on how this is solved.
  - forrestthewoods 1y ago
    You’re my hero for this. One of my side projects is building a toolchain to enable C++ cross-compile using the Zig header/source libs. I didn’t love Zig as a Clang++ replacement because it has a bit too much magic. And it might go ahead? But the underlying library code is a God send
- vlovich123 1y ago
  Which works if you use binutils ld. Does it work with mold or gold? And then how do you use this with languages other than c++/c like Go or Rust?
  - rjsw 1y ago
    I thought that Go invoked syscalls directly instead of going through libc.
    - jfuafdfwa 1y ago
      It's Go's best feature, a simple CGO_ENABLED=0 gives you freedom from the tyranny of libc.
  - sylware 1y ago
    Rust toolchain has already a bug since 2015: it cannot link statically libgcc, namely it does not have "-static-libgcc" option since 2015. I got the bug with "TinyGlade" video game (extremely good BTW), which is written in rust, and with the dev we hit that bug. Namely... better have a libgcc with the right ABI... and I can tell you, this has a been a HUGE issue since valve started to distribute games more than a decade ago.
    - vlovich123 1y ago
      As in you have to build on an old distro? Or something else?
- IX-103 1y ago
  Could you point to an example of a project doing it the right way? I'm a little unclear on the type of incantation needed for the "version script".
superkuh 1y ago
As an end user I often patch the glibc version incompatibility away with https://github.com/corsix/polyfill-glibc https://github.com/corsix/polyfill-glibc $ ./polyfill-glibc --target-glibc=2.17 /path/to/my-program This often leads to discovering new version incompatibilities in other libs. But as the article says others usually can be statically compiled.
- vlovich123 1y ago
  Wow. This is huge. This patch doesn’t risk weird runtime errors due to missing polyfills or anything? Does it risk polyfilling irrelevant functions that aren’t referenced and bloating?
- therein 1y ago
  Very cool. Glad someone finally made this.
Jeaye 1y ago
I don't understand why they don't just statically link their binaries. First, they said this: > Even if you managed to statically link GLIBC—or used an alternative like musl—your application would be unable to load any dynamic libraries at runtime. But then they immediately said they actually statically link all of their deps aside from libc. > Instead, we take a different approach: statically linking everything we can. If they're statically linking everything other than libc, then using musl or statically linking glibc will finish the job. Unless they have some need for loading share libs at runtime which they didn't already have linked into their binary (i.e. manual dlopen), this solves the portability problem on Linux. What am I missing (assuming I know of the security implications of statically linked binaries -- which they didn't mention as a concern)?
- masfuerte 1y ago
  Various things including name (DNS) resolution rely on dynamic linking.
  - Jeaye 1y ago
    Are you saying that a statically linked binary cannot make an HTTP request to `google.com` because it would be unable to resolve the domain name? There are entire distros, like alpine, built on musl. I find this very hard to believe.
    - masfuerte 1y ago
      The configuration of DNS resolution on Linux is quite complicated [1]. Musl just ignores all that. You can build a distro that works with musl, but a static musl binary dropped into an arbitrary Linux system won't necessarily work correctly. [1]: https://news.ycombinator.com/item?id=43451861 https://news.ycombinator.com/item?id=43451861
    - Spivak 1y ago
      You have to bundle your own resolver into your application. But here's the rub, users expect your application to respect nsswitch which requires loading shared libs which execute arbitrary code. How Go handles this is somewhat awkward. They parse /etc/nsswitch and decide if they can cheat and use their own resolver based on what modules they see[1]. Otherwise they farm out to cgo to go through glibc. [1] They're playing with fire here because you can't really assume to know for sure how the module 'dns' behaves. A user could replace the lib that backs it with their own that resolves everything to zombo.com. It would be one thing if nsswitch described behavior which was well defined and could be emulated but it doesn't, it specifies a specific implementation.
bee_rider 1y ago
Really the title should be something like “the difficulty of releasing binaries on Linux and how to work around it.” It isn’t really an atrocious state, the goal of a distro should be to get useful software into the repos. Software not distributed in an open source format doesn’t really help there.
James_K 1y ago
Here's a thought: just distribute source code. ABI issues should be mostly fixed. Most computers can compile source code fast enough for the user not to notice and cache the results so that it's never a problem again. If you want optimised code, you can do a source to source optimisation then zip and minify the file. You could compile such a library to approximately native speeds without much user-end lag using modern JIT methods, and maybe even run LTO in a background thread so that the exectuables outdo dynamically linked ones.
- RedShift1 1y ago
  I hate compiling. 9/10 something goes wrong. Sometimes I can fix it, other times I just abandon the effort and use something else. These days I just use packages or docker images and if that doesn't work out I'm moving on, ain't nobody got time for this. You really can't expect people who just want to use their computers and don't even know what a compiler is to get involved in a process like that.
  - ses1984 1y ago
    No, end users need not get involved, it could/should be handled by the operating system.
    - HKH2 1y ago
      Gentoo? Compiling big packages takes ages.
      - adrian_b 1y ago
        More than 20 years, with a single-core Pentium 4, it could take indeed something like 3 days of continuous compilation to compile an entire Gentoo distribution, in order to have a personal computer with every application that one might want. However, already after the appearance of the first dual-core AMD Athlon64, 20 years ago, that time could be reduced to not much more than a half of day, while nowadays, with a decent desktop CPU from 5 years ago, most Gentoo packages can be compiled and installed in less than a minute. There are only a few packages whose compilation and installation can take a noticeable time, of up to tens of minutes, depending on the chosen options and on the number of cores of the CPU, e.g. Firefox, LibreOffice, LLVM. There is only a single package whose compilation may take ages unless you have an expensive CPU and enough memory per core: Google Chromium (including its derivatives that use the same code base).
lukeh 1y ago
Windows having multiple C libraries has its own pain points, in particular it's difficult to ship binary libraries that return allocated memory to their consumer (you either need to have the library consumer allocate the memory, which probably explains why so many Win32 APIs have this behaviour, or allow alloc/free functions to be registered). Not to mention different C libraries having their own file handle, TLS, etc state. Unsurprisingly Microsoft now ships the Universal CRT (UCRT) as part of Windows.
- int_19h 1y ago
  Having consumer allocate the memory where possible has other benefits in that the consumer can use more efficient allocation techniques (stack, arenas etc), and given that Win32 API design goes all the way back to the earliest version of Windows running on very resource-limited PCs, that would be the main reason why. However, in cases where this wasn't consistently feasible - e.g. COM - Windows instead mandates the use of a common API to manage memory: CoGetMalloc etc.
BwackNinja 1y ago
There is no distinction between system and program libraries in Linux. We used to pretend there was one before usrmigration, but that was never good to take seriously. The distro as packager model ensures that everything is mixed together in the filesystem and is actively hostile to external packaging. Vendoring dependencies or static linking improves compatibility by choosing known working versions, but decreases incentive and ability for downstream (or users) to upgrade those dependencies. The libc stuff in this article is mostly glibc-specific, and you'd have fewer issues targeting musl. Mixing static linking and dlopen doesn't make much sense, as said here[1] which is an interesting thread. Even dns resolution on glibc implies dynamic linking due to nsswitch. Solutions like Snap, Flatpak, and AppImage work to contain the problem by reusing the same abstractions internally rather than introducing anything that directly addresses the issue. We won't have a clean solution until we collectively abandon the FHS for a decentralized filesystem layout where adding an application (not just a program binary) is as easy as extracting a package into a folder and integrates with the rest of the system. I've worked on this off and on for a while, but being so opinionated makes everything an uphill battle while accepting the current reality is easy. [1] https://musl.openwall.narkive.com/lW4KCyXd/static-linking-and-dlopen https://musl.openwall.narkive.com/lW4KCyXd/static-linking-an...
- veqq 1y ago
  > adding an application (not just a program binary) is as easy as extracting a package into a folder and integrates with the rest of the system I have fond memories of installed Warlords Battle Cry 3, Warcraft 3, AOE2 etc. directories on flash drives, distributed to 20+ kids in high school (all using the same key). Good days.
  - HideousKojima 1y ago
    Way off topic but you just reminded me of all the time I spent playing Warlords 3 (not Warlords Battlecry 3, the original Warlords games were turn-based). One cool feature it had that I'm surprised I haven't really seen other turn-based games do is a "play by email" option similar to correspondence chess, except you're just emailing save files back and forth and the game makes importing/exporting the save files via email a bit more streamlined.
jll29 1y ago
> More importantly, separating the dynamic linker from the C library itself would allow multiple versions of libc to coexist, eliminating a major source of compatibility issues. This is exactly how Windows handles it, which is one of the reasons Windows maintains such strong binary compatibility. You can still run decades-old Windows software today because Microsoft doesn’t force everything to be tied to a single, ever-changing libc. One of the questions of multiple versions on the same box is what about security issues of those older versions...
- moron4hire 1y ago
  What about the security issues of the old operating systems people keep around for their mission critical software that has no upgrade path?
  - pmarreck 1y ago
    isn't this tu quoque?
- m463 1y ago
  > Windows maintains such strong binary compatibility The REAL reason windows maintains binary compatibility is because it is commercial and nobody ships source code. In fact, many applications ship a whole boatload of DLLs, which I think is the commercial equivalent of static linking.
  - bruce511 1y ago
    It kinda is, and kinda isn't. It is, in the sense that the package is bigger, and the package ships "everything it needs". It isn't in the sense that those parts can be updated independently as long as the DLL interface is backward compatible. For example, I ship OpenSSL dlls with my app. Which means swapping in a later (compatible) OpenSSL can be done (by the user if necessary.) If I'm making a small utility I static link it - and I still use utilities daily I compiled 25 years ago. Obviously those dynamically link to KERNEL etc, but Microsoft has insane levels of compatibility there. And perhaps that's the fundamental root of the issue. Windows has one provider, very committed to the longevity of software. Linux, well, does not. That's OK. The world has room for different philosophies. And each one will have strengths and weaknesses.
knowaveragejoe 1y ago
No discussion of Cosmopolitan or APE?
- h4ck_th3_pl4n3t 1y ago
  Software and binary formats with a 100% chance of being blocked as malware doesn't count.
mrbluecoat 1y ago
> shipping software on Linux That's a surprisingly hard nut to crack when containers won't work for your use case. We found https://github.com/silitics/rugix https://github.com/silitics/rugix to work well in that situation.
brie22 1y ago
Absolutely ridiculous that scrollbars are disabled for this page. The kids are too cool for such pedestrian web parts! Grow up and stop forcing your shitty UX on people. Edit: Found it - it's black on black - even worse!
pmarreck 1y ago
Does Nix have any relevance here?
mikepavone 1y ago
So of the 3 glibc issues they link - one is about the format of symbol information in the actual ELF binaries which is only an issue if you are not using the standard libc functions for looking up symbols for some strange reason - one is an issue that impacts targeting a lower version of glibc from a higher one which is a configuration that was never supported (though usually fails more loudly) - the last one is a security policy change which is legitimately an ABI break, but mostly impacts programs that have their execstack flags set incorrectly glibc actually goes to a fair bit of effort to be compatible with old binaries unlike most of the rest of the Linux userspace. The binaries I built for my side project back in 2015 (BlastEm 0.3.0) still work fine on modern Linux and they dynamically link against glibc. This is just a hobby project, not a piece of professional software, and a build from before this JangaFX company even existed works fine. I find it really bizarre when people talk about Linux binary compat and then complain entirely about glibc rather than the sort of problems that the manylinux project has had to deal with. glibc is one of the few parts of userspace you can depend on. Yes, setting up your toolchain to build against an old glibc on a modern distro is a bit annoying. Sure, if you do something sufficiently weird you might find yourself outside what glibc considers part of their stable ABI. But from where I sit, it works pretty well.
- fooblaster 1y ago
  Is there a good summary of the problems manylinux had to deal with?
  - mikepavone 1y ago
    The PEP-600 [0] Rationale section touches on this a bit. The basic problem is that there are things beyond glibc that would be nice to use from the environment for a number of reasons (security updates, avoiding clashes between multiple wheels that depend on the same lib, etc.), but since most libs outside of glibc and libstdc++ don't really have an ABI policy and the distros don't necessarily have a policy on what libraries are guaranteed to be present you sort of have to guess and hope for the best. While the initial list in PEP-513 [1] was a pretty good guess, one of the libraries chosen (libcrypt.so.1) got dropped in Fedora 30 and replaced with an ABI incompatible version. Crypto libraries are an example of something that's actually important to keep up to date so I find this rather unfortunate. [0] https://peps.python.org/pep-0600/ https://peps.python.org/pep-0600/ [1] https://peps.python.org/pep-0513/ https://peps.python.org/pep-0513/
ryao 1y ago
> GLIBC is an example of a "system library" that cannot be bundled with your application because it includes the dynamic linker itself. This linker is responsible for loading other libraries, some of which may also depend on GLIBC—but not always. Running WordPerfect on modern Linux is done by shipping both of those components: https://github.com/taviso/wpunix https://github.com/taviso/wpunix
h4ck_th3_pl4n3t 1y ago
Isn't the compatibility promise what syscalls essentially are? Historically, they (almost) never break and they are steadily incremented to prevent overlaps in differences of parameters. As WASI is also implementing syscalls for WASM, I'd argue that the binary format doesn't really matter as long as it's using the same syscalls in the end. I understand this topic is mostly focussing on glibc/muslc problems, but if you want to develop stable software, use CGo free Go binaries. They likely will run in 10 years the same way they do today. C ABI compatibility is a mess on linux mostly because upstream maintainers don't give a damn about semantic versioning. Just take a look at the SO file headers, and how they differ from upstream "semantic" versions of the library. As long as shared objects differ in versions due to breaking changes, and as long as the C ecosystem doesn't enforce correct versioning, this won't change.
inftech 1y ago
I think the problem is that people treat Linux as OS instead of just a kernel. You should assume every Linux distro is a different OS so when you are shipping your app for Linux you're actually shipping your app for Debian, Fedora, Ubuntu etc
- rollcat 1y ago
  That's the sane way to tackle it. If you're the vendor, just target the top N (whatever value of N you can cope with). I don't mean disrespect towards people running Alpine (hi), Arch, or Gentoo, but you wouldn't be running these distros if you aren't ready to handle their quirks. TFA mostly talks about binary compat. Even if you can get away with statically linking everything, you still have to cope with the mess that is userspace fragmentation: <https://tailscale.com/blog/sisyphean-dns-client-linux https://tailscale.com/blog/sisyphean-dns-client-linux> So yeah, supporting the top N gets you approximately sqrt(N/(N+1))% of the way. (Assuming desktop Linux market share is about 1%.)
  - HideousKojima 1y ago
    >I don't mean disrespect towards people running Alpine (hi), Arch, or Gentoo, but you wouldn't be running these distros if you aren't ready to handle their quirks. You'd think so, but you'd be surprised how many gamers switching from Windows to Linux get recommended Arch and Manjaro
    - skyyler 1y ago
      I see Bazzite recommended more than Arch these days, but Arch is confusingly a very popular choice. I wonder if Arch being the basis for SteamOS is responsible.
- thewebguyd 1y ago
  This is definitely the root of it. Linux isn't the OS, your distro is. Somewhere along the way we seem to have collectively forgotten that the distro, via it's package manager, is responsible for delivering software, ensuring compatibility, and integrating it all - at least for open source software. That was one of Debian's biggest selling points "back in the day" - that they had a huge repository and big team of maintainers packaging everything under the sun so it works on Debian. Now we have app image, flatpak, snap, etc. All means of getting developers to ship binaries directly to users, bypassing the package managers, instead of shipping code to distros, but we shouldn't expect devs to ship to the 100s of different OSes in this ecosystem. They ship the code, distros pick it up (or don't) and make it work on their distro. The shift to direct-to-user binaries doesn't make sense for OSS. It should be packaged by the distro, not the dev (of course, the devs can help with packaging if they are so inclined). Obviously this doesn't apply to proprietary software, but like you said, you can just target on of the LTS enterprise distros and get a well-defined, stable base (for 10 years with Ubuntu and Red hat).
ecef9-8c0f-4374 1y ago
I bought 3 linux games on DVD between 2006 and 2016. I stopped buying linux games and instead started again buying windows games. Because there is no easy way to run them. On the other hand I can just run myst1993 and most of windows games without much hustle via wine. Wine is linux only stable abi
- SSLy 1y ago
  > Wine is linux only stable abi https://blog.hiler.eu/win32-the-only-stable-abi/ https://blog.hiler.eu/win32-the-only-stable-abi/
- guappa 1y ago
  As if a windows game from 2006 is going to run on windows 11 :)
  - surfaceofthesun 1y ago
    I was able to run Age of Empires 2 on Windows 11. It didn't like running on an ultrawide, but it worked well enough at smaller resolutions.
    - guappa 1y ago
      a 3d game…
      - SSLy 1y ago
        I've just installed and run Star Wars Empire at war (2005) from steam, it seems to run just fine in WQHD. The UI isn't even blurry
        guappa 1y ago
        On steam? Games that get updated are updated? Do you have any more groundbreaking revelations?
everdrive 1y ago
There’s no reason to believe that widespread Linux adoption would not irrevocably damage the experience. It would end up looking something like Android. The same thing happened to the internet. It was ruined by mass adoption. You can call this gatekeeping, but nothing positive has come from getting most of the world onto social media.
- thewebguyd 1y ago
  I'm inclined to agree with you here. As much as I'd love to see native apps from some big vendors (Adobe, in particular) if what it takes to get there is mass adoption of Linux on the desktop, I'm not sure I want it. In a way, we already have it in the form of ChromeOS, and I certainly don't want ChromeOS or Android to be the default experience, or for the community to start to cater to that model. All of the things Linux would need to become to get Windows level marketshare would strip away what makes it appealing in the first place, in particular, full user control. I don't want a nanny desktop, but that's exactly what it would become. Linux can stay niche for those who appreciate it as it is.
pizlonator 1y ago
This is a really great article about binary compatibility! I disagree with their idea for fixing it by splitting up glibc. I think it's a bad idea because it doesn't actually fix the problems that lead to compat breakage, and it's bad because it's harder than it seems. They cite these compat bugs as part of their reasoning for why glibc should be split up: - https://sourceware.org/bugzilla/show_bug.cgi?id=29456 https://sourceware.org/bugzilla/show_bug.cgi?id=29456 - https://sourceware.org/bugzilla/show_bug.cgi?id=32653 https://sourceware.org/bugzilla/show_bug.cgi?id=32653 - https://sourceware.org/bugzilla/show_bug.cgi?id=32786 https://sourceware.org/bugzilla/show_bug.cgi?id=32786 I don't see how a single one of these would be fixed by splitting up glibc. If their proposed libdl or libthread were updated and had one of these regressions, it would cause just as much of a bug as if a monolithic libc updates with one of these regressions. So, splitting up glibc wouldn't fix the issue. Also, splitting up glibc would be super nasty because of how the threading, loading, and syscall parts of libc are coupled (some syscalls are implemented with deep threading awareness, like the setxid calls, threads need to know about the loader and vice-versa, and other issues). I think the problem here is how releases are cut. In an ideal world, glibc devs would have caught all three of those bugs before shipping 2.41. Big corpos like Microsoft manage that by having a binary compatibility team that runs All The Apps on every new version of the OS. I'm guessing that glibc doesn't have (as much of) that kind of process.
grumpy_coder 1y ago
Related video about releasing games on linux, i.e. dlopen() all the things https://www.youtube.com/watch?v=MeMPCSqQ-34 https://www.youtube.com/watch?v=MeMPCSqQ-34
advisedwang 1y ago
golang seems to manage being 100% statically linked (unless you go out of your way, by using plugins). It just forgoes the stuff that the article mentions glibc doing dynamic linking for, and does it the simple and direct way.
- okanat 1y ago
  What do you use golang for? If your answer is only web-dev and if you don't touch anything GUI, system libraries, PAM or VPNs, you simply haven't compiled something complex enough.
DrNosferatu 1y ago
Why don’t the authors release a distro that actually implements their own proposal? Say, based on (perhaps on pieces of) musl, to be more straightforward to get things going?
- okanat 1y ago
  This is simply too complex. There are quite a bit dependencies on Glibc and other GNU-based behavior. What you're actually asking from them is creating a new Android or Chrome OS i.e. recreating the entire userspace. "Just split up musl" will not work. How the system headers work needs to be redefined. This is basically a complete redefinition of entire API of all Linux programs and libraries. Which library will supply unistd.h or any of the sys/whatever.h? Any interdependencies? I think even POSIX itself can be hostile against this. So the fix will have to deviate a lot from that (that's probably why Windows cannot be made POSIX easily either).
DrNosferatu 1y ago
Zig CC + Vulkan would solve most of these issues: 1. Zig's toolchain statically links with musl libc, producing binaries that depend only on the Linux kernel syscall ABI, not any specific glibc version. 2. This eliminates all the symbol versioning nightmares (`GLIBC_2.xx not found`) that plague distribution across different Linux systems. 3. Vulkan provides a standardized, vendor-neutral GPU API that's supported across all modern graphics hardware, eliminating driver-specific dependencies. 4. The resulting binary is completely self-contained - no external library dependencies, no version mismatches, no containerization needed. 5. You get forward AND backward compatibility - the binary will run on both ancient and cutting-edge distros without modification. The only real limitation is for NVIDIA CUDA-specific workloads, where you'd still need their proprietary stack. Furthermore, for those who prefer a higher-level API, Zig CC + WebGPU offers similar benefits with a simpler programming model, though with slightly less mature implementations and a possible small performance overhead.
dp-hackernews 1y ago
I'm surprised nobody has mentioned the "nix" package manager from NixOS as an alternative here...

201 points by generichuman 1y ago

This article missed a critical point which is "the right way" to select a glibc ABI version: see binutils ld documentation, second part of the page related to VERSION support. This must include glibc internal symbols. This will allow to craft ELF binaries on a modern distro which will run on "older" distros. This is critical for games and game engines. There is an significant upfront only-once work in order to select an "old" glibc ABI. The quick and dirty alternative being having a toolchain configured to link with an "old" glibc on the side. This article missed the -static-libstdc++ critical option for c++ applications (the c++ ABI is hell on earth), but did not miss the -static-libgcc and the dynamic loading of system interface shared libs.

api 1y ago
Is there a reason glibc can't just do a better job at keeping some legacy symbols around? It's not like it's big stuff. They're things like legacy string functions. We're talking a few kilobytes of code in most cases. The Linux kernel goes to a lot of effort to not break user space, at least for non-exotic core features and syscalls. It seems like a lot of user-space in Linux-land does not make the same effort. It's particularly bad when it's the C library doing this, since that's at the center of the dependency graph for almost everything.
- Conan_Kudo 1y ago
  The legacy symbols are still around. It's just that the linker won't use them by default.
- FitCodIa 1y ago
  > a lot of user-space in Linux-land does not make the same effort I believe that, what the article misses is that glibc is maintained and extended with an entirely different community and development model. Windows remains compatible over decades because Microsoft (a) is the sole distributor, and (b) puts an immense effort towards backwards compat. In Linux userspace, it's simply a non-goal across distributions. If you want to ship a binary for a particular distro, you need to build the binary on / for that distro; even within a distro, a major release bump (or especially a major release downgrade) may break a binary. Ultimately, it's a consequence of Conway’s Law. Microsoft is the sole distributor of Windows, so they can enforce compatibilty with an iron fist, and there are people working for Microsoft whose pay depends on said compatibility. With "Linux" in general, there is no common authority to appeal to, and (again) most vendors don't even promise a seamless userspace upgrade path from one major release to another. This is unfixable; it will never change -- as long as independent parties are permitted to distribute different operating systems yet call them all "Linux". Ship multiple binaries, or distribute the source code (and let users build it). EDIT: you'll notice that "ship multiple binaries" is what distros (especially commercial distros) do. They maintain separate branches, backport fixes to old branches, and employ software maintenance engineers to focus on this kind of work. If you want to target multiple major releases, this is what you have to do, too. If you (as a commercial ISV) target a commercial distro with long-term support, and can convince your users to use / license the same distro, you'll have a good, stable development experience. You only need to port like once every decade, when you jump major releases. The Linux user base / the Linux market is fragmented; that's the whole goal. The technical proliferation / inconsistency is just a consequence. Unless you take away the freedom of users to run their own flavors of "Linux", there won't be a uniform Linux target. In a way, it's funny to even expect otherwise. Why do you expect to ship the same binaries when the foundations are diverse, with no standardization authority that all Linux distributors recognize as such? And even POSIX is an API spec, not an ABI spec. And, any authority that controls binary aspects will immediately accrue political capital. This is exactly what shouldn't happen in Linux. The fact that anyone can fork (or start) a distro, and contribute to the chaos, is good for freedom.
  - cozzyd 1y ago
    The big Linux distros (EL, Ubuntu) in fact have the opposite incentive, to get proprietary vendors to rather their distribution specifically.
    - myk9001 1y ago
      This theory would check out if a proprietary vendor could easily get away with shipping a single binary package for all supported versions of, say, Ubuntu. Having to build and maintain a binary packege separately for each version of the same distro probably isn't that appealing to them.
      - cozzyd 1y ago
        For Ubuntu they would only target LTS releases, most likely. On EL it's easier, now you would just support 2 or 3 of EL7, EL8, and EL9. As an example of something I use, Xfdtd only officially supports one version of Ubuntu and 2 versions of EL https://www.remcom.com/system-requirements#xfdtd-system-requirements https://www.remcom.com/system-requirements#xfdtd-system-requ... In practice, it wasn't too hard to get it running on EL9 or Fedora either...
  - yjftsjthsd-h 1y ago
    > If you (as a commercial ISV) target a commercial distro with long-term support, and can convince your users to use / license the same distro, you'll have a good, stable development experience. You only need to port like once every decade, when you jump major releases. If things go well, it's even better than that: If you target ex. RHEL 8, there's a very good chance that your binaries will work on RHEL 9 and a decent shot at RHEL 10 with zero changes (though of course you should test all versions you want to work). And the same for Ubuntu 20.04/22.04/24.04/... and Debian/SUSE/whatever. Backwards incompatibilities can happen, but within a single stable distro they're not super common so the lazy ISV can probably only really port forward after more than a decade if they really want. (Incidentally, this isn't a hypothetical: I once had the joy of working on software that targeted RHEL 5, and those binaries ran on RHEL/CentOS 7 without any problems.)
  - 1y ago
- AshamedCaptain 1y ago
  The problem is the opposite: they are trying to run executables built using a newer glibc in a system that has an older glibc. glibc keeps all the old function definitions since practically forever. Frankly, I do not understand who would think glibc symbols themselves would be the challenge in this case. Even if you statically link glibc there's zero guarantee the syscalls will be present in the older Linux (cue .ABI-tag failures). Or even damn ELF format changes (e.g. gnu-style hashes). The simple solution is to build in the older Linux (&glibc). In my long experience with ancient binaries, glibc has almost never been the problem, and its ability to _run_ ancient binaries is all but excellent; even Linux is more of a problem than glibc is (for starters paths to everywhere in /proc, /sys change every other half-decade).
  - forrestthewoods 1y ago
    > executables built and using a newer glibc It’s an abomination that Linux uses system libraries when building. Catastrophically terrible and stupid decision. It should be trivial for any program to compile and specify any arbitrary previous version of glibc as the target. Linux got this so incredibly wildly wrong. It’s a shame.
    - AshamedCaptain 1y ago
      Ehm... you _can_ use the older toolchain even in the newer "Linux", in the same way you have to use a older Visual Studio to target Windows 8 from Windows 10.
      - myk9001 1y ago
        So maybe the difference here is between can and actually doing it? I tend to stay on the oldest supported version of Windows until they drop support and haven't ever seen an application that wouldn't run because it's built on a newer version of Windows.
      - forrestthewoods 1y ago
        I am saying that compiler toolchains on Linux should never ever under any circumstances ever rely on anything on the system for compiling. Compiling based on the system global version of glibc is stupid, bad, wrong, and Linus should be ashamed for letting it happen. It should be trivial for Windows to cross-compile for Linux for any distro and for any ancient version of glibc. It is not trivial. Here is a post describing the mountain range of bullshit that Zig had to move to enable trivial cross-compile and backwards targeting. https://andrewkelley.me/post/zig-cc-powerful-drop-in-replacement-gcc-clang.html https://andrewkelley.me/post/zig-cc-powerful-drop-in-replace... Linux is far and away the worst offender out of Linux, Mac, and Windows. By leaps and bounds.
AndyKelley 1y ago
One of the features Zig provides is ability to target any glibc version. See https://github.com/ziglang/glibc-abi-tool/ https://github.com/ziglang/glibc-abi-tool/ for more details on how this is solved.
- forrestthewoods 1y ago
  You’re my hero for this. One of my side projects is building a toolchain to enable C++ cross-compile using the Zig header/source libs. I didn’t love Zig as a Clang++ replacement because it has a bit too much magic. And it might go ahead? But the underlying library code is a God send
vlovich123 1y ago
Which works if you use binutils ld. Does it work with mold or gold? And then how do you use this with languages other than c++/c like Go or Rust?
- rjsw 1y ago
  I thought that Go invoked syscalls directly instead of going through libc.
  - jfuafdfwa 1y ago
    It's Go's best feature, a simple CGO_ENABLED=0 gives you freedom from the tyranny of libc.
- sylware 1y ago
  Rust toolchain has already a bug since 2015: it cannot link statically libgcc, namely it does not have "-static-libgcc" option since 2015. I got the bug with "TinyGlade" video game (extremely good BTW), which is written in rust, and with the dev we hit that bug. Namely... better have a libgcc with the right ABI... and I can tell you, this has a been a HUGE issue since valve started to distribute games more than a decade ago.
  - vlovich123 1y ago
    As in you have to build on an old distro? Or something else?
IX-103 1y ago
Could you point to an example of a project doing it the right way? I'm a little unclear on the type of incantation needed for the "version script".

superkuh 1y ago

As an end user I often patch the glibc version incompatibility away with https://github.com/corsix/polyfill-glibc https://github.com/corsix/polyfill-glibc $ ./polyfill-glibc --target-glibc=2.17 /path/to/my-program This often leads to discovering new version incompatibilities in other libs. But as the article says others usually can be statically compiled.

vlovich123 1y ago
Wow. This is huge. This patch doesn’t risk weird runtime errors due to missing polyfills or anything? Does it risk polyfilling irrelevant functions that aren’t referenced and bloating?
therein 1y ago
Very cool. Glad someone finally made this.

Jeaye 1y ago

I don't understand why they don't just statically link their binaries. First, they said this: > Even if you managed to statically link GLIBC—or used an alternative like musl—your application would be unable to load any dynamic libraries at runtime. But then they immediately said they actually statically link all of their deps aside from libc. > Instead, we take a different approach: statically linking everything we can. If they're statically linking everything other than libc, then using musl or statically linking glibc will finish the job. Unless they have some need for loading share libs at runtime which they didn't already have linked into their binary (i.e. manual dlopen), this solves the portability problem on Linux. What am I missing (assuming I know of the security implications of statically linked binaries -- which they didn't mention as a concern)?

masfuerte 1y ago
Various things including name (DNS) resolution rely on dynamic linking.
- Jeaye 1y ago
  Are you saying that a statically linked binary cannot make an HTTP request to `google.com` because it would be unable to resolve the domain name? There are entire distros, like alpine, built on musl. I find this very hard to believe.
  - masfuerte 1y ago
    The configuration of DNS resolution on Linux is quite complicated [1]. Musl just ignores all that. You can build a distro that works with musl, but a static musl binary dropped into an arbitrary Linux system won't necessarily work correctly. [1]: https://news.ycombinator.com/item?id=43451861 https://news.ycombinator.com/item?id=43451861
  - Spivak 1y ago
    You have to bundle your own resolver into your application. But here's the rub, users expect your application to respect nsswitch which requires loading shared libs which execute arbitrary code. How Go handles this is somewhat awkward. They parse /etc/nsswitch and decide if they can cheat and use their own resolver based on what modules they see[1]. Otherwise they farm out to cgo to go through glibc. [1] They're playing with fire here because you can't really assume to know for sure how the module 'dns' behaves. A user could replace the lib that backs it with their own that resolves everything to zombo.com. It would be one thing if nsswitch described behavior which was well defined and could be emulated but it doesn't, it specifies a specific implementation.

bee_rider 1y ago

Really the title should be something like “the difficulty of releasing binaries on Linux and how to work around it.” It isn’t really an atrocious state, the goal of a distro should be to get useful software into the repos. Software not distributed in an open source format doesn’t really help there.

James_K 1y ago

Here's a thought: just distribute source code. ABI issues should be mostly fixed. Most computers can compile source code fast enough for the user not to notice and cache the results so that it's never a problem again. If you want optimised code, you can do a source to source optimisation then zip and minify the file. You could compile such a library to approximately native speeds without much user-end lag using modern JIT methods, and maybe even run LTO in a background thread so that the exectuables outdo dynamically linked ones.

RedShift1 1y ago
I hate compiling. 9/10 something goes wrong. Sometimes I can fix it, other times I just abandon the effort and use something else. These days I just use packages or docker images and if that doesn't work out I'm moving on, ain't nobody got time for this. You really can't expect people who just want to use their computers and don't even know what a compiler is to get involved in a process like that.
- ses1984 1y ago
  No, end users need not get involved, it could/should be handled by the operating system.
  - HKH2 1y ago
    Gentoo? Compiling big packages takes ages.
    - adrian_b 1y ago
      More than 20 years, with a single-core Pentium 4, it could take indeed something like 3 days of continuous compilation to compile an entire Gentoo distribution, in order to have a personal computer with every application that one might want. However, already after the appearance of the first dual-core AMD Athlon64, 20 years ago, that time could be reduced to not much more than a half of day, while nowadays, with a decent desktop CPU from 5 years ago, most Gentoo packages can be compiled and installed in less than a minute. There are only a few packages whose compilation and installation can take a noticeable time, of up to tens of minutes, depending on the chosen options and on the number of cores of the CPU, e.g. Firefox, LibreOffice, LLVM. There is only a single package whose compilation may take ages unless you have an expensive CPU and enough memory per core: Google Chromium (including its derivatives that use the same code base).

lukeh 1y ago

Windows having multiple C libraries has its own pain points, in particular it's difficult to ship binary libraries that return allocated memory to their consumer (you either need to have the library consumer allocate the memory, which probably explains why so many Win32 APIs have this behaviour, or allow alloc/free functions to be registered). Not to mention different C libraries having their own file handle, TLS, etc state. Unsurprisingly Microsoft now ships the Universal CRT (UCRT) as part of Windows.

int_19h 1y ago
Having consumer allocate the memory where possible has other benefits in that the consumer can use more efficient allocation techniques (stack, arenas etc), and given that Win32 API design goes all the way back to the earliest version of Windows running on very resource-limited PCs, that would be the main reason why. However, in cases where this wasn't consistently feasible - e.g. COM - Windows instead mandates the use of a common API to manage memory: CoGetMalloc etc.

BwackNinja 1y ago

There is no distinction between system and program libraries in Linux. We used to pretend there was one before usrmigration, but that was never good to take seriously. The distro as packager model ensures that everything is mixed together in the filesystem and is actively hostile to external packaging. Vendoring dependencies or static linking improves compatibility by choosing known working versions, but decreases incentive and ability for downstream (or users) to upgrade those dependencies. The libc stuff in this article is mostly glibc-specific, and you'd have fewer issues targeting musl. Mixing static linking and dlopen doesn't make much sense, as said here[1] which is an interesting thread. Even dns resolution on glibc implies dynamic linking due to nsswitch. Solutions like Snap, Flatpak, and AppImage work to contain the problem by reusing the same abstractions internally rather than introducing anything that directly addresses the issue. We won't have a clean solution until we collectively abandon the FHS for a decentralized filesystem layout where adding an application (not just a program binary) is as easy as extracting a package into a folder and integrates with the rest of the system. I've worked on this off and on for a while, but being so opinionated makes everything an uphill battle while accepting the current reality is easy. [1] https://musl.openwall.narkive.com/lW4KCyXd/static-linking-and-dlopen https://musl.openwall.narkive.com/lW4KCyXd/static-linking-an...

veqq 1y ago
> adding an application (not just a program binary) is as easy as extracting a package into a folder and integrates with the rest of the system I have fond memories of installed Warlords Battle Cry 3, Warcraft 3, AOE2 etc. directories on flash drives, distributed to 20+ kids in high school (all using the same key). Good days.
- HideousKojima 1y ago
  Way off topic but you just reminded me of all the time I spent playing Warlords 3 (not Warlords Battlecry 3, the original Warlords games were turn-based). One cool feature it had that I'm surprised I haven't really seen other turn-based games do is a "play by email" option similar to correspondence chess, except you're just emailing save files back and forth and the game makes importing/exporting the save files via email a bit more streamlined.

jll29 1y ago

> More importantly, separating the dynamic linker from the C library itself would allow multiple versions of libc to coexist, eliminating a major source of compatibility issues. This is exactly how Windows handles it, which is one of the reasons Windows maintains such strong binary compatibility. You can still run decades-old Windows software today because Microsoft doesn’t force everything to be tied to a single, ever-changing libc. One of the questions of multiple versions on the same box is what about security issues of those older versions...

moron4hire 1y ago
What about the security issues of the old operating systems people keep around for their mission critical software that has no upgrade path?
- pmarreck 1y ago
  isn't this tu quoque?
m463 1y ago
> Windows maintains such strong binary compatibility The REAL reason windows maintains binary compatibility is because it is commercial and nobody ships source code. In fact, many applications ship a whole boatload of DLLs, which I think is the commercial equivalent of static linking.
- bruce511 1y ago
  It kinda is, and kinda isn't. It is, in the sense that the package is bigger, and the package ships "everything it needs". It isn't in the sense that those parts can be updated independently as long as the DLL interface is backward compatible. For example, I ship OpenSSL dlls with my app. Which means swapping in a later (compatible) OpenSSL can be done (by the user if necessary.) If I'm making a small utility I static link it - and I still use utilities daily I compiled 25 years ago. Obviously those dynamically link to KERNEL etc, but Microsoft has insane levels of compatibility there. And perhaps that's the fundamental root of the issue. Windows has one provider, very committed to the longevity of software. Linux, well, does not. That's OK. The world has room for different philosophies. And each one will have strengths and weaknesses.

knowaveragejoe 1y ago

No discussion of Cosmopolitan or APE?

h4ck_th3_pl4n3t 1y ago
Software and binary formats with a 100% chance of being blocked as malware doesn't count.

mrbluecoat 1y ago

> shipping software on Linux That's a surprisingly hard nut to crack when containers won't work for your use case. We found https://github.com/silitics/rugix https://github.com/silitics/rugix to work well in that situation.

brie22 1y ago

Absolutely ridiculous that scrollbars are disabled for this page. The kids are too cool for such pedestrian web parts! Grow up and stop forcing your shitty UX on people. Edit: Found it - it's black on black - even worse!

pmarreck 1y ago

Does Nix have any relevance here?

mikepavone 1y ago

So of the 3 glibc issues they link - one is about the format of symbol information in the actual ELF binaries which is only an issue if you are not using the standard libc functions for looking up symbols for some strange reason - one is an issue that impacts targeting a lower version of glibc from a higher one which is a configuration that was never supported (though usually fails more loudly) - the last one is a security policy change which is legitimately an ABI break, but mostly impacts programs that have their execstack flags set incorrectly glibc actually goes to a fair bit of effort to be compatible with old binaries unlike most of the rest of the Linux userspace. The binaries I built for my side project back in 2015 (BlastEm 0.3.0) still work fine on modern Linux and they dynamically link against glibc. This is just a hobby project, not a piece of professional software, and a build from before this JangaFX company even existed works fine. I find it really bizarre when people talk about Linux binary compat and then complain entirely about glibc rather than the sort of problems that the manylinux project has had to deal with. glibc is one of the few parts of userspace you can depend on. Yes, setting up your toolchain to build against an old glibc on a modern distro is a bit annoying. Sure, if you do something sufficiently weird you might find yourself outside what glibc considers part of their stable ABI. But from where I sit, it works pretty well.

fooblaster 1y ago
Is there a good summary of the problems manylinux had to deal with?
- mikepavone 1y ago
  The PEP-600 [0] Rationale section touches on this a bit. The basic problem is that there are things beyond glibc that would be nice to use from the environment for a number of reasons (security updates, avoiding clashes between multiple wheels that depend on the same lib, etc.), but since most libs outside of glibc and libstdc++ don't really have an ABI policy and the distros don't necessarily have a policy on what libraries are guaranteed to be present you sort of have to guess and hope for the best. While the initial list in PEP-513 [1] was a pretty good guess, one of the libraries chosen (libcrypt.so.1) got dropped in Fedora 30 and replaced with an ABI incompatible version. Crypto libraries are an example of something that's actually important to keep up to date so I find this rather unfortunate. [0] https://peps.python.org/pep-0600/ https://peps.python.org/pep-0600/ [1] https://peps.python.org/pep-0513/ https://peps.python.org/pep-0513/

ryao 1y ago

> GLIBC is an example of a "system library" that cannot be bundled with your application because it includes the dynamic linker itself. This linker is responsible for loading other libraries, some of which may also depend on GLIBC—but not always. Running WordPerfect on modern Linux is done by shipping both of those components: https://github.com/taviso/wpunix https://github.com/taviso/wpunix

h4ck_th3_pl4n3t 1y ago

Isn't the compatibility promise what syscalls essentially are? Historically, they (almost) never break and they are steadily incremented to prevent overlaps in differences of parameters. As WASI is also implementing syscalls for WASM, I'd argue that the binary format doesn't really matter as long as it's using the same syscalls in the end. I understand this topic is mostly focussing on glibc/muslc problems, but if you want to develop stable software, use CGo free Go binaries. They likely will run in 10 years the same way they do today. C ABI compatibility is a mess on linux mostly because upstream maintainers don't give a damn about semantic versioning. Just take a look at the SO file headers, and how they differ from upstream "semantic" versions of the library. As long as shared objects differ in versions due to breaking changes, and as long as the C ecosystem doesn't enforce correct versioning, this won't change.

inftech 1y ago

I think the problem is that people treat Linux as OS instead of just a kernel. You should assume every Linux distro is a different OS so when you are shipping your app for Linux you're actually shipping your app for Debian, Fedora, Ubuntu etc

rollcat 1y ago
That's the sane way to tackle it. If you're the vendor, just target the top N (whatever value of N you can cope with). I don't mean disrespect towards people running Alpine (hi), Arch, or Gentoo, but you wouldn't be running these distros if you aren't ready to handle their quirks. TFA mostly talks about binary compat. Even if you can get away with statically linking everything, you still have to cope with the mess that is userspace fragmentation: <https://tailscale.com/blog/sisyphean-dns-client-linux https://tailscale.com/blog/sisyphean-dns-client-linux> So yeah, supporting the top N gets you approximately sqrt(N/(N+1))% of the way. (Assuming desktop Linux market share is about 1%.)
- HideousKojima 1y ago
  >I don't mean disrespect towards people running Alpine (hi), Arch, or Gentoo, but you wouldn't be running these distros if you aren't ready to handle their quirks. You'd think so, but you'd be surprised how many gamers switching from Windows to Linux get recommended Arch and Manjaro
  - skyyler 1y ago
    I see Bazzite recommended more than Arch these days, but Arch is confusingly a very popular choice. I wonder if Arch being the basis for SteamOS is responsible.
thewebguyd 1y ago
This is definitely the root of it. Linux isn't the OS, your distro is. Somewhere along the way we seem to have collectively forgotten that the distro, via it's package manager, is responsible for delivering software, ensuring compatibility, and integrating it all - at least for open source software. That was one of Debian's biggest selling points "back in the day" - that they had a huge repository and big team of maintainers packaging everything under the sun so it works on Debian. Now we have app image, flatpak, snap, etc. All means of getting developers to ship binaries directly to users, bypassing the package managers, instead of shipping code to distros, but we shouldn't expect devs to ship to the 100s of different OSes in this ecosystem. They ship the code, distros pick it up (or don't) and make it work on their distro. The shift to direct-to-user binaries doesn't make sense for OSS. It should be packaged by the distro, not the dev (of course, the devs can help with packaging if they are so inclined). Obviously this doesn't apply to proprietary software, but like you said, you can just target on of the LTS enterprise distros and get a well-defined, stable base (for 10 years with Ubuntu and Red hat).

ecef9-8c0f-4374 1y ago

I bought 3 linux games on DVD between 2006 and 2016. I stopped buying linux games and instead started again buying windows games. Because there is no easy way to run them. On the other hand I can just run myst1993 and most of windows games without much hustle via wine. Wine is linux only stable abi

SSLy 1y ago
> Wine is linux only stable abi https://blog.hiler.eu/win32-the-only-stable-abi/ https://blog.hiler.eu/win32-the-only-stable-abi/
guappa 1y ago
As if a windows game from 2006 is going to run on windows 11 :)
- surfaceofthesun 1y ago
  I was able to run Age of Empires 2 on Windows 11. It didn't like running on an ultrawide, but it worked well enough at smaller resolutions.
  - guappa 1y ago
    a 3d game…
    - SSLy 1y ago
      I've just installed and run Star Wars Empire at war (2005) from steam, it seems to run just fine in WQHD. The UI isn't even blurry
      - guappa 1y ago
        On steam? Games that get updated are updated? Do you have any more groundbreaking revelations?

everdrive 1y ago

There’s no reason to believe that widespread Linux adoption would not irrevocably damage the experience. It would end up looking something like Android. The same thing happened to the internet. It was ruined by mass adoption. You can call this gatekeeping, but nothing positive has come from getting most of the world onto social media.

thewebguyd 1y ago
I'm inclined to agree with you here. As much as I'd love to see native apps from some big vendors (Adobe, in particular) if what it takes to get there is mass adoption of Linux on the desktop, I'm not sure I want it. In a way, we already have it in the form of ChromeOS, and I certainly don't want ChromeOS or Android to be the default experience, or for the community to start to cater to that model. All of the things Linux would need to become to get Windows level marketshare would strip away what makes it appealing in the first place, in particular, full user control. I don't want a nanny desktop, but that's exactly what it would become. Linux can stay niche for those who appreciate it as it is.

pizlonator 1y ago

This is a really great article about binary compatibility! I disagree with their idea for fixing it by splitting up glibc. I think it's a bad idea because it doesn't actually fix the problems that lead to compat breakage, and it's bad because it's harder than it seems. They cite these compat bugs as part of their reasoning for why glibc should be split up: - https://sourceware.org/bugzilla/show_bug.cgi?id=29456 https://sourceware.org/bugzilla/show_bug.cgi?id=29456 - https://sourceware.org/bugzilla/show_bug.cgi?id=32653 https://sourceware.org/bugzilla/show_bug.cgi?id=32653 - https://sourceware.org/bugzilla/show_bug.cgi?id=32786 https://sourceware.org/bugzilla/show_bug.cgi?id=32786 I don't see how a single one of these would be fixed by splitting up glibc. If their proposed libdl or libthread were updated and had one of these regressions, it would cause just as much of a bug as if a monolithic libc updates with one of these regressions. So, splitting up glibc wouldn't fix the issue. Also, splitting up glibc would be super nasty because of how the threading, loading, and syscall parts of libc are coupled (some syscalls are implemented with deep threading awareness, like the setxid calls, threads need to know about the loader and vice-versa, and other issues). I think the problem here is how releases are cut. In an ideal world, glibc devs would have caught all three of those bugs before shipping 2.41. Big corpos like Microsoft manage that by having a binary compatibility team that runs All The Apps on every new version of the OS. I'm guessing that glibc doesn't have (as much of) that kind of process.

grumpy_coder 1y ago

Related video about releasing games on linux, i.e. dlopen() all the things https://www.youtube.com/watch?v=MeMPCSqQ-34 https://www.youtube.com/watch?v=MeMPCSqQ-34

advisedwang 1y ago

golang seems to manage being 100% statically linked (unless you go out of your way, by using plugins). It just forgoes the stuff that the article mentions glibc doing dynamic linking for, and does it the simple and direct way.

okanat 1y ago
What do you use golang for? If your answer is only web-dev and if you don't touch anything GUI, system libraries, PAM or VPNs, you simply haven't compiled something complex enough.

DrNosferatu 1y ago

Why don’t the authors release a distro that actually implements their own proposal? Say, based on (perhaps on pieces of) musl, to be more straightforward to get things going?

okanat 1y ago
This is simply too complex. There are quite a bit dependencies on Glibc and other GNU-based behavior. What you're actually asking from them is creating a new Android or Chrome OS i.e. recreating the entire userspace. "Just split up musl" will not work. How the system headers work needs to be redefined. This is basically a complete redefinition of entire API of all Linux programs and libraries. Which library will supply unistd.h or any of the sys/whatever.h? Any interdependencies? I think even POSIX itself can be hostile against this. So the fix will have to deviate a lot from that (that's probably why Windows cannot be made POSIX easily either).

DrNosferatu 1y ago

Zig CC + Vulkan would solve most of these issues: 1. Zig's toolchain statically links with musl libc, producing binaries that depend only on the Linux kernel syscall ABI, not any specific glibc version. 2. This eliminates all the symbol versioning nightmares (`GLIBC_2.xx not found`) that plague distribution across different Linux systems. 3. Vulkan provides a standardized, vendor-neutral GPU API that's supported across all modern graphics hardware, eliminating driver-specific dependencies. 4. The resulting binary is completely self-contained - no external library dependencies, no version mismatches, no containerization needed. 5. You get forward AND backward compatibility - the binary will run on both ancient and cutting-edge distros without modification. The only real limitation is for NVIDIA CUDA-specific workloads, where you'd still need their proprietary stack. Furthermore, for those who prefer a higher-level API, Zig CC + WebGPU offers similar benefits with a simpler programming model, though with slightly less mature implementations and a possible small performance overhead.

dp-hackernews 1y ago

I'm surprised nobody has mentioned the "nix" package manager from NixOS as an alternative here...