6 ms·

It is a testament to the complete failure of the GNU/Linux userland that something like this seems at all attractive to spend time on (or, it seems, LLM tokens)

It is a testament to the complete failure of the GNU/Linux userland that something like this seems at all attractive to spend time on (or, it seems, LLM tokens). Actually, scratch that, because Windows and macOS have historically struggled with ABI compatibility as well (macOS less so, due to not caring about backward compatibility in the first place). How did we get to the point where people feel they need to go to the length of embedding an ELF loader in their binary (!!) rather than just linking with glibc?

pg83 29d ago
Glibc has a terrible history of binary incompatibility. If that's so hard to believe, try running binaries built on one distribution on other distributions. Linux has two stable ABIs: the kernel ABI for static programs, and, ironically, WINE.
- vlovich123 29d ago
  I haven’t heard of this and I don’t think you’re right. Glibc, for all its faults, as a general rule does backward compatibility well. The problem is if you compile against a newer glibc (common in CI by default) and try to run on a distro with an older (common in the wild). If your CI uses an older glibc you should be fine AFAIK.
  - pg83 29d ago
    https://bugzilla.redhat.com/show_bug.cgi?id=638477 https://bugzilla.redhat.com/show_bug.cgi?id=638477 is the most "famous" example. There are also much less well-known "little things" that regularly pop up here and there. > If your CI uses an older glibc you should be fine AFAIK. In any case, my binaries work not only under glibc, but also under Alpine, and (work in progress) under android/bionic.
    - vlovich123 29d ago
      Not sure what you’re trying to show with that bug report but it’s not a case of cross distro glibc issues. If I read correctly it’s a vanilla behavioral change that exposed preexisting UB in flash. Not sure how the comments about alpine or bionic relate either to my claim that cross distro glibc is fine.
      - pg83 29d ago
        It depends on how we define the ABI. I see it as a set of client-visible invariants that they rely on. In my world, glibc changed the client's visible invariants, breaking the client. The client works on one glibc-based host, but not on another. What is this if not "a case of cross-distro glibc issues?" Overall, both of our points of view on compatibility were discussed well in that thread; we probably shouldn't repeat ourselves. :)
        account42 29d ago
        ABI is not "whatever happens to work with this distro" but "what programs that comply with the API contract compile to". Overlapping memcpy arguments is an API contract violation and thus not something covered by the ABI either. This distinction is the entire reason why C has a separate memmove function. You can't just make up your own imaginary ABI contract and then blame the system when it doesn't fulfill it. That's going to result in self-inflicted plain on any OS.
        pg83 29d ago
        It's so good that Linus thinks differently!
        ninkendo 28d ago
        Even Linus’s comments in that bug hedge things a bit: > and that if people depend on interfaces we exported having side effects that weren't intentional, we try to fix things so that they still work unless there is a major reason not to. (Emphasis mine.) In the Flash case, they changed the direction memcpy works in, in a way that didn’t actually improve any performance (Linus did benchmarks to show it) and Linus’s point was that there’s no upside to the change, and only downsides, so on the whole it makes no sense to keep the changed memcpy. You could imagine plenty of scenarios where existing behavior truly is broken, and show that no popular software depends on the brokenness (which is why distros are in a great place to do these kinds of smoke tests, they have thousands of packages they can test), and make a judgement call. A “zero regressions no matter what” policy is impossible in practice due to Hyrum’s law, at some point you have to use common sense and draw the line somewhere sensible.
    - __turbobrew__ 29d ago
      > Quite frankly, I find your attitude to be annoying and downright stupid. - Linus Torvalds That bug report was a good read.
- diabllicseagull 29d ago
  according to appimage recommendations as long as you build against glibc with an earlier version than the system it's run on it should be fine. https://docs.appimage.org/reference/best-practices.html https://docs.appimage.org/reference/best-practices.html I hear you about WINE though.
  - Natalia724 29d ago
    [flagged]
  - deleted 29d ago
    [deleted]
- jezek2 29d ago
  This is not true. Glibc supports symbol versioning. You can use it to select old versions of used symbols. The result is a binary that can work on 20 year old distros the same as on the latest, compiled with latest compiler and Glibc. You can also compile using old distro and old Glibc to get similar effect. Though you would miss the advances of the newer compilers.
  - pg83 29d ago
    https://news.ycombinator.com/item?id=49358538 https://news.ycombinator.com/item?id=49358538
- cryptonector 28d ago
  That irony is delicious. Build to WIN32 APIs to ship on Linux.
diabllicseagull 29d ago
I'm mostly taken aback all the solutions devised to go around the issue, especially the container-based ones. I really disliked it when I grabbed the flatpak version of Blender only to find out that it can't have HIP support. (they might have fixed it by now but you get the point)
wmf 29d ago
Linux loves to leave papercuts unfixed or undocumented for decades. The solution is to build against an older version of glibc but no one tells you that or how to do it.
- pg83 29d ago
  It's a pretty poor solution, to be honest. 1) Why should I limit myself to the available APIs? 2) Not just glibc. For example, if I build against the latest libstdc++, it will automatically support the more recent glibc. And pinning the old libstdc++ -well, that's just not a good idea.
- emidoots 29d ago
  Luckily Zig makes this quite easy to do. In Mach[0] we are able to just `zig build -Dtarget=x86_64-linux-gnu.2.28` to build GUI apps against an ~8 year old glibc version for maximum compatibility. This is possible because Zig allows for targeting most glibc versions out of the box with its cross-compilation support. [0] https://machengine.org https://machengine.org
deleted 29d ago
[deleted]
krupan 29d ago
Complete failure is strong words when lots and lots of Linux boxes are running just fine. I think the disconnect is mostly people that can't decide if they want a stable distro or a rolling release distro. Most everyone uses a stable distro because it's stable, but then the want some up-to-date software that isn't ore-built for their (crusty old) stable distro and they get annoyed. My solution was to finally give in and embrace a rolling release distro (I use arch, btw). If there isn't a package for something I want, it's not hard to build something myself because all my build tools, kernel, and libs are up to date. Other reasons to want static linking is to distribute proprietary software with no source code available. Linux certainly does not cater to that scenario and I suppose some might call that a complete failure ¯ \ _ ( ツ ) _ / ¯
jeroenhd 29d ago
> How did we get to the point where people feel they need to go to the length of embedding an ELF loader in their binary (!!) rather than just linking with glibc? Most Linux distros have been built around the ability to compile their software together in a large repository, from source, so this was rarely ever an issue. Proprietary distribution or executing binaries from the internet like on Windows just wasn't really a common issue. The problem arises when you start combining distros (glibc and MUSL for instance) or if you try to do the Windows model of sharing software. Historically, projects just compiled different versions for different distros. When doing static compilation, just targetting an old version of glibc (which is generally forward compatible) also works. You can hack your way into using software like this (or rather, have an LLM hack its way in) but I don't think any real distro actually cares. This issue exists in a quite small space where people are trying to use proprietary software built for glibc in MUSL environments for whatever reason, and the usual compatibility tricks don't work. It's a niche use case for most Linux distros. It's not a "complete failure" of the GNU/Linux userland, it's the result of a couple of proprietary components not having MUSL builds available, or MUSL-based distributions not including libraries people want. I'd like glibc to change so that these hacks aren't necessary for these use cases anymore, but it's not really a problem in practice for the vast majority of Linux use cases.
- pg83 29d ago
  I disagree that this is a niche problem. Yes, on the one hand, each specific distribution doesn't have this problem because it can pick up everything it needs. But for us, independent developers of small programs, the problem is truly stark: we can't afford to build our programs for every distribution. And we can't afford to waste time navigating all the idiosyncrasies of various package repositories, both technical and political (not all repositories allow easy access). And we can't count on someone else packaging our work until we become incredibly popular.
  - skydhash 28d ago
    > But for us, independent developers of small programs, the problem is truly stark: we can't afford to build our programs for every distribution. Most people use the major distributions like Ubuntu, Debian, Fedora, Arch, alpine linux, void linux,… A build machine with VMs can easily take care of those. > And we can't afford to waste time navigating all the idiosyncrasies of various package repositories, both technical and political (not all repositories allow easy access). Most (if not all) package managers allow for custom repositories. No need to get access to the official ones.
account42 29d ago
> How did we get to the point where people feel they need to go to the length of embedding an ELF loader in their binary (!!) rather than just linking with glibc? Due to FUD, mostly. Sane people do just link with an old enough version of glibc.

pg83 29d ago

Glibc has a terrible history of binary incompatibility. If that's so hard to believe, try running binaries built on one distribution on other distributions. Linux has two stable ABIs: the kernel ABI for static programs, and, ironically, WINE.

vlovich123 29d ago
I haven’t heard of this and I don’t think you’re right. Glibc, for all its faults, as a general rule does backward compatibility well. The problem is if you compile against a newer glibc (common in CI by default) and try to run on a distro with an older (common in the wild). If your CI uses an older glibc you should be fine AFAIK.
- pg83 29d ago
  https://bugzilla.redhat.com/show_bug.cgi?id=638477 https://bugzilla.redhat.com/show_bug.cgi?id=638477 is the most "famous" example. There are also much less well-known "little things" that regularly pop up here and there. > If your CI uses an older glibc you should be fine AFAIK. In any case, my binaries work not only under glibc, but also under Alpine, and (work in progress) under android/bionic.
  - vlovich123 29d ago
    Not sure what you’re trying to show with that bug report but it’s not a case of cross distro glibc issues. If I read correctly it’s a vanilla behavioral change that exposed preexisting UB in flash. Not sure how the comments about alpine or bionic relate either to my claim that cross distro glibc is fine.
    - pg83 29d ago
      It depends on how we define the ABI. I see it as a set of client-visible invariants that they rely on. In my world, glibc changed the client's visible invariants, breaking the client. The client works on one glibc-based host, but not on another. What is this if not "a case of cross-distro glibc issues?" Overall, both of our points of view on compatibility were discussed well in that thread; we probably shouldn't repeat ourselves. :)
      - account42 29d ago
        ABI is not "whatever happens to work with this distro" but "what programs that comply with the API contract compile to". Overlapping memcpy arguments is an API contract violation and thus not something covered by the ABI either. This distinction is the entire reason why C has a separate memmove function. You can't just make up your own imaginary ABI contract and then blame the system when it doesn't fulfill it. That's going to result in self-inflicted plain on any OS.
        pg83 29d ago
        It's so good that Linus thinks differently!
        ninkendo 28d ago
        Even Linus’s comments in that bug hedge things a bit: > and that if people depend on interfaces we exported having side effects that weren't intentional, we try to fix things so that they still work unless there is a major reason not to. (Emphasis mine.) In the Flash case, they changed the direction memcpy works in, in a way that didn’t actually improve any performance (Linus did benchmarks to show it) and Linus’s point was that there’s no upside to the change, and only downsides, so on the whole it makes no sense to keep the changed memcpy. You could imagine plenty of scenarios where existing behavior truly is broken, and show that no popular software depends on the brokenness (which is why distros are in a great place to do these kinds of smoke tests, they have thousands of packages they can test), and make a judgement call. A “zero regressions no matter what” policy is impossible in practice due to Hyrum’s law, at some point you have to use common sense and draw the line somewhere sensible.
  - __turbobrew__ 29d ago
    > Quite frankly, I find your attitude to be annoying and downright stupid. - Linus Torvalds That bug report was a good read.
diabllicseagull 29d ago
according to appimage recommendations as long as you build against glibc with an earlier version than the system it's run on it should be fine. https://docs.appimage.org/reference/best-practices.html https://docs.appimage.org/reference/best-practices.html I hear you about WINE though.
- Natalia724 29d ago
  [flagged]
- deleted 29d ago
  [deleted]
jezek2 29d ago
This is not true. Glibc supports symbol versioning. You can use it to select old versions of used symbols. The result is a binary that can work on 20 year old distros the same as on the latest, compiled with latest compiler and Glibc. You can also compile using old distro and old Glibc to get similar effect. Though you would miss the advances of the newer compilers.
- pg83 29d ago
  https://news.ycombinator.com/item?id=49358538 https://news.ycombinator.com/item?id=49358538
cryptonector 28d ago
That irony is delicious. Build to WIN32 APIs to ship on Linux.

diabllicseagull 29d ago

I'm mostly taken aback all the solutions devised to go around the issue, especially the container-based ones. I really disliked it when I grabbed the flatpak version of Blender only to find out that it can't have HIP support. (they might have fixed it by now but you get the point)

wmf 29d ago

Linux loves to leave papercuts unfixed or undocumented for decades. The solution is to build against an older version of glibc but no one tells you that or how to do it.

pg83 29d ago
It's a pretty poor solution, to be honest. 1) Why should I limit myself to the available APIs? 2) Not just glibc. For example, if I build against the latest libstdc++, it will automatically support the more recent glibc. And pinning the old libstdc++ -well, that's just not a good idea.
emidoots 29d ago
Luckily Zig makes this quite easy to do. In Mach[0] we are able to just `zig build -Dtarget=x86_64-linux-gnu.2.28` to build GUI apps against an ~8 year old glibc version for maximum compatibility. This is possible because Zig allows for targeting most glibc versions out of the box with its cross-compilation support. [0] https://machengine.org https://machengine.org

deleted 29d ago

[deleted]

krupan 29d ago

Complete failure is strong words when lots and lots of Linux boxes are running just fine. I think the disconnect is mostly people that can't decide if they want a stable distro or a rolling release distro. Most everyone uses a stable distro because it's stable, but then the want some up-to-date software that isn't ore-built for their (crusty old) stable distro and they get annoyed. My solution was to finally give in and embrace a rolling release distro (I use arch, btw). If there isn't a package for something I want, it's not hard to build something myself because all my build tools, kernel, and libs are up to date. Other reasons to want static linking is to distribute proprietary software with no source code available. Linux certainly does not cater to that scenario and I suppose some might call that a complete failure ¯ \ _ ( ツ ) _ / ¯

jeroenhd 29d ago

> How did we get to the point where people feel they need to go to the length of embedding an ELF loader in their binary (!!) rather than just linking with glibc? Most Linux distros have been built around the ability to compile their software together in a large repository, from source, so this was rarely ever an issue. Proprietary distribution or executing binaries from the internet like on Windows just wasn't really a common issue. The problem arises when you start combining distros (glibc and MUSL for instance) or if you try to do the Windows model of sharing software. Historically, projects just compiled different versions for different distros. When doing static compilation, just targetting an old version of glibc (which is generally forward compatible) also works. You can hack your way into using software like this (or rather, have an LLM hack its way in) but I don't think any real distro actually cares. This issue exists in a quite small space where people are trying to use proprietary software built for glibc in MUSL environments for whatever reason, and the usual compatibility tricks don't work. It's a niche use case for most Linux distros. It's not a "complete failure" of the GNU/Linux userland, it's the result of a couple of proprietary components not having MUSL builds available, or MUSL-based distributions not including libraries people want. I'd like glibc to change so that these hacks aren't necessary for these use cases anymore, but it's not really a problem in practice for the vast majority of Linux use cases.

pg83 29d ago
I disagree that this is a niche problem. Yes, on the one hand, each specific distribution doesn't have this problem because it can pick up everything it needs. But for us, independent developers of small programs, the problem is truly stark: we can't afford to build our programs for every distribution. And we can't afford to waste time navigating all the idiosyncrasies of various package repositories, both technical and political (not all repositories allow easy access). And we can't count on someone else packaging our work until we become incredibly popular.
- skydhash 28d ago
  > But for us, independent developers of small programs, the problem is truly stark: we can't afford to build our programs for every distribution. Most people use the major distributions like Ubuntu, Debian, Fedora, Arch, alpine linux, void linux,… A build machine with VMs can easily take care of those. > And we can't afford to waste time navigating all the idiosyncrasies of various package repositories, both technical and political (not all repositories allow easy access). Most (if not all) package managers allow for custom repositories. No need to get access to the official ones.

account42 29d ago

> How did we get to the point where people feel they need to go to the length of embedding an ELF loader in their binary (!!) rather than just linking with glibc? Due to FUD, mostly. Sane people do just link with an old enough version of glibc.