9 ms·
The Holy Grail of Linux Binary Compatibility: Musl and Dlopen
- einpoklum 8mo agoThis seems interesting even regardless of go. Is it realistic to create an executable which would work on very different kinds of Linux distros? e.g. 32-bit and 64-bit? Or maybe some general framework/library for building an arbitrary program at least for "any libc"?
- deleted 8mo ago[deleted]
- quesomaster9000 8mo agoCosmopolitan goes one further: [binaries] that runs natively on Linux + Mac + Windows + FreeBSD + OpenBSD + NetBSD + BIOS on AMD64 and ARM64 https://justine.lol/cosmopolitan/ https://justine.lol/cosmopolitan/
- oguz-ismail2 8mo ago>Linux if you configure binfmt_misc >Windows if you disable Windows Defender >OpenBSD only older versions
- account42 8mo agoYeah while APE is a technically impressive trick, these issues far outweigh the minor convenience of having a single binary. For most cases, a single Windows exe that targets the oldest version you want to support plus a single Glibc binary that dynamically links against the oldest version you want to support and so on is still the best option.
- yjftsjthsd-h 8mo ago>> Linux > if you configure binfmt_misc I don't think that's a requirement, it'll just fall back to the shell script bootstrap without it.
- oguz-ismail2 8mo agoOn some distros, yes. On others it'll fire up Wine for whatever reason
- yjftsjthsd-h 8mo agoOkay, yes, if you configure binfmt_misc for WINE and not APE then PE-compatible binaries will get run with WINE and not APE. That feels unfair.
- oguz-ismail2 8mo ago>if you configure binfmt_misc for WINE It came preconfigured on Ubuntu 20.04 and 22.04, don't know about newer versions.
- dontdoxxme 8mo agoClearly a joke if it uses the .lol tld.
- account42 8mo agoIt's his personal website lol.
- hyperbolablabla 8mo agoJustine identifies as a woman.
- hofrogs 8mo ago"identifies as" is an unnecessarily dismissive choice of words. She is a woman.
- hyperbolablabla 8mo agoMy statement was a fact, and in my opinion not politically loaded, yet respectful to Justine. I chose my words carefully.
- sambuccid 8mo agoAppimage exists that packs linux applications into a single executable file that you just download and open. It works on most linux distros
- greyw 8mo agoI vaguely remember that Appimage-based programs would fail for me because of fuse and glibc symbol version incompatibilties. Gave up them afterwards. If I need to tweak dependencies might as well deal with the packet manager of my distro.
- deleted 8mo ago[deleted]
- iberator 8mo agoYup. Just compile it as static executable. Static binaries are very undervalued imo.
- flohofwoe 8mo agoThe "just" is doing a lot of heavylifting here (as detailed in the article), especially for anything that's not a trivial cmdline tool.
- qznc 8mo agoAck. I went down that rabbit hole to "just" build a static Python: https://beza1e1.tuxen.de/python_bazel.html https://beza1e1.tuxen.de/python_bazel.html
- Xraider72 8mo agoIn my experience it seems to be an issue caused by optimizations in legacy code that relied on dlopen to implement a plugin system, or help with startup, since you could lazy load said plugins on demand and start faster. If you forego the requirement of a runtime plugin system, is there anything realistically preventing greenfield projects from just being fully statically linked, assuming their dependencies dont rely on dlopen ?
- flohofwoe 8mo agoIt becomes tricky when you need to use system DLLs like X11 or GL/Vulkan (so you need to use the 'hacks' described in the article to work around that) - the problem is that those system DLLs then bring a dynamically linked glibc into the process, so suddenly you have two C stdlibs running side by side and the question is whether this works just fine or causes subtle breakage under the hood (e.g. the reason why MUSL doesn't implement dlopen). E.g. in my experience: command line tools are fine to link statically with MUSL, but as soon as you need a window and 3D rendering it's not worth the hassle.
- account42 8mo ago
- amelius 8mo agoIs there a tool that takes an executable, collects all the required .so files and produces either a static executable, or a package that runs everywhere?
- mdavid626 8mo agoYou can "package" all .so files you need into one file, there are many tools which do this (like a zip file). But you can't take .so files and make one "static" binary out of them.
- fc417fc802 8mo agoWell not a static binary in the sense that's commonly meant when speaking about static linking. But you can pack .so files into the executable as binary data and then dlopen the relevant memory ranges.
- mdavid626 8mo agoYes, that's true. But I'm always a bit sceptical about such approaches. They are not universal. You still need glibc/musl to be the same on the target system. Also, if you compile againt new glibc version, but try to run on old glibc version, it might not work. These are just strange and confusing from the end users' perspective.
- toast0 8mo ago> But I'm always a bit sceptical about such approaches. They are not universal. You still need glibc/musl to be the same on the target system. Also, if you compile againt new glibc version, but try to run on old glibc version, it might not work. Why would you include most of your dynamic libraries but not your libc? You could still run into problems if you (or your libraries) want to use syscalls that weren't available on older kernels or whatever.
- mdavid626 8mo ago
- Meneth 8mo agoThat seems mostly useful for proprietary programs. I don't like it.
- seba_dos1 8mo agoYeah, in my 20 years of using and developing on GNU/Linux the only binary compatibility issues I experienced that I can think of now were related to either Adobe Flash, Adobe Reader or games. Adobe stuff is of the kind that you'd prefer to not exist at all rather than have it fixed (and today you largely can pretend that it never existed already), and the situation for games has been pretty much fixed by Steam runtimes. It's fine that some people care about it and some solutions are really clever, but it just doesn't seem to be an actual issue you stumble on in practice much.
- whizzter 8mo agoThe solution to games is to load Windows games instead of Linux binaries. Basically the way for the year of the Linux desktop is to become Windows.
- seba_dos1 8mo agoThese days Linux binaries usually work fine, even older ones, and when they don't the reason is that they often don't get the same attention as their Windows counterparts.
- paddim8 8mo agoProbably because your distro purposefully keeps software out of date because it is too fragile otherwise. I don't think that is reasonable at all for desktop use.
- seba_dos1 8mo agoArch?...
- juliangmp 8mo ago
- athrowaway3z 8mo agoI'd never heard of detour. That's a pretty cool hack.
- ckbkr10 8mo agothey were prominent in game hacking 2005ish windows made hooking into game code much easier than before
- sidewndr46 8mo agoAren't all DLLs on the Windows platform compiled with an unusual instruction at the start of each function? This makes it possible to somehow hot patch the DLL after it is already in memory
- leeter 8mo agoI believe you're thinking of the x86 Hotpatching hook[1], which doesn't exist on x86-64[2] (in the same form, it uses a x86-64 safe one). [1] https://devblogs.microsoft.com/oldnewthing/20110921-00/?p=9583 https://devblogs.microsoft.com/oldnewthing/20110921-00/?p=95... [2] https://devblogs.microsoft.com/oldnewthing/20221109-00/?p=107373 https://devblogs.microsoft.com/oldnewthing/20221109-00/?p=10...
- sidewndr46 8mo agoyes, that's it. Thanks for clarifying
- mgaunard 8mo agoIt's funny how people insist on wanting to link everything statically when shared libraries were specifically designed to have a better alternative. Even worse is containers, which has the disadvantage of both.
- vv_ 8mo agoIt's easier to distribute software fully self-contained, if you ignore the pain of statically linking everything together :)
- diego_sandoval 8mo agoWhat's the pain?
- throwaway2046 8mo agoI'm guessing the pain of fighting the various build systems that insist on dynamic linking, sometimes against the user's explicit wishes.
- vv_ 8mo agoMost open source software tooling Were designed to be dynamically linked. It is non-standard to statically link things together, which causes various random issues.
- fc417fc802 8mo agoDynamic linking exists to make a specific set of tradeoffs. Neither better nor worse than static linking in the general sense.
- arghwhat 8mo agoDynamic libraries have been frowned upon since their inception as being a terrible solution to a non-existent problem, generally amplifying binary sizes and harming performance. Some fun quotes of quite notable characters on the matter here: https://harmful.cat-v.org/software/dynamic-linking/ https://harmful.cat-v.org/software/dynamic-linking/ In practice, a statically linked system is often smaller than a meticulously dynamically linked one - while there are many copies of common routines, programs only contain tightly packed, specifically optimized and sometimes inlined versions of the symbols they use. The space and performance gain per program is quite significant. Modern apps and containers are another issue entirely - linking doesn't help if your issue is gigabytes of graphical assets or using a container base image that includes the entire world.
- Rochus 8mo agoSo what we need is essentially a "libc virtualization". But Musl is only available on Linux, isn't it? Cosmopolitan (https://github.com/jart/cosmopolitan https://github.com/jart/cosmopolitan) goes further and is available also on Mac and Windows, and it uses e.g. SIMD and other performance related improvements. Unfortunately, one has to cut through the marketing "magic" to find the main engineering value; stripping away the "polyglot" shell-script hacks and the "Actually Portable Executable" container (which are undoubtedly innovative), the core benefit proposition of Cosmopolitan is indeed a platform-agnostic, statically-linked C standard (plus some Posix) library that performs runtime system call translation, so to say "the Musl we have been waiting for".
- sidewndr46 8mo agoAt the rate things are going we'll need a container virtualization layer as well, a docker for docker if you know what I mean
- weebull 8mo agoIf you're using dlopen(), you're just reimplementing the dynamic linker.
- 112233 8mo agothat's cute, but dismissive, sort of like "if you use popen(), you are reimplementing bash". There is so much hair in ld nobody wants to know about — parsing elf, ctors/dtors, ...
- pilif 8mo agoIsn't this asking for the exact trouble musl wanted so spare you from by disabling dlopen()?
- netbioserror 8mo agoI've been statically linking Nim binaries with musl. It's fantastic. Relatively easy to set up (just a few compiler flags and the musl toolchain), and I get an optimized binary that is indistinguishable from any other static C Linux binary. It runs on any machine we throw it at. For a newer-generation systems language, that is a massive selling point.
- zoobab 8mo agoI have an idea for a static linux distribution based on musl, with either an Alpine rebuild or Gentoo-musl: http://stalinux.wikidot.com http://stalinux.wikidot.com The documentation to make static binary with GLibc is sparce for a reason, they don't like static binaries.
- cb321 8mo agoYeah. I've been doing this for almost 10 years now. It's not APE/cosmopolitan (which also "kinda works" with Nim but has many lowest common denominator platform support issues, e.g. posix_fallocate). However, it does let you have very cross-Linux portable binaries. Maybe beyond Linux. Some might appreciate a concrete instance of this advice inline here. For `foo.nim`, you can just add a `foo.nim.cfg`: @if gcc: gcc.exe = "musl-gcc" gcc.linkerexe = "musl-gcc" passL = "-static -s" @end There is also a "NimScript" syntax you could use a `foo.nims`: if defined gcc: # nim.cfg runs faster than NimScript switch "gcc.exe" , "musl-gcc" switch "gcc.linkerexe", "musl-gcc" switch "passL" , "-static -s"
- ValdikSS 8mo ago`dlopen`'ing system libraries is an "easy" hack to try to maintain compatibility with wide variety of libraries/ABIs. It's barely used (I know only of SDL, Small HTTP Server, and now Godot). Without dlopen (with regular dynamic linking), it's much harder to compile for older distros, and I doubt you can easily implement glibc/musl cross-compatibility at all in general. Take a look what Valve does in a Steam Runtime: - https://gitlab.steamos.cloud/steamrt/steam-runtime-tools/-/blob/main/docs/pressure-vessel.md - https://gitlab.steamos.cloud/steamrt/steam-runtime-tools/-/blob/main/subprojects/libcapsule/doc/Capsules.txt
- surajrmal 8mo agoBinary comparability extends beyond the vide that runs in your process. These days a lot of functionality occurs by way of IPC which has a variety of wire protocols depending on the interface. For instance there is dbus, Wayland protocols, varlink, etc. Both the wire protocol, and the APIs built on top need to retain backwards comparability to ensure Binary compatibility. Otherwise you're not going to be able to run on various different Linux based platforms arbitrarily. And unlike the kernel, these userspace surfaces do not take backwards compatibility nearly as important. It's also much more difficult to target a subset of these APIs that are available on systems that are only 5 years old. I would argue API endpoints on the web have less risk here (although those break all the time as well)
- leni536 8mo agoDo I get this right that this effectively dlopens glibc (indirectly) into an executable that is statically linked to musl? How can the two runtimes coexist? What about malloc/free? AFAIK both libc's allocators take ownership of brk, that can't be good. What about malloc/free across the dynamic library interface? There are certainly libraries that hand out allocated objects and expect the user to free them, but that's probably uncommon in graphics.
- deleted 8mo ago[deleted]
- Splizard 8mo agoYou have to tell musl to use mmap instead of brk. You're right that it doesn't work in all cases but as long as you switch TLS on calls (and callbacks), at least with a project the size of Godot, you can approach a workable solution.
- dunder_cat 8mo agoRelated discussion (the actual project is mentioned in the issue): "Detour: Dynamic linking on Linux without Libc" https://news.ycombinator.com/item?id=45740241 https://news.ycombinator.com/item?id=45740241
- aspbee555 8mo agoI managed to get this combo going not too long ago with my musl rust app but I found that even after it all was compiled and loading the lib it did not function properly because the library I loaded still depended on libc functions. even with everything compiled into a huge monolithic musl binary it couldn't find something graphics related I eventually decided to keep the tiny musl app and make a companion app in a secondary process as needed (since the entire point of me compiling musl was cross platform linux compatibility/stability)
- tuhgdetzhh 8mo agoThe best binary compatibily you can get on Linux is via Wine. Source: https://blog.hiler.eu/win32-the-only-stable-abi/ https://blog.hiler.eu/win32-the-only-stable-abi/
- nektro 8mo agoyou'll likely be interested in [1] and [2] for further reading [1]: https://github.com/ziglang/zig/issues/7240 https://github.com/ziglang/zig/issues/7240 [2]: https://www.youtube.com/watch?v=pq1XqP4-qOo https://www.youtube.com/watch?v=pq1XqP4-qOo