8 ms·

I get why people don't bother replacing the default allocator from musl all the time (it's there, convenient). But in an application whose purpose is to be FAST

• vlovich123 2mo ago
  It’s a kernel bug. While I agree libc allocators suck for no good reason, it seems like this work of equally likely hit other application code including mimalloc and glibc.
  • Orphis 2mo ago
    Sure, but the problem isn't that the bug exists, is that it has surfaced in a performance application using musl and exerting code paths in a slow allocator. mallocng should not be used at all.
    • tripflag 2mo ago
      it's a tradeoff; mallocng does have benefits as well -- for example it uses less than half the amount of RAM compared to mimalloc and glibc for certain Python workloads. While yes it is a bit slower, I prefer mallocng in many cases.
      • Orphis 2mo ago
        ripgrep is there to be FAST. Trading any speed to improve memory efficiency over a longer period of time for a process with a short life-time doesn't make sense in this case. It's also a development tool. If your development machine is having RAM issues because it's doing a grep, you have bigger problems to solve. As for other workloads that might use less RAM with mallocng compared to other performant ones, I'm curious to know about the magnitudes we're talking about. From 10MB to 20MB or from 100MB to 2GB? How was the speed of the program? Was there any multithreading involved?
        • tripflag 2mo ago
          For context, the Python workload was a general-purpose fileserver with file indexing and image thumbnailing, largely IO-bound. Switching from mallocng to mimalloc resulted in a slight speedup (150% of baseline), but over twice the memory usage, going from 250 to 670 MiB, primarily for thumbnailing. Some pathological cases (libvips calling imagemagick to decode heif) was a 10x multiplier. There was multithreading, and yes, mallocng visibly became a bottleneck beyond 5 threads. However already at 3 threads there was diminishing returns for both allocators, so this was not an issue in this case. The speed gain was hardly noticeable in practice since the program was already plenty fast, but the additional memory usage was a very real inconvenience.
          • danudey 2mo ago
            Is that memory increase seen in peak usage or average/idle usage? For file servers, I could be willing to give up 350 MB of memory temporarily for large-scale indexing and thumbnailing operations if it doesn't happen often and makes the file serving/browsing more responsive.
            • 3eb7988a1663 2mo ago
              On the other hand, plenty of NAS/file server systems are going to have paltry CPU and memory. Running slower with a more modest memory foot print might be the only way to keep the system stable.
          • jcupitt 2mo ago
            libvips shouldn't be calling imagemagick for heif decode, it has a nice one built in. Unless you were using a very old libvips! I've found jemalloc works best for long running libvips processes, fwiw.
      • 3eb7988a1663 2mo ago
        I imagine there is no free lunch in the allocator space, but a series of trade-offs. Pick your poison on implementation which is going to be sub-optimal for some subset of use cases. Without a hugely compelling reason to switch, going with the default is reasonable.
    • inigyou 2mo ago
      Ah, no, the problem is that the bug exists. Another, separate, problem, noticed by people investigating this problem, is that ripgrep on musl uses a serializing allocator in the hot path.
  • karel-3d 2mo ago
    It's maybe a kernel bug.
• masklinn 2mo ago
  > I get why people don't bother replacing the default allocator from musl all the time (it's there, convenient). ripgrep actually sets jemalloc as global allocator when built for 64b musl: https://github.com/BurntSushi/ripgrep/blob/435f59fc4b43af3ab32f34d53fa34978f393fe52/crates/core/main.rs#L20-L41 https://github.com/BurntSushi/ripgrep/blob/435f59fc4b43af3ab...
  • Orphis 2mo ago
    That's good! But I wonder why it wasn't then enabled for that configuration or why the override wasn't "global" enough and the default allocator was still partly used.
    • tialaramex 2mo ago
      Rust doesn't get to override musl internals. Ripgrep uses opendir, a POSIX library feature implemented in musl to look at er, directories. The stack trace suggests we blew up when Rust's std::sys::fs::unix::readdir internal detail called opendir, and it in turn allocated. On Linux it would be possible for ripgrep to talk directly to the kernel via documented system calls without libc, but that wouldn't work on any other popular OS.
      • inigyou 2mo ago
        I assume it already doesn't work on Windows. At some point you have to define your compatibility boundary. And high performance often coincides with mediocre compatibility.
        • galangalalgol 2mo ago
          A good example is go. On linux you can use a from scratch image fairly easily because it only uses syscalls. But for windows or mac the moving target wasn't maintainable so they link against shared objects. Linus enforcing the don't break userspace rule is what made that possible. That definitely has tradeoffs. At some point relibc or something similar will allow the same (stably) for rust. But using posix as that compatibility boundary gets you a much larger set of OS and only occasionally has a performance penalty. Often, a posix api tuned to the kernel is more performant. Musl is an exception precisely because it makes an openbsd-esque trade of performance for simple small attack surface.
• deleted 2mo ago
  [deleted]
• mort96 2mo ago
  The way most programs achieve being fast is by re-using allocations. You don't need a fast allocator if you don't allocate. Nothing of what ripgrep does inherently requires frequent allocations.
  • entrope 2mo ago
    The ISO C definition of opendir() requires an allocation in practice because it returns a DIR* and it's bad practice for the library to arbitrarily limit how many of those an application has at a time. Maybe a C library could preallocate several DIRs and only use the heap when those are exhausted, but this ripgrep use case (lots of threads running in parallel on a large tree) would still be likely to trigger that.
    • mort96 2mo ago
      There is no ISO C definition of opendir(), that's a POSIX thing. But yeah, those DIR objects which opendir() returns a pointer to are probably some kind of heap allocated. But we're talking about a system call involving the filesystem here. The time taken by the allocator is gonna be dwarfed by the time spent in the syscall even with the slowest allocator. Ripgrep reads through every byte of most files and matches it against a regex. That is the tight inner loop where you want to avoid allocations. Not that you even need to call the C functions. I don't think ripgrep would gain anything from it, but the syscall to read directory contents just needs a file descriptor.
      • entrope 2mo ago
        I think ripgrep does avoid allocations in that innermost loop. That's why this report says all the crashes are associated with opendir() and its memory allocation: the opendir() call is outside of ripgrep's innermost loop but still runs often enough to trigger the race condition.
        • mort96 2mo ago
          And that's why musl's allocator's performance isn't a huge concern for ripgrep
• zeuxcg 2mo ago
  If you look at the sigsegv stack, the allocation comes from opendir which is in musl libc as well. The allocation override mechanism used in Rust doesn’t replace the allocator process-wide; it merely replaces the allocator Rust code talks to. It does seem like ripgrep should probably avoid using opendir from libc if it allocates using an allocator with a global lock though.
• searealist 2mo ago
  There is a real tradeoff: - The musl allocator is only slow with multi-threading. - Almost all other allocators have trouble reclaiming memory when using multi-threading. This often results in multiples more RSS than single threaded or musl's allocator. Agree with you on mimalloc. It can even be configured to be aggressive in memory reclaim at the cost of performance.
• lrvick 2mo ago
  We use musl+mimalloc by default for our entire production operating systems: https://stagex.tools https://stagex.tools
  • NewJazz 2mo ago
    Interesting. Not sure if you are aware, but your toolchain looks nearly identical to another linux distribution.. You might have luck looking for patches there if you ever need them. https://chimera-linux.org/about/#alternative-userland https://chimera-linux.org/about/#alternative-userland
    • lrvick 2mo ago
      While we are building for a dramatically different threat model, we use quite a few of their llvm patches and are very thankful for their work, and alpine making llvm/musl a viable option for us.
• potus_kushner 2mo ago
  this use of musl's mallocng actually lead to the discovery of a kernel bug, thanks to its hardening. without it, this might have gone unnoticed for months, silently corrupting memory in the meantime.