9 ms·

Compile your software

46 points by wowi42 5y ago

powersnail 5y ago
Please label your bar graph axis, with units. It’s kind of counterproductive to look at a benchmark graph without knowing whether more or less is better.
- n8ta 5y ago
  Agreed. I thought this was an interesting post but was baffled by every y axis.
  - wowi42 5y ago
    Noted!
- yellow_lead 5y ago
  And better titles. "Performance" is confusing. I think he means "Performance of different base images for Redis"?
- wowi42 5y ago
  Thanks for the feedback. Let me find a better way to make it more clear!
prpl 5y ago
The argument here I suppose is live on the bleeding edge? Or swap compilers/allocators as necessary? Or use O3? I'm not entirely sure. It wraps up with: >>> COMPILE YOUR SOFTWARES. It's going to help you understand it, make your app faster, more secure (by removing the mail gateway of NGiNX for example), and it will show you which softwares are easily maintanable and reliable. It is PITA is keeping up with compiler changes and library changes of third party software. In this case, throwing in different malloc implementations in there too, not to mention different libc implementations. With all those variables for each component you deploy you're probably less likely to understand what's going in your software. Maybe for a small app with 2-3 extra pieces or something that needs to really be optimized this is good advice, but it sounds like a lot of work for significant footprints. It would be nice if the official docker images had some tags that were better optimized, within reason.
- l0b0 5y ago
  So much advice in software development presupposes that everyone has the same single goal and at least one inexhaustible resource. In this case, time: - To read and understand all relevant parts of the build pipelines for each performance-critical part of the infrastructure. In a relatively simple web application that could mean at least a web server, a caching framework, a database and a message bus. - To debug any build failures, which could be plentiful and hard to parse until you're very familiar with the build infrastructure for that particular piece of software. - To benchmark the build outputs in comparable ways with realistic configuration and inputs. - To repeat the above whenever any part of the stack changes significantly.
- ec109685 5y ago
  It also could introduce bugs given your company might be the only one earth with that combination of compiler, kernel and libraries.
  - coliveira 5y ago
    You mean, expose bugs. If the software doesn't support a certain combination, this is a bug in the software.
    - indianboy42 5y ago
      Is it required for open source software to support every combination of compiler, kernel, libc etc? We wouldn't say it has to support every version of any other dependency (not provided by system) It's definitely a nice to have, but I think it's also valid for some project to say we support the version we used in CI and for releases. On the other hand you could say being stuck on a specific version is skin to relying on implementation details, which is a code smell atleast.
      - wowi42 5y ago
        That's what people are doing with Alpine and Musl!
      - 5y ago
latenightcoding 5y ago
On my side projects I compile everything myself, but I do not completely agree with this post because Redis is one of easiest/fastest mainstream databases to compile, it can get very time consuming and the returns are not always there.
- wowi42 5y ago
  that's true, but without some crazy guy like me to check it, you would not know! :-)
Macha 5y ago
I wonder how x86-64-v3 for Arch/v2 for fedora in the near future will change this calculus. Currently you're basically compiling for a Core 2/Athlon 64 era chip, so there's clear wins to be had, but I wonder how much of the benefit can be had just by using software requiring Haswell/Zen1 at minimum
- binarybanana 5y ago
  -march=native will still give you better performance. It's not just about the instruction set, but also heuristics taking into account cache size, latencies, topology and other things. Intel for example has this quirk that aligning functions (and other jump targets) at 32 byte boundaries speeds up funct8 calls and jumps. I haven't tested it but I suspect you'd gain more from -mtune=native with the generic x86_64 target than -march=native. Some loops that can be autovectorized with AVX instructions will probably be faster though. But cache size especially is important for deciding if some optimization is beneficial or just leads to stalls due to thrashing.
mathfailure 5y ago
Why would I read some tosser?
- wowi42 5y ago
  Nice one, I never heard it!
- wowi42 5y ago
  https://www.larousse.fr/dictionnaires/francais/tosser/78590 https://www.larousse.fr/dictionnaires/francais/tosser/78590
  - Zuider 5y ago
    https://www.etymonline.com/search?q=tosser https://www.etymonline.com/search?q=tosser
Aeolun 5y ago
I don’t follow, where did this person compile anything? The only thing I saw him do was use pregenerated docker images?
- wowi42 5y ago
  Read the article. I compiled redis with Zig, O3 native, and mimalloc!
  - Aeolun 5y ago
    I did read the article, but that wasn’t immediately clear to me. Maybe something about the formatting of pre-generated docker images in the same line as the compiled versions? Anyway, thanks for the clarification!
brigandish 5y ago
The sad fact is that a lot of software is difficult to compile; isn't documented well, something that is worse for building; won't work well if installed in a non-standard way, whether that is final location, different supporting libs, or different platform; and can take a long time. I'm happy nowadays when I see there's a binary available, no mucking around with gcc/clang/llvm - just trying to work out which one, let alone which version! - no diving down a rabbit hole of compiling dependencies that then need other dependencies compiled… no deciphering Makefiles that were written in a way that only a C guru can grok, with no comments. Whatever the benefits are, I prefer sanity.
- Rendello 5y ago
  This is one thing where I see Rust (and probably Zig) making headway. A lot of the newer Rust software isn't in package repos yet, but I don't mind doing a Cargo build. It might take a while to compile, but it always seems to just work.
- lnxg33k1 5y ago
  https://hub.docker.com/r/gentoo/stage3 https://hub.docker.com/r/gentoo/stage3
- wowi42 5y ago
  > The sad fact is that a lot of software is difficult to compile; isn't documented well, something that is worse for building; won't work well if installed in a non-standard way, whether that is final location, different supporting libs, or different platform; and can take a long time. So are you ready to deploy in prod a software that is so hard to compile? > I'm happy nowadays when I see there's a binary available, no mucking around with gcc/clang/llvm - just trying to work out which one, let alone which version! - no diving down a rabbit hole of compiling dependencies that then need other dependencies compiled… no deciphering Makefiles that were written in a way that only a C guru can grok, with no comments. But that's my job, as SRE/DevOps/whatever new fancy name! > Whatever the benefits are, I prefer sanity. Sanity of having a very old software, with backported features that are only on this distrib? I prefer to trust the engineers from the software that I deploy.
binarybanana 5y ago
Other cool things you can do if you compile yourself is use features like auto parallelization[1]. I wouldn't recommend to enable it system wide because it causes issues with programs that fork() due to limitations in gcc's OpenMP library[2], but other than that it works pretty well. For example, I can fully load my 4C/8T CPU using 3 clang processes because compilation is magically spread over multiple threads. I've seen a "single threaded" program (qemu-img) suddenly start using more than a single core to convert disk images into other formats, leading to speedups. Also things like PGO/FDO in combination with workload specific profiling data can easily give you 10% or more if you are CPU bound. [1]: https://gcc.gnu.org/wiki/AutoParInGCC https://gcc.gnu.org/wiki/AutoParInGCC [2]: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=42624 https://gcc.gnu.org/bugzilla/show_bug.cgi?id=42624 (There was a patch to fix this, but it never got merged and doesn't apply to the current version any more, sadly)
b215826 5y ago
Please avoid GIFs and memes in your article. It adds nothing to the actual information in the article, but takes the seriousness away and makes it less readable.

46 points by wowi42 5y ago

Please label your bar graph axis, with units. It’s kind of counterproductive to look at a benchmark graph without knowing whether more or less is better.

n8ta 5y ago
Agreed. I thought this was an interesting post but was baffled by every y axis.
- wowi42 5y ago
  Noted!
yellow_lead 5y ago
And better titles. "Performance" is confusing. I think he means "Performance of different base images for Redis"?
wowi42 5y ago
Thanks for the feedback. Let me find a better way to make it more clear!

prpl 5y ago

The argument here I suppose is live on the bleeding edge? Or swap compilers/allocators as necessary? Or use O3? I'm not entirely sure. It wraps up with: >>> COMPILE YOUR SOFTWARES. It's going to help you understand it, make your app faster, more secure (by removing the mail gateway of NGiNX for example), and it will show you which softwares are easily maintanable and reliable. It is PITA is keeping up with compiler changes and library changes of third party software. In this case, throwing in different malloc implementations in there too, not to mention different libc implementations. With all those variables for each component you deploy you're probably less likely to understand what's going in your software. Maybe for a small app with 2-3 extra pieces or something that needs to really be optimized this is good advice, but it sounds like a lot of work for significant footprints. It would be nice if the official docker images had some tags that were better optimized, within reason.

l0b0 5y ago
So much advice in software development presupposes that everyone has the same single goal and at least one inexhaustible resource. In this case, time: - To read and understand all relevant parts of the build pipelines for each performance-critical part of the infrastructure. In a relatively simple web application that could mean at least a web server, a caching framework, a database and a message bus. - To debug any build failures, which could be plentiful and hard to parse until you're very familiar with the build infrastructure for that particular piece of software. - To benchmark the build outputs in comparable ways with realistic configuration and inputs. - To repeat the above whenever any part of the stack changes significantly.
ec109685 5y ago
It also could introduce bugs given your company might be the only one earth with that combination of compiler, kernel and libraries.
- coliveira 5y ago
  You mean, expose bugs. If the software doesn't support a certain combination, this is a bug in the software.
  - indianboy42 5y ago
    Is it required for open source software to support every combination of compiler, kernel, libc etc? We wouldn't say it has to support every version of any other dependency (not provided by system) It's definitely a nice to have, but I think it's also valid for some project to say we support the version we used in CI and for releases. On the other hand you could say being stuck on a specific version is skin to relying on implementation details, which is a code smell atleast.
    - wowi42 5y ago
      That's what people are doing with Alpine and Musl!
    - 5y ago

latenightcoding 5y ago

On my side projects I compile everything myself, but I do not completely agree with this post because Redis is one of easiest/fastest mainstream databases to compile, it can get very time consuming and the returns are not always there.

wowi42 5y ago
that's true, but without some crazy guy like me to check it, you would not know! :-)

Macha 5y ago

I wonder how x86-64-v3 for Arch/v2 for fedora in the near future will change this calculus. Currently you're basically compiling for a Core 2/Athlon 64 era chip, so there's clear wins to be had, but I wonder how much of the benefit can be had just by using software requiring Haswell/Zen1 at minimum

binarybanana 5y ago
-march=native will still give you better performance. It's not just about the instruction set, but also heuristics taking into account cache size, latencies, topology and other things. Intel for example has this quirk that aligning functions (and other jump targets) at 32 byte boundaries speeds up funct8 calls and jumps. I haven't tested it but I suspect you'd gain more from -mtune=native with the generic x86_64 target than -march=native. Some loops that can be autovectorized with AVX instructions will probably be faster though. But cache size especially is important for deciding if some optimization is beneficial or just leads to stalls due to thrashing.

mathfailure 5y ago

Why would I read some tosser?

wowi42 5y ago
Nice one, I never heard it!
wowi42 5y ago
https://www.larousse.fr/dictionnaires/francais/tosser/78590 https://www.larousse.fr/dictionnaires/francais/tosser/78590
- Zuider 5y ago
  https://www.etymonline.com/search?q=tosser https://www.etymonline.com/search?q=tosser

Aeolun 5y ago

I don’t follow, where did this person compile anything? The only thing I saw him do was use pregenerated docker images?

wowi42 5y ago
Read the article. I compiled redis with Zig, O3 native, and mimalloc!
- Aeolun 5y ago
  I did read the article, but that wasn’t immediately clear to me. Maybe something about the formatting of pre-generated docker images in the same line as the compiled versions? Anyway, thanks for the clarification!

brigandish 5y ago

The sad fact is that a lot of software is difficult to compile; isn't documented well, something that is worse for building; won't work well if installed in a non-standard way, whether that is final location, different supporting libs, or different platform; and can take a long time. I'm happy nowadays when I see there's a binary available, no mucking around with gcc/clang/llvm - just trying to work out which one, let alone which version! - no diving down a rabbit hole of compiling dependencies that then need other dependencies compiled… no deciphering Makefiles that were written in a way that only a C guru can grok, with no comments. Whatever the benefits are, I prefer sanity.

Rendello 5y ago
This is one thing where I see Rust (and probably Zig) making headway. A lot of the newer Rust software isn't in package repos yet, but I don't mind doing a Cargo build. It might take a while to compile, but it always seems to just work.
lnxg33k1 5y ago
https://hub.docker.com/r/gentoo/stage3 https://hub.docker.com/r/gentoo/stage3
wowi42 5y ago
> The sad fact is that a lot of software is difficult to compile; isn't documented well, something that is worse for building; won't work well if installed in a non-standard way, whether that is final location, different supporting libs, or different platform; and can take a long time. So are you ready to deploy in prod a software that is so hard to compile? > I'm happy nowadays when I see there's a binary available, no mucking around with gcc/clang/llvm - just trying to work out which one, let alone which version! - no diving down a rabbit hole of compiling dependencies that then need other dependencies compiled… no deciphering Makefiles that were written in a way that only a C guru can grok, with no comments. But that's my job, as SRE/DevOps/whatever new fancy name! > Whatever the benefits are, I prefer sanity. Sanity of having a very old software, with backported features that are only on this distrib? I prefer to trust the engineers from the software that I deploy.

binarybanana 5y ago

Other cool things you can do if you compile yourself is use features like auto parallelization[1]. I wouldn't recommend to enable it system wide because it causes issues with programs that fork() due to limitations in gcc's OpenMP library[2], but other than that it works pretty well. For example, I can fully load my 4C/8T CPU using 3 clang processes because compilation is magically spread over multiple threads. I've seen a "single threaded" program (qemu-img) suddenly start using more than a single core to convert disk images into other formats, leading to speedups. Also things like PGO/FDO in combination with workload specific profiling data can easily give you 10% or more if you are CPU bound. [1]: https://gcc.gnu.org/wiki/AutoParInGCC https://gcc.gnu.org/wiki/AutoParInGCC [2]: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=42624 https://gcc.gnu.org/bugzilla/show_bug.cgi?id=42624 (There was a patch to fix this, but it never got merged and doesn't apply to the current version any more, sadly)

b215826 5y ago

Please avoid GIFs and memes in your article. It adds nothing to the actual information in the article, but takes the seriousness away and makes it less readable.