8 ms·

My Go executable files are still getting larger

264 points by alexbilbie 5y ago

haberman 5y ago
> The sum of the sizes reported by go tool nm does not add up to the final size of the Go executable. > At this time, I do not have a satisfying explanation for this “dark” file usage. The author's journey of starting with "nm --size", discovering "dark" bytes, and wanting to attribute them properly, is exactly what led me to create and invest so much effort into Bloaty McBloatface: https://github.com/google/bloaty https://github.com/google/bloaty Bloaty's core principle is that every byte of the file should be attributed to something, so that the sum of the parts always adds up to the total file size. If we can't get detailed symbol, etc. information for a given region of the file, we can at least fall back to describing what section the bytes were in. Attributing all of the bytes requires parsing much more than just the symbol table. Bloaty parses many different sections of the binary, including unwind information, relocation information, debug info, and the data section itself in an attempt to attribute every part of the binary to the function/data that emitted it. It will even disassemble the binary looking for references to anonymous data (some data won't make it into the symbol table, especially things like string literals). I wrote up some details of how Bloaty works here: https://github.com/google/bloaty/blob/master/doc/how-bloaty-works.md https://github.com/google/bloaty/blob/master/doc/how-bloaty-.... The section on the "Symbols" data source is particularly relevant here: > I excerpted two symbols from the report. Between these two symbols, Bloaty has found seven distinct kinds of data that contributed to these two symbols. If you wrote a tool that naively just parsed the symbol table, you would only find the first of these seven:" The author's contention that these "dark" bytes are "non-useful" is not quite fair. There are plenty of things a binary contains that are useful even though they are not literally executable code. For example, making a binary position-independent (which is good for security) requires emitting relocations into the binary so that globals with pointer values can be relocated at program load time, once the base address of the binary is chosen. I don't know if Go does this or not, but it's just one example. On the other hand, I do agree that the ability to produce slim binaries is an important and often undervalued property of modern compiler toolchains. All else being equal, I much prefer a toolchain that can make the smallest binaries. Bloaty should work reasonably well for Go binaries, though I have gotten some bug reports about things Bloaty is not yet handling properly for Go: https://github.com/google/bloaty/issues/204 https://github.com/google/bloaty/issues/204 Bloaty is just a side thing for me, so I often don't get as much time as I'd like to fix bugs like this.
brainzap 5y ago
Is there a bug report for this?
- ainar-g 5y ago
  Probably falls under the umbrella issue filed by Rob Pike… back in 2013[1]. But that's the umbrella issue, and it's probably better for the Cockroach people to file a new one mentioning their findings, but don't quote me on that. [1]: https://github.com/golang/go/issues/6853 https://github.com/golang/go/issues/6853
- jordanlewis 5y ago
  Rob Pike filed this issue for it after the first incarnation of the article was released: https://github.com/golang/go/issues/36313 https://github.com/golang/go/issues/36313
marcus_holmes 5y ago
I always wonder if this is the flip side of the fast compilation? It would be nice to be able to decide on those trade-offs ourselves. I mostly write web servers in Go, which (as the article says) are executed rarely, so init time really doesn't matter to me. But I've been looking at writing some desktop apps in Go, and then init time will matter.
- throwaway894345 5y ago
  > I mostly write web servers in Go, which (as the article says) are executed rarely, so init time really doesn't matter to me. Presumably a few mb of disk usage also doesn't matter then? I also write web services, but I do care about the init time precisely because I want to be deployments to take as little time as possible so that we can deploy (and rollback) many times per day with relatively simple automation. That said, the bottleneck to fast deployments isn't the binary starting up, but the machine pulling the executable artifact, so the binary sizes do matter to me. That said, very often these executable artifacts are a Docker image, which tend to come with a lot more bloat than one will find in a Go binary, so step 1 is getting your Go binary on a scratch image.
  - cpuguy83 5y ago
    Unless you are running these in containers, then that time adds up (assuming you are using go-based container runtime implementations, as most people are).
  - marcus_holmes 5y ago
    Yeah, it's a trade-off. At the moment the longest part of my deploy is copying the new executable to the server. I'd trade a couple of seconds of init time for a smaller executable because that would result in a faster deploy. I don't use Docker to deploy, because it's just a single executable file (and a bunch of templates, though I'm looking at embedding those). One of the reasons I'm reluctant to go down the Docker road is because it's going to add more time to my deployment.
    - 5y ago
londons_explore 5y ago
It's time for debug info like this to be sent to "onlinesymbolserver.com", encrypted with a hash of the binary. Then, whenever a debugger connects to a binary, it can simply download the symbols as required. And for the 99.9% who don't need debug info, it isn't needlessly shipped. Microsoft invented this in the 90's...
- pjmlp 5y ago
  Around 1992 Borland C++ would use macro based code generation for generic code in BIDS 1.0, sounds familiar? I guess it is just a tradition with Go, rediscovering history.
  - agumonkey 5y ago
    On the memory capacity of social structures.pdf
- the_mitsuhiko 5y ago
  Sadly go doesn’t care about DWARF which would allow them to reuse already existing infrastructure.
  - icholy 5y ago
    Go uses DWARF.
    - the_mitsuhiko 5y ago
      Not for unwinding or line programs.
      - icholy 5y ago
        What's your point exactly?
        the_mitsuhiko 5y ago
deleted 5y ago
[deleted]
sciflare 5y ago
Great blog! Thanks for sharing this blog. Today's market is based on mobile users, Developing a mobile application helps businesses in covering a larger market segment. We are the best mobile app development company that thinks out of the box and build award-winning mobile applications globally. Our mobile development team is capable of handling applications for such a market base and our expertise in building custom apps, sports, iOS apps, android apps which helps the clients to put their idea into a digital process, To build a mobile application https://www.sciflare.com/android-application-development/ https://www.sciflare.com/android-application-development/
parhamn 5y ago
> In other words, the Go team decided to make executable files larger to save up on initialization time. I mean... Im genuinely curious if this is a "we have extra engineering resources and can explore/complain about this" or "we have a client who is running cockroachdb and can't handle a 172mb binary install for a database server". Is there really someone out there who installs Cockroach (a global distributed auto-sharded database) and thinks twice about 172mb of disk space? Sure, it'd be nice to have smaller binaries but outside of some embedded applications Go's binaries sizes are well within the nothing-burger range for most compute systems.
- gtirloni 5y ago
  It affects the download and instantiation time for containers.
  - henvic 5y ago
    One word: trade-offs.
  - mikepurvis 5y ago
    I worry about that for 10gb omnibus containers, not so much for <500mb.
  - parhamn 5y ago
    The article says they don't really care about initialization time though, which is right. Remember: cockroachdb is always synchronizing data across the cluster, that 175mb of ingress to start up a DB node, probably pales in comparison to data synchronization/relocations that happen on a cluster. Which is why worrying about ingress/egress costs over binary size is nonsense here too. The bandwidth you need to run a distributed database cluster could download 172mb binary in milliseconds. If your node initiation time for DB failovers needs anything faster, you're doing something wrong. There are stakeholders to this problem, Cockroach probably isn't one.
    - spockz 5y ago
      For production, yes. But it also effects startup and download time on developer machines. Want to have multiple versions installed? Now it takes more space. Takes longer to download on 4g while on the road or on crappy corporate/conference WiFi, etc . In the end this all ends up because it is for all go binaries. I’ve come appreciate attention for leanness because in the end it does add up.
kreetx 5y ago
Couldn't the dark bytes just be shipped as a separate file - for those who need it?
- skywhopper 5y ago
  Is it clear that the “dark bytes” are “useless” debug information? It sounds like it’s just stuff not in the symbol table.
  - tyingq 5y ago
    I did keep waiting for some point in the blog post where they would null out the "dark bytes" and see if/how the binary runs.
    - jeffbee 5y ago
      It won't. If you try to remove this information your program will crash as soon as the GC runs or someone calls systemstack, i.e. pretty much instantly.
stabbles 5y ago
> Every time, 70% of a couple hundred megabytes are copied around for no good reason and someone needs to pay ingress/egress networking costs for these file copies. That is quite some money being burned for no good reason! Meanwhile half of the world is pushing images by nvidia, intel and amd around for their machine learning software: Intel OneAPI runtime libraries: 4.74GB (or 18.4GB for compilers) CUDA runtime libraries: 1.92GB (or 4.2GB for compilers) These go binaries are still relatively small
- bbatha 5y ago
  Most of the time, especially with the docker hub rate limiting changes people are or should be using their cloud provider's mirror or running their own mirror. Actually ingressing the docker image into your network should happen once in most production setups.
boredpandas777 5y ago
Maybe Go optimizing for serverless with reduced init time?
blinkingled 5y ago
> We can call this the “dark file usage” of Go binaries, and it occupies between 15% and 33% of the total file size inside CockroachDB. > Sadly, the removal of pclntab in Go 1.16 actually transferred the payload to the “dark” bytes. I surely would have expect better from programming language designers/developers than this. Sounds like they just moved the problem from one place to another.
- rob74 5y ago
  I'll hold my judgement until someone manages to actually find out what this "dark" space is (quote from the article: "At this time, I do not have a satisfying explanation for this “dark” file usage").
tedunangst 5y ago
> Every time, 70% of a couple hundred megabytes are copied around for no good reason and someone needs to pay ingress/egress networking costs for these file copies. Just zero them out and they'll compress to nothing. Even better, with a sparse file aware tool like tar, they won't even use disk space.
bradfitz 5y ago
To promote my own tool, https://github.com/bradfitz/shotizam https://github.com/bradfitz/shotizam lets you drill down into why Go binaries are large without having to make up terms like "dark bytes".
- knz42 5y ago
  This code, if I'm reading it right, uses the symbol table and ELF section headers. As explained in OP, the sum of sizes advertised in the symtable and ELF section headers does not add up to the final binary size. The shotizam tool is thus blind to that difference.
  - nappy-doo 5y ago
    This sounds like a bug to me, not nefarious intent.
    - knz42 5y ago
      What in my reply suggests I assumed nefarious intent? Brad is a good person, it's the code we're talking about here.
- arp242 5y ago
  If physicists can have dark matter and dark energy, then why can't we have dark bytes? Why should we let the physicists have all the dark fun?
  - saati 5y ago
    Because it's genuinely unknown what those are, figuring out what these bytes are is just a deep dive into the opensource go compiler at worst or just asking around on some go channels for much less effort.
    - arp242 5y ago
      You're taking a joke too serious :-)
njuw 5y ago
> starting in Go 1.16, the pclntab is not present any more, and instead is re-computed from other data in the executable file. Does anyone have a source for this? As it still appears to be there - Go 1.15 https://i.imgur.com/3YlZGOk.png https://i.imgur.com/3YlZGOk.png - Go 1.16 https://i.imgur.com/gGYsj32.png https://i.imgur.com/gGYsj32.png
- nappy-doo 5y ago
  No source needed -- you're right. The author's looking in the symbol table, and in 1.16 the Go linker set the size of runtime.pclntab to 0. If the OP used nm to look at their binary, they'd have seen that pclntab is still there. They author has apparently revised their article to address some of this (yet still draw some incorrect conclusions). (This is derived from Russ' discussion above.)
daitangio 5y ago
Go static linking is a great happy idea for a Java guy trapped in the Classpath Dependency Hell (or C# / DLL Hell). It is a very annoying thing for a C++ programmer, which can dynamically link operating system libraries at will.
- skrebbel 5y ago
  You mean a C++ programmer who does not care whether their program works on anybody else's computer?
  - synergy20 5y ago
    in that case you can happily static-link c++ programs, which probably will be smaller than go binary.
    - anthk 5y ago
      happily and "statically link c++" don't match together well.
      - pjmlp 5y ago
        They do, there are more C++ compilers out there than gcc with glibc.
- kstenerud 5y ago
  "DLL Hell" was coined in the early days of Windows (before C#), and originally referred to C and C++ dynamic library problems.
- anthk 5y ago
  Go's static linking idea comes from plan9 C compilers, a few years before Java. We owe a lot from plan9: - Go's design, based on both C compilers and Inferno's limbo - /proc - utf-8 - 9p
  - 5y ago
kissgyorgy 5y ago
Python is still more wildly used / popular language, but I never seen a Python container for a real project which was less than 1GB.
- rataata_jr 5y ago
  I've seen ones in MBs with alpine
- simtel20 5y ago
  Have you tried basing one on distroless? That has turned out pretty small python containers for me.
- dbt00 5y ago
  I'm guessing you're doing ML here? Because the core python runtime is far smaller than that.
  - kornholi 5y ago
    Yeah, ML libraries are a huge offender. The Torch 1.8.1 manylinux wheel is 1.6G unpacked, with 1.1G going to CUDA support.
- masklinn 5y ago
  Alpine containers for P3 go down to about 60MB for the baseline. Everything else is what you added in. CPython doesn't generate anything, and the binary installers (including docs and all) are about 30MB. Hell, the 64b embeddable package for windows (https://docs.python.org/3/using/windows.html#the-embeddable-package https://docs.python.org/3/using/windows.html#the-embeddable-...) is 16MB uncompressed.
rsc 5y ago
This article is full of misinformation. Just a few representative things: - The expansion of pclntab in Go 1.2 dramatically improved startup time and reduced memory footprint, by letting the OS demand-page this critical table that is used any time a stack must be walked (in particular, during garbage collection). See https://golang.org/s/go12symtab https://golang.org/s/go12symtab for details. - We (the Go team) did not “recompress” pclntab in Go 1.15. We did not remove pclntab in Go 1.16. Nor do we have plans to do either. Consequently, we never claimed “pclntab has been reduced to zero”, which is presented in the article as if a direct quote. - If the 73% of the binary diagnosed as “not useful” were really not useful, a reasonable demonstration would be to delete it from the binary and see the binary still run. It clearly would not. - The big table seems to claim that a 40 MB Go 1.8 binary has grown to a 289 MB Go 1.16 binary. That’s certainly not the case. More is changing from line to line in that table than the Go version. Overall, the claim of “dark bytes” or “non-useful bytes” strikes me as similar to the claims of “junk DNA”. They’re not dark or non-useful. It turns out that having the necessary metadata for garbage collection and reflection in a statically-compiled language takes up a significant amount of space, which we’ve worked over time at reducing. But the dynamic possibilities in reflection and interface assertions mean that fewer bytes can be dropped than you’d hope. We track binary size work in https://golang.org/issue/6853 https://golang.org/issue/6853. An unfortunate article.
- rsc 5y ago
  An easily obtained apples-to-apples¹ table: $ for i in $(seq 3 16); do curl -sLo go1.$i.tgz https://golang.org/dl/go1.$i.linux-amd64.tar.gz tar xzf go1.$i.tgz go/bin/gofmt size=$(ls -l go/bin/gofmt | awk '{print $5}') strip go/bin/gofmt size2=$(ls -l go/bin/gofmt | awk '{print $5}') echo go1.$i $size $size2 done go1.3 3496520 2528664 go1.4² 14398336 13139184 go1.5 3937888 2765696 go1.6 3894568 2725376 go1.7 3036195 1913704 go1.8 3481554 2326760 go1.9 3257829 2190792 go1.10 3477807 2166536 go1.11 3369391 2441288 go1.12 3513529 2506632 go1.13 3543823 2552632 go1.14 3587746 2561208 go1.15 3501176 2432248 go1.16 3448663 2443736 $ Size fluctuates from release to release, but the overall trendline is flat: Go 1.16 binaries are roughly where Go 1.3 binaries were. At the moment, it looks like Go 1.17 binaries will get a bit smaller thanks to the new register ABI making executable code smaller (and faster). ¹ Well, not completely. The gofmt code itself was changing from release to release, but not much. Most of the binary is the libraries and runtime, though, so it's still accurate for trends. ² Turns out we shipped the go 1.4 gofmt binary built with the race detector enabled! Oops.
u678u 5y ago
We need more initContainers where the big shared system libraries are in a separate container that is cached locally. I feel like history is repeating.
EdiX 5y ago
From the same data scientist that concluded non-linear growth from two single data points...
jeffbee 5y ago
This article should be renamed “what is Chesterton‘s fence?” And the author should have realized their mistake right after typing “at this time I don’t know what it is used for”.
tediousdemise 5y ago
It seems to be a challenge to add zero-overhead features to programming languages. Poor design decisions result in a language that gets extremely bloated over time, forcing you to use features that you don’t want to. The better approach is to make these features optional, such as through a standard library.
arp242 5y ago
> Moreover, consider that these executable files fly around as container images, and/or are copied between VMs in the cloud, thousands of times per day! Every time, 70% of a couple hundred megabytes are copied around for no good reason and someone needs to pay ingress/egress networking costs for these file copies. That is quite some money being burned for no good reason! Does the author think the Go authors are stupid blubbering idiots who someone missed this huge elephant-sized low-hanging fruit? Binary sizes have been a point of attention for years, and somehow missing 70% wasted space would be staggeringly incompetent. Reminds me of the time in high school when one of the my classmates ended up with a 17A doorbell in some calculations. I think he used the wrong formula or swapped some numbers. The teacher, quite rightfully, berated him for not actually looking at the result of his calculation and judging if it's roughly in the right ballpark, as 17A is a ludicrous amount of power for a doorbell. Anyone can see that's just widely wrong. If this story had ended up with 0.7%, sure, I can believe that. 7%? Unlikely and I'd be skeptical, but still possible I suppose. *70%* Yeah nah, that's just as silly as a 17A doorbell. This huge 70% number should have been a clue to the author themselves too that they've missed something.
- superdisk 5y ago
  In a world where electron reigns supreme, 70% waste isn't unthinkable
jeffbee 5y ago
After looking into the size of the CockroachDB binary, the magnitude of the plank in the author's eye becomes clear. This iceberg is ridiculously bloated. Much of the space is coming from the static data of geographic projections that, I assume, basically nobody needs. This includes a single init function that is 1.4MB of machine code from 6MB of auto-generated source code. Then there's the entire AWS SDK, with a static definition of every service AWS offers, by name, and in what regions, by name. Nevermind the Azure SDK. There are three implementations of protocol buffers in here: gogo in Go and Google's in both Go and C++. There are at least four SQL parsers in here, including vitess's and another one for crdb in Go. Last but by no means least there are in total 13MB of autogenerated functions of the colexec package, each of which is over 100KB long, which are autogenerated and share virtually all of their code. These are an obscene waste of code space and undoubtedly de-deuplicating this code would not just reduce code size but also speed up the program, due to icache trashing.
orangechairs 5y ago
Hey all -- Cockroach Labs blog editor here. Based in part on the feedback we received from this community, we have retracted the post. The blog post link above will take you the retraction, where we share what we've learned from this experience.

264 points by alexbilbie 5y ago

> The sum of the sizes reported by go tool nm does not add up to the final size of the Go executable. > At this time, I do not have a satisfying explanation for this “dark” file usage. The author's journey of starting with "nm --size", discovering "dark" bytes, and wanting to attribute them properly, is exactly what led me to create and invest so much effort into Bloaty McBloatface: https://github.com/google/bloaty https://github.com/google/bloaty Bloaty's core principle is that every byte of the file should be attributed to something, so that the sum of the parts always adds up to the total file size. If we can't get detailed symbol, etc. information for a given region of the file, we can at least fall back to describing what section the bytes were in. Attributing all of the bytes requires parsing much more than just the symbol table. Bloaty parses many different sections of the binary, including unwind information, relocation information, debug info, and the data section itself in an attempt to attribute every part of the binary to the function/data that emitted it. It will even disassemble the binary looking for references to anonymous data (some data won't make it into the symbol table, especially things like string literals). I wrote up some details of how Bloaty works here: https://github.com/google/bloaty/blob/master/doc/how-bloaty-works.md https://github.com/google/bloaty/blob/master/doc/how-bloaty-.... The section on the "Symbols" data source is particularly relevant here: > I excerpted two symbols from the report. Between these two symbols, Bloaty has found seven distinct kinds of data that contributed to these two symbols. If you wrote a tool that naively just parsed the symbol table, you would only find the first of these seven:" The author's contention that these "dark" bytes are "non-useful" is not quite fair. There are plenty of things a binary contains that are useful even though they are not literally executable code. For example, making a binary position-independent (which is good for security) requires emitting relocations into the binary so that globals with pointer values can be relocated at program load time, once the base address of the binary is chosen. I don't know if Go does this or not, but it's just one example. On the other hand, I do agree that the ability to produce slim binaries is an important and often undervalued property of modern compiler toolchains. All else being equal, I much prefer a toolchain that can make the smallest binaries. Bloaty should work reasonably well for Go binaries, though I have gotten some bug reports about things Bloaty is not yet handling properly for Go: https://github.com/google/bloaty/issues/204 https://github.com/google/bloaty/issues/204 Bloaty is just a side thing for me, so I often don't get as much time as I'd like to fix bugs like this.

brainzap 5y ago

Is there a bug report for this?

ainar-g 5y ago
Probably falls under the umbrella issue filed by Rob Pike… back in 2013[1]. But that's the umbrella issue, and it's probably better for the Cockroach people to file a new one mentioning their findings, but don't quote me on that. [1]: https://github.com/golang/go/issues/6853 https://github.com/golang/go/issues/6853
jordanlewis 5y ago
Rob Pike filed this issue for it after the first incarnation of the article was released: https://github.com/golang/go/issues/36313 https://github.com/golang/go/issues/36313

marcus_holmes 5y ago

I always wonder if this is the flip side of the fast compilation? It would be nice to be able to decide on those trade-offs ourselves. I mostly write web servers in Go, which (as the article says) are executed rarely, so init time really doesn't matter to me. But I've been looking at writing some desktop apps in Go, and then init time will matter.

throwaway894345 5y ago
> I mostly write web servers in Go, which (as the article says) are executed rarely, so init time really doesn't matter to me. Presumably a few mb of disk usage also doesn't matter then? I also write web services, but I do care about the init time precisely because I want to be deployments to take as little time as possible so that we can deploy (and rollback) many times per day with relatively simple automation. That said, the bottleneck to fast deployments isn't the binary starting up, but the machine pulling the executable artifact, so the binary sizes do matter to me. That said, very often these executable artifacts are a Docker image, which tend to come with a lot more bloat than one will find in a Go binary, so step 1 is getting your Go binary on a scratch image.
- cpuguy83 5y ago
  Unless you are running these in containers, then that time adds up (assuming you are using go-based container runtime implementations, as most people are).
- marcus_holmes 5y ago
  Yeah, it's a trade-off. At the moment the longest part of my deploy is copying the new executable to the server. I'd trade a couple of seconds of init time for a smaller executable because that would result in a faster deploy. I don't use Docker to deploy, because it's just a single executable file (and a bunch of templates, though I'm looking at embedding those). One of the reasons I'm reluctant to go down the Docker road is because it's going to add more time to my deployment.
  - 5y ago

londons_explore 5y ago

It's time for debug info like this to be sent to "onlinesymbolserver.com", encrypted with a hash of the binary. Then, whenever a debugger connects to a binary, it can simply download the symbols as required. And for the 99.9% who don't need debug info, it isn't needlessly shipped. Microsoft invented this in the 90's...

pjmlp 5y ago
Around 1992 Borland C++ would use macro based code generation for generic code in BIDS 1.0, sounds familiar? I guess it is just a tradition with Go, rediscovering history.
- agumonkey 5y ago
  On the memory capacity of social structures.pdf
the_mitsuhiko 5y ago
Sadly go doesn’t care about DWARF which would allow them to reuse already existing infrastructure.
- icholy 5y ago
  Go uses DWARF.
  - the_mitsuhiko 5y ago
    Not for unwinding or line programs.
    - icholy 5y ago
      What's your point exactly?
      - the_mitsuhiko 5y ago

deleted 5y ago

[deleted]

sciflare 5y ago

Great blog! Thanks for sharing this blog. Today's market is based on mobile users, Developing a mobile application helps businesses in covering a larger market segment. We are the best mobile app development company that thinks out of the box and build award-winning mobile applications globally. Our mobile development team is capable of handling applications for such a market base and our expertise in building custom apps, sports, iOS apps, android apps which helps the clients to put their idea into a digital process, To build a mobile application https://www.sciflare.com/android-application-development/ https://www.sciflare.com/android-application-development/

parhamn 5y ago

> In other words, the Go team decided to make executable files larger to save up on initialization time. I mean... Im genuinely curious if this is a "we have extra engineering resources and can explore/complain about this" or "we have a client who is running cockroachdb and can't handle a 172mb binary install for a database server". Is there really someone out there who installs Cockroach (a global distributed auto-sharded database) and thinks twice about 172mb of disk space? Sure, it'd be nice to have smaller binaries but outside of some embedded applications Go's binaries sizes are well within the nothing-burger range for most compute systems.

gtirloni 5y ago
It affects the download and instantiation time for containers.
- henvic 5y ago
  One word: trade-offs.
- mikepurvis 5y ago
  I worry about that for 10gb omnibus containers, not so much for <500mb.
- parhamn 5y ago
  The article says they don't really care about initialization time though, which is right. Remember: cockroachdb is always synchronizing data across the cluster, that 175mb of ingress to start up a DB node, probably pales in comparison to data synchronization/relocations that happen on a cluster. Which is why worrying about ingress/egress costs over binary size is nonsense here too. The bandwidth you need to run a distributed database cluster could download 172mb binary in milliseconds. If your node initiation time for DB failovers needs anything faster, you're doing something wrong. There are stakeholders to this problem, Cockroach probably isn't one.
  - spockz 5y ago
    For production, yes. But it also effects startup and download time on developer machines. Want to have multiple versions installed? Now it takes more space. Takes longer to download on 4g while on the road or on crappy corporate/conference WiFi, etc . In the end this all ends up because it is for all go binaries. I’ve come appreciate attention for leanness because in the end it does add up.

kreetx 5y ago

Couldn't the dark bytes just be shipped as a separate file - for those who need it?

skywhopper 5y ago
Is it clear that the “dark bytes” are “useless” debug information? It sounds like it’s just stuff not in the symbol table.
- tyingq 5y ago
  I did keep waiting for some point in the blog post where they would null out the "dark bytes" and see if/how the binary runs.
  - jeffbee 5y ago
    It won't. If you try to remove this information your program will crash as soon as the GC runs or someone calls systemstack, i.e. pretty much instantly.

stabbles 5y ago

> Every time, 70% of a couple hundred megabytes are copied around for no good reason and someone needs to pay ingress/egress networking costs for these file copies. That is quite some money being burned for no good reason! Meanwhile half of the world is pushing images by nvidia, intel and amd around for their machine learning software: Intel OneAPI runtime libraries: 4.74GB (or 18.4GB for compilers) CUDA runtime libraries: 1.92GB (or 4.2GB for compilers) These go binaries are still relatively small

bbatha 5y ago
Most of the time, especially with the docker hub rate limiting changes people are or should be using their cloud provider's mirror or running their own mirror. Actually ingressing the docker image into your network should happen once in most production setups.

boredpandas777 5y ago

Maybe Go optimizing for serverless with reduced init time?

blinkingled 5y ago

> We can call this the “dark file usage” of Go binaries, and it occupies between 15% and 33% of the total file size inside CockroachDB. > Sadly, the removal of pclntab in Go 1.16 actually transferred the payload to the “dark” bytes. I surely would have expect better from programming language designers/developers than this. Sounds like they just moved the problem from one place to another.

rob74 5y ago
I'll hold my judgement until someone manages to actually find out what this "dark" space is (quote from the article: "At this time, I do not have a satisfying explanation for this “dark” file usage").

tedunangst 5y ago

> Every time, 70% of a couple hundred megabytes are copied around for no good reason and someone needs to pay ingress/egress networking costs for these file copies. Just zero them out and they'll compress to nothing. Even better, with a sparse file aware tool like tar, they won't even use disk space.

bradfitz 5y ago

To promote my own tool, https://github.com/bradfitz/shotizam https://github.com/bradfitz/shotizam lets you drill down into why Go binaries are large without having to make up terms like "dark bytes".

knz42 5y ago
This code, if I'm reading it right, uses the symbol table and ELF section headers. As explained in OP, the sum of sizes advertised in the symtable and ELF section headers does not add up to the final binary size. The shotizam tool is thus blind to that difference.
- nappy-doo 5y ago
  This sounds like a bug to me, not nefarious intent.
  - knz42 5y ago
    What in my reply suggests I assumed nefarious intent? Brad is a good person, it's the code we're talking about here.
arp242 5y ago
If physicists can have dark matter and dark energy, then why can't we have dark bytes? Why should we let the physicists have all the dark fun?
- saati 5y ago
  Because it's genuinely unknown what those are, figuring out what these bytes are is just a deep dive into the opensource go compiler at worst or just asking around on some go channels for much less effort.
  - arp242 5y ago
    You're taking a joke too serious :-)

njuw 5y ago

> starting in Go 1.16, the pclntab is not present any more, and instead is re-computed from other data in the executable file. Does anyone have a source for this? As it still appears to be there - Go 1.15 https://i.imgur.com/3YlZGOk.png https://i.imgur.com/3YlZGOk.png - Go 1.16 https://i.imgur.com/gGYsj32.png https://i.imgur.com/gGYsj32.png

nappy-doo 5y ago
No source needed -- you're right. The author's looking in the symbol table, and in 1.16 the Go linker set the size of runtime.pclntab to 0. If the OP used nm to look at their binary, they'd have seen that pclntab is still there. They author has apparently revised their article to address some of this (yet still draw some incorrect conclusions). (This is derived from Russ' discussion above.)

daitangio 5y ago

Go static linking is a great happy idea for a Java guy trapped in the Classpath Dependency Hell (or C# / DLL Hell). It is a very annoying thing for a C++ programmer, which can dynamically link operating system libraries at will.

skrebbel 5y ago
You mean a C++ programmer who does not care whether their program works on anybody else's computer?
- synergy20 5y ago
  in that case you can happily static-link c++ programs, which probably will be smaller than go binary.
  - anthk 5y ago
    happily and "statically link c++" don't match together well.
    - pjmlp 5y ago
      They do, there are more C++ compilers out there than gcc with glibc.
kstenerud 5y ago
"DLL Hell" was coined in the early days of Windows (before C#), and originally referred to C and C++ dynamic library problems.
anthk 5y ago
Go's static linking idea comes from plan9 C compilers, a few years before Java. We owe a lot from plan9: - Go's design, based on both C compilers and Inferno's limbo - /proc - utf-8 - 9p
- 5y ago

kissgyorgy 5y ago

Python is still more wildly used / popular language, but I never seen a Python container for a real project which was less than 1GB.

rataata_jr 5y ago
I've seen ones in MBs with alpine
simtel20 5y ago
Have you tried basing one on distroless? That has turned out pretty small python containers for me.
dbt00 5y ago
I'm guessing you're doing ML here? Because the core python runtime is far smaller than that.
- kornholi 5y ago
  Yeah, ML libraries are a huge offender. The Torch 1.8.1 manylinux wheel is 1.6G unpacked, with 1.1G going to CUDA support.
masklinn 5y ago
Alpine containers for P3 go down to about 60MB for the baseline. Everything else is what you added in. CPython doesn't generate anything, and the binary installers (including docs and all) are about 30MB. Hell, the 64b embeddable package for windows (https://docs.python.org/3/using/windows.html#the-embeddable-package https://docs.python.org/3/using/windows.html#the-embeddable-...) is 16MB uncompressed.

rsc 5y ago

This article is full of misinformation. Just a few representative things: - The expansion of pclntab in Go 1.2 dramatically improved startup time and reduced memory footprint, by letting the OS demand-page this critical table that is used any time a stack must be walked (in particular, during garbage collection). See https://golang.org/s/go12symtab https://golang.org/s/go12symtab for details. - We (the Go team) did not “recompress” pclntab in Go 1.15. We did not remove pclntab in Go 1.16. Nor do we have plans to do either. Consequently, we never claimed “pclntab has been reduced to zero”, which is presented in the article as if a direct quote. - If the 73% of the binary diagnosed as “not useful” were really not useful, a reasonable demonstration would be to delete it from the binary and see the binary still run. It clearly would not. - The big table seems to claim that a 40 MB Go 1.8 binary has grown to a 289 MB Go 1.16 binary. That’s certainly not the case. More is changing from line to line in that table than the Go version. Overall, the claim of “dark bytes” or “non-useful bytes” strikes me as similar to the claims of “junk DNA”. They’re not dark or non-useful. It turns out that having the necessary metadata for garbage collection and reflection in a statically-compiled language takes up a significant amount of space, which we’ve worked over time at reducing. But the dynamic possibilities in reflection and interface assertions mean that fewer bytes can be dropped than you’d hope. We track binary size work in https://golang.org/issue/6853 https://golang.org/issue/6853. An unfortunate article.

rsc 5y ago
An easily obtained apples-to-apples¹ table: $ for i in $(seq 3 16); do curl -sLo go1.$i.tgz https://golang.org/dl/go1.$i.linux-amd64.tar.gz tar xzf go1.$i.tgz go/bin/gofmt size=$(ls -l go/bin/gofmt | awk '{print $5}') strip go/bin/gofmt size2=$(ls -l go/bin/gofmt | awk '{print $5}') echo go1.$i $size $size2 done go1.3 3496520 2528664 go1.4² 14398336 13139184 go1.5 3937888 2765696 go1.6 3894568 2725376 go1.7 3036195 1913704 go1.8 3481554 2326760 go1.9 3257829 2190792 go1.10 3477807 2166536 go1.11 3369391 2441288 go1.12 3513529 2506632 go1.13 3543823 2552632 go1.14 3587746 2561208 go1.15 3501176 2432248 go1.16 3448663 2443736 $ Size fluctuates from release to release, but the overall trendline is flat: Go 1.16 binaries are roughly where Go 1.3 binaries were. At the moment, it looks like Go 1.17 binaries will get a bit smaller thanks to the new register ABI making executable code smaller (and faster). ¹ Well, not completely. The gofmt code itself was changing from release to release, but not much. Most of the binary is the libraries and runtime, though, so it's still accurate for trends. ² Turns out we shipped the go 1.4 gofmt binary built with the race detector enabled! Oops.

u678u 5y ago

We need more initContainers where the big shared system libraries are in a separate container that is cached locally. I feel like history is repeating.

EdiX 5y ago

From the same data scientist that concluded non-linear growth from two single data points...

jeffbee 5y ago

This article should be renamed “what is Chesterton‘s fence?” And the author should have realized their mistake right after typing “at this time I don’t know what it is used for”.

tediousdemise 5y ago

It seems to be a challenge to add zero-overhead features to programming languages. Poor design decisions result in a language that gets extremely bloated over time, forcing you to use features that you don’t want to. The better approach is to make these features optional, such as through a standard library.

arp242 5y ago

> Moreover, consider that these executable files fly around as container images, and/or are copied between VMs in the cloud, thousands of times per day! Every time, 70% of a couple hundred megabytes are copied around for no good reason and someone needs to pay ingress/egress networking costs for these file copies. That is quite some money being burned for no good reason! Does the author think the Go authors are stupid blubbering idiots who someone missed this huge elephant-sized low-hanging fruit? Binary sizes have been a point of attention for years, and somehow missing 70% wasted space would be staggeringly incompetent. Reminds me of the time in high school when one of the my classmates ended up with a 17A doorbell in some calculations. I think he used the wrong formula or swapped some numbers. The teacher, quite rightfully, berated him for not actually looking at the result of his calculation and judging if it's roughly in the right ballpark, as 17A is a ludicrous amount of power for a doorbell. Anyone can see that's just widely wrong. If this story had ended up with 0.7%, sure, I can believe that. 7%? Unlikely and I'd be skeptical, but still possible I suppose. *70%* Yeah nah, that's just as silly as a 17A doorbell. This huge 70% number should have been a clue to the author themselves too that they've missed something.

superdisk 5y ago
In a world where electron reigns supreme, 70% waste isn't unthinkable

jeffbee 5y ago

After looking into the size of the CockroachDB binary, the magnitude of the plank in the author's eye becomes clear. This iceberg is ridiculously bloated. Much of the space is coming from the static data of geographic projections that, I assume, basically nobody needs. This includes a single init function that is 1.4MB of machine code from 6MB of auto-generated source code. Then there's the entire AWS SDK, with a static definition of every service AWS offers, by name, and in what regions, by name. Nevermind the Azure SDK. There are three implementations of protocol buffers in here: gogo in Go and Google's in both Go and C++. There are at least four SQL parsers in here, including vitess's and another one for crdb in Go. Last but by no means least there are in total 13MB of autogenerated functions of the colexec package, each of which is over 100KB long, which are autogenerated and share virtually all of their code. These are an obscene waste of code space and undoubtedly de-deuplicating this code would not just reduce code size but also speed up the program, due to icache trashing.

orangechairs 5y ago

Hey all -- Cockroach Labs blog editor here. Based in part on the feedback we received from this community, we have retracted the post. The blog post link above will take you the retraction, where we share what we've learned from this experience.