7 ms·

Might make sense if you use Go or any other language with a straightforward tooling landscape (compilers, package management etc.). That changes drastically if

by simfoo 6y ago

Might make sense if you use Go or any other language with a straightforward tooling landscape (compilers, package management etc.). That changes drastically if your codebase contains a lot of C++ and your SW model doesn't quite match the way Bazel tries to push you into. For instance doing any of the following things will quickly turn into a nightmare when using the Bazel C++ rules: * dynamically linking libraries * using a containerized approach for library includes (so transitive relative include paths) * using different toolchains and cross compiling * interfacing with thirdparty libraries We are talking seasoned build engineers ending up frustrated after literally months of trying to achieve something that is easy as pie in CMake. In addition there is still no real IDE support. The CLion plugin is permanently broken and lags behind versions. No real VSCode support. Using custom rules makes this even worse, to the point where CLion will refuse to sync and not having a way to produce a compilation database json. There are so many bugs open on those projects and no progress or answers. I can not recommend Bazel if C++ or C is what you care about.

maybeiambatman 6y ago
FWIW, Uber ATG uses the same monorepo culture with Bazel. It has for some time now. And most of the stuff is written in C++.
q3k 6y ago
> We are talking seasoned build engineers ending up frustrated after literally months of trying to achieve something that is easy as pie in CMake. But it's not the same as CMake. A properly set up Bazel project gives you so much more: organization-wide incremental builds, build cache and build farm. Also, actual full hermeticity of builds (no, taking ambient deps from a Docker container doesn't replace that). Comparing CMake to Bazel is like comparing some barely-working bash scripts on a single box to a kubernetes deployment. Maybe you're okay with just bash scripts, but some of us aren't, and that's where Bazel comes in.
- simfoo 6y ago
  Completely agree that you can't compare Bazel and CMake directly, they are not on the same level. But for compiling C++ Bazel will have to compare itself to CMake, which is plain and simple the defacto standard. All the complex things that Bazel brings into the game (distributed builds and caches primarily) simply don't have any value when the main task that you're trying to make faster is not done well. I would much rather like to see more tooling that does these things but builds upon layers of battle-tested existing software.
- keithwinstein 6y ago
  I've been confused for a while about the common claim that Bazel gives "full hermeticity" of builds -- it doesn't seem to be true in practice (at least for packages with system dependencies). Maybe you can help me clear it up. E.g. Google's protobuf libraries [1] can be built with Bazel, and they depend heavily on system headers outside the repository, e.g. <iostream> and <stdio.h> and lots more. If those headers subsequently change, Bazel will not pick up on this and will not know to rebuild the parts of the build that depend on them. To reproduce: run `bazel build -c opt //:protoc_lib` and then put random garbage in your /usr/include/stdio.h and /usr/include/c++/<version>/iostream and then rerun the bazel command -- it will not know to invalidate the build cache. If you `bazel clean` and then build again, you'll get different results. Bazel does a lot of really nice things and I can believe that within a google3-like environment (where the source code never references a system header?), it effectively provides hermetic builds, but in practice as used outside Google (or even in Google's public OSS releases) it doesn't seem to really match this description or enforce a hermetic seal. What am I missing? [1] https://github.com/protocolbuffers/protobuf https://github.com/protocolbuffers/protobuf
  - kornholi 6y ago
    I wish Bazel was more explicit about things leaking in from outside the workspace, but you can always vendor the toolchain to be more hermetic. For example see https://github.com/grailbio/bazel-toolchain https://github.com/grailbio/bazel-toolchain which sets up LLVM and Clang for you. https://github.com/bazelbuild/bazel/issues/4558 https://github.com/bazelbuild/bazel/issues/4558
    - keithwinstein 6y ago
      Ok, but that decision would have to be made by the project maintainer, in this case Google, not the person using Bazel to compile protobuf. (And not particular to Bazel -- a developer can make any build system effectively hermetic by vendoring everything.) In my view the challenge here is that a dependency changes (e.g. /usr/include/stdio.h is upgraded by the system package manager, or two users sharing a cache have different versions of a system library) and Bazel doesn't realize that it needs to rebuild. It would be a pretty heavy hammer if the way to fix that requires every OSS project to include the whole user-space OS (C++ compiler, system headers, libraries) in the repo or via submodule and then be careful that no include path or any part of the build system accidentally references any header or library outside the repository. And maybe this issue just doesn't need to be fixed (it's not like automake produces build rules that explicitly depend on system headers either!) -- my quibble was with the notion that Bazel, unlike CMake or whatever, provides fully hermetic builds, or tracks dependencies carefully enough to provide an organization-wide build cache across diversely configured/upgraded systems.
majormajor 6y ago
I think the biggest thing to remember is that changing your build tool for a large project is going to be a massive effort, you're gonna need a LOT of hours to throw at it, and you might not find many benefits for a long time. Starting a new project with it, so it can help you do things the "right way" is a much more attractive proposition. That said, the IDE/tooling situation for Bazel was disappointing to me in the Java world too. It definitely lags and is incomplete.
throwaway894345 6y ago
I was keeping tabs on the Python plugin for a couple of years (but stopped checking in about a year ago) and Python3 was advertised as "supported" the whole time, but no one was able to get it working. There were a number of open issues, none with any action on them. And while Nix has an insane learning curve, figuring out how to extend Bazel (and consequently build one's own Python plugin) was a nightmare. The documentation was unhelpful/incomplete and the codebase was incomprehensible and poorly organized, with inheritance sprinkled on everything for its own sake (which is to say, par for the course for Java apps). I know... "Don't look a gift horse in the mouth! / Don't complain about open source software!" Fair enough, but I don't have to use it, and I'm free to caution others not to as well.
- sterlind 6y ago
  One of my side projects was setting up a K8s cluster that ran Hydra in containers, used Hydra to build packages, and Kubenix to build and deploy services to the cluster. It was a little closed loop.. my goal was to have it building and deploying itself fully. Unfortunately Kubenix was too immature, my knowledge of Nix insufficient, and Hydra required privileged containers due to Nix build sandboxing. Have you used nix as a primary build system rather than a meta build system? If it works, I might have the energy to revive my project.
  - throwaway894345 6y ago
    I'm not sure what you mean by "a meta build system" vs "a primary build system", but for many years Nix was what our company used to build and package our software. Unfortunately, it was a very bad experience; Nix doesn't strive to be a build system, it strives to be a package manager. The learning curve for writing packages is too steep to expect every engineer to learn it, and when things broke it was often hours and hours debugging into some obscure C library way down the dependency tree that no one in your organization has ever heard of. Worse, very little is documented well or accurately. Trying to figure out, for example, how Nix's python packaging libraries work is frustrating--partly because there is no obvious way to find the defining file for a given package dependency and partly because the package definitions in nixpkgs are rarely documented. These are all tractable problems, but Nix doesn't (seem to) aspire to solve them.
ajconway 6y ago
CMake is a natural fit for C/C++ (even though I strongly dislike the syntax). It has a good IDE project generation, and you can use ninja to build super fast. But it isn’t pre-configured for hermetic builds out of the box, remote, team-wide builds and caching are much further than a command line option away.
- kortex 6y ago
  You can use sccache with Cmake easily. On mobile so I can only give you breadcrumbs. 1. Set up sccache backing server (I use docker redis) 2. Export sccache config. Set SCCACHE_REDIS to a Redis url in format redis://[:<passwd>@]<hostname>[:port][/<db>] Set CMAKE_<LANG>_COMPILER_LAUNCHER (cc, cxx) to sccache.
gravypod 6y ago
> There are so many bugs open on those projects and no progress or answers. I can not recommend Bazel if C++ or C is what you care about. I think this is something that's slowly, but surely, changing. There's a lot of adoption of Bazel from some big players out there (Uber, Bloomberg, etc). The Blaze team @ Google has also started incorporating outside OSS maintainers into their OSS software (specifically rules_python). Things are improving. Many of the things you've addressed (third party libraries, cross compiling, dynamically linking things) have been talked about by many people in Bazel's slack. It might not be flawless currently but Bazel's abstractions of how builds work are cool, RBE/caching is amazing for large projects, and a single build system with the promise of gracefully handling "every" language is something our field has been needing for some time. In 3 to 10 years I can imagine being able to start writing code and never think about which language some library I want to use is written in, which package manager and toolchain I need to use to build everything, how to package my code to be deployable in production, how to setup unit tests, how to get autocompltions and docs and everything in my IDE, etc. The answer could just be "bazel, a language server, an IDE that speaks lsp".
m0zg 6y ago
Bazel has extensive support for cross compilation. You can find pre-made toolchains. Intefacing with third party libs is as easy as writing a BUILD file.

by simfoo 6y ago

maybeiambatman 6y ago

FWIW, Uber ATG uses the same monorepo culture with Bazel. It has for some time now. And most of the stuff is written in C++.

q3k 6y ago

> We are talking seasoned build engineers ending up frustrated after literally months of trying to achieve something that is easy as pie in CMake. But it's not the same as CMake. A properly set up Bazel project gives you so much more: organization-wide incremental builds, build cache and build farm. Also, actual full hermeticity of builds (no, taking ambient deps from a Docker container doesn't replace that). Comparing CMake to Bazel is like comparing some barely-working bash scripts on a single box to a kubernetes deployment. Maybe you're okay with just bash scripts, but some of us aren't, and that's where Bazel comes in.

simfoo 6y ago
Completely agree that you can't compare Bazel and CMake directly, they are not on the same level. But for compiling C++ Bazel will have to compare itself to CMake, which is plain and simple the defacto standard. All the complex things that Bazel brings into the game (distributed builds and caches primarily) simply don't have any value when the main task that you're trying to make faster is not done well. I would much rather like to see more tooling that does these things but builds upon layers of battle-tested existing software.
keithwinstein 6y ago
I've been confused for a while about the common claim that Bazel gives "full hermeticity" of builds -- it doesn't seem to be true in practice (at least for packages with system dependencies). Maybe you can help me clear it up. E.g. Google's protobuf libraries [1] can be built with Bazel, and they depend heavily on system headers outside the repository, e.g. <iostream> and <stdio.h> and lots more. If those headers subsequently change, Bazel will not pick up on this and will not know to rebuild the parts of the build that depend on them. To reproduce: run `bazel build -c opt //:protoc_lib` and then put random garbage in your /usr/include/stdio.h and /usr/include/c++/<version>/iostream and then rerun the bazel command -- it will not know to invalidate the build cache. If you `bazel clean` and then build again, you'll get different results. Bazel does a lot of really nice things and I can believe that within a google3-like environment (where the source code never references a system header?), it effectively provides hermetic builds, but in practice as used outside Google (or even in Google's public OSS releases) it doesn't seem to really match this description or enforce a hermetic seal. What am I missing? [1] https://github.com/protocolbuffers/protobuf https://github.com/protocolbuffers/protobuf
- kornholi 6y ago
  I wish Bazel was more explicit about things leaking in from outside the workspace, but you can always vendor the toolchain to be more hermetic. For example see https://github.com/grailbio/bazel-toolchain https://github.com/grailbio/bazel-toolchain which sets up LLVM and Clang for you. https://github.com/bazelbuild/bazel/issues/4558 https://github.com/bazelbuild/bazel/issues/4558
  - keithwinstein 6y ago
    Ok, but that decision would have to be made by the project maintainer, in this case Google, not the person using Bazel to compile protobuf. (And not particular to Bazel -- a developer can make any build system effectively hermetic by vendoring everything.) In my view the challenge here is that a dependency changes (e.g. /usr/include/stdio.h is upgraded by the system package manager, or two users sharing a cache have different versions of a system library) and Bazel doesn't realize that it needs to rebuild. It would be a pretty heavy hammer if the way to fix that requires every OSS project to include the whole user-space OS (C++ compiler, system headers, libraries) in the repo or via submodule and then be careful that no include path or any part of the build system accidentally references any header or library outside the repository. And maybe this issue just doesn't need to be fixed (it's not like automake produces build rules that explicitly depend on system headers either!) -- my quibble was with the notion that Bazel, unlike CMake or whatever, provides fully hermetic builds, or tracks dependencies carefully enough to provide an organization-wide build cache across diversely configured/upgraded systems.

majormajor 6y ago

I think the biggest thing to remember is that changing your build tool for a large project is going to be a massive effort, you're gonna need a LOT of hours to throw at it, and you might not find many benefits for a long time. Starting a new project with it, so it can help you do things the "right way" is a much more attractive proposition. That said, the IDE/tooling situation for Bazel was disappointing to me in the Java world too. It definitely lags and is incomplete.

throwaway894345 6y ago

I was keeping tabs on the Python plugin for a couple of years (but stopped checking in about a year ago) and Python3 was advertised as "supported" the whole time, but no one was able to get it working. There were a number of open issues, none with any action on them. And while Nix has an insane learning curve, figuring out how to extend Bazel (and consequently build one's own Python plugin) was a nightmare. The documentation was unhelpful/incomplete and the codebase was incomprehensible and poorly organized, with inheritance sprinkled on everything for its own sake (which is to say, par for the course for Java apps). I know... "Don't look a gift horse in the mouth! / Don't complain about open source software!" Fair enough, but I don't have to use it, and I'm free to caution others not to as well.

sterlind 6y ago
One of my side projects was setting up a K8s cluster that ran Hydra in containers, used Hydra to build packages, and Kubenix to build and deploy services to the cluster. It was a little closed loop.. my goal was to have it building and deploying itself fully. Unfortunately Kubenix was too immature, my knowledge of Nix insufficient, and Hydra required privileged containers due to Nix build sandboxing. Have you used nix as a primary build system rather than a meta build system? If it works, I might have the energy to revive my project.
- throwaway894345 6y ago
  I'm not sure what you mean by "a meta build system" vs "a primary build system", but for many years Nix was what our company used to build and package our software. Unfortunately, it was a very bad experience; Nix doesn't strive to be a build system, it strives to be a package manager. The learning curve for writing packages is too steep to expect every engineer to learn it, and when things broke it was often hours and hours debugging into some obscure C library way down the dependency tree that no one in your organization has ever heard of. Worse, very little is documented well or accurately. Trying to figure out, for example, how Nix's python packaging libraries work is frustrating--partly because there is no obvious way to find the defining file for a given package dependency and partly because the package definitions in nixpkgs are rarely documented. These are all tractable problems, but Nix doesn't (seem to) aspire to solve them.

ajconway 6y ago

CMake is a natural fit for C/C++ (even though I strongly dislike the syntax). It has a good IDE project generation, and you can use ninja to build super fast. But it isn’t pre-configured for hermetic builds out of the box, remote, team-wide builds and caching are much further than a command line option away.

kortex 6y ago
You can use sccache with Cmake easily. On mobile so I can only give you breadcrumbs. 1. Set up sccache backing server (I use docker redis) 2. Export sccache config. Set SCCACHE_REDIS to a Redis url in format redis://[:<passwd>@]<hostname>[:port][/<db>] Set CMAKE_<LANG>_COMPILER_LAUNCHER (cc, cxx) to sccache.

gravypod 6y ago

> There are so many bugs open on those projects and no progress or answers. I can not recommend Bazel if C++ or C is what you care about. I think this is something that's slowly, but surely, changing. There's a lot of adoption of Bazel from some big players out there (Uber, Bloomberg, etc). The Blaze team @ Google has also started incorporating outside OSS maintainers into their OSS software (specifically rules_python). Things are improving. Many of the things you've addressed (third party libraries, cross compiling, dynamically linking things) have been talked about by many people in Bazel's slack. It might not be flawless currently but Bazel's abstractions of how builds work are cool, RBE/caching is amazing for large projects, and a single build system with the promise of gracefully handling "every" language is something our field has been needing for some time. In 3 to 10 years I can imagine being able to start writing code and never think about which language some library I want to use is written in, which package manager and toolchain I need to use to build everything, how to package my code to be deployable in production, how to setup unit tests, how to get autocompltions and docs and everything in my IDE, etc. The answer could just be "bazel, a language server, an IDE that speaks lsp".

m0zg 6y ago

Bazel has extensive support for cross compilation. You can find pre-made toolchains. Intefacing with third party libs is as easy as writing a BUILD file.