14 ms·

Chromium Switching from Ninja to Siso

117 points by hortense 1y ago

jefftk 1y ago
I think this makes Chrome build system number four: scons, gyp, ninja, siso.
- enqk 1y ago
  gyp to gn gyp could produce ninja files. gn is short for generate ninja. Now gn produces files compatible with siso
  - csiegert 1y ago
    Please add punctuation. I had to read it twice to understand.
  - jefftk 1y ago
    Producing ninja files was an intermediate phase in the transition to gn. For most of its lifetime it was producing makefile.
- deleted 1y ago
  [deleted]
RainyDayTmrw 1y ago
Kinda impressive and terrifying that Chromium needs its own build system. Kinda strange that Bazel was right there, also from Google, and they not only choose not to use it, but also reference it in the name of the new tool.
- operator-name 1y ago
  If Android was anything to go by, migrating build systems is a risky endeavour. Miso claims compatibility with ninja, so I’m guessing this route was deemed easier to make incremental improvements.
- jinwoo68 1y ago
  To be fair, Bazel didn't exist when Chrome started.
  - dijit 1y ago
    Even so, a build system migration of any kind being anything other than Bazel given the design goals of Bazel and the heritage of Chrome is an implicit indictment of Bazel itself. I say this as a fan of Bazel.
    - arccy 1y ago
      moving to bazel might be a lot more work vs a drop in replacement which is what siso looks like atm.
      - dijit 1y ago
        Yeah, thats fair, but if I understood right- this is a custom built tool to be compatible with Ninja. That work building “yet another build tool” could have gone in to programmatically generating bazel BUILD files. So, there was an active choice here somewhere; we just don’t know all the information as to why effort was diverted away from Bazel and toward building a new tool. I trust them to make good decisions, so I would like to understand more. :) Seems like Siso supports Starlark, so maybe its a step in Bazels direction after all.
        intexpress 1y ago
        > That work building “yet another build tool” could have gone in to programmatically generating bazel BUILD files. Google did try that for Android (AOSP). They made a tool that generated thousands? of BUILD files to get AOSP building with Bazel. But the migration to Bazel was aborted for unknown reasons.
        skybrian 1y ago
        It reminds me of how Blaze (which became Bazel) was designed to be mostly compatible with the build files of a previous build system written in Python.
- tonyedgecombe 1y ago
  >Kinda impressive and terrifying that Chromium needs its own build system. This is one area where I think it makes some sense to build your own. Most projects only use a fraction of the capability of a typical build system. The last time I did this I managed the whole thing in just 300 lines of code.
  - bluGill 1y ago
    I stronly disagree. While you use a fraction of the features you often mistake it works for it is right and to something subtle is wrong. You end up either not working or dedicating a lot of effort fixing weird corner cases that you didn't think of
    - tonyedgecombe 1y ago
      >You end up either not working or dedicating a lot of effort fixing weird corner cases that you didn't think of That's not my experience at all. Part of my job is to think about weird corner cases, I'm not sure why a build script should be any worse than what I normally deal with.
      - bluGill 1y ago
        It isn't worse than other corner cases - but for almost everyone it isn't what their boss wants them to spend all their time on. Even if working for your own fun you probably want to do something else.
      - bluGill 1y ago
        Now that I'm on a real computer (keyboard) maybe I can elaborate a bit. It isn't hard to create a basic build system. However there are a lot more rare corner cases than anyone who sets out to create one as a side project realizes and so in almost all cases their build systems work for the easy/common situations, but in lots of weird cases it doesn't work. Often those corner cases will happen on setups that make sense for one exactly person in the world but for everyone else is stupid. (think cross compiling on haiku-os for a vax running netbsd) Which is to say if you want to create a build system that needs to be your only goal. You can maybe make the goal create the build system that some project needs (Chromium in this case) needs, but you won't be doing any hacking on the project (Chromium) itself because all your time will be spent on the build system. Since everyone I know thinks of build systems as a side project they should just choose one that works. I use cmake, which works well despite the warts. I've heard good things about build2 and bazel, but I have no experience with either. I have used ninja as well (that is handwritten ninja files), but it isn't intended for that purpose and so in general I wouldn't recommend it.
charcircuit 1y ago
I wonder if AOSP will also move over to Siso. Since it is advertised as a drop in replacement it would take less resources than the Bazel migration which got canceled. The readme explicitly calls out a feature used by AOSP, so it is plausible that thought was put into it.
mrlonglong 1y ago
They've added a dependency on Go with the addition of siso.
- lordofgibbons 1y ago
  How so? you don't need the go build toolchain installed to use Go binaries.
  - throwaway81523 1y ago
    If I'm going to compile Chromium from source, I probably want to compile the build system from source.
    - ahahahahah 1y ago
      Why? Would you compile the compiler from source as well? From what? You need to compile the compiler's compiler from source as well, right? Where does it stop? And why is that location more valid of a decision than the one that doesn't require building the build system from source?
      - kokada 1y ago
        Same can be extended for other tools that are generally used in builds like make. I never heard someone say that they need to build make from source so they can build X, unless of course you're using something like Linux From Scratch.
        fc417fc802 1y ago
        I build cmake from source but TBF that's because projects sometimes depend on specific version ranges for features that they use (either brand new or recently removed).
      - PhilipRoman 1y ago
        Not really relevant here, but this is actually exactly how it's done in embedded systems like Yocto, everything from gcc, make, etc. is built from source (I believe the host compiler is used in a 3-stage bootstrapping process for gcc). And in these cases you really see the impact of internal dependencies (building rust/llvm takes around 30-40% of the entire build). The upside is that you can patch and debug absolutely any part of the system.
txdv 1y ago
Is there an explanation what siso provides that ninja does not?
- operator-name 1y ago
  https://pkg.go.dev/go.chromium.org/infra/build/siso#readme-key-differences-from-ninja https://pkg.go.dev/go.chromium.org/infra/build/siso#readme-k...
- nikanj 1y ago
  It’s new
- out-of-ideas 1y ago
  > Siso is developed by the Chrome Build Infra Team at Google as a drop-in replacement for Ninja to support remote execution natively.
- IshKebab 1y ago
  Yes, if you click the article and read it it tells you.
operator-name 1y ago
https://pkg.go.dev/go.chromium.org/infra/build/siso#section-readme https://pkg.go.dev/go.chromium.org/infra/build/siso#section-... Ninja compatible, for the flags that chromium uses. There’s some behaviour they’ve tuned for their use case.
kosolam 1y ago
Looks like the Google developer in the discussion is clueless.
- kosolam 1y ago
  [flagged]
  - tomhow 1y ago
    Please don't post insinuations about astroturfing, shilling, brigading, foreign agents, and the like. It degrades discussion and is usually mistaken. If you're worried about abuse, email hn@ycombinator.com and we'll look at the data. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- kosolam 1y ago
  I just wanted to point out after reading the thread of the Chromium maintainers that it looked to me like they rushed out this change without proper consideration of all possible implications. Other comments there show that the developer isn’t intimately familiar in depth with the Chromium project at large, yet making changes with very wide impact. I apologize if anyone got me wrong, especially if there are any Googler here that took this personally.
kookamamie 1y ago
If Siso is a drop-in alternative to Ninja, are there builds of Siso usable for building Ninja-projects, in general?
intexpress 1y ago
Note that Siso uses Starlark (https://chromium.googlesource.com/infra/infra/+/refs/heads/main/go/src/infra/build/siso/docs/starlark_config.md https://chromium.googlesource.com/infra/infra/+/refs/heads/m...) which is the build language of Bazel I wonder if the end goal is to use Bazel for Chromium and Siso is an incremental step to get there
- oldmanhorton 1y ago
  A handful of other areas are configured using Starlark in chromium. This particular use is in a very different capacity than Bazel - the Bazel equivalent in chromium is GN, and I have not seen any signs that GN will be replaced any time soon.
  - tomjakubowski 1y ago
    GN at least used to generate Ninja files. So I suppose now it will be generating Siso files? edit: asked and answered, Siso is a "drop-in" replacement for Ninja, so presumably it can read .ninja files, and so GN probably didn't need to change much to accommodate it.
pjmlp 1y ago
Another NIH build tool? Never heard of it, and ninja is the only way to build C++20 modules alongside CMake. Then again, that is something that Chromium probably will never bother with.
- kichik 1y ago
  Wasn't ninja invented by a Google employee?
  - pjmlp 1y ago
    Doesn't change the situation of being yet another NIH tool, because Google departments have so few of them.
  - rdtsc 1y ago
    Sure, but only that team gets to put “designed and implemented new build system” on their resume. See how many meet/hangout/alo variants came out of Google. In companies of that size the “here” in NIH is a lot more localized to smaller units.
curtisszmania 1y ago
[dead]
ahartmetz 1y ago
They want to "improve their build processes", huh? Yeah, there are some problems, like ginormous build times, for which there are some very well-known and pedestrian solutions ("physical code design"). But these aren't impressive enough to get promoted. Just Google things.

117 points by hortense 1y ago

jefftk 1y ago

I think this makes Chrome build system number four: scons, gyp, ninja, siso.

enqk 1y ago
gyp to gn gyp could produce ninja files. gn is short for generate ninja. Now gn produces files compatible with siso
- csiegert 1y ago
  Please add punctuation. I had to read it twice to understand.
- jefftk 1y ago
  Producing ninja files was an intermediate phase in the transition to gn. For most of its lifetime it was producing makefile.
deleted 1y ago
[deleted]

RainyDayTmrw 1y ago

Kinda impressive and terrifying that Chromium needs its own build system. Kinda strange that Bazel was right there, also from Google, and they not only choose not to use it, but also reference it in the name of the new tool.

operator-name 1y ago
If Android was anything to go by, migrating build systems is a risky endeavour. Miso claims compatibility with ninja, so I’m guessing this route was deemed easier to make incremental improvements.
jinwoo68 1y ago
To be fair, Bazel didn't exist when Chrome started.
- dijit 1y ago
  Even so, a build system migration of any kind being anything other than Bazel given the design goals of Bazel and the heritage of Chrome is an implicit indictment of Bazel itself. I say this as a fan of Bazel.
  - arccy 1y ago
    moving to bazel might be a lot more work vs a drop in replacement which is what siso looks like atm.
    - dijit 1y ago
      Yeah, thats fair, but if I understood right- this is a custom built tool to be compatible with Ninja. That work building “yet another build tool” could have gone in to programmatically generating bazel BUILD files. So, there was an active choice here somewhere; we just don’t know all the information as to why effort was diverted away from Bazel and toward building a new tool. I trust them to make good decisions, so I would like to understand more. :) Seems like Siso supports Starlark, so maybe its a step in Bazels direction after all.
      - intexpress 1y ago
        > That work building “yet another build tool” could have gone in to programmatically generating bazel BUILD files. Google did try that for Android (AOSP). They made a tool that generated thousands? of BUILD files to get AOSP building with Bazel. But the migration to Bazel was aborted for unknown reasons.
      - skybrian 1y ago
        It reminds me of how Blaze (which became Bazel) was designed to be mostly compatible with the build files of a previous build system written in Python.
tonyedgecombe 1y ago
>Kinda impressive and terrifying that Chromium needs its own build system. This is one area where I think it makes some sense to build your own. Most projects only use a fraction of the capability of a typical build system. The last time I did this I managed the whole thing in just 300 lines of code.
- bluGill 1y ago
  I stronly disagree. While you use a fraction of the features you often mistake it works for it is right and to something subtle is wrong. You end up either not working or dedicating a lot of effort fixing weird corner cases that you didn't think of
  - tonyedgecombe 1y ago
    >You end up either not working or dedicating a lot of effort fixing weird corner cases that you didn't think of That's not my experience at all. Part of my job is to think about weird corner cases, I'm not sure why a build script should be any worse than what I normally deal with.
    - bluGill 1y ago
      It isn't worse than other corner cases - but for almost everyone it isn't what their boss wants them to spend all their time on. Even if working for your own fun you probably want to do something else.
    - bluGill 1y ago
      Now that I'm on a real computer (keyboard) maybe I can elaborate a bit. It isn't hard to create a basic build system. However there are a lot more rare corner cases than anyone who sets out to create one as a side project realizes and so in almost all cases their build systems work for the easy/common situations, but in lots of weird cases it doesn't work. Often those corner cases will happen on setups that make sense for one exactly person in the world but for everyone else is stupid. (think cross compiling on haiku-os for a vax running netbsd) Which is to say if you want to create a build system that needs to be your only goal. You can maybe make the goal create the build system that some project needs (Chromium in this case) needs, but you won't be doing any hacking on the project (Chromium) itself because all your time will be spent on the build system. Since everyone I know thinks of build systems as a side project they should just choose one that works. I use cmake, which works well despite the warts. I've heard good things about build2 and bazel, but I have no experience with either. I have used ninja as well (that is handwritten ninja files), but it isn't intended for that purpose and so in general I wouldn't recommend it.

charcircuit 1y ago

I wonder if AOSP will also move over to Siso. Since it is advertised as a drop in replacement it would take less resources than the Bazel migration which got canceled. The readme explicitly calls out a feature used by AOSP, so it is plausible that thought was put into it.

mrlonglong 1y ago

They've added a dependency on Go with the addition of siso.

lordofgibbons 1y ago
How so? you don't need the go build toolchain installed to use Go binaries.
- throwaway81523 1y ago
  If I'm going to compile Chromium from source, I probably want to compile the build system from source.
  - ahahahahah 1y ago
    Why? Would you compile the compiler from source as well? From what? You need to compile the compiler's compiler from source as well, right? Where does it stop? And why is that location more valid of a decision than the one that doesn't require building the build system from source?
    - kokada 1y ago
      Same can be extended for other tools that are generally used in builds like make. I never heard someone say that they need to build make from source so they can build X, unless of course you're using something like Linux From Scratch.
      - fc417fc802 1y ago
        I build cmake from source but TBF that's because projects sometimes depend on specific version ranges for features that they use (either brand new or recently removed).
    - PhilipRoman 1y ago
      Not really relevant here, but this is actually exactly how it's done in embedded systems like Yocto, everything from gcc, make, etc. is built from source (I believe the host compiler is used in a 3-stage bootstrapping process for gcc). And in these cases you really see the impact of internal dependencies (building rust/llvm takes around 30-40% of the entire build). The upside is that you can patch and debug absolutely any part of the system.

txdv 1y ago

Is there an explanation what siso provides that ninja does not?

operator-name 1y ago
https://pkg.go.dev/go.chromium.org/infra/build/siso#readme-key-differences-from-ninja https://pkg.go.dev/go.chromium.org/infra/build/siso#readme-k...
nikanj 1y ago
It’s new
out-of-ideas 1y ago
> Siso is developed by the Chrome Build Infra Team at Google as a drop-in replacement for Ninja to support remote execution natively.
IshKebab 1y ago
Yes, if you click the article and read it it tells you.

operator-name 1y ago

https://pkg.go.dev/go.chromium.org/infra/build/siso#section-readme https://pkg.go.dev/go.chromium.org/infra/build/siso#section-... Ninja compatible, for the flags that chromium uses. There’s some behaviour they’ve tuned for their use case.

kosolam 1y ago

Looks like the Google developer in the discussion is clueless.

kosolam 1y ago
[flagged]
- tomhow 1y ago
  Please don't post insinuations about astroturfing, shilling, brigading, foreign agents, and the like. It degrades discussion and is usually mistaken. If you're worried about abuse, email hn@ycombinator.com and we'll look at the data. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
kosolam 1y ago
I just wanted to point out after reading the thread of the Chromium maintainers that it looked to me like they rushed out this change without proper consideration of all possible implications. Other comments there show that the developer isn’t intimately familiar in depth with the Chromium project at large, yet making changes with very wide impact. I apologize if anyone got me wrong, especially if there are any Googler here that took this personally.

kookamamie 1y ago

If Siso is a drop-in alternative to Ninja, are there builds of Siso usable for building Ninja-projects, in general?

intexpress 1y ago

Note that Siso uses Starlark (https://chromium.googlesource.com/infra/infra/+/refs/heads/main/go/src/infra/build/siso/docs/starlark_config.md https://chromium.googlesource.com/infra/infra/+/refs/heads/m...) which is the build language of Bazel I wonder if the end goal is to use Bazel for Chromium and Siso is an incremental step to get there

oldmanhorton 1y ago
A handful of other areas are configured using Starlark in chromium. This particular use is in a very different capacity than Bazel - the Bazel equivalent in chromium is GN, and I have not seen any signs that GN will be replaced any time soon.
- tomjakubowski 1y ago
  GN at least used to generate Ninja files. So I suppose now it will be generating Siso files? edit: asked and answered, Siso is a "drop-in" replacement for Ninja, so presumably it can read .ninja files, and so GN probably didn't need to change much to accommodate it.

pjmlp 1y ago

Another NIH build tool? Never heard of it, and ninja is the only way to build C++20 modules alongside CMake. Then again, that is something that Chromium probably will never bother with.

kichik 1y ago
Wasn't ninja invented by a Google employee?
- pjmlp 1y ago
  Doesn't change the situation of being yet another NIH tool, because Google departments have so few of them.
- rdtsc 1y ago
  Sure, but only that team gets to put “designed and implemented new build system” on their resume. See how many meet/hangout/alo variants came out of Google. In companies of that size the “here” in NIH is a lot more localized to smaller units.

curtisszmania 1y ago

[dead]

ahartmetz 1y ago

They want to "improve their build processes", huh? Yeah, there are some problems, like ginormous build times, for which there are some very well-known and pedestrian solutions ("physical code design"). But these aren't impressive enough to get promoted. Just Google things.