9 ms·

Zb: An Early-Stage Build System

254 points by zombiezen 2y ago

evanjrowley 2y ago
This looks really exciting and I absolutely must give it a try. Well done! At face value the vision and design choices appear to be great.
- zombiezen 2y ago
  Thank you! <3
ramon156 2y ago
I'd definitely write a build systen in lua, looks promising!
Iceland_jack 2y ago
I appreciate the link to https://dl.acm.org/doi/10.1145/3236774 https://dl.acm.org/doi/10.1145/3236774
- o11c 2y ago
  Definitely interesting, but it's flat-out wrong about the limitations of `make`. In particular, the `release.txt` task is trivial by adding a dummy rule to generate and include dependencies; see https://www.gnu.org/software/make/manual/html_node/Remaking-Makefiles.html https://www.gnu.org/software/make/manual/html_node/Remaking-... (be sure to add empty rules to handle the case of deleted dynamic dependencies). You can use hashes instead of file modification times by adding a different kind of dummy rule. The only downside is that you have to think about the performance a little. I imagine it's possible for a project to have some kind of dynamic dependencies that GNU make can't handle, but I dare say that any such dependency tree is hard to understand for humans too, and thus should be avoided regardless. By contrast, in many other build tools it is impossible to handle some of the things that are trivial in `make`. (if you're not using GNU make, you are the problem; do not blame `make`)
  - bjourne 2y ago
    I guess you aren't keen on Java then? Complex dynamic dependency graphs aren't difficult for humans to handle or many build tools other than make.
    - throwaway2037 2y ago
      > I guess you aren't keen on Java then? Can you explain more? I don't follow.
      - bjourne 2y ago
        Java dependencies are too complicated for make. See https://www.oreilly.com/library/view/managing-projects-with/0596006101/ch09.html https://www.oreilly.com/library/view/managing-projects-with/...
    - o11c 2y ago
      I'm not keen on Java for other reasons. The fact that a single .java file can generate multiple .class files is annoying but not something tools can't handle (it's actually similar to .h files for C/C++ - remember, we only need the complete dependency graph for the rebuild, not the current build). The main part that's difficult for humans is if there's a non-public class at top level rather than nested (I forget all the Java-specific terminology for the various kinds of class nesting).
packetlost 2y ago
One request that I would make of a project like this is to support distributed builds out of the box. Like, really basic support for identical builder hosts (this is much easier now than in the past with containers) and caching of targets. Otherwise, this looks great! Big fan of the choice of Lua, though the modifications to strings might make it difficult to onboard new users depending on how the modification was made.
- zombiezen 2y ago
  Yup, remote building and caching is on my radar. I expect it will work much in the same way Nix does now, although I'm being a bit more deliberate in creating an RPC layer so build coordinators and other such tools are more straightforward to build. The string tweak is transparent to users broadly speaking. IME with Nix this thing works the way people expect (i.e if you use a dependency variable in your build target, it adds a dependency).
- Glacia 2y ago
  Xmake?
bsnnkv 2y ago
You had my interest at Windows support! I'll carve out some time this weekend to see if I can write a build for komorebi
- zombiezen 2y ago
  Nice! It might be a little too rough until I've got a working C compiler for Windows: https://github.com/256lights/zb/issues/6 https://github.com/256lights/zb/issues/6 (and Linux for that matter: https://github.com/256lights/zb/issues/30 https://github.com/256lights/zb/issues/30)
steeleduncan 2y ago
Looks great, Nix-with-Lua that also supports Windows would be amazing. Two questions if I may - Does this sandbox builds the way flakes do? - What is MinGW used for on Windows? Does this rely on the MinGW userland, or is it just because it would be painful to write a full bootstrap for a windows compiler while also developing Zb? Also, its great to see the live-bootstrap in there. I love the purity of how Guix's packages are built, and I like the idea Zb will be that way from the start
- zombiezen 2y ago
  Nix sandboxes derivation runs on Linux even without flakes, and I'm planning on implementing that, yes: https://github.com/256lights/zb/issues/29 https://github.com/256lights/zb/issues/29 and https://github.com/256lights/zb/issues/31 https://github.com/256lights/zb/issues/31 MinGW is used to build Lua using cgo. I'd like to remove that part, see https://github.com/256lights/zb/issues/28 https://github.com/256lights/zb/issues/28 I haven't started the userspace for Windows yet (https://github.com/256lights/zb/issues/6 https://github.com/256lights/zb/issues/6), but I suspect that it will be more "download the Visual C++ compiler binary from this URL" than the Linux source bootstrap. Yeah, I'm happy with live-bootstrap, too! I tried emulating Guix's bootstrap, but it depended a little too much on Scheme for me to use as-is. live-bootstrap has mostly worked out-of-the-box, which was a great validation test for this approach.
  - steeleduncan 2y ago
    Thanks for answering and I really hope it works out. A Nix alternative with less friction would be very welcome!
laurentlb 2y ago
I'd like to know more about the "Support for non-determinism" and how that differs from other build systems. Usually, build systems rerun actions when at least one of the inputs has changed. Are non-deterministic targets rerun all the time? Also, I'm curious to know if you've considered using Starlark or the build file syntax used in multiple other recent build systems (Bazel, Buck, Please, Pants).
- zombiezen 2y ago
  (Hi! I recognize your name from Bazel mailing lists but I forget whether we've talked before.) I'm mostly contrasting from Nix, which has difficulty with poisoning cache when faced with non-deterministic build steps when using input-addressing (the default mode). If zb encounters a build target with multiple cached outputs for the same inputs, it rebuilds and then relies on content-addressing to obtain build outputs for subsequent steps if possible. (I have an open issue for marking a target as intentionally non-deterministic and always triggering this re-run behavior: https://github.com/256lights/zb/issues/33 https://github.com/256lights/zb/issues/33) I'll admit I haven't done my research into how Bazel handles non-determinism, especially nowadays, so I can't remark there. I know from my Google days that even writing genrules you had to be careful about introducing non-determinism, but I forget how that failure mode plays out. If you have a good link (or don't mind giving a quick summary), I'd love to read up. I have considered Starlark, and still might end up using it. The critical feature I wanted to bolt in from Nix was having strings carrying dependency information (see https://github.com/NixOS/nix/blob/2f678331d59451dd6f1d9512cb6d92e4ecb9750f/src/libexpr/value.hh#L192-L211 https://github.com/NixOS/nix/blob/2f678331d59451dd6f1d9512cb... for a description of the feature). In my prototyping, this was pretty simple to bolt on to Lua, but I'm not sure how disruptive that would be to Starlark. Nix configurations tend to be a bit more complex than Bazel ones, so having a more full-featured language felt more appropriate. Still exploring the design space!
  - aseipp 2y ago
    I mean, to be fair, Nix is nothing more than a big ass pile of genrule() calls, at the end of the day. Everything is really just genrule. Nix just makes it all work with the sandbox it puts all builds in. Bazel has an equivalent sandbox and I'm pretty sure you can sandbox genrule so it's in a nice, hermetic container. (Side note, but one of my biggest pet peeves is that Nix without the sandbox is actually fundamentally _broken_, yet we let people install it without the sandbox. I have no idea why "Install this thing in a broken way!" is even offered as an option. Ridiculous.) The way Nix-like systems achieve hermetic sandboxing isn't so much a technical feat, in my mind. That's part of it -- sure, you need to get rid of /dev devices, and every build always has to look like it happens at /tmp/build within a mount namespace, and you need to set SOURCE_EPOCH_DATE and blah blah, stuff like that. But it's also a social one, because with Nix you are expected to wrap arbitrary build systems and package mechanisms and "go where they are." That means you have to bludgeon every random hostile badly written thing into working inside the sandbox you designed, carve out exceptions, and write ptaches for things that don't -- and get them working in a deterministic way. For example, you have to change the default search paths for nearly every single tool to look inside calculated Nix store path. That's not a technical feat, it's mostly just a huge amount of hard work to write all the abstractions, like buildRustPackage or makeDerivation. You need to patch every build system like CMake or Scons in order to alleviate some of their assumptions, and so on and so forth. Bazel and Buck like systems do not avoid this pain but they do pay for it in a different way. They don't "go where they are", they expect everyone to "come to them." Culturally, Bazel users do not accept "just run Make under a sandbox" nearly as much. The idea is to write everything as a BUILD file rule, from scratch rewriting the build system, and those BUILD files instead should perform the build "natively" in a way that is designed to work hermetically. So you don't run ./configure, you actually pick an exact set of configuration options and build with that 100% of the time. Therefore, the impurities in the build are removed "by design", which makes the strict requirements on a sandbox somewhat more lenient. You still need the sandbox, but by definition your builds are much more robust anyway. So you are trading the pain of wrapping every system for the pain of integrating every system manually. They're not the same thing but have a lot of overlap. So the answer is, yes you can write impure genrules, but the vast majority of impurity is totally encapsulated in a way that forces it to be pure, just like Nix, so it's mostly just a small nit rather than truly fundamental. The real question is a matter of when you want to pay the pied piper.
msvan 2y ago
As a current Nix user, what I would really like is a statically typed language to define builds. Recreating Nix without addressing that feels like a missed opportunity.
- zombiezen 2y ago
  The Lua VSCode extension adds a type system that works really well IME
  - 0cf8612b2e1e 2y ago
    There are Lua flavors with typing. Teal is one I have heard that compiles down to regular Lua like a typescript
- Rucadi 2y ago
  For me the killer feature is Windows Support, Ericsson is doing a great job bringing nix into Windows, but the process it's understandably slow, If this project is similar enough to nix that I can kind-off translate easily the zb derivations to nix derivations, I'm willing to use it in windows (It's not like nix has windows programs in the nixpkgs either way I have to bring them in my own). The problem for me is that I see no benefit on using this over nix language (which I kinda like a lot right now)
  - droelf 2y ago
    We're working on rattler-build (https://github.com/prefix-dev/rattler-build/ https://github.com/prefix-dev/rattler-build/) - which is a build system inspired by Apko / conda-build and uses YAML files to statically define dependencies. It works really well with pixi (our package manager) but also any other conda compatible package managers (mamba, conda). And it has Windows support, of course. It can also be used to build your own distribution (e.g. here is one for a bunch of Rust utilities: https://github.com/wolfv/rust-forge https://github.com/wolfv/rust-forge)
  - hamandcheese 2y ago
    > Ericsson is doing a great job bringing nix into Windows Is this Ericsson... the corporation? Windows support for nix is something I don't hear much about, but if there is progress being made (even slowly) I'd love to know more.
mikepurvis 2y ago
Whoa, nifty. Can you speak more to the interop issues with Nix? I've been working on a pretty large Nix deployment in the robotics space for the past 3ish years, and the infrastructure side is the biggest pain point: * Running a bare `nix build` in your CI isn't really enough— no hosted logs, lack of proper prioritization, may end up double-building things. * Running your own instance of Hydra is a gigantic pain; it's a big ball of perl and has compiled components that link right into Nix internals, and architectural fiasco. * SaaS solutions are limited and lack maturity (Hercules CI is Github-only, nixbuild.net is based in Europe and last I checked was still missing some features I needed). * Tvix is cool but not ready for primetime, and the authors oppose flakes, which is a deal-breaker for me. Something that's a barebones capable of running these builds and could be wrapped in a sane REST API and simple web frontend would be very appealing.
- Rucadi 2y ago
  Why are flakes such a deal-breaker? While not ideal, you can still tag your versions in the .nix file instead of the lockfile. I even had to avoid flakes in a system I developed used by ~200 developers since it involved a non-nixos OS and it involved user secrets (Tokens etc...) So with flakes I had to keep track of the secrets (and was a pain point, since they obviously didn't have to push them into the git repo) but nix flakes doesn't handle well omitting files on git (it ignores them also on nix commands). In the end, the workarounds were too messy and had to drop flakes entirely.
  - mikepurvis 2y ago
    As a new user, I learned flakes first, and the tie-in with git tags/branches and the corresponding cli ergonomics aren’t something I’d be able to give up.
    - JasonSage 2y ago
      How do you handle flakes pushing an entire copy of the repo into the nix store? Is this not an issue for you somehow?
imiric 2y ago
Happy to see someone inspired by Nix, but wanting to carve their own path. Nix popularized some powerful ideas in the Linux world, but it has a steep learning curve and a very unfriendly UI, so there is plenty of room for improvement there. I'm not sure if Lua is the right choice, though. A declarative language seems like a better fit for reproducibility. The goal of supporting non-deterministic builds also seems to go against this. But I'm interested to know how this would work in practice. Good luck!
- hinkley 2y ago
  If you design it like SCons, it'll look imperative but behave more declaratively. If I understand the architecture correctly, the imperative calls in the config file don't actually run the build process. They run a Builder Pattern that sets up the state machine necessary for the builds to happen. So it's a bit like LINQ in C# (but older). I have no idea how that plays out single-step debugging build problems though. That depends on how it's implemented and a lot of abstractions (especially frameworks) seem to forget that breakpoints are things other people want to use as well.
  - zombiezen 2y ago
    That's accurate (unless the config file attempts to read something from the build process, that will trigger a build). It's a good point about debugging build problems. This is an issue I've experienced in Nix and Bazel as well. I'm not convinced that I have a great solution yet, but at least for my own debugging while using the system, I've included a `zb derivation env` command which spits out a .env file that matches the environment the builder runs under. I'd like to extend that to pop open a shell.
    - skybrian 2y ago
      One thing I like to see is a 'dry run' like 'make -n'. Although, maybe that's not possible in all cases. Another possibility might be to output a something like a shell script that would do a rebuild the same way, so you can see what it did and hack it when debugging.
IshKebab 2y ago
Interesting. I feel like I would have gone with Starlark over Lua, but I guess it's good to have options. Does it support sandboxing?
- zombiezen 2y ago
  Not yet, but I've hacked up most of the Linux sandboxing: https://github.com/256lights/zb/issues/29 https://github.com/256lights/zb/issues/29 I want to introduce Windows sandboxing, too, but I'm not as confident about how to do that: https://github.com/256lights/zb/issues/31 https://github.com/256lights/zb/issues/31
  - zombiezen 2y ago
    Oh and as for Starlark, I went into more detail over in this thread: https://news.ycombinator.com/item?id=41596426 https://news.ycombinator.com/item?id=41596426
- israrkhan 2y ago
  You need bazel if you need starlark & sandboxing
  - IshKebab 2y ago
    Well yeah. Starlark and sandboxing are the best things about Bazel, but it could still definitely be improved. So I'm still curious about other build systems. I think making a new build system without sandboxing (or at least a plan for it) would be pretty stupid. Fortunately he is planning it.
alxmng 2y ago
Did you consider writing a nicer language that compiles to Nix? A "friendly" tool on the outside with Nix inside.
- zombiezen 2y ago
  Yup, that was how I built the prototype: https://www.zombiezen.com/blog/2024/06/zb-build-system-prototype/ https://www.zombiezen.com/blog/2024/06/zb-build-system-proto... The last commit using that approach was https://github.com/256lights/zb/tree/558c6f52b7ef915428c9af9a9997edca84dfd831 https://github.com/256lights/zb/tree/558c6f52b7ef915428c9af9... if you want to try it out. And actually, I haven't touched the Lua frontend much since I swapped out the backend: the .drv files it writes are the same. The motivation behind replacing the backend was content-addressibility and Windows support, which have been slow to be adopted in Nix core.
- Rucadi 2y ago
  I don't think nix is that awful, while there are some tasks that are more difficult or can be a little bit verbose (if you want to play a lot with the attribute sets / lists or string manip) When using nix most of the time you'll end up just writing bash or using it as a templating language.
droelf 2y ago
Cool that this space is getting more attention - I just came from the reproducible builds summit in Hamburg. We're working on similar low level build system tools with rattler-build and pixi. Would love to have a chat and potentially figure out if collaboration is possible.
- zombiezen 2y ago
  Cool! Contact info is in my profile and on my website. :)
greener_grass 2y ago
I'm excited by this! Quick question: if the build graph can be dynamic (I think they call it monadic in the paper), then does it become impossible to reason about the build statically? I think this is why Bazel has a static graph and why it scales so well.
- zombiezen 2y ago
  According to Build systems à la carte, "it is not possible to express dynamic dependencies in [Bazel's] user-defined build rules; however some of the pre-defined build rules require dynamic dependencies and the internal build engine can cope with them by using a restarting task scheduler, which is similar to that of Excel but does not use the calc chain." (p6) IME import-from-derivation and similar in Nix is usually used for importing build configurations from remote repositories. Bazel has a repository rule system that is similar: https://bazel.build/extending/repo https://bazel.build/extending/repo So to answer your question: yes from the strictest possible definition, but in practice, I believe the tradeoffs are acceptable.
  - Ericson2314 2y ago
    You should look at Nix's experimental dynamic derivations, which provide functionality entirely at the level of derivation language / store layer.
    - zombiezen 2y ago
      Interesting! Thanks, hadn't seen that yet. (For anyone else curious, the RFC is here: https://github.com/NixOS/rfcs/blob/master/rfcs/0092-plan-dynamism.md https://github.com/NixOS/rfcs/blob/master/rfcs/0092-plan-dyn...)
- aseipp 2y ago
  Buck2 can express dynamic dependencies, so it can capture dynamic compilation problems like C++ modules, OCaml/Fortran modules, etc. in "user space" without built-in support like Bazel requires. The secret to why is twofold. One, your internal build graph can be fully dynamic at the implementation level; rather, it's a matter of how much expressivity you expose to the user in letting them leverage and control the dynamic graph. Just because you have a Monad, doesn't mean you have to have to expose it. You can just expose an Applicative. And actually, if you take the view that build systems are a form of staged programming, then all build systems are monadic because the first stage is building the graph at all, and the second stage is evaluating it. Make, for example, has to parse the Makefiles, and during this phase it constructs the graph... dynamically! Based on the input source code! Rather it is during the second phase done later, when rules are evaluated, and that is now the time when the graph is static and all edges must be known. See some notes from Neil Mitchell about that.[1] The other key is in a system like Buck or Bazel, there are actually two graphs that are clearly defined. There is the target graph where you have abstract dependencies between things (a cxx_binary depends on a cxx_library), and there is the action graph (the command gcc must run before the ld command can run). You cannot have dynamic nodes in the target graph. Target graph construction MUST be deterministic and "complete" in the sense it captures all nodes. This is really important because it breaks features like target determination: given a list of changed files, what changed targets need to be rebuilt? You cannot know the complete list of targets when the target graph is dynamic, and evaluation can produce new nodes. That's what everyone means when they say it's "scalable." That you can detect, only given a list of input files from version control, what the difference between these two build graphs are. And then you can go build those targets exactly and skip everything else. So, if you make a small change to a monumentally sized codebase, you don't have to rebuild everything. Just a very small, locally impacted part of the whole pie. In other words, "small changes to the code should have small changes in the resulting build." That's incremental programming in a nutshell. OK, so there's no target graph dynamism. But you can have dynamic actions in the action graph, where the edges to those dynamic actions are well defined. For example, compiling an OCaml module first requires you to build a .m file, then read it, then run some set of commands in an order dictated by the .m file. The output is an .a file. So you always know the in/out edges for these actions, but you just don't know what order you need to run compiler commands in. That dynamic action can be captured without breaking the other stuff. There are some more notes from Neil about this.[2] Under this interpretation, Nix also defines a static target graph in the sense that every store path/derivation is a node represented as term in the pure, lazy lambda calculus (with records). When you evaluate a Nix expression, it produces a fully closed term, and terms that are already evaluated previously (packaged and built) are shared and reused. The sharing is how "target determination" is achieved; you actually evaluate everything and anything that is shared is "free." And under this same interpretation, the pure subset of Zb programs should, by definition, also construct a static target graph. It's not enough to just sandboxing I/O but also some other things; for example if you construct hash tables with undefined iteration order you might screw the pooch somewhere down the line. Or you could just make up things out of thin air I guess. But if you restrict yourself to the pure subset of Zb programs, you should in theory be fine (and that pure subset is arguably the actual valuable, useful subset, so it's maybe fine.) [1] https://ndmitchell.com/downloads/paper-implementing_applicative_build_systems_monadically-01_jan_2022.pdf https://ndmitchell.com/downloads/paper-implementing_applicat... [2] https://ndmitchell.com/downloads/slides-somewhat_dynamic_build_systems-07_mar_2024.pdf https://ndmitchell.com/downloads/slides-somewhat_dynamic_bui...
kortex 2y ago
How do you pronounce "Zb"? Zee-bee?
- zombiezen 2y ago
  Heh, I think I need to add something to the README. I've been pronouncing it as "zeeb" in my head as in the first syllable Zebesian Space Pirate from Metroid, but TIL that that's canonically "Zay-bay-zee-uhn" so idk. Naming is hard.
  - kortex 2y ago
    I kinda dig zeeb! Naming is hard. Really awesome project by the way! Should have mentioned that first. Build systems are neat. I've always wanted to try building a build system, in a "learn how it works" sense, not so much "yet another build tool".
    - zombiezen 2y ago
      Thanks! And go for it, it's a good learning experience! It's a really interesting problem domain and there's a lot of different directions you can take it.
Ericson2314 2y ago
Nice to see Windows support. We/I are working on that with upstream Nix too. Also I hope we can keep the store layer compatible. It would be good to replace ATerm with JSON, for example. We should coordinate that!
- zombiezen 2y ago
  Rad! Yes, please keep me in the loop!
  - Ericson2314 2y ago
    Will do!
stmonty 2y ago
This looks awesome. I've had this same exact idea for a build system, but I haven't had the time to build it yet. Cool to see someone basically build what I had imagined!
jiggawatts 2y ago
From the Build Systems à la Carte paper: Topological. The topological scheduler pre-computes a linear order of tasks, which when followed, ensures the build result is correct regardless of the initial store. Given a task description and the output key, you can compute the linear order by first finding the (acyclic) graph of the key’s reachable dependencies, and then computing a topological sort. However this rules out dynamic dependencies. Restarting. To handle dynamic dependencies we can use the following approach: build tasks in an arbitrary initial order, discovering their dependencies on the fly; whenever a task calls fetch on an out-of-date key dep, abort the task, and switch to building the dependency dep; eventually the previously aborted task is restarted and makes further progress thanks to dep now being up to date. This approach requires a way to abort tasks that have failed due to out-of-date dependencies. It is also not minimal in the sense that a task may start, do some meaningful work, and then abort. Suspending. An alternative approach, utilised by the busy build system and Shake, is to simply build dependencies when they are requested, suspending the currently running task. By combining that with tracking the keys that have already been built, one can obtain a minimal build system with dynamic dependencies. This approach requires that a task may be started and then suspended until another task is complete. Suspending can be done with cheap green threads and blocking (the original approach of Shake) or using continuation-passing style (what Shake currently does).
aseipp 2y ago
Whatever choices this project makes (I have some opinions, but I think they're not too important) I don't see it mentioning one of the most absolutely critical choices Nix made that was absolutely key to its insane success (at least, IMO, as a hardcore contributor and user for like 10+ years): the monorepo, containing all of the packages and all the libraries for use by everyone downstream, and all contributions trying to go there. Please do not give into the temptation to just write a version manager and stitch together some hodgepodge and throw the hard problem over the fence to the "community", a set of balkanized repositories to make everything work. It is really really really hard to overstate how much value Nixpkgs gets from going the monorepo route and how much the project has been able to improve, adapt, and overcome things thanks to it. It feels like Nixpkgs regularly pulls off major code-wide changes on an average Tuesday that other projects would balk at. (It's actually a benefit early on to just keep everything in one repo too, because you can just... clean up all the code in one spot if you do something like make a major breaking change. Huge huge benefit!) Finally: as a die hard Nix user, I also have been using Buck2 as a kind of thing-that-is-hermetic-cloud-based-and-supports-Windows tool, and it competes in the same space as Zb; a monorepo containing all BUILD files is incredibly important for things to work reliably and it's what I'm exploring right now and seeing if that can be viable. I'm even exploring the possibility of starting from stage0-posix as well. Good luck! There's still work to be done in this space and Nix isn't the final answer, even if I love it.
- theLiminator 2y ago
  Buck2 looks very principled. Will definitely be interesting as it gets mature in the open source world. I'm personally convinced monorepo is strictly superior (provided you have the right tooling to support it).
- deleted 2y ago
  [deleted]
- deleted 2y ago
  [deleted]
deleted 2y ago
[deleted]
teddyh 2y ago
It’s amazing the lengths some people will go to in order to avoid scary parentheses.
- mtndew4brkfst 2y ago
  It's not Guile I want to avoid, it's GNU ideologues who insist on every freedom except "use proprietary software and hardware" and shame people for doing so.
Nikhilbidkar1 2y ago
This looks awesome....
fizlebit 2y ago
I can't help but wonder whether the major problem is actually API changing from version to version of software and keeping everything compatible. If the build language is LUA, doesn't it support top level variables. It probably just takes a few folks manipulating top level variables before the build steps and build logic is no longer hermetic, but instead plagued by side effects. I think you need to build inside very effective sandboxes to stop build side effects and then you need your sandboxes to be very fast. Anyway, nice to see attempts at more innovation in the build space. I imagine a kind of merging between build systems, deployment systems, and running systems. Somehow a manageable sea of distributed processes running on a distributed operating system. I suspect Alan Kay thought that smalltalk might evolve in that direction, but there are many things to solve including billing, security, and somehow making the sea of objects comprehensible. It has the hope of everything being data driven, aka structured, schemad, versions, json like data rather than the horrendous mess that is unix configuration files and system information. There was an interested talk on Developer Voice perhaps related to a merger of Ocaml and Erlang that moved a little in that direction.
anacrolix 2y ago
Russ? Roxy? Is an outstanding developer. Keen to see how this goes.
- zombiezen 2y ago
  Thank you! My name is Roxy. https://www.zombiezen.com/blog/2024/07/my-name-is-roxy/ https://www.zombiezen.com/blog/2024/07/my-name-is-roxy/
nvlled 2y ago
https://github.com/256lights/zb/blob/102795d6cb383a919dd378d9e257c5883c62d245/demo/hello.lua https://github.com/256lights/zb/blob/102795d6cb383a919dd378d... TIL I can also use semicolons on lua tables, not just commas: return derivation { name = "hello.txt"; ["in"] = path "hello.txt"; builder = "/bin/sh"; system = "x86_64-linux"; args = {"-c", "while read line; do echo \"$line\"; done < $in > $out"}; } I like using lua as a DSL, now I like it even more! I've using lua as a html templating language that looks like this: DIV { id="id"; class="class; H1 "heading"; P [[ Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor ]] / EM incididunt / [[ ut labore et dolore magna aliqua. ]]; PRE ^ CODE [[ this is <code> tag inside <pre> ]]; }
- deleted 2y ago
  [deleted]
weitzj 2y ago
Great idea. Just a tip. You can wrap your lua part into cosmopolitan C. This way you get lua on many architectures and os. Also cosmopolitan can be bootstrapped with tiny cc I guess. And personally wrapping your lua code in https://fennel-lang.org/ https://fennel-lang.org/ would be nice. This way with libcosmopolitan, you could just checkin a copy of your build tool in a project, to be self sufficient. Think of it like gradlew( the gradle bash/bat wrapper) but completely self contained and air gapped https://github.com/jart/cosmopolitan https://github.com/jart/cosmopolitan
- zcam 2y ago
  +1 for fennel
PROMISE_237 2y ago
[dead]
bjourne 2y ago
I've been using WAF for ages so naturally I wonder how this system compares to WAF? My experience with build systems is that they all get the easy parts rights. You can compile C and C++ code and they successfully scan header files for dependencies. But FEW get the hard parts rights. E.g., compiling LaTeX with multiple figures, custom fonts and classes, and multiple bib files. It requires correctly interfacing with pdfatex which is a complete PITA as it spews intermediate files everywhere and puts constraints on the current directory. Most build tools can't. What I want in a build tool is universality. Sometimes a whole directory tree is the dependency of a target. Sometimes it's an url and the build tool should correctly download and cache that url. Sometimes the pre-requisite is training an ML model.
- rwmj 2y ago
  I wrote an experimental make replacement some years ago that understands that not every target is a file. eg. You can have targets be a remote URL (for an action of uploading to a fileserver). http://git.annexia.org/?p=goals.git;a=summary http://git.annexia.org/?p=goals.git;a=summary http://oirase.annexia.org/2020-02-rjones-goals-tech-talk.mp4 http://oirase.annexia.org/2020-02-rjones-goals-tech-talk.mp4
- johnny22 2y ago
  is it impossible to fix that issue in pdflatex?
  - bjourne 2y ago
    My point is that build systems must be able to deal with tools with insanely stupid interfaces like pdflatex. Btw, WAF's strategy of dealing with pdflatex is to rerun the command "until output files stop changing". That's how dumb it is.
- ladams 2y ago
  latexrun does a pretty reasonable job with LaTeX files, and only runs when needed, etc. Would be nice to have this integrated into a build system for plots, data generation, etc.
- 2y ago
jjuliano 2y ago
I made a graph-based orchestrator - https://github.com/jjuliano/runner https://github.com/jjuliano/runner - It uses declarative YAML, and preflight, postflight and skip conditions. I think it can also be a full-fledge build system.
mdaniel 2y ago
I see the instructions discuss $(mkdir /zb) <https://github.com/256lights/zb#linux-or-macos https://github.com/256lights/zb#linux-or-macos> and after seeing references to Nix I wanted to ensure this wasn't a hard-and-fast directory choice since macOS has immutable / and it causes no end to the Nix stupidity on macOS
- zombiezen 2y ago
  Good point. Opened https://github.com/256lights/zb/issues/47 https://github.com/256lights/zb/issues/47 to track this idea.

254 points by zombiezen 2y ago

This looks really exciting and I absolutely must give it a try. Well done! At face value the vision and design choices appear to be great.

zombiezen 2y ago
Thank you! <3

ramon156 2y ago

I'd definitely write a build systen in lua, looks promising!

Iceland_jack 2y ago

I appreciate the link to https://dl.acm.org/doi/10.1145/3236774 https://dl.acm.org/doi/10.1145/3236774

o11c 2y ago
Definitely interesting, but it's flat-out wrong about the limitations of `make`. In particular, the `release.txt` task is trivial by adding a dummy rule to generate and include dependencies; see https://www.gnu.org/software/make/manual/html_node/Remaking-Makefiles.html https://www.gnu.org/software/make/manual/html_node/Remaking-... (be sure to add empty rules to handle the case of deleted dynamic dependencies). You can use hashes instead of file modification times by adding a different kind of dummy rule. The only downside is that you have to think about the performance a little. I imagine it's possible for a project to have some kind of dynamic dependencies that GNU make can't handle, but I dare say that any such dependency tree is hard to understand for humans too, and thus should be avoided regardless. By contrast, in many other build tools it is impossible to handle some of the things that are trivial in `make`. (if you're not using GNU make, you are the problem; do not blame `make`)
- bjourne 2y ago
  I guess you aren't keen on Java then? Complex dynamic dependency graphs aren't difficult for humans to handle or many build tools other than make.
  - throwaway2037 2y ago
    > I guess you aren't keen on Java then? Can you explain more? I don't follow.
    - bjourne 2y ago
      Java dependencies are too complicated for make. See https://www.oreilly.com/library/view/managing-projects-with/0596006101/ch09.html https://www.oreilly.com/library/view/managing-projects-with/...
  - o11c 2y ago
    I'm not keen on Java for other reasons. The fact that a single .java file can generate multiple .class files is annoying but not something tools can't handle (it's actually similar to .h files for C/C++ - remember, we only need the complete dependency graph for the rebuild, not the current build). The main part that's difficult for humans is if there's a non-public class at top level rather than nested (I forget all the Java-specific terminology for the various kinds of class nesting).

packetlost 2y ago

One request that I would make of a project like this is to support distributed builds out of the box. Like, really basic support for identical builder hosts (this is much easier now than in the past with containers) and caching of targets. Otherwise, this looks great! Big fan of the choice of Lua, though the modifications to strings might make it difficult to onboard new users depending on how the modification was made.

zombiezen 2y ago
Yup, remote building and caching is on my radar. I expect it will work much in the same way Nix does now, although I'm being a bit more deliberate in creating an RPC layer so build coordinators and other such tools are more straightforward to build. The string tweak is transparent to users broadly speaking. IME with Nix this thing works the way people expect (i.e if you use a dependency variable in your build target, it adds a dependency).
Glacia 2y ago
Xmake?

bsnnkv 2y ago

You had my interest at Windows support! I'll carve out some time this weekend to see if I can write a build for komorebi

zombiezen 2y ago
Nice! It might be a little too rough until I've got a working C compiler for Windows: https://github.com/256lights/zb/issues/6 https://github.com/256lights/zb/issues/6 (and Linux for that matter: https://github.com/256lights/zb/issues/30 https://github.com/256lights/zb/issues/30)

steeleduncan 2y ago

Looks great, Nix-with-Lua that also supports Windows would be amazing. Two questions if I may - Does this sandbox builds the way flakes do? - What is MinGW used for on Windows? Does this rely on the MinGW userland, or is it just because it would be painful to write a full bootstrap for a windows compiler while also developing Zb? Also, its great to see the live-bootstrap in there. I love the purity of how Guix's packages are built, and I like the idea Zb will be that way from the start

zombiezen 2y ago
Nix sandboxes derivation runs on Linux even without flakes, and I'm planning on implementing that, yes: https://github.com/256lights/zb/issues/29 https://github.com/256lights/zb/issues/29 and https://github.com/256lights/zb/issues/31 https://github.com/256lights/zb/issues/31 MinGW is used to build Lua using cgo. I'd like to remove that part, see https://github.com/256lights/zb/issues/28 https://github.com/256lights/zb/issues/28 I haven't started the userspace for Windows yet (https://github.com/256lights/zb/issues/6 https://github.com/256lights/zb/issues/6), but I suspect that it will be more "download the Visual C++ compiler binary from this URL" than the Linux source bootstrap. Yeah, I'm happy with live-bootstrap, too! I tried emulating Guix's bootstrap, but it depended a little too much on Scheme for me to use as-is. live-bootstrap has mostly worked out-of-the-box, which was a great validation test for this approach.
- steeleduncan 2y ago
  Thanks for answering and I really hope it works out. A Nix alternative with less friction would be very welcome!

laurentlb 2y ago

I'd like to know more about the "Support for non-determinism" and how that differs from other build systems. Usually, build systems rerun actions when at least one of the inputs has changed. Are non-deterministic targets rerun all the time? Also, I'm curious to know if you've considered using Starlark or the build file syntax used in multiple other recent build systems (Bazel, Buck, Please, Pants).

zombiezen 2y ago
(Hi! I recognize your name from Bazel mailing lists but I forget whether we've talked before.) I'm mostly contrasting from Nix, which has difficulty with poisoning cache when faced with non-deterministic build steps when using input-addressing (the default mode). If zb encounters a build target with multiple cached outputs for the same inputs, it rebuilds and then relies on content-addressing to obtain build outputs for subsequent steps if possible. (I have an open issue for marking a target as intentionally non-deterministic and always triggering this re-run behavior: https://github.com/256lights/zb/issues/33 https://github.com/256lights/zb/issues/33) I'll admit I haven't done my research into how Bazel handles non-determinism, especially nowadays, so I can't remark there. I know from my Google days that even writing genrules you had to be careful about introducing non-determinism, but I forget how that failure mode plays out. If you have a good link (or don't mind giving a quick summary), I'd love to read up. I have considered Starlark, and still might end up using it. The critical feature I wanted to bolt in from Nix was having strings carrying dependency information (see https://github.com/NixOS/nix/blob/2f678331d59451dd6f1d9512cb6d92e4ecb9750f/src/libexpr/value.hh#L192-L211 https://github.com/NixOS/nix/blob/2f678331d59451dd6f1d9512cb... for a description of the feature). In my prototyping, this was pretty simple to bolt on to Lua, but I'm not sure how disruptive that would be to Starlark. Nix configurations tend to be a bit more complex than Bazel ones, so having a more full-featured language felt more appropriate. Still exploring the design space!
- aseipp 2y ago
  I mean, to be fair, Nix is nothing more than a big ass pile of genrule() calls, at the end of the day. Everything is really just genrule. Nix just makes it all work with the sandbox it puts all builds in. Bazel has an equivalent sandbox and I'm pretty sure you can sandbox genrule so it's in a nice, hermetic container. (Side note, but one of my biggest pet peeves is that Nix without the sandbox is actually fundamentally _broken_, yet we let people install it without the sandbox. I have no idea why "Install this thing in a broken way!" is even offered as an option. Ridiculous.) The way Nix-like systems achieve hermetic sandboxing isn't so much a technical feat, in my mind. That's part of it -- sure, you need to get rid of /dev devices, and every build always has to look like it happens at /tmp/build within a mount namespace, and you need to set SOURCE_EPOCH_DATE and blah blah, stuff like that. But it's also a social one, because with Nix you are expected to wrap arbitrary build systems and package mechanisms and "go where they are." That means you have to bludgeon every random hostile badly written thing into working inside the sandbox you designed, carve out exceptions, and write ptaches for things that don't -- and get them working in a deterministic way. For example, you have to change the default search paths for nearly every single tool to look inside calculated Nix store path. That's not a technical feat, it's mostly just a huge amount of hard work to write all the abstractions, like buildRustPackage or makeDerivation. You need to patch every build system like CMake or Scons in order to alleviate some of their assumptions, and so on and so forth. Bazel and Buck like systems do not avoid this pain but they do pay for it in a different way. They don't "go where they are", they expect everyone to "come to them." Culturally, Bazel users do not accept "just run Make under a sandbox" nearly as much. The idea is to write everything as a BUILD file rule, from scratch rewriting the build system, and those BUILD files instead should perform the build "natively" in a way that is designed to work hermetically. So you don't run ./configure, you actually pick an exact set of configuration options and build with that 100% of the time. Therefore, the impurities in the build are removed "by design", which makes the strict requirements on a sandbox somewhat more lenient. You still need the sandbox, but by definition your builds are much more robust anyway. So you are trading the pain of wrapping every system for the pain of integrating every system manually. They're not the same thing but have a lot of overlap. So the answer is, yes you can write impure genrules, but the vast majority of impurity is totally encapsulated in a way that forces it to be pure, just like Nix, so it's mostly just a small nit rather than truly fundamental. The real question is a matter of when you want to pay the pied piper.

msvan 2y ago

As a current Nix user, what I would really like is a statically typed language to define builds. Recreating Nix without addressing that feels like a missed opportunity.

zombiezen 2y ago
The Lua VSCode extension adds a type system that works really well IME
- 0cf8612b2e1e 2y ago
  There are Lua flavors with typing. Teal is one I have heard that compiles down to regular Lua like a typescript
Rucadi 2y ago
For me the killer feature is Windows Support, Ericsson is doing a great job bringing nix into Windows, but the process it's understandably slow, If this project is similar enough to nix that I can kind-off translate easily the zb derivations to nix derivations, I'm willing to use it in windows (It's not like nix has windows programs in the nixpkgs either way I have to bring them in my own). The problem for me is that I see no benefit on using this over nix language (which I kinda like a lot right now)
- droelf 2y ago
  We're working on rattler-build (https://github.com/prefix-dev/rattler-build/ https://github.com/prefix-dev/rattler-build/) - which is a build system inspired by Apko / conda-build and uses YAML files to statically define dependencies. It works really well with pixi (our package manager) but also any other conda compatible package managers (mamba, conda). And it has Windows support, of course. It can also be used to build your own distribution (e.g. here is one for a bunch of Rust utilities: https://github.com/wolfv/rust-forge https://github.com/wolfv/rust-forge)
- hamandcheese 2y ago
  > Ericsson is doing a great job bringing nix into Windows Is this Ericsson... the corporation? Windows support for nix is something I don't hear much about, but if there is progress being made (even slowly) I'd love to know more.

mikepurvis 2y ago

Whoa, nifty. Can you speak more to the interop issues with Nix? I've been working on a pretty large Nix deployment in the robotics space for the past 3ish years, and the infrastructure side is the biggest pain point: * Running a bare `nix build` in your CI isn't really enough— no hosted logs, lack of proper prioritization, may end up double-building things. * Running your own instance of Hydra is a gigantic pain; it's a big ball of perl and has compiled components that link right into Nix internals, and architectural fiasco. * SaaS solutions are limited and lack maturity (Hercules CI is Github-only, nixbuild.net is based in Europe and last I checked was still missing some features I needed). * Tvix is cool but not ready for primetime, and the authors oppose flakes, which is a deal-breaker for me. Something that's a barebones capable of running these builds and could be wrapped in a sane REST API and simple web frontend would be very appealing.

Rucadi 2y ago
Why are flakes such a deal-breaker? While not ideal, you can still tag your versions in the .nix file instead of the lockfile. I even had to avoid flakes in a system I developed used by ~200 developers since it involved a non-nixos OS and it involved user secrets (Tokens etc...) So with flakes I had to keep track of the secrets (and was a pain point, since they obviously didn't have to push them into the git repo) but nix flakes doesn't handle well omitting files on git (it ignores them also on nix commands). In the end, the workarounds were too messy and had to drop flakes entirely.
- mikepurvis 2y ago
  As a new user, I learned flakes first, and the tie-in with git tags/branches and the corresponding cli ergonomics aren’t something I’d be able to give up.
  - JasonSage 2y ago
    How do you handle flakes pushing an entire copy of the repo into the nix store? Is this not an issue for you somehow?

imiric 2y ago

Happy to see someone inspired by Nix, but wanting to carve their own path. Nix popularized some powerful ideas in the Linux world, but it has a steep learning curve and a very unfriendly UI, so there is plenty of room for improvement there. I'm not sure if Lua is the right choice, though. A declarative language seems like a better fit for reproducibility. The goal of supporting non-deterministic builds also seems to go against this. But I'm interested to know how this would work in practice. Good luck!

hinkley 2y ago
If you design it like SCons, it'll look imperative but behave more declaratively. If I understand the architecture correctly, the imperative calls in the config file don't actually run the build process. They run a Builder Pattern that sets up the state machine necessary for the builds to happen. So it's a bit like LINQ in C# (but older). I have no idea how that plays out single-step debugging build problems though. That depends on how it's implemented and a lot of abstractions (especially frameworks) seem to forget that breakpoints are things other people want to use as well.
- zombiezen 2y ago
  That's accurate (unless the config file attempts to read something from the build process, that will trigger a build). It's a good point about debugging build problems. This is an issue I've experienced in Nix and Bazel as well. I'm not convinced that I have a great solution yet, but at least for my own debugging while using the system, I've included a `zb derivation env` command which spits out a .env file that matches the environment the builder runs under. I'd like to extend that to pop open a shell.
  - skybrian 2y ago
    One thing I like to see is a 'dry run' like 'make -n'. Although, maybe that's not possible in all cases. Another possibility might be to output a something like a shell script that would do a rebuild the same way, so you can see what it did and hack it when debugging.

IshKebab 2y ago

Interesting. I feel like I would have gone with Starlark over Lua, but I guess it's good to have options. Does it support sandboxing?

zombiezen 2y ago
Not yet, but I've hacked up most of the Linux sandboxing: https://github.com/256lights/zb/issues/29 https://github.com/256lights/zb/issues/29 I want to introduce Windows sandboxing, too, but I'm not as confident about how to do that: https://github.com/256lights/zb/issues/31 https://github.com/256lights/zb/issues/31
- zombiezen 2y ago
  Oh and as for Starlark, I went into more detail over in this thread: https://news.ycombinator.com/item?id=41596426 https://news.ycombinator.com/item?id=41596426
israrkhan 2y ago
You need bazel if you need starlark & sandboxing
- IshKebab 2y ago
  Well yeah. Starlark and sandboxing are the best things about Bazel, but it could still definitely be improved. So I'm still curious about other build systems. I think making a new build system without sandboxing (or at least a plan for it) would be pretty stupid. Fortunately he is planning it.

alxmng 2y ago

Did you consider writing a nicer language that compiles to Nix? A "friendly" tool on the outside with Nix inside.

zombiezen 2y ago
Yup, that was how I built the prototype: https://www.zombiezen.com/blog/2024/06/zb-build-system-prototype/ https://www.zombiezen.com/blog/2024/06/zb-build-system-proto... The last commit using that approach was https://github.com/256lights/zb/tree/558c6f52b7ef915428c9af9a9997edca84dfd831 https://github.com/256lights/zb/tree/558c6f52b7ef915428c9af9... if you want to try it out. And actually, I haven't touched the Lua frontend much since I swapped out the backend: the .drv files it writes are the same. The motivation behind replacing the backend was content-addressibility and Windows support, which have been slow to be adopted in Nix core.
Rucadi 2y ago
I don't think nix is that awful, while there are some tasks that are more difficult or can be a little bit verbose (if you want to play a lot with the attribute sets / lists or string manip) When using nix most of the time you'll end up just writing bash or using it as a templating language.

droelf 2y ago

Cool that this space is getting more attention - I just came from the reproducible builds summit in Hamburg. We're working on similar low level build system tools with rattler-build and pixi. Would love to have a chat and potentially figure out if collaboration is possible.

zombiezen 2y ago
Cool! Contact info is in my profile and on my website. :)

greener_grass 2y ago

I'm excited by this! Quick question: if the build graph can be dynamic (I think they call it monadic in the paper), then does it become impossible to reason about the build statically? I think this is why Bazel has a static graph and why it scales so well.

zombiezen 2y ago
According to Build systems à la carte, "it is not possible to express dynamic dependencies in [Bazel's] user-defined build rules; however some of the pre-defined build rules require dynamic dependencies and the internal build engine can cope with them by using a restarting task scheduler, which is similar to that of Excel but does not use the calc chain." (p6) IME import-from-derivation and similar in Nix is usually used for importing build configurations from remote repositories. Bazel has a repository rule system that is similar: https://bazel.build/extending/repo https://bazel.build/extending/repo So to answer your question: yes from the strictest possible definition, but in practice, I believe the tradeoffs are acceptable.
- Ericson2314 2y ago
  You should look at Nix's experimental dynamic derivations, which provide functionality entirely at the level of derivation language / store layer.
  - zombiezen 2y ago
    Interesting! Thanks, hadn't seen that yet. (For anyone else curious, the RFC is here: https://github.com/NixOS/rfcs/blob/master/rfcs/0092-plan-dynamism.md https://github.com/NixOS/rfcs/blob/master/rfcs/0092-plan-dyn...)
aseipp 2y ago
Buck2 can express dynamic dependencies, so it can capture dynamic compilation problems like C++ modules, OCaml/Fortran modules, etc. in "user space" without built-in support like Bazel requires. The secret to why is twofold. One, your internal build graph can be fully dynamic at the implementation level; rather, it's a matter of how much expressivity you expose to the user in letting them leverage and control the dynamic graph. Just because you have a Monad, doesn't mean you have to have to expose it. You can just expose an Applicative. And actually, if you take the view that build systems are a form of staged programming, then all build systems are monadic because the first stage is building the graph at all, and the second stage is evaluating it. Make, for example, has to parse the Makefiles, and during this phase it constructs the graph... dynamically! Based on the input source code! Rather it is during the second phase done later, when rules are evaluated, and that is now the time when the graph is static and all edges must be known. See some notes from Neil Mitchell about that.[1] The other key is in a system like Buck or Bazel, there are actually two graphs that are clearly defined. There is the target graph where you have abstract dependencies between things (a cxx_binary depends on a cxx_library), and there is the action graph (the command gcc must run before the ld command can run). You cannot have dynamic nodes in the target graph. Target graph construction MUST be deterministic and "complete" in the sense it captures all nodes. This is really important because it breaks features like target determination: given a list of changed files, what changed targets need to be rebuilt? You cannot know the complete list of targets when the target graph is dynamic, and evaluation can produce new nodes. That's what everyone means when they say it's "scalable." That you can detect, only given a list of input files from version control, what the difference between these two build graphs are. And then you can go build those targets exactly and skip everything else. So, if you make a small change to a monumentally sized codebase, you don't have to rebuild everything. Just a very small, locally impacted part of the whole pie. In other words, "small changes to the code should have small changes in the resulting build." That's incremental programming in a nutshell. OK, so there's no target graph dynamism. But you can have dynamic actions in the action graph, where the edges to those dynamic actions are well defined. For example, compiling an OCaml module first requires you to build a .m file, then read it, then run some set of commands in an order dictated by the .m file. The output is an .a file. So you always know the in/out edges for these actions, but you just don't know what order you need to run compiler commands in. That dynamic action can be captured without breaking the other stuff. There are some more notes from Neil about this.[2] Under this interpretation, Nix also defines a static target graph in the sense that every store path/derivation is a node represented as term in the pure, lazy lambda calculus (with records). When you evaluate a Nix expression, it produces a fully closed term, and terms that are already evaluated previously (packaged and built) are shared and reused. The sharing is how "target determination" is achieved; you actually evaluate everything and anything that is shared is "free." And under this same interpretation, the pure subset of Zb programs should, by definition, also construct a static target graph. It's not enough to just sandboxing I/O but also some other things; for example if you construct hash tables with undefined iteration order you might screw the pooch somewhere down the line. Or you could just make up things out of thin air I guess. But if you restrict yourself to the pure subset of Zb programs, you should in theory be fine (and that pure subset is arguably the actual valuable, useful subset, so it's maybe fine.) [1] https://ndmitchell.com/downloads/paper-implementing_applicative_build_systems_monadically-01_jan_2022.pdf https://ndmitchell.com/downloads/paper-implementing_applicat... [2] https://ndmitchell.com/downloads/slides-somewhat_dynamic_build_systems-07_mar_2024.pdf https://ndmitchell.com/downloads/slides-somewhat_dynamic_bui...

kortex 2y ago

How do you pronounce "Zb"? Zee-bee?

zombiezen 2y ago
Heh, I think I need to add something to the README. I've been pronouncing it as "zeeb" in my head as in the first syllable Zebesian Space Pirate from Metroid, but TIL that that's canonically "Zay-bay-zee-uhn" so idk. Naming is hard.
- kortex 2y ago
  I kinda dig zeeb! Naming is hard. Really awesome project by the way! Should have mentioned that first. Build systems are neat. I've always wanted to try building a build system, in a "learn how it works" sense, not so much "yet another build tool".
  - zombiezen 2y ago
    Thanks! And go for it, it's a good learning experience! It's a really interesting problem domain and there's a lot of different directions you can take it.

Ericson2314 2y ago

Nice to see Windows support. We/I are working on that with upstream Nix too. Also I hope we can keep the store layer compatible. It would be good to replace ATerm with JSON, for example. We should coordinate that!

zombiezen 2y ago
Rad! Yes, please keep me in the loop!
- Ericson2314 2y ago
  Will do!

stmonty 2y ago

This looks awesome. I've had this same exact idea for a build system, but I haven't had the time to build it yet. Cool to see someone basically build what I had imagined!

jiggawatts 2y ago

From the Build Systems à la Carte paper: Topological. The topological scheduler pre-computes a linear order of tasks, which when followed, ensures the build result is correct regardless of the initial store. Given a task description and the output key, you can compute the linear order by first finding the (acyclic) graph of the key’s reachable dependencies, and then computing a topological sort. However this rules out dynamic dependencies. Restarting. To handle dynamic dependencies we can use the following approach: build tasks in an arbitrary initial order, discovering their dependencies on the fly; whenever a task calls fetch on an out-of-date key dep, abort the task, and switch to building the dependency dep; eventually the previously aborted task is restarted and makes further progress thanks to dep now being up to date. This approach requires a way to abort tasks that have failed due to out-of-date dependencies. It is also not minimal in the sense that a task may start, do some meaningful work, and then abort. Suspending. An alternative approach, utilised by the busy build system and Shake, is to simply build dependencies when they are requested, suspending the currently running task. By combining that with tracking the keys that have already been built, one can obtain a minimal build system with dynamic dependencies. This approach requires that a task may be started and then suspended until another task is complete. Suspending can be done with cheap green threads and blocking (the original approach of Shake) or using continuation-passing style (what Shake currently does).

aseipp 2y ago

Whatever choices this project makes (I have some opinions, but I think they're not too important) I don't see it mentioning one of the most absolutely critical choices Nix made that was absolutely key to its insane success (at least, IMO, as a hardcore contributor and user for like 10+ years): the monorepo, containing all of the packages and all the libraries for use by everyone downstream, and all contributions trying to go there. Please do not give into the temptation to just write a version manager and stitch together some hodgepodge and throw the hard problem over the fence to the "community", a set of balkanized repositories to make everything work. It is really really really hard to overstate how much value Nixpkgs gets from going the monorepo route and how much the project has been able to improve, adapt, and overcome things thanks to it. It feels like Nixpkgs regularly pulls off major code-wide changes on an average Tuesday that other projects would balk at. (It's actually a benefit early on to just keep everything in one repo too, because you can just... clean up all the code in one spot if you do something like make a major breaking change. Huge huge benefit!) Finally: as a die hard Nix user, I also have been using Buck2 as a kind of thing-that-is-hermetic-cloud-based-and-supports-Windows tool, and it competes in the same space as Zb; a monorepo containing all BUILD files is incredibly important for things to work reliably and it's what I'm exploring right now and seeing if that can be viable. I'm even exploring the possibility of starting from stage0-posix as well. Good luck! There's still work to be done in this space and Nix isn't the final answer, even if I love it.

theLiminator 2y ago
Buck2 looks very principled. Will definitely be interesting as it gets mature in the open source world. I'm personally convinced monorepo is strictly superior (provided you have the right tooling to support it).
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]

deleted 2y ago

[deleted]

teddyh 2y ago

It’s amazing the lengths some people will go to in order to avoid scary parentheses.

mtndew4brkfst 2y ago
It's not Guile I want to avoid, it's GNU ideologues who insist on every freedom except "use proprietary software and hardware" and shame people for doing so.

Nikhilbidkar1 2y ago

This looks awesome....

fizlebit 2y ago

I can't help but wonder whether the major problem is actually API changing from version to version of software and keeping everything compatible. If the build language is LUA, doesn't it support top level variables. It probably just takes a few folks manipulating top level variables before the build steps and build logic is no longer hermetic, but instead plagued by side effects. I think you need to build inside very effective sandboxes to stop build side effects and then you need your sandboxes to be very fast. Anyway, nice to see attempts at more innovation in the build space. I imagine a kind of merging between build systems, deployment systems, and running systems. Somehow a manageable sea of distributed processes running on a distributed operating system. I suspect Alan Kay thought that smalltalk might evolve in that direction, but there are many things to solve including billing, security, and somehow making the sea of objects comprehensible. It has the hope of everything being data driven, aka structured, schemad, versions, json like data rather than the horrendous mess that is unix configuration files and system information. There was an interested talk on Developer Voice perhaps related to a merger of Ocaml and Erlang that moved a little in that direction.

anacrolix 2y ago

Russ? Roxy? Is an outstanding developer. Keen to see how this goes.

zombiezen 2y ago
Thank you! My name is Roxy. https://www.zombiezen.com/blog/2024/07/my-name-is-roxy/ https://www.zombiezen.com/blog/2024/07/my-name-is-roxy/

nvlled 2y ago

https://github.com/256lights/zb/blob/102795d6cb383a919dd378d9e257c5883c62d245/demo/hello.lua https://github.com/256lights/zb/blob/102795d6cb383a919dd378d... TIL I can also use semicolons on lua tables, not just commas: return derivation { name = "hello.txt"; ["in"] = path "hello.txt"; builder = "/bin/sh"; system = "x86_64-linux"; args = {"-c", "while read line; do echo \"$line\"; done < $in > $out"}; } I like using lua as a DSL, now I like it even more! I've using lua as a html templating language that looks like this: DIV { id="id"; class="class; H1 "heading"; P [[ Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor ]] / EM incididunt / [[ ut labore et dolore magna aliqua. ]]; PRE ^ CODE [[ this is <code> tag inside <pre> ]]; }

deleted 2y ago
[deleted]

weitzj 2y ago

Great idea. Just a tip. You can wrap your lua part into cosmopolitan C. This way you get lua on many architectures and os. Also cosmopolitan can be bootstrapped with tiny cc I guess. And personally wrapping your lua code in https://fennel-lang.org/ https://fennel-lang.org/ would be nice. This way with libcosmopolitan, you could just checkin a copy of your build tool in a project, to be self sufficient. Think of it like gradlew( the gradle bash/bat wrapper) but completely self contained and air gapped https://github.com/jart/cosmopolitan https://github.com/jart/cosmopolitan

zcam 2y ago
+1 for fennel

PROMISE_237 2y ago

[dead]

bjourne 2y ago

I've been using WAF for ages so naturally I wonder how this system compares to WAF? My experience with build systems is that they all get the easy parts rights. You can compile C and C++ code and they successfully scan header files for dependencies. But FEW get the hard parts rights. E.g., compiling LaTeX with multiple figures, custom fonts and classes, and multiple bib files. It requires correctly interfacing with pdfatex which is a complete PITA as it spews intermediate files everywhere and puts constraints on the current directory. Most build tools can't. What I want in a build tool is universality. Sometimes a whole directory tree is the dependency of a target. Sometimes it's an url and the build tool should correctly download and cache that url. Sometimes the pre-requisite is training an ML model.

rwmj 2y ago
I wrote an experimental make replacement some years ago that understands that not every target is a file. eg. You can have targets be a remote URL (for an action of uploading to a fileserver). http://git.annexia.org/?p=goals.git;a=summary http://git.annexia.org/?p=goals.git;a=summary http://oirase.annexia.org/2020-02-rjones-goals-tech-talk.mp4 http://oirase.annexia.org/2020-02-rjones-goals-tech-talk.mp4
johnny22 2y ago
is it impossible to fix that issue in pdflatex?
- bjourne 2y ago
  My point is that build systems must be able to deal with tools with insanely stupid interfaces like pdflatex. Btw, WAF's strategy of dealing with pdflatex is to rerun the command "until output files stop changing". That's how dumb it is.
ladams 2y ago
latexrun does a pretty reasonable job with LaTeX files, and only runs when needed, etc. Would be nice to have this integrated into a build system for plots, data generation, etc.
2y ago

jjuliano 2y ago

I made a graph-based orchestrator - https://github.com/jjuliano/runner https://github.com/jjuliano/runner - It uses declarative YAML, and preflight, postflight and skip conditions. I think it can also be a full-fledge build system.

mdaniel 2y ago

I see the instructions discuss $(mkdir /zb) <https://github.com/256lights/zb#linux-or-macos https://github.com/256lights/zb#linux-or-macos> and after seeing references to Nix I wanted to ensure this wasn't a hard-and-fast directory choice since macOS has immutable / and it causes no end to the Nix stupidity on macOS

zombiezen 2y ago
Good point. Opened https://github.com/256lights/zb/issues/47 https://github.com/256lights/zb/issues/47 to track this idea.