12 ms·

Goodbye to the C++ Implementation of Zig

528 points by nyberg 4y ago

Decabytes 4y ago
I have honestly been more excited about Wasm for desktop than I am for the web. And I'm really excited about it for the Web. Really cool to see this use case pop up right as I'm trying to integrate it into my stack!
- throwaway894345 4y ago
  I've never really thought about wasm for the desktop (I've thought about it for server and of course browser), can you elaborate on your excitement? Is it just for this sort of bootstrapping application, or are there other benefits?
  - als0 4y ago
    Write once, run anywhere… but this time the dream will come true!
    - typon 4y ago
      The difference is Oracle vs. No Oracle.
      - pjmlp 4y ago
        You get Apple, Google and Microsoft instead (FF hardly counts).
    - kllrnohj 4y ago
      Nearly every desktop is still x86 and just a couple years ago that was entirely true, and yet write-once-run-anywhere wasn't remotely close to true. WASM as a result isn't changing anything here, since the assembly is very extremely not remotely close to the issue with having a portable binary.
      - int_19h 4y ago
        Wasm is an opportunity for all platforms to adopt a single container format for architecture-independent binary code, and with uniforms APIs being defined on top of that; they are already standardizing a subset of POSIX. None of the bits here are really new, but the big difference this time is that it's not owned by any one of the major players (like e.g. Java and .NET were), and the design process is truly collaborative across the entire industry. This makes it politically viable.
        pjmlp 4y ago
        They were standardizing a subset of POSIX, until they found out why POSIX is mostly legacy nowadays. So, now the effort is being rebooted with WASM Components. https://github.com/WebAssembly/component-model https://github.com/WebAssembly/component-model
        kllrnohj 4y ago
        Neither Apple nor Microsoft are supporting WASM on the desktop. Being able to run basic command line programs on top of POSIX? Sure, that'll be possible. But that's also already possible without WASM, and yet still very rarely done. Having actually portable APIs and abstractions such that you don't need to be concerned with differences in file paths, system behaviors, system services, audio, video, windowing, input, etc...? Yeah, WASM isn't the answer to that and it's not trying to be either. The reason Java/.NET "failed" had nothing to do with being driven by only one player and everything to do with the abstractions are so high level at that point as to be problematic. Which you either need to fully embrace, and end up with Electron, or you end up trying to reinvent WxWidgets or AWT and the clunky compromises that result where it's really just easier to build a platform-specific binary Also both Google and Microsoft have (or currently have) their own Java bytecode runtimes. So that's not even locked up by one player, and just compare Android vs. desktop JVM to see the inevitable future of "portable WASM"
      - deleted 4y ago
        [deleted]
    - exDM69 4y ago
      More like compile once and distribute one blob, run anywhere. In theory. Write once, run anywhere is true with cross compilers and native executables without any bytecode intermediate formats. Or even things like APE executables and cosmopolitan-libc. The hard part is finding portable libraries to actually do anything interesting. Networking, graphics, GUI, peripherals. WASM is not helping here and maybe even makes things a bit worse by introducing yet another platform to the portability matrix. I do see the allure of using WASM for sandboxing, plugins and running untrusted code. Things where the distribution part matters.
- pjmlp 4y ago
  Nah, this use case is why Niklaus Wirth created P-Code for Pascal, and how UCSD created a full Pascal based OS that had P-Code based binaries, and some models even had a primitive JIT/AOT compiler for it. WASM is just another reboot of bytecode based binaries that keeps poping up in multiple ways since at least 1961, when Burroughs Large Systems got released.
  - vanderZwan 4y ago
    You're right, but even so one can still be excited it's popping up again. This time with a lot of support from various parties. And it's cool that zig goes with this solution too. I will say that I'm mildly disappointed that there is no mention of Wirth in this article though. I guess Andrew didn't get around to read his work yet. I'd would expect him to love it; they'd probably agree on many things.
WalterBright 4y ago
It was a good day when we finally removed 100% of the C and C++ code from the D compiler and all of the runtime library (including the memory manager). The assembler code uses D's inline assembler. The test suite has C code in it, because of course D can compile C code.
- xwolfi 4y ago
- dahfizz 4y ago
  Something I've always wondered about compilers written in their own language.... What is your process for compiling a new compiler? Let's say you make a code change to the compiler. You have a compiled version of the previous compiler you can run to compile the new compiler. But, by definition, the new compiler is different from the old one. Do you re-run the compilation with the new compiler? How many times?
  - moonchild 4y ago
    > How many times? Once.
  - atorodius 4y ago
    I think the keyword to look for is „bootstrapping compilers“
  - WalterBright 4y ago
    1. Compile the new compiler with the old compiler 2. Compile the new compiler with the result of (1) 3. Compile the new compiler with the result of (2) 4. Verify that (2) and (3) produce identical results
    - titzer 4y ago
      Indeed, this is part of Virgil's default test run (and presumably most self-hosted compilers'). If the question is about adding new features to the language, then the process is: 1. Add the new feature to the (source of the) new compiler in a way that doesn't break any existing feature. 2. Cement the new feature in with extensive tests. 3. Bootstrap the new compiler and stable-rev it (in Virgil, that means checking in the new compiler's binary into the repo). 4. Work on other things for a while; either optimizations in the compiler or applications, to shake out bugs. 5. Bootstrap and stable-rev again. 6. Gently start using the new feature in the compiler source itself.
Kukumber 4y ago
- deleted 4y ago
  [deleted]
- jesse__ 4y ago
  I don't follow the zig community particularly closely, but from what I can tell 'too lazy' is not an accurate description of many people that are part of it, and certainly not the compiler author(s).
- lifthrasiir 4y ago
  Assuming you are genuinely not aware of language implementation processes, this is called bootstrapping. You may want to implement a language A in A (a very common goal!) but this is generally impossible when you don't have a compiler for A, so you first write an implementation of A in an already implemented language B, and then use that implementation to write an A implementation in A, abandoning the initial implementation at the end. Zig just did this.
  - kaba0 4y ago
    I'm not the parent commenter, but I will be honest -- I fail to understand the purpose of bootstrapping low-level languages. Like, if you just were to given a task to write a compiler, would you honestly choose Zig? No. Then why don't have the compiler be written in Haskell or whatever high-level language where writing code is actually productive and not error prone, since it is not a performance-critical application? EDIT: since my performance critical sentence is getting misunderstood: https://news.ycombinator.com/item?id=33914718 https://news.ycombinator.com/item?id=33914718
    - dralley 4y ago
      The Zig developers would strongly disagree that a compiler is not a performance-critical application, and they would also probably disagree that Zig doesn't bring anything to the table when it comes to writing compilers. As a generalization, the people who are motivated enough to work on a language, want to use that language. It's only natural that they would want to write their compiler in that language too, if practical. Contributors to the Zig project would on average probably be more proficient and productive in Zig than they would be in a language they don't care about so much. It's also just helpful to have the people who are designing the language working in that language regularly in the context of a sizable and nontrivial project.
capitalasset 4y ago
wdb 4y ago
Nice, the biggest achievement for a new programming language is met :D
spullara 4y ago
The Zig implementation is 3x the C++ implementation? That is surprising.
- cosmic_quanta 4y ago
  I imagine it has more features. Maybe more optimizations, for example.
- civopsec 4y ago
  Apparently it is just used to build the main compiler from source, so perhaps less featureful.
- acdha 4y ago
  I wouldn't be surprised if it had some more advanced optimizations or similar things which don't affect compatibility but also note there's one trailing clause in the description: “plus sharing Zig code with the new one”. I'd be curious exactly how much code could be reused across the two like that — it doesn't seem like it should be _that_ much because they were trying to do this to avoid commonly needing to implement things in two places.
- nektro 4y ago
  there was a lot of code shared between the implementations
- AndyKelley 4y ago
  Features that the new compiler has which the C++ implementation lacked: * The ability to translate C code into Zig * A caching system * A Mach-O linker, ELF linker, COFF linker, and WebAssembly linker * Logic to build musl libc, mingw-w64 libc, and (dynamic) glibc from source, as well as libunwind, libc++, libc++abi * Liveness analysis * A documentation generation system (Autodoc) * An x86_64 backend, aarch64 backend, WebAssembly backend, RISCV-64 backend, arm backend, SPIR-V backend, and C backend
  - cryptonector 4y ago
    Where would I read more about Zig? Not that I have cycles to spare, but I think a fair bit about moving from C. For example, I maintain an ASN.1 compiler written in C, and I hate C, so I made it emit a JSON AST of ASN.1 modules, and now a friend of mine just wrote a backend in Swift that takes that JSON AST output and produces Swift code / templates. Leaving C behind requires a good path for porting legacy C to the new thing, or else tons of time and mindshare to get new things built. So D and Zig are very appealing.
KingLancelot 4y ago
compiler-guy 4y ago
One reason (among many) to do this is because compilers require complex and demanding source code across a wide-range of theory and algorithms. And so make good tests for both the source language (is it sufficiently expressive to do this cleanly?) and for the various analysis, optimization, and code-generation passes.
Decabytes 4y ago
I really like how when Andrew makes a decision about something related to Zig, he outlines how other programming languages do it and gives his thoughts. My question is. I feel like Zig is trying to do a lot of things that GO set out to do. To reduce a lot of complexity of programs by removing hidden control flow, macros etc. But how will Zig keep itself from repeating the mistakes GO made that make people dislike it?
- AndyKelley 4y ago
  Thank you for the compliment. I have some interesting news for you... Go is a smashing success, wildly popular, and eating Java's lunch. It is an objectively incorrect generalization to say that people dislike Go.
  - acedTrex 4y ago
    > It is an objectively incorrect generalization to say that people dislike Go It is not, people USE go, they do not like go really.
    - lordgroff 4y ago
      Really? I like Go, I might even love it.
    - deleted 4y ago
      [deleted]
    - phinnaeus 4y ago
      I like Go
    - thadt 4y ago
      This is true, I don't really like Go. After years of writing code in Assembly, C, C++, Perl, Basic, C#, Java, Python, Rust, JavaScript, TypeScript, Lua, Zig, and Go I find that what I really like is good tooling, and code that is easy to read and reason about. Go the language and toolset happen to do this really rather well at the moment though.
garganzol 4y ago
This use-case shows a big potential of WASM. Just imagine how we would run a 50 years old software in year 2072 thanks to WASM and WASI standards.
- kristoff_it 4y ago
  It just so happens WASM is the one VM target that LLVM supports. Sure, it's a nice VM that can be implemented without too much fuss, and ditto for WASI, but that's it. It's just the most convenient VM to target for us.
kristoff_it 4y ago
If after reading the post you're still unsure about why we're going through this process, I made a video that focuses more on the reasons from the perspective of a Zig contributor, showing how the bootstrapping process helps contributors on their day to day tasks. https://youtu.be/MCfD7aIl-_E https://youtu.be/MCfD7aIl-_E
- pavon 4y ago
  Thanks, the explanation about how the Zig compiler uses compile-time code execution to implement multiplatform support in the compile-to-c-backend helped me understand why WASM has better trade-offs for Zig than other bootstrapping options. For others, the start of the video is discussing boostrapping in general, and the current compiler state, and then the discussion about "Why WASM" starts at around minute seven.
henry_viii 4y ago
Could someone explain to me why Zig is getting hyped so much on HN? From a quick glance it looks like Zig is memory-unsafe like C/C++. I thought the macro trend was moving onto memory-safe languages: https://news.ycombinator.com/item?id=33819616 https://news.ycombinator.com/item?id=33819616 https://news.ycombinator.com/item?id=33560227 https://news.ycombinator.com/item?id=33560227 https://news.ycombinator.com/item?id=32905885 https://news.ycombinator.com/item?id=32905885 What innovation does Zig bring that I'm missing?
- detaro 4y ago
  Not everyone thinks that macro trend is the most important thing ever and Zig is an interesting spin on a low-level language/a better C.
- nektro 4y ago
  it's perfectly possible to make memory safe programs in Zig. there are many an innovation that Zig brings to the table
  - AaronFriel 4y ago
    That isn't really an interesting statement, it's perfectly possible to make memory safe programs in C or assembly. The question is, how easy is it it to ensure a program is memory safe in Zig? The trend toward memory safety is marked by languages and tools making it harder to inadvertently write exploitable code, and easier to verify that the program is not exploitable. My understanding of Zig is that while it does some of the same things as so-called memory safe languages, it is not a "memory safe language" in the same sense as they use the term.
- adamgordonbell 4y ago
  This article here deserves attention because its interesting and counter-intuitive even if you don't use Zig. It's a story of problem solving.
- kristoff_it 4y ago
  > I thought the macro trend was moving onto memory-safe languages I guess some people like to Zig when others zag :^)
1vuio0pswjnm7 4y ago
Why not provide either hex editor or wasm as options. Let the user choose. The former is not trendy, it's time-tested and has no ties to a commercial entity or the online advertising "business". Whereas the later has only been around since 2015 and was introduced by a company that subsists off an agreement with a deviant online advertising company. Not to mention it targets "the web", which is only one use for computer programming, and one that is overwhelmingly under the control of a handful of large corporations.
- nektro 4y ago
  you might be interested in https://github.com/oriansj/stage0 https://github.com/oriansj/stage0 if you haven't seen it before
- kristoff_it 4y ago
  > Let the user choose. The main user of this bootstrapping process are core contributors, normal users are still supposed to download prebuilt executables from the official website. Distro maintainers also are not the target user of this bootstrapping process, since it involves a binary blob provided by us. The real users of this procedure are Zig contributors, so that they can trivially build latest zig always, and without the annoyance of having to keep a C++ version of the compiler in sync with the main one. That's it.
- CharlesW 4y ago
  > Whereas the later has only been around since 2015 and was created by a company that subsists off an agreement with a deviant online advertising company. Mozilla created a precursor technology, but I thought Wasm was developed via the W3C standards process from the start. From the notes of the first meeting, you can see attendees from Adobe, Apple, Arm, Autodesk, Google, Intel, Mozilla, Stanford, and more. https://github.com/WebAssembly/meetings/blob/main/main/2017/CG-05.md#attendees https://github.com/WebAssembly/meetings/blob/main/main/2017/... Additionally, Wasm has been a W3C standard since 2019.
- projektfu 4y ago
  https://en.wikipedia.org/wiki/MLX_(software) https://en.wikipedia.org/wiki/MLX_(software) Example of use: https://lparchive.org/Computes-Gazette/Update%2008/ https://lparchive.org/Computes-Gazette/Update%2008/
kalkin 4y ago
This is cool. I'm surprised that compiling a partial Zig backend to WASM and then compressing that ends up meaningfully smaller than compiling to C and compressing the C, when you include also the C partial WASM implementation and zstd decoder. This sounds kind of like a general strategy for compressing C code which I would not have expected to work well, but cool that it does! If AndyKelley ends up reading this - did you end up doing a direct comparison of "zig1.c.zstd + zstd.c" size vs the "zig1.wasm.zstd + zstd.c + wasm.c.zstd" set that you ended up with? If so, how did it turn out?
- AndyKelley 4y ago
  I ran the command locally just now: zig1-x86_64-linux.c: 86 MiB zig1-x86_64-linux.c.zstd: 3.5 MiB
- kristoff_it 4y ago
  I think it boils down to how bloated is the C code generated by the C backend, which to some degree has to be, since it's generated programmatically. My undestanding is what ends up happening is that the wasm step acts as a form of semantic compression that brings its own benefits over zstd (and which can still be combined with zstd by compressing the wasm file).
- nektro 4y ago
  this is likely due to LLVM being involved in the WASM generation and it being able to perform all of its optimization steps before outputting code. whereas Zig's C backend has not yet gained the ability to perform all the same optimizations.
iskander 4y ago
Naive question about Zig: is there any tooling for embedding it within larger Python codebases akin to Maturin (for Rust) or Nimporter (for Nim)? I have seen examples where the Zig code imports Python.h and uses low-level Python C API calls but I want something very lightweight for accelerating computational bottlenecks without worrying about unwrapping/wrapping data.
- nurbl 4y ago
  Not sure exactly what you need, but since Zig is C compatible, it's easy to build a zig library and import it from python using ctypes. I guess if you need something more sophisticated you could use cffi (haven't tried it). Zig is even available as a convenient python package: https://pypi.org/project/ziglang/ https://pypi.org/project/ziglang/
  - iskander 4y ago
    I'm looking for something that fits into a setup.py file (or, like Maturin creates a multi-language package config) which (1) automatically compiles zig source into a Python extension module for me, so that I can (2) just import zig code into Python and call it without writing any type conversion logic.
deleted 4y ago
[deleted]
deepsun 4y ago
> There is exactly one VM target available to Zig that is both OS-agnostic and subject to LLVM’s state-of-the-art optimization passes, and that is WebAssembly. Honestly, sounds like old Java would also fit their requirements. There was a time when multiple languages ran for multi-platform, which is eased nowadays with containers and remote developer environments. So if their main concern is multi-platform, then feels like they want to look at the technologies developed at that time.
- kristoff_it 4y ago
  > Honestly, sounds like old Java would also fit their requirements. Yes, it would, as would any other VM target. That said WASM is extremely convenient because it's a target that LLVM supports and because writing a VM for it (or something that compiles it to C) is easy. Java from this perspective seems way less convenient, as it would require us first to build a Zig backend for it, and then we would have to implement our own java intrepreter / aot compiler / ... for it.
- brundolf 4y ago
  From my understanding, the JVM is much more opinionated than webassembly (it has built-in GC, and I've heard it even has a notion of classes and related concepts at the bytecode level). Particularly for a low-level C-like language like Zig, it seems like a pretty bad match.
  - deepsun 4y ago
    We're talking about compiler, right? Sorry don't see why classes would be a bad match to compiling Zig code to LLVM representation. As long as we don't do lot of numbers crunching (where Fortran, R, Julia, MatLab shine), then structs/classes are ok.
    - brundolf 4y ago
      > We're talking about compiler, right? Yes, and Zig's compiler is now written in Zig Zig operates at a low level where it cares about things like manual memory-management. Compiling it to target the JVM instead of webassembly (assuming that's what you're suggesting) would be a really rough abstraction, because the JVM is higher-level. Webassembly is designed to accommodate lower-level languages adjacent to C that manage their own memory, etc And that's not even mentioning the fact that (it sounds like) Zig's compiler already has an LLVM back-end, which means they get wasm support "for free"
deleted 4y ago
[deleted]
shp0ngle 4y ago
> We provide a minimal WASI interpreter implementation that is built from C source, and then used to translate the Zig self-hosted compiler source code into C code. The C code is then compiled and linked, again by the system C compiler, into a stage2 binary. The stage2 binary can then be used repeatedly with zig build to build from source from that point on. Nope, no matter how many times I read this, I’m still lost. But then I never needed to care about VMs, compilers and bootstraps.
- Arnavion 4y ago
  Read the "To summarize:" section a little after the part you quoted.
- gavinray 4y ago
  Okay so we start from some C source code, and we build an interpreter for WASM + WASI from that (WASM that has access to system calls) C source -> WASM interpreter w/ system access Now we can take the Zig self-hosted compiler (the one in .zig), which has been compiled to .wasm/.wasi files. Since we have an interpreter for those now, we can do this: Zig compiler as .wasi instead of .exe --> WASM interpreter --> Zig's "translate-to-c" function, for the .zig file sources of the Zig compiler E.G. $ run-webassembly "zig-compiler-as-wasm.wasi" --translate-c <source code to zig compiler> At this point, we have the Zig compiler as .c files. Now you can use GCC/clang or whatnot, to build a regular binary for the compiler Output of Zig's "translate-to-c" from previous step --> GCC/clang --> Zig compiler but NOT AS WASM, as a regular binary
  - MrBuddyCasino 4y ago
    how do you compile the Zig self-hosted compiler to wasm?
    - TUSF 4y ago
      Defen explained explains this well here: https://news.ycombinator.com/item?id=33914969 https://news.ycombinator.com/item?id=33914969
  - 4y ago
randyrand 4y ago
> We provide a minimal WASI interpreter implementation that is built from C source, and then used to translate the Zig self-hosted compiler source code into C code. What do you use to compile the Zig source into C code? Wouldn't you need a Zig Compiler? I would have expected this? > We provide a minimal WASI interpreter implementation that is built from C source (i.e. so we don't need a Web Browser), then used to translate Zig Self-Hosted Compiler WASM code into C code. The Zig Self-Hosted Compiler WASM code is committed to the code base each time it changes, so when building a commit you already have the WASM source to a Zig compiler right there. > Of course, in the context of bootstrapping, this Zig Self-Hosted Compiler WASM source needed to be generated the first time at some point. For that first time, we used the C++ compiler to compile the Zig Self-Hosted Compiler from Zig into WASM.
- Yujf 4y ago
  The Zig compiler gets compiled to WASM. The WASI interpreter runs the WASM binary to compile Zig to C.
  - randyrand 4y ago
    > The Zig compiler gets compiled to WASM This is the part I don't understand. In the context of bootstrapping, where does the Zig to WASM compiler come from?
    - Laremere 4y ago
      WASM is platform agnostic, so it is one of the things you start with, along with the compiler source code. It is built on a different computer before the bootstrapping process begins.
    - defen 4y ago
      This is a method of ensuring that future builds do not depend on the existence of a Zig compiler; it's not a way to go from 0 to Zig without a Zig compiler ever having existed. Technically this already existed in the form of a Zig compiler written in C++; the point of this exercise was to stop using C++. So: Presume the existence of a compiler that can compile Zig. Use that compiler to compile the written-in-Zig Zig compiler to WASM. Now you have a big chunk of WASM, so you also need a WASI interpreter. Write that in 4,000 lines of highly portable C. Then use that WASI interpreter to run your big chunk of WASM code and give it your written-in-Zig Zig compiler, and tell it to output C. Then compile that C code with your system compiler, and then use that native executable to recompile the written-in-Zig Zig compiler. At this point you should be at a fixed point and further recompilations of the Zig compiler will yield the same binary.
titzer 4y ago
This is great. The more self-contained (really self-hosted) a language is, the more implementation freedom and ability to evolve it gets. Virgil version I&II were a Virgil->C compiler written in Java. Later, I wrote an interpreter for Virgil III in Java and then began writing a compiler in Virgil III. When that compiler could compile version III (including itself), I checked in the first "stable" compiler as a jar. Then periodically when enough new features and bugs were fixed, I checked in a new stable binary (jar). Later, I developed and eventually fully switched to native backends for 32- and 64-bit x86 on MacOS and Darwin. Today, 5 stable binaries are checked in: jar, x86-darwin, x86-64-darwin, x86-linux, and x86-64-linux. There is also a Wasm backend, which can bootstrap the compiler too, but I did not check in a stable binary for it. Initially I was worried that a codegen bug would prevent bootstrapping from a compiler binary and that I'd need to fall back to running on an interpreter. So far, there's never been a codegen bug bad enough to break bootstrapping, so I am not worried about this. The compiler never needs to bootstrap from an interpreter.
- funny_falcon 4y ago
  Why did you abandon "translate-to-C" backend? It would be good to have high level language translated to C (aside from Vala and Nim).
ajnin 4y ago
> The idea here is to use a minimal wasm binary as a stage1 kernel that is committed to source control and therefore can be used to build any commit from source. We provide a minimal WASI interpreter implementation that is built from C source, and then used to translate the Zig self-hosted compiler source code into C code. The C code is then compiled and linked, again by the system C compiler, into a stage2 binary. The stage2 binary can then be used repeatedly with zig build to build from source from that point on. 1/ Wouldn't that be considered "cheating" to basically commit precompiled compiler binaries to source control ? 2/ I don't understand how that solves the "features need to be implemented twice" problem. Wouldn't you need to implement new Zig language features into that WASM kernel whenever they are used in the Zig compiler source ?
- AndyKelley 4y ago
  1. Yes it is cheating. That is the downside of this approach. Contributors to Zig and users of Zig don't care about such cheating, but distribution maintainers such as Debian Developers do (rightly) care. This decision is a tradeoff that favors contributors and users at the expense of system package maintainers. I am counting on a third party implementation of Zig to arise someday and solve the bootstrapping problem for system package maintainers. But in the short term, it's more important to prioritize the needs of users and contributors. 2. Whenever this happens, the contributor runs `zig build update-zig1` and commits the updated wasm kernel to the repository.
  - Arnavion 4y ago
    Perhaps the distro devs could maintain their own golden WASM blobs that they compiled themselves and thus trust. Could be the same process as SecureBoot / package signing keys.
    - edwintorok 4y ago
      Can the stage1 wasm binary be reproduced by the stage2 or stage3 executable? Aside from a "trusting trust" type of attack that seems fine, and every modern distro relies on some bootstrap binary for C compilers anyway (usually older versions of them), so it wouldn't be that much different of a bootstrapping problem than bootstrapping GCC itself. (See the GNU Mes project which attempts to bootstrap from just a very small hex interpreter)
deleted 4y ago
[deleted]
u8 4y ago
If you’re interested in trying Zig out and want an easy way to update/use multiple versions I’ve been working on a Zig Version Manager for the past few weeks. It works on Windows, Mac, Linux, a smattering of BSD’s and Plan 9. Arm and x86. https://github.com/tristanisham/zvm https://github.com/tristanisham/zvm
- ptato 4y ago
  On Windows one can use https://scoop.sh https://scoop.sh too. There's a "zig" package for numbered releases, and a "zig-dev" package for nightly.
  - sitkack 4y ago
    There is also `pip install ziglang`
- Cloudef 4y ago
  With nix you can use https://github.com/Cloudef/nix-zig-stdenv https://github.com/Cloudef/nix-zig-stdenv see the versions.nix
yyyk 4y ago
This is clever. There's one thing that I do not understand: Why the step 3 compiler has only the C backend enabled? In theory one could enable all the backends and skip to step 6? The step 5 comment says something about 'correct final logic', but I'm unsure what it means?
- kristoff_it 4y ago
  That's the Zig compiler that is implemented inside the wasm blob. Since we commit that blob to the repository, we want to keep is as small as possible, which is why it only contains the C backend and nothing else.
waynecochran 4y ago
Zig originally used the same strategy as the D compiler - not freeing memory until process exit wait ... what!? impractical in C/C++ because of language footguns C and C++ are now very different languages. You might as well say objective-c/swift
- verdagon 4y ago
  Not freeing memory is a fairly common approach, in compilers and command line tools as well. If an AST hangs around until the end of the program anyway, why not let the OS take care of blasting it away? free() is expensive after all.
  - waynecochran 4y ago
    I am guessing LLVM does not do this.
    - JonChesterfield 4y ago
      Functions like Type::getInt32() or whatever it's called return the same heap allocated pointer each time so pointer equality can be used for value equality. That's a nice trick that only really works optimally if you leak the pointer, or at least don't free it until very near the end. I've seen cleaning up memory at the end of a program take 20% of the run time and that was indeed patched to just exit & leak as a result. With a flag to clean up so we could still run valgrind on it usefully.
    - sanxiyn 4y ago
      LLVM has JIT users, so LLVM can't do this.
      - Arnavion 4y ago
        Yes, any compiler-as-a-library software can't do this. And compilers-as-libraries are becoming more popular these days since being a library makes it easier to integrate the compiler with tooling, like linters and IDEs / language servers.
synergy20 4y ago
what's zig's advantage over nim? I have decided to stick with main stream languages after playing with various new languages in the past including ziglang, it's fun but in the end, more of a waste of time. in practice a language is really an ecosystem, from compiler, tools, editors, libraries, field testing...if you want to get things done, you just have to use the main stream ones.
- tmtvl 4y ago
  Indeed, we should all be using ALGOL, LISP, and COBOL.
  - synergy20 4y ago
    In the early days when CS and languages are new things, we need evolve faster. When things kind of settling down, we need avoid NIH. Time is different. we don't reinvent new languages to compete against English, Chinese, Spanish,etc nowadays, I'm sure that's different in the early days when human was figuring out how to communicate and how to create the language they need.
- GeorgeTirebiter 4y ago
  This is my personal experience, ymmv, and maybe somebody needed coffee or something, but! I've found the zig community more friendly and open-minded than the Nim community. Again, one person, one experience, I like them both, yada yada. I suggest you have a look at both of them, and decide. The Nim book is very good. I would NOT call using zig or nim a waste of time; yes, the ecosystem matters. But decent languages + good libraries for what you need to do = Total Win. IHMO, this is why Python wins so bigly in the (increasing) influence it has. C wins, and will always win, I think, because it's the closest we have to a portable assembly language. We'll see, re: wasm. Maybe wasm will be the 'pdp-11' of computing for the 21st century.
- ptato 4y ago
  from only a glance and without having ever used nim, it seems to be more abstracted from the machine, whereas zig is closer. nim code should be shorter, nicer, closer to ideal pseudocode. safer too, i imagine. zig code should be more explicit, and if you do it right, more efficient in time and memory. zig also has a philosophy of, quoted from https://ziglang.org https://ziglang.org: "No hidden control flow. No hidden memory allocations. No preprocessor, no macros.". this also should make zig code more explicit, but probably more verbose too. i could also be completely wrong. like i said, i know nothing of nim other than what the homepage says. don't listen to me.
adrianmonk 4y ago
> It is then further optimized with wasm-opt -Oz --enable-bulk-memory bringing the total down to 2.4 MiB. Finally, it is compressed with zstd, bringing the total down to 637 KB. This is offset by the size of the zstd decoder implementation in C, however it is worth it because the zstd implementation will change rarely if ever, saving a total of 1.8 MiB every time the wasm binary is updated. Is the goal here to save space in the Git repo, by compressing before committing? I wouldn't assume using zstd is necessarily worth the complication. It could even make things worse. As I understand it, Git stores objects in packfiles[1], and these are both delta-fied and compressed with zlib. Your zstd reduces the 2.4MiB .wasm file to 637K. But Git's zlib should reduce 2.4MB to 800K (according to a quick test I just did). So at best, you only save 163K, not 1.8 MiB. But if Git's delta-fication works, you may actually use more space. Git should try to use its binary diff algorithm[2] to compare your various committed versions of zig1.wasm. If that algorithm is effective against Wasm files (my guess is yes), it will be able to store one version as a full copy and other versions as (somewhat? much?) smaller deltas against the full one. If you store .wasm.zst files, since compression tends to obscure commonalities, my guess is Git won't be able to do deltas and will have to store full copies of every version. On a side note, Git is said to be bad at handling binaries, and that's somewhat true, but there's some nuance. Binary files get in the way of easy branching and merging because Git can't merge them. So Git is bad at binary files in that way, but that's not relevant here. Also a lot of binary formats (like JPEG) are very much not amenable to binary diff, but my bet is that's not relevant here either. --- [1] See: https://git-scm.com/docs/git-pack-objects https://git-scm.com/docs/git-pack-objects https://git-scm.com/docs/pack-format https://git-scm.com/docs/pack-format https://git-scm.com/book/en/v2/Git-Internals-Packfiles https://git-scm.com/book/en/v2/Git-Internals-Packfiles [2] "inspired by" LibXDiff, according to https://github.com/git/git/blob/master/diff-delta.c https://github.com/git/git/blob/master/diff-delta.c
- O_H_E 4y ago
  I'd imagine distributing tarballs is also an important use case.
  - adrianmonk 4y ago
    Actually zstd makes that worse too, somewhat paradoxically. At least in this case, because Zig uses xz for their tarballs. (If they used gzip, it would be the other way around.) The reason is that compression algorithms usually can't make further reductions when re-compressing already-compressed files. And xz has a higher compression ratio than zstd, so when you stick zig1.wasm.zst into a tar.xz file, xz is deprived of the opportunity to work its more powerful magic. As a test, I got zig-0.11.0-dev.638+5c67f9ce7.tar.xz from https://ziglang.org/download/ https://ziglang.org/download/ , extracted it, and rebuilt the tar.xz myself. Then I replaced stage1/zig1.wasm.zst with stage/zig1.wasm and rebuilt the tar.xz again. Results: $ du -sk *tar* 168136 zig.new.tar 14500 zig.new.tar.xz 166416 zig.orig.tar 14568 zig.orig.tar.xz So, zig.orig.tar is the uncompressed tarball that contains zig1.wasm.zst, and it is indeed smaller than zig.new.tar. But the .tar.xz files are the other way around. Not using zstd saves 68K. =-=-= Also, in the process, I accidentally discovered something else that makes a bigger difference. Since I knew the order of files within a tar archive can affect the compression ratio (due to data locality), while doing my test, I used "tar tf" to list my tar file's contents and compare it with what I downloaded. It didn't match, so I knew I wasn't doing an apples to apples comparison. So I added "--sort=name" to my tar commands. And both of my tar files ended up smaller than the one I downloaded: $ du -sk zig-0.11.0-dev.638+5c67f9ce7.tar.xz 15152 zig-0.11.0-dev.638+5c67f9ce7.tar.xz Just adding the "--sort=name" option to tar saves 584K! That's around 4% of the entire tar file. Locality matters more than I thought.
beltsazar 4y ago
> One big downside is losing the ability to build any commit from source without meta-complexity creeping in. For example, let’s say that you are trying to do git bisect. At some point, git checks out an older commit, but the script fails to build from source because the binary that is being used to build the compiler is now the wrong version. Sure, this can be addressed, but this introduces unwanted complexity that contributors would rather not deal with. If it's the main concern of using a prior build of the compiler, an alternative solution is to develop a tool for contributors to automate and ease the process. For example, Rust has this: https://github.com/rust-lang/cargo-bisect-rustc https://github.com/rust-lang/cargo-bisect-rustc
cryptonector 4y ago
> Now, there is this WebAssembly binary, which is not source code, but is in fact a build artifact. Some people, rightly, take these things very seriously [...]. Regarding this concern, well, you have to commit that build artifact because you're moving fast, but eventually you could do what the OpenJDK does: to build OpenJDK version N you need OpenJDK versions N-1 or N, and you can get OpenJDK version N-1 from your distro or from any number of places (like AdoptOpenJDK). You're essentially doing that now, but with unnamed versions -- you have to know which commits are like JDK version boundaries, and the clue is that the commit updates that one build artifact. TFA is a very good read.
teo_zero 4y ago
I'm surprised nobody has raised the trust argument... Who guarantees that the WASI blob has no hidden backdoor?
pabs3 4y ago
Thats an unfortunate development for the bootstrappability of Zig solely from source code. https://bootstrappable.org https://bootstrappable.org
sjmulder 4y ago
From the perspective of a package maintainer (I don't deal with the core infrastructure of our packaging system, I just package and patch things): While this unusual bootstrap with a WASM stage and a C WASI interpreter doesn't satisfy "everything from source" it's so much better than sitting on a non-Intel/ARM or non-Windows/Mac/Linux machine and having no other option but to maintain 5 different ancient versions of a compiler for a bootstrap sequence, or worse, being required to cross-compile from another host. Thanks :)
jokoon 4y ago
I like zig better than rust, but zig is still a bit too sophisticated for me.

528 points by nyberg 4y ago

I have honestly been more excited about Wasm for desktop than I am for the web. And I'm really excited about it for the Web. Really cool to see this use case pop up right as I'm trying to integrate it into my stack!

throwaway894345 4y ago
I've never really thought about wasm for the desktop (I've thought about it for server and of course browser), can you elaborate on your excitement? Is it just for this sort of bootstrapping application, or are there other benefits?
- als0 4y ago
  Write once, run anywhere… but this time the dream will come true!
  - typon 4y ago
    The difference is Oracle vs. No Oracle.
    - pjmlp 4y ago
      You get Apple, Google and Microsoft instead (FF hardly counts).
  - kllrnohj 4y ago
    Nearly every desktop is still x86 and just a couple years ago that was entirely true, and yet write-once-run-anywhere wasn't remotely close to true. WASM as a result isn't changing anything here, since the assembly is very extremely not remotely close to the issue with having a portable binary.
    - int_19h 4y ago
      Wasm is an opportunity for all platforms to adopt a single container format for architecture-independent binary code, and with uniforms APIs being defined on top of that; they are already standardizing a subset of POSIX. None of the bits here are really new, but the big difference this time is that it's not owned by any one of the major players (like e.g. Java and .NET were), and the design process is truly collaborative across the entire industry. This makes it politically viable.
      - pjmlp 4y ago
        They were standardizing a subset of POSIX, until they found out why POSIX is mostly legacy nowadays. So, now the effort is being rebooted with WASM Components. https://github.com/WebAssembly/component-model https://github.com/WebAssembly/component-model
      - kllrnohj 4y ago
        Neither Apple nor Microsoft are supporting WASM on the desktop. Being able to run basic command line programs on top of POSIX? Sure, that'll be possible. But that's also already possible without WASM, and yet still very rarely done. Having actually portable APIs and abstractions such that you don't need to be concerned with differences in file paths, system behaviors, system services, audio, video, windowing, input, etc...? Yeah, WASM isn't the answer to that and it's not trying to be either. The reason Java/.NET "failed" had nothing to do with being driven by only one player and everything to do with the abstractions are so high level at that point as to be problematic. Which you either need to fully embrace, and end up with Electron, or you end up trying to reinvent WxWidgets or AWT and the clunky compromises that result where it's really just easier to build a platform-specific binary Also both Google and Microsoft have (or currently have) their own Java bytecode runtimes. So that's not even locked up by one player, and just compare Android vs. desktop JVM to see the inevitable future of "portable WASM"
    - deleted 4y ago
      [deleted]
  - exDM69 4y ago
    More like compile once and distribute one blob, run anywhere. In theory. Write once, run anywhere is true with cross compilers and native executables without any bytecode intermediate formats. Or even things like APE executables and cosmopolitan-libc. The hard part is finding portable libraries to actually do anything interesting. Networking, graphics, GUI, peripherals. WASM is not helping here and maybe even makes things a bit worse by introducing yet another platform to the portability matrix. I do see the allure of using WASM for sandboxing, plugins and running untrusted code. Things where the distribution part matters.
pjmlp 4y ago
Nah, this use case is why Niklaus Wirth created P-Code for Pascal, and how UCSD created a full Pascal based OS that had P-Code based binaries, and some models even had a primitive JIT/AOT compiler for it. WASM is just another reboot of bytecode based binaries that keeps poping up in multiple ways since at least 1961, when Burroughs Large Systems got released.
- vanderZwan 4y ago
  You're right, but even so one can still be excited it's popping up again. This time with a lot of support from various parties. And it's cool that zig goes with this solution too. I will say that I'm mildly disappointed that there is no mention of Wirth in this article though. I guess Andrew didn't get around to read his work yet. I'd would expect him to love it; they'd probably agree on many things.

WalterBright 4y ago

It was a good day when we finally removed 100% of the C and C++ code from the D compiler and all of the runtime library (including the memory manager). The assembler code uses D's inline assembler. The test suite has C code in it, because of course D can compile C code.

xwolfi 4y ago
dahfizz 4y ago
Something I've always wondered about compilers written in their own language.... What is your process for compiling a new compiler? Let's say you make a code change to the compiler. You have a compiled version of the previous compiler you can run to compile the new compiler. But, by definition, the new compiler is different from the old one. Do you re-run the compilation with the new compiler? How many times?
- moonchild 4y ago
  > How many times? Once.
- atorodius 4y ago
  I think the keyword to look for is „bootstrapping compilers“
- WalterBright 4y ago
  1. Compile the new compiler with the old compiler 2. Compile the new compiler with the result of (1) 3. Compile the new compiler with the result of (2) 4. Verify that (2) and (3) produce identical results
  - titzer 4y ago
    Indeed, this is part of Virgil's default test run (and presumably most self-hosted compilers'). If the question is about adding new features to the language, then the process is: 1. Add the new feature to the (source of the) new compiler in a way that doesn't break any existing feature. 2. Cement the new feature in with extensive tests. 3. Bootstrap the new compiler and stable-rev it (in Virgil, that means checking in the new compiler's binary into the repo). 4. Work on other things for a while; either optimizations in the compiler or applications, to shake out bugs. 5. Bootstrap and stable-rev again. 6. Gently start using the new feature in the compiler source itself.

Kukumber 4y ago

deleted 4y ago
[deleted]
jesse__ 4y ago
I don't follow the zig community particularly closely, but from what I can tell 'too lazy' is not an accurate description of many people that are part of it, and certainly not the compiler author(s).
lifthrasiir 4y ago
Assuming you are genuinely not aware of language implementation processes, this is called bootstrapping. You may want to implement a language A in A (a very common goal!) but this is generally impossible when you don't have a compiler for A, so you first write an implementation of A in an already implemented language B, and then use that implementation to write an A implementation in A, abandoning the initial implementation at the end. Zig just did this.
- kaba0 4y ago
  I'm not the parent commenter, but I will be honest -- I fail to understand the purpose of bootstrapping low-level languages. Like, if you just were to given a task to write a compiler, would you honestly choose Zig? No. Then why don't have the compiler be written in Haskell or whatever high-level language where writing code is actually productive and not error prone, since it is not a performance-critical application? EDIT: since my performance critical sentence is getting misunderstood: https://news.ycombinator.com/item?id=33914718 https://news.ycombinator.com/item?id=33914718
  - dralley 4y ago
    The Zig developers would strongly disagree that a compiler is not a performance-critical application, and they would also probably disagree that Zig doesn't bring anything to the table when it comes to writing compilers. As a generalization, the people who are motivated enough to work on a language, want to use that language. It's only natural that they would want to write their compiler in that language too, if practical. Contributors to the Zig project would on average probably be more proficient and productive in Zig than they would be in a language they don't care about so much. It's also just helpful to have the people who are designing the language working in that language regularly in the context of a sizable and nontrivial project.

capitalasset 4y ago

wdb 4y ago

Nice, the biggest achievement for a new programming language is met :D

spullara 4y ago

The Zig implementation is 3x the C++ implementation? That is surprising.

cosmic_quanta 4y ago
I imagine it has more features. Maybe more optimizations, for example.
civopsec 4y ago
Apparently it is just used to build the main compiler from source, so perhaps less featureful.
acdha 4y ago
I wouldn't be surprised if it had some more advanced optimizations or similar things which don't affect compatibility but also note there's one trailing clause in the description: “plus sharing Zig code with the new one”. I'd be curious exactly how much code could be reused across the two like that — it doesn't seem like it should be _that_ much because they were trying to do this to avoid commonly needing to implement things in two places.
nektro 4y ago
there was a lot of code shared between the implementations
AndyKelley 4y ago
Features that the new compiler has which the C++ implementation lacked: * The ability to translate C code into Zig * A caching system * A Mach-O linker, ELF linker, COFF linker, and WebAssembly linker * Logic to build musl libc, mingw-w64 libc, and (dynamic) glibc from source, as well as libunwind, libc++, libc++abi * Liveness analysis * A documentation generation system (Autodoc) * An x86_64 backend, aarch64 backend, WebAssembly backend, RISCV-64 backend, arm backend, SPIR-V backend, and C backend
- cryptonector 4y ago
  Where would I read more about Zig? Not that I have cycles to spare, but I think a fair bit about moving from C. For example, I maintain an ASN.1 compiler written in C, and I hate C, so I made it emit a JSON AST of ASN.1 modules, and now a friend of mine just wrote a backend in Swift that takes that JSON AST output and produces Swift code / templates. Leaving C behind requires a good path for porting legacy C to the new thing, or else tons of time and mindshare to get new things built. So D and Zig are very appealing.

KingLancelot 4y ago

compiler-guy 4y ago

One reason (among many) to do this is because compilers require complex and demanding source code across a wide-range of theory and algorithms. And so make good tests for both the source language (is it sufficiently expressive to do this cleanly?) and for the various analysis, optimization, and code-generation passes.

Decabytes 4y ago

I really like how when Andrew makes a decision about something related to Zig, he outlines how other programming languages do it and gives his thoughts. My question is. I feel like Zig is trying to do a lot of things that GO set out to do. To reduce a lot of complexity of programs by removing hidden control flow, macros etc. But how will Zig keep itself from repeating the mistakes GO made that make people dislike it?

AndyKelley 4y ago
Thank you for the compliment. I have some interesting news for you... Go is a smashing success, wildly popular, and eating Java's lunch. It is an objectively incorrect generalization to say that people dislike Go.
- acedTrex 4y ago
  > It is an objectively incorrect generalization to say that people dislike Go It is not, people USE go, they do not like go really.
  - lordgroff 4y ago
    Really? I like Go, I might even love it.
  - deleted 4y ago
    [deleted]
  - phinnaeus 4y ago
    I like Go
  - thadt 4y ago
    This is true, I don't really like Go. After years of writing code in Assembly, C, C++, Perl, Basic, C#, Java, Python, Rust, JavaScript, TypeScript, Lua, Zig, and Go I find that what I really like is good tooling, and code that is easy to read and reason about. Go the language and toolset happen to do this really rather well at the moment though.

garganzol 4y ago

This use-case shows a big potential of WASM. Just imagine how we would run a 50 years old software in year 2072 thanks to WASM and WASI standards.

kristoff_it 4y ago
It just so happens WASM is the one VM target that LLVM supports. Sure, it's a nice VM that can be implemented without too much fuss, and ditto for WASI, but that's it. It's just the most convenient VM to target for us.

kristoff_it 4y ago

If after reading the post you're still unsure about why we're going through this process, I made a video that focuses more on the reasons from the perspective of a Zig contributor, showing how the bootstrapping process helps contributors on their day to day tasks. https://youtu.be/MCfD7aIl-_E https://youtu.be/MCfD7aIl-_E

pavon 4y ago
Thanks, the explanation about how the Zig compiler uses compile-time code execution to implement multiplatform support in the compile-to-c-backend helped me understand why WASM has better trade-offs for Zig than other bootstrapping options. For others, the start of the video is discussing boostrapping in general, and the current compiler state, and then the discussion about "Why WASM" starts at around minute seven.

henry_viii 4y ago

Could someone explain to me why Zig is getting hyped so much on HN? From a quick glance it looks like Zig is memory-unsafe like C/C++. I thought the macro trend was moving onto memory-safe languages: https://news.ycombinator.com/item?id=33819616 https://news.ycombinator.com/item?id=33819616 https://news.ycombinator.com/item?id=33560227 https://news.ycombinator.com/item?id=33560227 https://news.ycombinator.com/item?id=32905885 https://news.ycombinator.com/item?id=32905885 What innovation does Zig bring that I'm missing?

detaro 4y ago
Not everyone thinks that macro trend is the most important thing ever and Zig is an interesting spin on a low-level language/a better C.
nektro 4y ago
it's perfectly possible to make memory safe programs in Zig. there are many an innovation that Zig brings to the table
- AaronFriel 4y ago
  That isn't really an interesting statement, it's perfectly possible to make memory safe programs in C or assembly. The question is, how easy is it it to ensure a program is memory safe in Zig? The trend toward memory safety is marked by languages and tools making it harder to inadvertently write exploitable code, and easier to verify that the program is not exploitable. My understanding of Zig is that while it does some of the same things as so-called memory safe languages, it is not a "memory safe language" in the same sense as they use the term.
adamgordonbell 4y ago
This article here deserves attention because its interesting and counter-intuitive even if you don't use Zig. It's a story of problem solving.
kristoff_it 4y ago
> I thought the macro trend was moving onto memory-safe languages I guess some people like to Zig when others zag :^)

1vuio0pswjnm7 4y ago

Why not provide either hex editor or wasm as options. Let the user choose. The former is not trendy, it's time-tested and has no ties to a commercial entity or the online advertising "business". Whereas the later has only been around since 2015 and was introduced by a company that subsists off an agreement with a deviant online advertising company. Not to mention it targets "the web", which is only one use for computer programming, and one that is overwhelmingly under the control of a handful of large corporations.

nektro 4y ago
you might be interested in https://github.com/oriansj/stage0 https://github.com/oriansj/stage0 if you haven't seen it before
kristoff_it 4y ago
> Let the user choose. The main user of this bootstrapping process are core contributors, normal users are still supposed to download prebuilt executables from the official website. Distro maintainers also are not the target user of this bootstrapping process, since it involves a binary blob provided by us. The real users of this procedure are Zig contributors, so that they can trivially build latest zig always, and without the annoyance of having to keep a C++ version of the compiler in sync with the main one. That's it.
CharlesW 4y ago
> Whereas the later has only been around since 2015 and was created by a company that subsists off an agreement with a deviant online advertising company. Mozilla created a precursor technology, but I thought Wasm was developed via the W3C standards process from the start. From the notes of the first meeting, you can see attendees from Adobe, Apple, Arm, Autodesk, Google, Intel, Mozilla, Stanford, and more. https://github.com/WebAssembly/meetings/blob/main/main/2017/CG-05.md#attendees https://github.com/WebAssembly/meetings/blob/main/main/2017/... Additionally, Wasm has been a W3C standard since 2019.
projektfu 4y ago
https://en.wikipedia.org/wiki/MLX_(software) https://en.wikipedia.org/wiki/MLX_(software) Example of use: https://lparchive.org/Computes-Gazette/Update%2008/ https://lparchive.org/Computes-Gazette/Update%2008/

kalkin 4y ago

This is cool. I'm surprised that compiling a partial Zig backend to WASM and then compressing that ends up meaningfully smaller than compiling to C and compressing the C, when you include also the C partial WASM implementation and zstd decoder. This sounds kind of like a general strategy for compressing C code which I would not have expected to work well, but cool that it does! If AndyKelley ends up reading this - did you end up doing a direct comparison of "zig1.c.zstd + zstd.c" size vs the "zig1.wasm.zstd + zstd.c + wasm.c.zstd" set that you ended up with? If so, how did it turn out?

AndyKelley 4y ago
I ran the command locally just now: zig1-x86_64-linux.c: 86 MiB zig1-x86_64-linux.c.zstd: 3.5 MiB
kristoff_it 4y ago
I think it boils down to how bloated is the C code generated by the C backend, which to some degree has to be, since it's generated programmatically. My undestanding is what ends up happening is that the wasm step acts as a form of semantic compression that brings its own benefits over zstd (and which can still be combined with zstd by compressing the wasm file).
nektro 4y ago
this is likely due to LLVM being involved in the WASM generation and it being able to perform all of its optimization steps before outputting code. whereas Zig's C backend has not yet gained the ability to perform all the same optimizations.

iskander 4y ago

Naive question about Zig: is there any tooling for embedding it within larger Python codebases akin to Maturin (for Rust) or Nimporter (for Nim)? I have seen examples where the Zig code imports Python.h and uses low-level Python C API calls but I want something very lightweight for accelerating computational bottlenecks without worrying about unwrapping/wrapping data.

nurbl 4y ago
Not sure exactly what you need, but since Zig is C compatible, it's easy to build a zig library and import it from python using ctypes. I guess if you need something more sophisticated you could use cffi (haven't tried it). Zig is even available as a convenient python package: https://pypi.org/project/ziglang/ https://pypi.org/project/ziglang/
- iskander 4y ago
  I'm looking for something that fits into a setup.py file (or, like Maturin creates a multi-language package config) which (1) automatically compiles zig source into a Python extension module for me, so that I can (2) just import zig code into Python and call it without writing any type conversion logic.

deleted 4y ago

[deleted]

deepsun 4y ago

> There is exactly one VM target available to Zig that is both OS-agnostic and subject to LLVM’s state-of-the-art optimization passes, and that is WebAssembly. Honestly, sounds like old Java would also fit their requirements. There was a time when multiple languages ran for multi-platform, which is eased nowadays with containers and remote developer environments. So if their main concern is multi-platform, then feels like they want to look at the technologies developed at that time.

kristoff_it 4y ago
> Honestly, sounds like old Java would also fit their requirements. Yes, it would, as would any other VM target. That said WASM is extremely convenient because it's a target that LLVM supports and because writing a VM for it (or something that compiles it to C) is easy. Java from this perspective seems way less convenient, as it would require us first to build a Zig backend for it, and then we would have to implement our own java intrepreter / aot compiler / ... for it.
brundolf 4y ago
From my understanding, the JVM is much more opinionated than webassembly (it has built-in GC, and I've heard it even has a notion of classes and related concepts at the bytecode level). Particularly for a low-level C-like language like Zig, it seems like a pretty bad match.
- deepsun 4y ago
  We're talking about compiler, right? Sorry don't see why classes would be a bad match to compiling Zig code to LLVM representation. As long as we don't do lot of numbers crunching (where Fortran, R, Julia, MatLab shine), then structs/classes are ok.
  - brundolf 4y ago
    > We're talking about compiler, right? Yes, and Zig's compiler is now written in Zig Zig operates at a low level where it cares about things like manual memory-management. Compiling it to target the JVM instead of webassembly (assuming that's what you're suggesting) would be a really rough abstraction, because the JVM is higher-level. Webassembly is designed to accommodate lower-level languages adjacent to C that manage their own memory, etc And that's not even mentioning the fact that (it sounds like) Zig's compiler already has an LLVM back-end, which means they get wasm support "for free"

deleted 4y ago

[deleted]

shp0ngle 4y ago

> We provide a minimal WASI interpreter implementation that is built from C source, and then used to translate the Zig self-hosted compiler source code into C code. The C code is then compiled and linked, again by the system C compiler, into a stage2 binary. The stage2 binary can then be used repeatedly with zig build to build from source from that point on. Nope, no matter how many times I read this, I’m still lost. But then I never needed to care about VMs, compilers and bootstraps.

Arnavion 4y ago
Read the "To summarize:" section a little after the part you quoted.
gavinray 4y ago
Okay so we start from some C source code, and we build an interpreter for WASM + WASI from that (WASM that has access to system calls) C source -> WASM interpreter w/ system access Now we can take the Zig self-hosted compiler (the one in .zig), which has been compiled to .wasm/.wasi files. Since we have an interpreter for those now, we can do this: Zig compiler as .wasi instead of .exe --> WASM interpreter --> Zig's "translate-to-c" function, for the .zig file sources of the Zig compiler E.G. $ run-webassembly "zig-compiler-as-wasm.wasi" --translate-c <source code to zig compiler> At this point, we have the Zig compiler as .c files. Now you can use GCC/clang or whatnot, to build a regular binary for the compiler Output of Zig's "translate-to-c" from previous step --> GCC/clang --> Zig compiler but NOT AS WASM, as a regular binary
- MrBuddyCasino 4y ago
  how do you compile the Zig self-hosted compiler to wasm?
  - TUSF 4y ago
    Defen explained explains this well here: https://news.ycombinator.com/item?id=33914969 https://news.ycombinator.com/item?id=33914969
- 4y ago

randyrand 4y ago

> We provide a minimal WASI interpreter implementation that is built from C source, and then used to translate the Zig self-hosted compiler source code into C code. What do you use to compile the Zig source into C code? Wouldn't you need a Zig Compiler? I would have expected this? > We provide a minimal WASI interpreter implementation that is built from C source (i.e. so we don't need a Web Browser), then used to translate Zig Self-Hosted Compiler WASM code into C code. The Zig Self-Hosted Compiler WASM code is committed to the code base each time it changes, so when building a commit you already have the WASM source to a Zig compiler right there. > Of course, in the context of bootstrapping, this Zig Self-Hosted Compiler WASM source needed to be generated the first time at some point. For that first time, we used the C++ compiler to compile the Zig Self-Hosted Compiler from Zig into WASM.

Yujf 4y ago
The Zig compiler gets compiled to WASM. The WASI interpreter runs the WASM binary to compile Zig to C.
- randyrand 4y ago
  > The Zig compiler gets compiled to WASM This is the part I don't understand. In the context of bootstrapping, where does the Zig to WASM compiler come from?
  - Laremere 4y ago
    WASM is platform agnostic, so it is one of the things you start with, along with the compiler source code. It is built on a different computer before the bootstrapping process begins.
  - defen 4y ago
    This is a method of ensuring that future builds do not depend on the existence of a Zig compiler; it's not a way to go from 0 to Zig without a Zig compiler ever having existed. Technically this already existed in the form of a Zig compiler written in C++; the point of this exercise was to stop using C++. So: Presume the existence of a compiler that can compile Zig. Use that compiler to compile the written-in-Zig Zig compiler to WASM. Now you have a big chunk of WASM, so you also need a WASI interpreter. Write that in 4,000 lines of highly portable C. Then use that WASI interpreter to run your big chunk of WASM code and give it your written-in-Zig Zig compiler, and tell it to output C. Then compile that C code with your system compiler, and then use that native executable to recompile the written-in-Zig Zig compiler. At this point you should be at a fixed point and further recompilations of the Zig compiler will yield the same binary.

titzer 4y ago

This is great. The more self-contained (really self-hosted) a language is, the more implementation freedom and ability to evolve it gets. Virgil version I&II were a Virgil->C compiler written in Java. Later, I wrote an interpreter for Virgil III in Java and then began writing a compiler in Virgil III. When that compiler could compile version III (including itself), I checked in the first "stable" compiler as a jar. Then periodically when enough new features and bugs were fixed, I checked in a new stable binary (jar). Later, I developed and eventually fully switched to native backends for 32- and 64-bit x86 on MacOS and Darwin. Today, 5 stable binaries are checked in: jar, x86-darwin, x86-64-darwin, x86-linux, and x86-64-linux. There is also a Wasm backend, which can bootstrap the compiler too, but I did not check in a stable binary for it. Initially I was worried that a codegen bug would prevent bootstrapping from a compiler binary and that I'd need to fall back to running on an interpreter. So far, there's never been a codegen bug bad enough to break bootstrapping, so I am not worried about this. The compiler never needs to bootstrap from an interpreter.

funny_falcon 4y ago
Why did you abandon "translate-to-C" backend? It would be good to have high level language translated to C (aside from Vala and Nim).

ajnin 4y ago

> The idea here is to use a minimal wasm binary as a stage1 kernel that is committed to source control and therefore can be used to build any commit from source. We provide a minimal WASI interpreter implementation that is built from C source, and then used to translate the Zig self-hosted compiler source code into C code. The C code is then compiled and linked, again by the system C compiler, into a stage2 binary. The stage2 binary can then be used repeatedly with zig build to build from source from that point on. 1/ Wouldn't that be considered "cheating" to basically commit precompiled compiler binaries to source control ? 2/ I don't understand how that solves the "features need to be implemented twice" problem. Wouldn't you need to implement new Zig language features into that WASM kernel whenever they are used in the Zig compiler source ?

AndyKelley 4y ago
1. Yes it is cheating. That is the downside of this approach. Contributors to Zig and users of Zig don't care about such cheating, but distribution maintainers such as Debian Developers do (rightly) care. This decision is a tradeoff that favors contributors and users at the expense of system package maintainers. I am counting on a third party implementation of Zig to arise someday and solve the bootstrapping problem for system package maintainers. But in the short term, it's more important to prioritize the needs of users and contributors. 2. Whenever this happens, the contributor runs `zig build update-zig1` and commits the updated wasm kernel to the repository.
- Arnavion 4y ago
  Perhaps the distro devs could maintain their own golden WASM blobs that they compiled themselves and thus trust. Could be the same process as SecureBoot / package signing keys.
  - edwintorok 4y ago
    Can the stage1 wasm binary be reproduced by the stage2 or stage3 executable? Aside from a "trusting trust" type of attack that seems fine, and every modern distro relies on some bootstrap binary for C compilers anyway (usually older versions of them), so it wouldn't be that much different of a bootstrapping problem than bootstrapping GCC itself. (See the GNU Mes project which attempts to bootstrap from just a very small hex interpreter)

deleted 4y ago

[deleted]

u8 4y ago

If you’re interested in trying Zig out and want an easy way to update/use multiple versions I’ve been working on a Zig Version Manager for the past few weeks. It works on Windows, Mac, Linux, a smattering of BSD’s and Plan 9. Arm and x86. https://github.com/tristanisham/zvm https://github.com/tristanisham/zvm

ptato 4y ago
On Windows one can use https://scoop.sh https://scoop.sh too. There's a "zig" package for numbered releases, and a "zig-dev" package for nightly.
- sitkack 4y ago
  There is also `pip install ziglang`
Cloudef 4y ago
With nix you can use https://github.com/Cloudef/nix-zig-stdenv https://github.com/Cloudef/nix-zig-stdenv see the versions.nix

yyyk 4y ago

This is clever. There's one thing that I do not understand: Why the step 3 compiler has only the C backend enabled? In theory one could enable all the backends and skip to step 6? The step 5 comment says something about 'correct final logic', but I'm unsure what it means?

kristoff_it 4y ago
That's the Zig compiler that is implemented inside the wasm blob. Since we commit that blob to the repository, we want to keep is as small as possible, which is why it only contains the C backend and nothing else.

waynecochran 4y ago

Zig originally used the same strategy as the D compiler - not freeing memory until process exit wait ... what!? impractical in C/C++ because of language footguns C and C++ are now very different languages. You might as well say objective-c/swift

verdagon 4y ago
Not freeing memory is a fairly common approach, in compilers and command line tools as well. If an AST hangs around until the end of the program anyway, why not let the OS take care of blasting it away? free() is expensive after all.
- waynecochran 4y ago
  I am guessing LLVM does not do this.
  - JonChesterfield 4y ago
    Functions like Type::getInt32() or whatever it's called return the same heap allocated pointer each time so pointer equality can be used for value equality. That's a nice trick that only really works optimally if you leak the pointer, or at least don't free it until very near the end. I've seen cleaning up memory at the end of a program take 20% of the run time and that was indeed patched to just exit & leak as a result. With a flag to clean up so we could still run valgrind on it usefully.
  - sanxiyn 4y ago
    LLVM has JIT users, so LLVM can't do this.
    - Arnavion 4y ago
      Yes, any compiler-as-a-library software can't do this. And compilers-as-libraries are becoming more popular these days since being a library makes it easier to integrate the compiler with tooling, like linters and IDEs / language servers.

synergy20 4y ago

what's zig's advantage over nim? I have decided to stick with main stream languages after playing with various new languages in the past including ziglang, it's fun but in the end, more of a waste of time. in practice a language is really an ecosystem, from compiler, tools, editors, libraries, field testing...if you want to get things done, you just have to use the main stream ones.

tmtvl 4y ago
Indeed, we should all be using ALGOL, LISP, and COBOL.
- synergy20 4y ago
  In the early days when CS and languages are new things, we need evolve faster. When things kind of settling down, we need avoid NIH. Time is different. we don't reinvent new languages to compete against English, Chinese, Spanish,etc nowadays, I'm sure that's different in the early days when human was figuring out how to communicate and how to create the language they need.
GeorgeTirebiter 4y ago
This is my personal experience, ymmv, and maybe somebody needed coffee or something, but! I've found the zig community more friendly and open-minded than the Nim community. Again, one person, one experience, I like them both, yada yada. I suggest you have a look at both of them, and decide. The Nim book is very good. I would NOT call using zig or nim a waste of time; yes, the ecosystem matters. But decent languages + good libraries for what you need to do = Total Win. IHMO, this is why Python wins so bigly in the (increasing) influence it has. C wins, and will always win, I think, because it's the closest we have to a portable assembly language. We'll see, re: wasm. Maybe wasm will be the 'pdp-11' of computing for the 21st century.
ptato 4y ago
from only a glance and without having ever used nim, it seems to be more abstracted from the machine, whereas zig is closer. nim code should be shorter, nicer, closer to ideal pseudocode. safer too, i imagine. zig code should be more explicit, and if you do it right, more efficient in time and memory. zig also has a philosophy of, quoted from https://ziglang.org https://ziglang.org: "No hidden control flow. No hidden memory allocations. No preprocessor, no macros.". this also should make zig code more explicit, but probably more verbose too. i could also be completely wrong. like i said, i know nothing of nim other than what the homepage says. don't listen to me.

adrianmonk 4y ago

> It is then further optimized with wasm-opt -Oz --enable-bulk-memory bringing the total down to 2.4 MiB. Finally, it is compressed with zstd, bringing the total down to 637 KB. This is offset by the size of the zstd decoder implementation in C, however it is worth it because the zstd implementation will change rarely if ever, saving a total of 1.8 MiB every time the wasm binary is updated. Is the goal here to save space in the Git repo, by compressing before committing? I wouldn't assume using zstd is necessarily worth the complication. It could even make things worse. As I understand it, Git stores objects in packfiles[1], and these are both delta-fied and compressed with zlib. Your zstd reduces the 2.4MiB .wasm file to 637K. But Git's zlib should reduce 2.4MB to 800K (according to a quick test I just did). So at best, you only save 163K, not 1.8 MiB. But if Git's delta-fication works, you may actually use more space. Git should try to use its binary diff algorithm[2] to compare your various committed versions of zig1.wasm. If that algorithm is effective against Wasm files (my guess is yes), it will be able to store one version as a full copy and other versions as (somewhat? much?) smaller deltas against the full one. If you store .wasm.zst files, since compression tends to obscure commonalities, my guess is Git won't be able to do deltas and will have to store full copies of every version. On a side note, Git is said to be bad at handling binaries, and that's somewhat true, but there's some nuance. Binary files get in the way of easy branching and merging because Git can't merge them. So Git is bad at binary files in that way, but that's not relevant here. Also a lot of binary formats (like JPEG) are very much not amenable to binary diff, but my bet is that's not relevant here either. --- [1] See: https://git-scm.com/docs/git-pack-objects https://git-scm.com/docs/git-pack-objects https://git-scm.com/docs/pack-format https://git-scm.com/docs/pack-format https://git-scm.com/book/en/v2/Git-Internals-Packfiles https://git-scm.com/book/en/v2/Git-Internals-Packfiles [2] "inspired by" LibXDiff, according to https://github.com/git/git/blob/master/diff-delta.c https://github.com/git/git/blob/master/diff-delta.c

O_H_E 4y ago
I'd imagine distributing tarballs is also an important use case.
- adrianmonk 4y ago
  Actually zstd makes that worse too, somewhat paradoxically. At least in this case, because Zig uses xz for their tarballs. (If they used gzip, it would be the other way around.) The reason is that compression algorithms usually can't make further reductions when re-compressing already-compressed files. And xz has a higher compression ratio than zstd, so when you stick zig1.wasm.zst into a tar.xz file, xz is deprived of the opportunity to work its more powerful magic. As a test, I got zig-0.11.0-dev.638+5c67f9ce7.tar.xz from https://ziglang.org/download/ https://ziglang.org/download/ , extracted it, and rebuilt the tar.xz myself. Then I replaced stage1/zig1.wasm.zst with stage/zig1.wasm and rebuilt the tar.xz again. Results: $ du -sk *tar* 168136 zig.new.tar 14500 zig.new.tar.xz 166416 zig.orig.tar 14568 zig.orig.tar.xz So, zig.orig.tar is the uncompressed tarball that contains zig1.wasm.zst, and it is indeed smaller than zig.new.tar. But the .tar.xz files are the other way around. Not using zstd saves 68K. =-=-= Also, in the process, I accidentally discovered something else that makes a bigger difference. Since I knew the order of files within a tar archive can affect the compression ratio (due to data locality), while doing my test, I used "tar tf" to list my tar file's contents and compare it with what I downloaded. It didn't match, so I knew I wasn't doing an apples to apples comparison. So I added "--sort=name" to my tar commands. And both of my tar files ended up smaller than the one I downloaded: $ du -sk zig-0.11.0-dev.638+5c67f9ce7.tar.xz 15152 zig-0.11.0-dev.638+5c67f9ce7.tar.xz Just adding the "--sort=name" option to tar saves 584K! That's around 4% of the entire tar file. Locality matters more than I thought.

beltsazar 4y ago

> One big downside is losing the ability to build any commit from source without meta-complexity creeping in. For example, let’s say that you are trying to do git bisect. At some point, git checks out an older commit, but the script fails to build from source because the binary that is being used to build the compiler is now the wrong version. Sure, this can be addressed, but this introduces unwanted complexity that contributors would rather not deal with. If it's the main concern of using a prior build of the compiler, an alternative solution is to develop a tool for contributors to automate and ease the process. For example, Rust has this: https://github.com/rust-lang/cargo-bisect-rustc https://github.com/rust-lang/cargo-bisect-rustc

cryptonector 4y ago

> Now, there is this WebAssembly binary, which is not source code, but is in fact a build artifact. Some people, rightly, take these things very seriously [...]. Regarding this concern, well, you have to commit that build artifact because you're moving fast, but eventually you could do what the OpenJDK does: to build OpenJDK version N you need OpenJDK versions N-1 or N, and you can get OpenJDK version N-1 from your distro or from any number of places (like AdoptOpenJDK). You're essentially doing that now, but with unnamed versions -- you have to know which commits are like JDK version boundaries, and the clue is that the commit updates that one build artifact. TFA is a very good read.

teo_zero 4y ago

I'm surprised nobody has raised the trust argument... Who guarantees that the WASI blob has no hidden backdoor?

pabs3 4y ago

Thats an unfortunate development for the bootstrappability of Zig solely from source code. https://bootstrappable.org https://bootstrappable.org

sjmulder 4y ago

From the perspective of a package maintainer (I don't deal with the core infrastructure of our packaging system, I just package and patch things): While this unusual bootstrap with a WASM stage and a C WASI interpreter doesn't satisfy "everything from source" it's so much better than sitting on a non-Intel/ARM or non-Windows/Mac/Linux machine and having no other option but to maintain 5 different ancient versions of a compiler for a bootstrap sequence, or worse, being required to cross-compile from another host. Thanks :)

jokoon 4y ago

I like zig better than rust, but zig is still a bit too sophisticated for me.