6 ms·

Author here. I'm surprised to see my hobby project on Hacker News. I know this kind of stuff spark the ``it's meaningless to rewrite everything (especially Lin

by nuta 5y ago

Author here. I'm surprised to see my hobby project on Hacker News. I know this kind of stuff spark the ``it's meaningless to rewrite everything (especially Linux) in Rust'' debate. I agree 100% that rewriting Linux in Rust (or your favorite language) is just a waste of time and such a project won't live long. That said, what I'd say here is that it's fun. Really fun. Implementing ABI compatibility requires you to understand under the hood of the kernel. You can learn how printf(3) works. You can learn what happens before main(). Especially, in Rust, it's inspiring to explore ways to write a kernel safely and expressively. One of my favorite part is the implementation of read system call [1]. Lastly, You can try this OS on SSH. Hope you enjoy :) ssh root@kerla-demo.seiya.me [1]: https://github.com/nuta/kerla/blob/main/kernel/syscalls/read.rs https://github.com/nuta/kerla/blob/main/kernel/syscalls/read...

rakoo 5y ago
> I know this kind of stuff spark the ``it's meaningless to rewrite everything (especially Linux) in Rust'' debate. I agree 100% that rewriting Linux in Rust (or your favorite language) is just a waste of time and such a project won't live long. Considering that this is exactly how Linux was born (just a hobby project for fun), I wouldn't assume so fast that it's useless. Moreover, you don't need to justify yourself if you just want to have fun !
- uuddlrlr 5y ago
  Any rust project like this inevitably gets sardonic replies about rewrite-in-rust fanaticism. I agree they shouldn't have to justify themselves, but it is handy to preempt that.
  - Iv 5y ago
    Also, the rust community is too nice to openly admit it, but it's goal IS to displace C as the main low level language especially for the kernel. This is doomed to attract a lot of attention
- nuta 5y ago
  Thanks! I'll keep doing Just for Fun :D
- hnlmorg 5y ago
  I don't see why we need to pretend that every hobby has the potential for greatness. Being a hobby is a good enough end to itself. In fact in this instance I think it's a little disingenuous to quote Linux and say it could happen again. The industry is totally different now. There's much more competition than there was when Linux was released and that competition is much more mature too. Plus the bar for a production-quality kernel is a lot higher than it was when Linux was released.
  - _joel 5y ago
    > Being a hobby is a good enough end to itself. Indeed, there's a word for it which I find particularly lovely, autotelic.
  - elihu 5y ago
    > There's much more competition than there was when Linux was released and that competition is much more mature too. I think it's the opposite. When Linux was released, basically every major tech company had their own variant of Unix. Now almost everyone has migrated to Linux or Windows. Competing with modern Linux by creating a drop-in replacement for Linux would be a pretty difficult task given that people have been optimizing Linux for decades now, but on the other hand I think there are ways to improve dramatically on the traditional POSIX-style API that Linux mostly adheres to. For a random example, why can't a given process have more than one "current working directory" at a time? That seems like an arbitrary limitation imposed by an implementation detail in early Unix system, and it causes problems for modularity. There are many other little details like that. I think if Linux is replaced by something eventually, it'll most likely be because the new thing has a cleaner, more powerful, or more generally usable API. (Kerla is apparently not trying to invent a new API; I'm just saying that's the direction I would recommend to anyone project with a goal of seriously competing with Linux.)
    - elihu 5y ago
      Follow up to add: another way to compete with Linux is on security. The Linux kernel generally has a pretty good security record I think, but there have been plenty of serious bugs over the years. How many exploitable array-out-of-bounds errors or use-after-free errors remain in the Linux kernel? No one knows. If you can rule those out by using a safer language, that might be compelling to a lot of users who care about security above other concerns. Of course that's hard to pull off in practice. Linux might have classes of errors that wouldn't exist if it were written in Rust, but even if using Rust eliminates three fourths of the code defects, the end result could be less secure if Linux gets ten or a hundred times as much scrutiny from people actively looking through the code for bugs to fix.
0xTJ 5y ago
Wow, I really like the implementation of that syscall. I've got my own toy OS project in C, for the M68k, and my `read` is "a little" less clean [0, 1]. [0]: https://gitlab.com/0xTJ/mosys/-/blob/master/src/vfs.c#L469 https://gitlab.com/0xTJ/mosys/-/blob/master/src/vfs.c#L469 [1]: https://gitlab.com/0xTJ/mosys/-/blob/master/src/vfs.c#L907 https://gitlab.com/0xTJ/mosys/-/blob/master/src/vfs.c#L907
- nuta 5y ago
  Thanks! By the way, your MUTEX_WITH macro looks pretty interesting to me. I've never seen the idea.
qwerty456127 5y ago
Even "rewriting Linux in Rust" is far from a waste if you make it run and if your code quality is good enough for others to join. If only you could write an entirely new (meant to be better, for some cases at least) kernel compatible with Linux hardware drivers - this would be not waste at all but in fact fantastic.
- nuta 5y ago
  The most challenging point is, as others said, the lack of the compatibility with Linux Driver API. I believe it would be really hard to implement and keep following changes in Linux. An idea in my mind is to use Kerla for virtualized environments like KVM where only limited device drivers are needed (virtio for example).
  - phkahler 5y ago
    >> The most challenging point is, as others said, the lack of the compatibility with Linux Driver API. IMHO getting a minimal set that works is good enough to get people on board. This is still nontrivial. But for example, getting FUSE working would be useful. Even if the driver itself was not ABI compatible, it would bring functionality and someone might then aim for ABI compatibility afterward which would open even more doors.
  - yjftsjthsd-h 5y ago
    > The most challenging point is, as others said, the lack of the compatibility with Linux Driver API. I believe it would be really hard to implement and keep following changes in Linux. Especially since, as I understand it, even Linux isn't compatible with the Linux driver API across versions; they can and will change internals at will and just update in-tree drivers to match. Hence some of the difficulty doing things like getting a newer kernel on assorted embedded devices (ex. 99% of phones) because you have to port the vendor's drivers to the new version and both sides changed stuff.
baktubi 5y ago
Spending your time how you want to is never meaningless. Especially if you’re creating value in the world.
bitcoinmoney 5y ago
How long did it take you to write this?
- nuta 5y ago
  About 3-4 months in total: took 1.5 month to run a simple Hello World program, 1 month to implement bunch of system calls, and another 1 month to implement features essential to run Dropbear SSH server (tty/pry, bug fixes, …).
stjohnswarts 5y ago
It doesn't matter if it's meaningless to the hackernews crowd, it really only matters if it has meaning to you and if you're learning.
bpye 5y ago
I honestly think there is real value here for firecracker style containers. Running memory safe code for your whole stack, with a minimal number of virtio devices? Fantastic! The permissive license would also be interesting in some applications.
monopoledance 5y ago
Thanks for checking in. Makes me happy to see your excitement for learning the inner workings of things. What do you think about other Rust OS projects like Redox? https://www.redox-os.org/ https://www.redox-os.org/
- nuta 5y ago
  In addition to its huge contribution to OS development in Rust, as a microkernel enthusiast, it sounds exciting to writing a microkernel in Rust, in the "Everything is a URL" principle. Moreover, it can run a good-looking GUI on a real machines [1]! I know it's very hard to be realized. Aside from Redox, I'd mention Tock [2], an embedded operating system. It introduces a novel components isolation using Rust's power. I believe this field is where Rust shines and am looking forward to seeing it in production. [1]: https://www.redox-os.org/screens/ https://www.redox-os.org/screens/ [2]: https://www.tockos.org https://www.tockos.org
  - panick21_ 5y ago
    There are more good efforts, the BeTrusted guys are working on Xous, its a microkernel for a phone like device called the Precurser. https://github.com/betrusted-io/xous-core https://github.com/betrusted-io/xous-core As a embedded service processor OS for a big server rack, Oxide Computer is working on 'HubrisOS'. They seem to have not released it yet, but that will be open sourced. https://github.com/oxidecomputer https://github.com/oxidecomputer Those are two efforts where I know real resources are going into.
    - surajrmal 5y ago
      Is Hubris targeted at BMCs? Or is something different?
      - panick21_ 5y ago
        I think its an FPGA with a costume Open-Source Titan chip on it (RISC-V). It is not really a traditional BMC, its more like a service processor that does secure boot and gets you in the OS. It does a few other things but I think they really want it to have minimum functionality. This is a great talk about what they do and why: https://www.youtube.com/watch?v=vvZA9n3e5pc https://www.youtube.com/watch?v=vvZA9n3e5pc
nix23 5y ago
Really cool that you mentioned OSv on your github, really not enough eye's on it, i think.
deleted 5y ago
[deleted]
a-dub 5y ago
nice project! curious how the rust ownership model works in a monolithic kernel context. are the lifetimes of kernel structures associated with a process "owned" by the process itself somehow?
- nuta 5y ago
  I’m not sure this answers what you asked, objects that are referenced from multiple objects are simply wrapped with Arc<T>.
  - a-dub 5y ago
    sortof. i'm just kinda curious if any of rust's cool memory management features work particularly well or have to be bypassed in a monolithic kernel context.
jedimastert 5y ago
Aside from "rewrite in rust" debate that I was unaware of, there seems to be this pervasive attitude among the HN hivemind that the end-goal for all projects, side- or main-, is "launch", and therefore needs a market analysis to decide the "worth" of such an idea, and it ends up being rather silly. For example, I've written a Mandelbrot visualizer so many times I've lost count. Not because the world needs another poorly written or optimized rainbow-ladybug-simulator, but because it serves as a slightly-non-trivial hello-world. For example, it's the first end-to-end thing I made in Common Lisp. https://git.sr.ht/~amtunlimited/mandelbrot-plot https://git.sr.ht/~amtunlimited/mandelbrot-plot
- pjmlp 5y ago
  I have a A* search algorithm and a toy compiler that I use exactly for the same purpose. I just rewrite them all the time as means to get a feeling about programming languages. I dumped a couple of other stuff on GitHub so that HR people are happy to get a link that they never read anyway. Then I get back to Java and .NET at the office. :)
  - jedimastert 5y ago
    I've actually been considering adding a toy compiler to my collection of getting-up-to-speed projects, do you mind sharing what you think are good features?
    - pjmlp 5y ago
      Try to follow along the Tiger Book, in the variant that appeals to you. https://www.cs.princeton.edu/~appel/modern/ https://www.cs.princeton.edu/~appel/modern/ It covers most areas quite alright even if a bit aged.
    - spoiler 5y ago
      Not OP, but there's also "crafting interpreters". In the second half of the book you emit bytecode for whatever language you designed in the first half, and also implement a VM for said bytecode.
allochthon 5y ago
I'm new to Rust and am far less along than you are, as evidenced by this project. I noticed that in the few files I spot checked, there aren't many/any tests. Without any context or opinion, I'm curious whether unit and integration testing are hard for a project like this.
- nuta 5y ago
  Good point. As you say Kerla lacks unit tests I focused on running a SSH server as soon as possible. IMO, writing and running tests in the kernel space is pretty easy thanks to Rust’s flexible testing feature [1]. [1]: https://os.phil-opp.com/testing/ https://os.phil-opp.com/testing/
matheusmoreira 5y ago
> That said, what I'd say here is that it's fun. Really fun. Implementing ABI compatibility requires you to understand under the hood of the kernel. > You can learn how printf(3) works. You can learn what happens before main(). Yes! It's such a wonderful experience. This is exactly what I most enjoy doing, just seeing how things work, maybe making my own version. I hope you have lots of fun. You're reimplementing Linux's kernel-userspace binary interface, right? The system call interface is stable and language agnostic, it's really nice. Some pointers for anyone who'd like to know more: https://man7.org/linux/man-pages/man2/syscalls.2.html https://man7.org/linux/man-pages/man2/syscalls.2.html https://man7.org/linux/man-pages/man2/syscall.2.html https://man7.org/linux/man-pages/man2/syscall.2.html https://github.com/torvalds/linux/blob/master/Documentation/ABI/stable/syscalls https://github.com/torvalds/linux/blob/master/Documentation/...
sophacles 5y ago
This is really cool. Ignore the haters: * Some people bake bread even though there's a good bakery nearby. * Some people grow gardens even though there's a farmers' market down the street. * Some people restore old cars even though there's a good restoration professional in town. * Some people make k8s clusters on rpi even though they could rent that for cheap. You're making an OS that's linux compatible even though there's already linux. And that is awesome!
- teekert 5y ago
  Yes, and some people write an OS that is similar to Minix and won't ever be ported to anything beyond 386 and AT harddisks ;)
sandGorgon 5y ago
How did you set-up the ssh system? Where a new vm is spawned on ssh ?

by nuta 5y ago

rakoo 5y ago

> I know this kind of stuff spark the ``it's meaningless to rewrite everything (especially Linux) in Rust'' debate. I agree 100% that rewriting Linux in Rust (or your favorite language) is just a waste of time and such a project won't live long. Considering that this is exactly how Linux was born (just a hobby project for fun), I wouldn't assume so fast that it's useless. Moreover, you don't need to justify yourself if you just want to have fun !

uuddlrlr 5y ago
Any rust project like this inevitably gets sardonic replies about rewrite-in-rust fanaticism. I agree they shouldn't have to justify themselves, but it is handy to preempt that.
- Iv 5y ago
  Also, the rust community is too nice to openly admit it, but it's goal IS to displace C as the main low level language especially for the kernel. This is doomed to attract a lot of attention
nuta 5y ago
Thanks! I'll keep doing Just for Fun :D
hnlmorg 5y ago
I don't see why we need to pretend that every hobby has the potential for greatness. Being a hobby is a good enough end to itself. In fact in this instance I think it's a little disingenuous to quote Linux and say it could happen again. The industry is totally different now. There's much more competition than there was when Linux was released and that competition is much more mature too. Plus the bar for a production-quality kernel is a lot higher than it was when Linux was released.
- _joel 5y ago
  > Being a hobby is a good enough end to itself. Indeed, there's a word for it which I find particularly lovely, autotelic.
- elihu 5y ago
  > There's much more competition than there was when Linux was released and that competition is much more mature too. I think it's the opposite. When Linux was released, basically every major tech company had their own variant of Unix. Now almost everyone has migrated to Linux or Windows. Competing with modern Linux by creating a drop-in replacement for Linux would be a pretty difficult task given that people have been optimizing Linux for decades now, but on the other hand I think there are ways to improve dramatically on the traditional POSIX-style API that Linux mostly adheres to. For a random example, why can't a given process have more than one "current working directory" at a time? That seems like an arbitrary limitation imposed by an implementation detail in early Unix system, and it causes problems for modularity. There are many other little details like that. I think if Linux is replaced by something eventually, it'll most likely be because the new thing has a cleaner, more powerful, or more generally usable API. (Kerla is apparently not trying to invent a new API; I'm just saying that's the direction I would recommend to anyone project with a goal of seriously competing with Linux.)
  - elihu 5y ago
    Follow up to add: another way to compete with Linux is on security. The Linux kernel generally has a pretty good security record I think, but there have been plenty of serious bugs over the years. How many exploitable array-out-of-bounds errors or use-after-free errors remain in the Linux kernel? No one knows. If you can rule those out by using a safer language, that might be compelling to a lot of users who care about security above other concerns. Of course that's hard to pull off in practice. Linux might have classes of errors that wouldn't exist if it were written in Rust, but even if using Rust eliminates three fourths of the code defects, the end result could be less secure if Linux gets ten or a hundred times as much scrutiny from people actively looking through the code for bugs to fix.

0xTJ 5y ago

Wow, I really like the implementation of that syscall. I've got my own toy OS project in C, for the M68k, and my `read` is "a little" less clean [0, 1]. [0]: https://gitlab.com/0xTJ/mosys/-/blob/master/src/vfs.c#L469 https://gitlab.com/0xTJ/mosys/-/blob/master/src/vfs.c#L469 [1]: https://gitlab.com/0xTJ/mosys/-/blob/master/src/vfs.c#L907 https://gitlab.com/0xTJ/mosys/-/blob/master/src/vfs.c#L907

nuta 5y ago
Thanks! By the way, your MUTEX_WITH macro looks pretty interesting to me. I've never seen the idea.

qwerty456127 5y ago

Even "rewriting Linux in Rust" is far from a waste if you make it run and if your code quality is good enough for others to join. If only you could write an entirely new (meant to be better, for some cases at least) kernel compatible with Linux hardware drivers - this would be not waste at all but in fact fantastic.

nuta 5y ago
The most challenging point is, as others said, the lack of the compatibility with Linux Driver API. I believe it would be really hard to implement and keep following changes in Linux. An idea in my mind is to use Kerla for virtualized environments like KVM where only limited device drivers are needed (virtio for example).
- phkahler 5y ago
  >> The most challenging point is, as others said, the lack of the compatibility with Linux Driver API. IMHO getting a minimal set that works is good enough to get people on board. This is still nontrivial. But for example, getting FUSE working would be useful. Even if the driver itself was not ABI compatible, it would bring functionality and someone might then aim for ABI compatibility afterward which would open even more doors.
- yjftsjthsd-h 5y ago
  > The most challenging point is, as others said, the lack of the compatibility with Linux Driver API. I believe it would be really hard to implement and keep following changes in Linux. Especially since, as I understand it, even Linux isn't compatible with the Linux driver API across versions; they can and will change internals at will and just update in-tree drivers to match. Hence some of the difficulty doing things like getting a newer kernel on assorted embedded devices (ex. 99% of phones) because you have to port the vendor's drivers to the new version and both sides changed stuff.

baktubi 5y ago

Spending your time how you want to is never meaningless. Especially if you’re creating value in the world.

bitcoinmoney 5y ago

How long did it take you to write this?

nuta 5y ago
About 3-4 months in total: took 1.5 month to run a simple Hello World program, 1 month to implement bunch of system calls, and another 1 month to implement features essential to run Dropbear SSH server (tty/pry, bug fixes, …).

stjohnswarts 5y ago

It doesn't matter if it's meaningless to the hackernews crowd, it really only matters if it has meaning to you and if you're learning.

bpye 5y ago

I honestly think there is real value here for firecracker style containers. Running memory safe code for your whole stack, with a minimal number of virtio devices? Fantastic! The permissive license would also be interesting in some applications.

monopoledance 5y ago

Thanks for checking in. Makes me happy to see your excitement for learning the inner workings of things. What do you think about other Rust OS projects like Redox? https://www.redox-os.org/ https://www.redox-os.org/

nuta 5y ago
In addition to its huge contribution to OS development in Rust, as a microkernel enthusiast, it sounds exciting to writing a microkernel in Rust, in the "Everything is a URL" principle. Moreover, it can run a good-looking GUI on a real machines [1]! I know it's very hard to be realized. Aside from Redox, I'd mention Tock [2], an embedded operating system. It introduces a novel components isolation using Rust's power. I believe this field is where Rust shines and am looking forward to seeing it in production. [1]: https://www.redox-os.org/screens/ https://www.redox-os.org/screens/ [2]: https://www.tockos.org https://www.tockos.org
- panick21_ 5y ago
  There are more good efforts, the BeTrusted guys are working on Xous, its a microkernel for a phone like device called the Precurser. https://github.com/betrusted-io/xous-core https://github.com/betrusted-io/xous-core As a embedded service processor OS for a big server rack, Oxide Computer is working on 'HubrisOS'. They seem to have not released it yet, but that will be open sourced. https://github.com/oxidecomputer https://github.com/oxidecomputer Those are two efforts where I know real resources are going into.
  - surajrmal 5y ago
    Is Hubris targeted at BMCs? Or is something different?
    - panick21_ 5y ago
      I think its an FPGA with a costume Open-Source Titan chip on it (RISC-V). It is not really a traditional BMC, its more like a service processor that does secure boot and gets you in the OS. It does a few other things but I think they really want it to have minimum functionality. This is a great talk about what they do and why: https://www.youtube.com/watch?v=vvZA9n3e5pc https://www.youtube.com/watch?v=vvZA9n3e5pc

nix23 5y ago

Really cool that you mentioned OSv on your github, really not enough eye's on it, i think.

deleted 5y ago

[deleted]

a-dub 5y ago

nice project! curious how the rust ownership model works in a monolithic kernel context. are the lifetimes of kernel structures associated with a process "owned" by the process itself somehow?

nuta 5y ago
I’m not sure this answers what you asked, objects that are referenced from multiple objects are simply wrapped with Arc<T>.
- a-dub 5y ago
  sortof. i'm just kinda curious if any of rust's cool memory management features work particularly well or have to be bypassed in a monolithic kernel context.

jedimastert 5y ago

Aside from "rewrite in rust" debate that I was unaware of, there seems to be this pervasive attitude among the HN hivemind that the end-goal for all projects, side- or main-, is "launch", and therefore needs a market analysis to decide the "worth" of such an idea, and it ends up being rather silly. For example, I've written a Mandelbrot visualizer so many times I've lost count. Not because the world needs another poorly written or optimized rainbow-ladybug-simulator, but because it serves as a slightly-non-trivial hello-world. For example, it's the first end-to-end thing I made in Common Lisp. https://git.sr.ht/~amtunlimited/mandelbrot-plot https://git.sr.ht/~amtunlimited/mandelbrot-plot

pjmlp 5y ago
I have a A* search algorithm and a toy compiler that I use exactly for the same purpose. I just rewrite them all the time as means to get a feeling about programming languages. I dumped a couple of other stuff on GitHub so that HR people are happy to get a link that they never read anyway. Then I get back to Java and .NET at the office. :)
- jedimastert 5y ago
  I've actually been considering adding a toy compiler to my collection of getting-up-to-speed projects, do you mind sharing what you think are good features?
  - pjmlp 5y ago
    Try to follow along the Tiger Book, in the variant that appeals to you. https://www.cs.princeton.edu/~appel/modern/ https://www.cs.princeton.edu/~appel/modern/ It covers most areas quite alright even if a bit aged.
  - spoiler 5y ago
    Not OP, but there's also "crafting interpreters". In the second half of the book you emit bytecode for whatever language you designed in the first half, and also implement a VM for said bytecode.

allochthon 5y ago

I'm new to Rust and am far less along than you are, as evidenced by this project. I noticed that in the few files I spot checked, there aren't many/any tests. Without any context or opinion, I'm curious whether unit and integration testing are hard for a project like this.

nuta 5y ago
Good point. As you say Kerla lacks unit tests I focused on running a SSH server as soon as possible. IMO, writing and running tests in the kernel space is pretty easy thanks to Rust’s flexible testing feature [1]. [1]: https://os.phil-opp.com/testing/ https://os.phil-opp.com/testing/

matheusmoreira 5y ago

> That said, what I'd say here is that it's fun. Really fun. Implementing ABI compatibility requires you to understand under the hood of the kernel. > You can learn how printf(3) works. You can learn what happens before main(). Yes! It's such a wonderful experience. This is exactly what I most enjoy doing, just seeing how things work, maybe making my own version. I hope you have lots of fun. You're reimplementing Linux's kernel-userspace binary interface, right? The system call interface is stable and language agnostic, it's really nice. Some pointers for anyone who'd like to know more: https://man7.org/linux/man-pages/man2/syscalls.2.html https://man7.org/linux/man-pages/man2/syscalls.2.html https://man7.org/linux/man-pages/man2/syscall.2.html https://man7.org/linux/man-pages/man2/syscall.2.html https://github.com/torvalds/linux/blob/master/Documentation/ABI/stable/syscalls https://github.com/torvalds/linux/blob/master/Documentation/...

sophacles 5y ago

This is really cool. Ignore the haters: * Some people bake bread even though there's a good bakery nearby. * Some people grow gardens even though there's a farmers' market down the street. * Some people restore old cars even though there's a good restoration professional in town. * Some people make k8s clusters on rpi even though they could rent that for cheap. You're making an OS that's linux compatible even though there's already linux. And that is awesome!

teekert 5y ago
Yes, and some people write an OS that is similar to Minix and won't ever be ported to anything beyond 386 and AT harddisks ;)

sandGorgon 5y ago

How did you set-up the ssh system? Where a new vm is spawned on ssh ?