7 ms·

A NetBSD/amd64 guest can now boot in 18ms

112 points by bleusink 3y ago

pantantrant 3y ago
What's the firecracker command for this? (and also for Linux)
- datadeft 3y ago
  I am assuming your asking about how to boot up the image. You can poke around here: https://dev.l1x.be/posts/2020/12/13/diving-into-firecracker-with-alpine/#starting-firecracker https://dev.l1x.be/posts/2020/12/13/diving-into-firecracker-... If you are asking for the NetBSD image I am not sure.
- doakes 3y ago
  I like this resource for starting a VM from a container image: https://github.com/alexellis/firecracker-init-lab https://github.com/alexellis/firecracker-init-lab
alberth 3y ago
How does this compare to other BSDs? Wasn't there recently a lot of work to reduce the time on FreeBSD (and it's 20ms) https://www.usenix.org/publications/loginonline/freebsd-firecracker#:~:text=FreeBSD%20boots%20under%20Firecracker%20—%20and,process%20appears%20in%20Figure%201 https://www.usenix.org/publications/loginonline/freebsd-fire....
- jedberg 3y ago
  This work is based on the FreeBSD work.
- dchest 3y ago
  "I think the fastest I got the FreeBSD kernel booting in Firecracker was 21 ms. NetBSD is now at 18 ms... I need to go back and address some more of the issues I noticed but didn't get around to fixing. Anyone know what the current record for Linux is? Last I heard was ~50 ms." - @cperciva at https://twitter.com/cperciva/status/1747270461095043532 https://twitter.com/cperciva/status/1747270461095043532
  - alberth 3y ago
    > “I need to go back and address some more of the issues I noticed but didn't get around to fixing.” I wonder if this is related to what Netflix found as a regression. Starting at slide #18 below https://people.freebsd.org/~gallatin/talks/OpenFest2023.pdf https://people.freebsd.org/~gallatin/talks/OpenFest2023.pdf
    - cperciva 3y ago
      The bug Netflix tripped over was something I introduced while shaving off milliseconds, yes. The "other issues I didn't get around to fixing" are things like precomputing lookup tables (we can wait and do them on demand, or not at all if they turn out to never get used) and an O(n^2) issue registering names of sysctls.
myself248 3y ago
[in mice]
- cperciva 3y ago
  With 1 CPU and 128 MB of RAM. Sure, it's a small system, but it's enough to be useful for some purposes -- more useful than medical discoveries in mice, at least.
  - myself248 3y ago
    And in a VM, not bare metal. "Boot" isn't even a sensible concept in a VM, IMHO, you could just toss an image into RAM. There's no hardware to initialize, the caches are probably already warm, etc.
    - cperciva 3y ago
      You do need to shake hands with the virtio devices; that's a bit easier than most hardware, but it's not trivial. The "take a snapshot after you finish booting and resume that" approach can work (Lambda does it) but it only works if you have the same type and number of CPUs, the same amount of RAM, the same filesystem on disk, and even the same MAC addresses on your network interfaces. So it's not like FreeBSD can ship useful "pre-booted" images.
csdvrx 3y ago
That's nice, personally it takes me about 400 ms to start a linux kernel with qemu (more time is spent by qemu initializing its network interfaces that starting the kernel and the init!) I'll see which tricks can be reused on linux, as I'd love to cut that by 1 order of magnitude!
lateralux 3y ago
Very interesting
wang_li 3y ago
I'm curious as to why this isn't 0ms for a VM. The entire state of the system can be known ahead of time. Why does the kernel need to do any kind of dynamic initialization? Why aren't all data structures and variables statically assigned to proper values for the given VM. So at VM start up time it simply enters the main dispatch loop.
- tristor 3y ago
  Mostly because that's not how the kernel is designed. While what you're saying is true, it would require a fundamentally different kernel design. The kernel as it is today is designed to work on a variety of devices, not just VMs, so it behaves in a way in which it doesn't have specific knowledge of the device prior to initialization. To do what you're suggesting, the kernel would need to have a special mode where it supports injection of this information from the VM host in advance of initialization, and there's probably a bunch of possible security issues involved as well.
  - anthk 3y ago
    NetBSD can just trim the kernel down to Virtio support and not much more.
  - deleted 3y ago
    [deleted]
  - alexey-salmin 3y ago
    You don't need kernel support, just freeze it's state right after the boot and clone it afterwards. Boot time will be the setup of shadow page tables and other hypervisor structures at new offsets. Plus a few tricks to avoid reproducible RNGs.
    - inkyoto 3y ago
      > […] freeze it's state right after the boot and clone it afterwards. You can't do that easily as devices can be attached, detached or reattached dynamically whilst the VM is running and when it is shut down. In lieu of the hardware / VM support for device trees, with the existing design, the kernel has to probe each device upon every boot.

112 points by bleusink 3y ago

pantantrant 3y ago

What's the firecracker command for this? (and also for Linux)

datadeft 3y ago
I am assuming your asking about how to boot up the image. You can poke around here: https://dev.l1x.be/posts/2020/12/13/diving-into-firecracker-with-alpine/#starting-firecracker https://dev.l1x.be/posts/2020/12/13/diving-into-firecracker-... If you are asking for the NetBSD image I am not sure.
doakes 3y ago
I like this resource for starting a VM from a container image: https://github.com/alexellis/firecracker-init-lab https://github.com/alexellis/firecracker-init-lab

alberth 3y ago

How does this compare to other BSDs? Wasn't there recently a lot of work to reduce the time on FreeBSD (and it's 20ms) https://www.usenix.org/publications/loginonline/freebsd-firecracker#:~:text=FreeBSD%20boots%20under%20Firecracker%20—%20and,process%20appears%20in%20Figure%201 https://www.usenix.org/publications/loginonline/freebsd-fire....

jedberg 3y ago
This work is based on the FreeBSD work.
dchest 3y ago
"I think the fastest I got the FreeBSD kernel booting in Firecracker was 21 ms. NetBSD is now at 18 ms... I need to go back and address some more of the issues I noticed but didn't get around to fixing. Anyone know what the current record for Linux is? Last I heard was ~50 ms." - @cperciva at https://twitter.com/cperciva/status/1747270461095043532 https://twitter.com/cperciva/status/1747270461095043532
- alberth 3y ago
  > “I need to go back and address some more of the issues I noticed but didn't get around to fixing.” I wonder if this is related to what Netflix found as a regression. Starting at slide #18 below https://people.freebsd.org/~gallatin/talks/OpenFest2023.pdf https://people.freebsd.org/~gallatin/talks/OpenFest2023.pdf
  - cperciva 3y ago
    The bug Netflix tripped over was something I introduced while shaving off milliseconds, yes. The "other issues I didn't get around to fixing" are things like precomputing lookup tables (we can wait and do them on demand, or not at all if they turn out to never get used) and an O(n^2) issue registering names of sysctls.

myself248 3y ago

[in mice]

cperciva 3y ago
With 1 CPU and 128 MB of RAM. Sure, it's a small system, but it's enough to be useful for some purposes -- more useful than medical discoveries in mice, at least.
- myself248 3y ago
  And in a VM, not bare metal. "Boot" isn't even a sensible concept in a VM, IMHO, you could just toss an image into RAM. There's no hardware to initialize, the caches are probably already warm, etc.
  - cperciva 3y ago
    You do need to shake hands with the virtio devices; that's a bit easier than most hardware, but it's not trivial. The "take a snapshot after you finish booting and resume that" approach can work (Lambda does it) but it only works if you have the same type and number of CPUs, the same amount of RAM, the same filesystem on disk, and even the same MAC addresses on your network interfaces. So it's not like FreeBSD can ship useful "pre-booted" images.

csdvrx 3y ago

That's nice, personally it takes me about 400 ms to start a linux kernel with qemu (more time is spent by qemu initializing its network interfaces that starting the kernel and the init!) I'll see which tricks can be reused on linux, as I'd love to cut that by 1 order of magnitude!

lateralux 3y ago

Very interesting

wang_li 3y ago

I'm curious as to why this isn't 0ms for a VM. The entire state of the system can be known ahead of time. Why does the kernel need to do any kind of dynamic initialization? Why aren't all data structures and variables statically assigned to proper values for the given VM. So at VM start up time it simply enters the main dispatch loop.

tristor 3y ago
Mostly because that's not how the kernel is designed. While what you're saying is true, it would require a fundamentally different kernel design. The kernel as it is today is designed to work on a variety of devices, not just VMs, so it behaves in a way in which it doesn't have specific knowledge of the device prior to initialization. To do what you're suggesting, the kernel would need to have a special mode where it supports injection of this information from the VM host in advance of initialization, and there's probably a bunch of possible security issues involved as well.
- anthk 3y ago
  NetBSD can just trim the kernel down to Virtio support and not much more.
- deleted 3y ago
  [deleted]
- alexey-salmin 3y ago
  You don't need kernel support, just freeze it's state right after the boot and clone it afterwards. Boot time will be the setup of shadow page tables and other hypervisor structures at new offsets. Plus a few tricks to avoid reproducible RNGs.
  - inkyoto 3y ago
    > […] freeze it's state right after the boot and clone it afterwards. You can't do that easily as devices can be attached, detached or reattached dynamically whilst the VM is running and when it is shut down. In lieu of the hardware / VM support for device trees, with the existing design, the kernel has to probe each device upon every boot.