6 ms·

Ask HN: what workload is not suitable for microVMs? Can I use them like a regular VM?

tsujp 4y ago
These specific microVMs are managed by: https://github.com/containers/libkrun#goals-and-non-goals https://github.com/containers/libkrun#goals-and-non-goals (linked directly to project scopes). In summary though (others redacted): # Goals - Be compatible with a reasonable amount of workloads. # Non-goals - Be compatible with all kinds of workloads.
0xbadcafebee 4y ago
Since it's a VM, it's ideal for workloads with a set amount of resource use and that need strong isolation guarantees. Regular containers are better to share a pool of resources whose usage varies widely, and when you don't need strong isolation guarantees. Depending on how I/O is handled, container I/O can be very slow, whereas a dedicated disk snapshot without CoW/overlays would be much faster. Since this also uses TSI for networking, you will need a patched Linux kernel to use networking in the guest at all, and raw sockets don't work at all.
- staticassertion 4y ago
  > Depending on how I/O is handled, container I/O can be very slow, whereas a dedicated disk snapshot without CoW/overlays would be much faster. Do you mean VM I/O can be very slow? I don't think containers should have any overhead, please correct me if I'm wrong though.
  - 0xbadcafebee 4y ago
    Container file I/O is very slow. It unpacks the OCI image layers onto the regular host filesystem, then adds overlay filesystems, does copy-on-write, and references files between each layer. For example, doing 10 containerized nodejs app builds simultaneously will swamp the host with iowait. A common hack to is to put the OCI file tree / overlays on a dedicated disk with much higher iops than the boot disk.
    - staticassertion 4y ago
      Thank you, I'll have to look into this. I was thinking from a file namespacing perspective there shouldn't be overhead, but it makes sense that adding the overlay filesystems and mounts would impact performance.
    - rascul 4y ago
      > It unpacks the OCI image layers onto the regular host filesystem, then adds overlay filesystems, does copy-on-write, and references files between each layer. That's just Docker though, right? Does LXC or systemd-nspawn do that?
staticassertion 4y ago
That depends on the microvm. Device support in Firecracker, like GPUs, doesn't exist, which also makes Firecracker suitable for multitenant workloads. Something like QEMU has far more device support but is also significantly easier to escape out of.

These specific microVMs are managed by: https://github.com/containers/libkrun#goals-and-non-goals https://github.com/containers/libkrun#goals-and-non-goals (linked directly to project scopes). In summary though (others redacted): # Goals - Be compatible with a reasonable amount of workloads. # Non-goals - Be compatible with all kinds of workloads.

0xbadcafebee 4y ago

Since it's a VM, it's ideal for workloads with a set amount of resource use and that need strong isolation guarantees. Regular containers are better to share a pool of resources whose usage varies widely, and when you don't need strong isolation guarantees. Depending on how I/O is handled, container I/O can be very slow, whereas a dedicated disk snapshot without CoW/overlays would be much faster. Since this also uses TSI for networking, you will need a patched Linux kernel to use networking in the guest at all, and raw sockets don't work at all.

staticassertion 4y ago
> Depending on how I/O is handled, container I/O can be very slow, whereas a dedicated disk snapshot without CoW/overlays would be much faster. Do you mean VM I/O can be very slow? I don't think containers should have any overhead, please correct me if I'm wrong though.
- 0xbadcafebee 4y ago
  Container file I/O is very slow. It unpacks the OCI image layers onto the regular host filesystem, then adds overlay filesystems, does copy-on-write, and references files between each layer. For example, doing 10 containerized nodejs app builds simultaneously will swamp the host with iowait. A common hack to is to put the OCI file tree / overlays on a dedicated disk with much higher iops than the boot disk.
  - staticassertion 4y ago
    Thank you, I'll have to look into this. I was thinking from a file namespacing perspective there shouldn't be overhead, but it makes sense that adding the overlay filesystems and mounts would impact performance.
  - rascul 4y ago
    > It unpacks the OCI image layers onto the regular host filesystem, then adds overlay filesystems, does copy-on-write, and references files between each layer. That's just Docker though, right? Does LXC or systemd-nspawn do that?

staticassertion 4y ago

That depends on the microvm. Device support in Firecracker, like GPUs, doesn't exist, which also makes Firecracker suitable for multitenant workloads. Something like QEMU has far more device support but is also significantly easier to escape out of.