5 ms·

> I don't know if this exists or not, but I'd like to try something like a fuse filesystem which can transparently copy a file to a fast scratch SSD when it is

by Mayzie 2y ago

> I don't know if this exists or not, but I'd like to try something like a fuse filesystem which can transparently copy a file to a fast scratch SSD when it is first accessed. You may be interested in checking out bcache[1] or bcachefs[2]. [1] https://www.kernel.org/doc/html/latest/admin-guide/bcache.html https://www.kernel.org/doc/html/latest/admin-guide/bcache.ht... [2] https://bcachefs.org/ https://bcachefs.org/

theblazehen 2y ago
lvm-cache works as well, if you're already using LVM. https://github.com/45Drives/autotier https://github.com/45Drives/autotier is exactly what they were asking for as well
- magicalhippo 2y ago
  I've done some testing with using ZFS on top of LVM with dm-writecache. Worked well enough on the small scale, but sadly haven't had the time or hardware to test it in a more production-like environment. Also, it was starting to feel a bit like a Jenga tower, increasing the chances of bugs and other weird issues to strike.
  - kaliszad 2y ago
    I wouldn't recommend combining those two. It's only begging for problems.
    - magicalhippo 2y ago
      Yeah that's my worry. Still, got 6x old 3TB disks that still work and a few spare NVMEs, so would be fun to try it for teh lulz.
      - kaliszad 2y ago
        Rather build a L2ARC or metadata special device hybrid setup using ZFS or skip ZFS and go for lvmcache/ mdadm style RAID with XFS or something.
        magicalhippo 2y ago
        But L2ARC only helps read speed. The idea with dm-writecache is to improve write speed. I started thinking about this when considering using a SAN for the disks, so that write speed was limited by the 10GbE network I had. A local NVMe could then absorb write bursts, maintaining performance. That said, it's not something I'd want to use in production that's for sure. There was some work being done on writeback caching for ZFS[1], sadly it seems to have remained closed-source. [1]: https://openzfs.org/wiki/Writeback_Cache https://openzfs.org/wiki/Writeback_Cache
        kaliszad 2y ago
        That's what SLOG is for if the writes are synchronous or if you have many small files/ want to optimize the metadata speed look at the metadata special device, which can also store small files of configurable size. ZFS of course has its limits too. But in my experience I feel much more confident (re)configuring it. You can tune the real world performance well enough especially if you can utilize some of the advanced features of ZFS like snapshots/ bookmarks + zfs-send/recv for backups. Because with LVM/ XFS you can certainly hack something together which will work pretty reliably too but with ZFS it's all integrated and well tested (because it is a common use case).
        magicalhippo 2y ago
        As I mentioned in my other[1] post, the SLOG isn't really a write-back cache. My workloads are mostly async, so SLOG wouldn't help unless I force sync=always which isn't great either. I love ZFS overall, it's been rock solid for me in the almost 15 years I've used it. This is just that one area where I feel could do with some improvements. [1]: https://news.ycombinator.com/item?id=41670945 https://news.ycombinator.com/item?id=41670945
LtdJorge 2y ago
Yeah, bcache is exactly that
hamandcheese 2y ago
My assumption with bcache is that it operates on blocks rather than entire files. Am I wrong?

by Mayzie 2y ago

theblazehen 2y ago

lvm-cache works as well, if you're already using LVM. https://github.com/45Drives/autotier https://github.com/45Drives/autotier is exactly what they were asking for as well

magicalhippo 2y ago
I've done some testing with using ZFS on top of LVM with dm-writecache. Worked well enough on the small scale, but sadly haven't had the time or hardware to test it in a more production-like environment. Also, it was starting to feel a bit like a Jenga tower, increasing the chances of bugs and other weird issues to strike.
- kaliszad 2y ago
  I wouldn't recommend combining those two. It's only begging for problems.
  - magicalhippo 2y ago
    Yeah that's my worry. Still, got 6x old 3TB disks that still work and a few spare NVMEs, so would be fun to try it for teh lulz.
    - kaliszad 2y ago
      Rather build a L2ARC or metadata special device hybrid setup using ZFS or skip ZFS and go for lvmcache/ mdadm style RAID with XFS or something.
      - magicalhippo 2y ago
        But L2ARC only helps read speed. The idea with dm-writecache is to improve write speed. I started thinking about this when considering using a SAN for the disks, so that write speed was limited by the 10GbE network I had. A local NVMe could then absorb write bursts, maintaining performance. That said, it's not something I'd want to use in production that's for sure. There was some work being done on writeback caching for ZFS[1], sadly it seems to have remained closed-source. [1]: https://openzfs.org/wiki/Writeback_Cache https://openzfs.org/wiki/Writeback_Cache
        kaliszad 2y ago
        That's what SLOG is for if the writes are synchronous or if you have many small files/ want to optimize the metadata speed look at the metadata special device, which can also store small files of configurable size. ZFS of course has its limits too. But in my experience I feel much more confident (re)configuring it. You can tune the real world performance well enough especially if you can utilize some of the advanced features of ZFS like snapshots/ bookmarks + zfs-send/recv for backups. Because with LVM/ XFS you can certainly hack something together which will work pretty reliably too but with ZFS it's all integrated and well tested (because it is a common use case).
        magicalhippo 2y ago
        As I mentioned in my other[1] post, the SLOG isn't really a write-back cache. My workloads are mostly async, so SLOG wouldn't help unless I force sync=always which isn't great either. I love ZFS overall, it's been rock solid for me in the almost 15 years I've used it. This is just that one area where I feel could do with some improvements. [1]: https://news.ycombinator.com/item?id=41670945 https://news.ycombinator.com/item?id=41670945

LtdJorge 2y ago

Yeah, bcache is exactly that

hamandcheese 2y ago

My assumption with bcache is that it operates on blocks rather than entire files. Am I wrong?