7 ms·

Before Bluestore, we ran Ceph on ZFS with the ZFS Intent Log on NVDIMM (basically non-volatile RAM backed by a battery). The performance was extremely good. Tod

Before Bluestore, we ran Ceph on ZFS with the ZFS Intent Log on NVDIMM (basically non-volatile RAM backed by a battery). The performance was extremely good. Today, we run Bluestore on ZVOLs on the same setup and if the zpool is a "hybrid" pool we put the Ceph OSD databases on an all-NVMe zpool. Ceph WAL wants a disk slice for each OSD, so we don't do Ceph WAL and consolidate incoming writes on the ZiL/SLOG on NVDIMM.

nightfly 1y ago
Why ceph on ZVOLs and not bare disks?
- acidmath 1y ago
  In the servers we have only 16gb to 64gb of NVDIMM, depending on density of NVDIMM and how many slots are populated with NVDIMM. Whatever raw NVDIMM is, usable is half because we mirror the contents for physical redundancy (if we lose a transaction it is fatal to our business). NVMe is amazing, but not everything should be NVMe, like petabyte scale object storage for example does not need to be on all NVMe (which is super pricey). In newer DDR5 servers where we can't get NVDIMM, the alternative battery backed RAM options leave us with even less to work with. Where we have counts of HDDs or SATA/SAS SSDs in the hundreds, we still want the performance improvements provided by WAL (or functional equivalent such as ZiL/SLOG) on NVDIMM and some layer-2 (where layer-1 is RAM) caching with NVMe. Ceph OSDs want a dedicated WAL device. Some places use OpenCAS to make "hybrid" devices out of HDDs by pairing them with SSDs where the SSDs can accelerate reads for that HDD and the Ceph OSD goes on a logical OpenCAS device. OpenCAS is really great, but the devices acting as "caching layer" often end up underutilized. By placing "big" Ceph OSDs on ZVOLs, we don't have individual disk slices for WAL (or equivalent) or individual disks for layer-2 read caching, but a consolidated layer in the form of ZFS Intent Log on "Separate Log" (NVDIMM) and another consolidated layer in the ZFS disk pool's L2ARC (layer-2 adaptive readback cache). The ZVOLs are striped across multiple relatively large RAIDz3 arrays. Yeah, it's "less efficient" in some ways, but the tradeoff is worth it for us. https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#devices https://open-cas.com/
  - __turbobrew__ 1y ago
    Do you have any recommendations or warnings about running ceph clusters?
    - Agingcoder 1y ago
      Find people who understand it. I’ve seen epic failures when things grow , you lose a DC and hell rains on you. It’s not magic , you will need people who get it ( source : unstable cluster of a few petabytes where I work ).
    - acidmath 1y ago
      Just off the top of my head: Run Ceph on https://rook.io/ https://rook.io/ ; don't bother with Cephadm. Running Rook provides very helpful guard rails. Put the logs for Ceph Rook into Elasticsearch+Kibana on its own small (three or four node) dedicated Ceph Rook cluster. Which Kubernetes distro this runs on matters more than anything. Recently we are looking at using https://www.parseable.com/ https://www.parseable.com/ instead of Elasticsearch+Kibana. And we had somewhat recently started moving things from Elaticsearch+Kibana to OpenSearch+OpenSearchDashboards due to the license change. The requirement outlined by Ceph documentation to dedicate layer-1 paths (can be same switches, but must be different ports) to Ceph replication is not about "performance" but about normal functionality. If you have any pointed questions feel free to email "section two thirty audit@mail2tor dot com" (where "two thirty" are the three digits rather than spelled out).
      - __turbobrew__ 1y ago
        I already set things up with Rook as we are super heavily invested into kubernetes, and things are working well so far. I built out a test cluster to 1PiB and was able to push more than a terrabit/second through the cluster which was good. I also set up topology aware replication so pg’s can be spread across racks/datacenters. My main worry now is disaster recovery. From what I have seen, object recovery is quite manual if you lose any. I would like to write some scripts so we can bulk mark objects which we know are actually lost. We already have a loki setup, so ceph logs just get put into there.

nightfly 1y ago

Why ceph on ZVOLs and not bare disks?

acidmath 1y ago
In the servers we have only 16gb to 64gb of NVDIMM, depending on density of NVDIMM and how many slots are populated with NVDIMM. Whatever raw NVDIMM is, usable is half because we mirror the contents for physical redundancy (if we lose a transaction it is fatal to our business). NVMe is amazing, but not everything should be NVMe, like petabyte scale object storage for example does not need to be on all NVMe (which is super pricey). In newer DDR5 servers where we can't get NVDIMM, the alternative battery backed RAM options leave us with even less to work with. Where we have counts of HDDs or SATA/SAS SSDs in the hundreds, we still want the performance improvements provided by WAL (or functional equivalent such as ZiL/SLOG) on NVDIMM and some layer-2 (where layer-1 is RAM) caching with NVMe. Ceph OSDs want a dedicated WAL device. Some places use OpenCAS to make "hybrid" devices out of HDDs by pairing them with SSDs where the SSDs can accelerate reads for that HDD and the Ceph OSD goes on a logical OpenCAS device. OpenCAS is really great, but the devices acting as "caching layer" often end up underutilized. By placing "big" Ceph OSDs on ZVOLs, we don't have individual disk slices for WAL (or equivalent) or individual disks for layer-2 read caching, but a consolidated layer in the form of ZFS Intent Log on "Separate Log" (NVDIMM) and another consolidated layer in the ZFS disk pool's L2ARC (layer-2 adaptive readback cache). The ZVOLs are striped across multiple relatively large RAIDz3 arrays. Yeah, it's "less efficient" in some ways, but the tradeoff is worth it for us. https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#devices https://open-cas.com/
- __turbobrew__ 1y ago
  Do you have any recommendations or warnings about running ceph clusters?
  - Agingcoder 1y ago
    Find people who understand it. I’ve seen epic failures when things grow , you lose a DC and hell rains on you. It’s not magic , you will need people who get it ( source : unstable cluster of a few petabytes where I work ).
  - acidmath 1y ago
    Just off the top of my head: Run Ceph on https://rook.io/ https://rook.io/ ; don't bother with Cephadm. Running Rook provides very helpful guard rails. Put the logs for Ceph Rook into Elasticsearch+Kibana on its own small (three or four node) dedicated Ceph Rook cluster. Which Kubernetes distro this runs on matters more than anything. Recently we are looking at using https://www.parseable.com/ https://www.parseable.com/ instead of Elasticsearch+Kibana. And we had somewhat recently started moving things from Elaticsearch+Kibana to OpenSearch+OpenSearchDashboards due to the license change. The requirement outlined by Ceph documentation to dedicate layer-1 paths (can be same switches, but must be different ports) to Ceph replication is not about "performance" but about normal functionality. If you have any pointed questions feel free to email "section two thirty audit@mail2tor dot com" (where "two thirty" are the three digits rather than spelled out).
    - __turbobrew__ 1y ago
      I already set things up with Rook as we are super heavily invested into kubernetes, and things are working well so far. I built out a test cluster to 1PiB and was able to push more than a terrabit/second through the cluster which was good. I also set up topology aware replication so pg’s can be spread across racks/datacenters. My main worry now is disaster recovery. From what I have seen, object recovery is quite manual if you lose any. I would like to write some scripts so we can bulk mark objects which we know are actually lost. We already have a loki setup, so ceph logs just get put into there.