5 ms·

We use Barman inside Kubernetes via CloudNativePG's plugin, as it is the default backup plugin. Barman has always been solid for backup and restore, however co

We use Barman inside Kubernetes via CloudNativePG's plugin, as it is the default backup plugin. Barman has always been solid for backup and restore, however configuring backup in CNPG is a little more interesting - WAL limits need to be set carefully or you just end up filling WAL volumes and the database becoming unavailable.

subhobroto 5mo ago
> We use Barman inside Kubernetes via CloudNativePG's plugin, as it is the default backup plugin. right and here's why CloudNativePG chose Barman over pgBackRest: https://github.com/cloudnative-pg/cloudnative-pg/issues/3077 https://github.com/cloudnative-pg/cloudnative-pg/issues/3077 > WAL limits need to be set carefully or you just end up filling WAL volumes and the database becoming unavailable. This is true. For anyone getting alarmed that this is due to a bug in PostgreSQL, it's not - it's PostgreSQL protecting the customer from attempting to write data that it cannot durable commit - "I am going to go unavailable because I don't have enough space to save more data". There are multiple ways to handle this, the easiest, most hands on way is to keep a monitor and alert that watches the WAL size like a hawk and then alerts OPS the moment it breaches a threshold.
- Tostino 5mo ago
  I thought that link was actually going to have a discussion on why they chose it. No such discussion exists.
  - gbartolini 5mo ago
    Barman Cloud was a convenient choice for CloudNativePG (CNPG) because it was developed by the same team that created Barman originally (I am part of both teams). When we started CNPG, we never anticipated it would become so popular, which has obviously resulted in some technical debt. The issue you mentioned concerns our decision not to integrate pgBackRest into CNPG's core, as we aimed to develop a pluggable interface (CNPG-I). As a community, we have decided to support volume snapshot backups and offer the Barman Cloud plugin to ensure we provide the same level of service. Our aim is to encourage other organisations or developers to create plugins for their preferred backup solutions. Currently, as maintainers of CNPG, we must concentrate on the core capabilities and allow the ecosystem to grow with both community and, potentially, commercial solutions based on CNPG-I.
    - Tostino 5mo ago
      I appreciate the response! Completely understand your motivations considering the lineage, and I appreciate that you guys put in the effort for the backup plug-in system. I think you guys made the right choices for that project. I appreciate it much more than the Crunchy k8s project which didn't have open images. I was definitely hoping that a pgbackrest plug-in would mature for cnpg, but I didn't realize there were troubles on the horizon for that project.
    - ninjaoxygen 4mo ago
      Great to see you posting on HN! I still remember making the decision to replace Crunchy Data PGO. CNPG was newer and smaller at the time, so maybe riskier, but seeing the way your team had responded to issues, had a real engineering mindset and clear knowledge of how PostgreSQL should be operated made it an easy choice! Thank you for your efforts, I hope the balance of commercial and community works out well for you, the product is great.
  - subhobroto 5mo ago
    The way I read that issue and the linked discussion was, that pgBackRest handles a lot of details itself that's otherwise handled by Kubernetes. Hence, a lot of functionality in pgBackRest is not only redundant but incompatible with how Kubernetes CSI could be used to provide incremental and differential backups. Hence, Barman and `barman-cloud` plugins are a better, natural fit for a Kubernetes environment than pgBackRest.
Synthetic7346 5mo ago
Can the plugin do non objectstore backups? E.g. If I don't want to use S3 / minio / whatever blobstore for my homelab
- mkroman 5mo ago
  The plugin is specifically for handling backups via object storage. CloudNativePG also implements support for volume snapshots: https://cloudnative-pg.io/docs/1.29/backup#backup-methods https://cloudnative-pg.io/docs/1.29/backup#backup-methods

subhobroto 5mo ago

> We use Barman inside Kubernetes via CloudNativePG's plugin, as it is the default backup plugin. right and here's why CloudNativePG chose Barman over pgBackRest: https://github.com/cloudnative-pg/cloudnative-pg/issues/3077 https://github.com/cloudnative-pg/cloudnative-pg/issues/3077 > WAL limits need to be set carefully or you just end up filling WAL volumes and the database becoming unavailable. This is true. For anyone getting alarmed that this is due to a bug in PostgreSQL, it's not - it's PostgreSQL protecting the customer from attempting to write data that it cannot durable commit - "I am going to go unavailable because I don't have enough space to save more data". There are multiple ways to handle this, the easiest, most hands on way is to keep a monitor and alert that watches the WAL size like a hawk and then alerts OPS the moment it breaches a threshold.

Tostino 5mo ago
I thought that link was actually going to have a discussion on why they chose it. No such discussion exists.
- gbartolini 5mo ago
  Barman Cloud was a convenient choice for CloudNativePG (CNPG) because it was developed by the same team that created Barman originally (I am part of both teams). When we started CNPG, we never anticipated it would become so popular, which has obviously resulted in some technical debt. The issue you mentioned concerns our decision not to integrate pgBackRest into CNPG's core, as we aimed to develop a pluggable interface (CNPG-I). As a community, we have decided to support volume snapshot backups and offer the Barman Cloud plugin to ensure we provide the same level of service. Our aim is to encourage other organisations or developers to create plugins for their preferred backup solutions. Currently, as maintainers of CNPG, we must concentrate on the core capabilities and allow the ecosystem to grow with both community and, potentially, commercial solutions based on CNPG-I.
  - Tostino 5mo ago
    I appreciate the response! Completely understand your motivations considering the lineage, and I appreciate that you guys put in the effort for the backup plug-in system. I think you guys made the right choices for that project. I appreciate it much more than the Crunchy k8s project which didn't have open images. I was definitely hoping that a pgbackrest plug-in would mature for cnpg, but I didn't realize there were troubles on the horizon for that project.
  - ninjaoxygen 4mo ago
    Great to see you posting on HN! I still remember making the decision to replace Crunchy Data PGO. CNPG was newer and smaller at the time, so maybe riskier, but seeing the way your team had responded to issues, had a real engineering mindset and clear knowledge of how PostgreSQL should be operated made it an easy choice! Thank you for your efforts, I hope the balance of commercial and community works out well for you, the product is great.
- subhobroto 5mo ago
  The way I read that issue and the linked discussion was, that pgBackRest handles a lot of details itself that's otherwise handled by Kubernetes. Hence, a lot of functionality in pgBackRest is not only redundant but incompatible with how Kubernetes CSI could be used to provide incremental and differential backups. Hence, Barman and `barman-cloud` plugins are a better, natural fit for a Kubernetes environment than pgBackRest.

Synthetic7346 5mo ago

Can the plugin do non objectstore backups? E.g. If I don't want to use S3 / minio / whatever blobstore for my homelab

mkroman 5mo ago
The plugin is specifically for handling backups via object storage. CloudNativePG also implements support for volume snapshots: https://cloudnative-pg.io/docs/1.29/backup#backup-methods https://cloudnative-pg.io/docs/1.29/backup#backup-methods