5 ms·

What are you doing for DB backups? Do you have a replica/standby? Or is it just hourly or something like that? Because with a single-server setup like this, I

What are you doing for DB backups? Do you have a replica/standby? Or is it just hourly or something like that? Because with a single-server setup like this, I'd imagine that hardware (e.g. SSD) failure brings down your app, and in the case of SSD failure, you then have hours or days downtime while you set everything up again.

kro 5mo ago
Hetzner normally advertises their hardware servers as 2x 1 TB SSD, because it's strongly recommended to run them in SWraid1 for net 1TB. (Their image installer will default to that) Once the first SSD fails after some years, and your monitoring catches that, you can either migrate to a new box, find another intermediate solution/replica, or let them hotswap it while the other drive takes on. Of course though, going to physical servers loses redundency of the cloud, but that's something you need to price in when looking at the savings and deciding your risk model. And yes, running this without also at least daily snapshotting/backup to remote storage is insane - that applies to cloud aswell, albeit easier to setup there.
- linsomniac 5mo ago
  For over a decade I ran a small scale dedicated and virtual hosting business (hundreds of machines) and the sort of setup you describe works very well. Software RAID across 2 devices, redundant power supplies, backups. We never had a significant data loss event that I recall (significant = beyond user accidentally removing files). For quite a while we ran single power supplies because they were pretty high quality, but then Supermicro went through a ~6 month period where basically every power supply in machines we got during that time failed within a year, and replacements were hard to come by (because of high demand, because of failures), and we switched to redundant. This was all cost savings trade-offs. When running single power supplies, we had in-rack Auto Transfer Switches, so that the single power supplies could survive A or B side power failure. But, and this is important, we were monitoring the systems for drive failures and replacing them within 24 hours. Ditto for power supplies. If you don't monitor your hardware for failure, redundancy doesn't mean anything.
  - iririririr 5mo ago
    [dead]
  - Dylan16807 5mo ago
    > But, and this is important, we were monitoring the systems for drive failures and replacing them within 24 hours. Ditto for power supplies. If you don't monitor your hardware for failure, redundancy doesn't mean anything. It does still mean something. If you have a 5% annual chance of failure and no redundancy, your five year failure chance is 23%. If you have redundancy and literally never check for five years, your five year failure chance is 5%. That's already a huge improvement. If you do an inventory of broken parts twice a year, still no proper monitoring, it goes down to 0.6% For 2% the numbers are: 10% 1% 0.1% For 10% the numbers are: 41% 17% 2.6% (The approximations for small percents are x*5, x²*25, and x²*2.5)
- faangguyindia 5mo ago
  Do not rely on raid alone. Have 2x servers atleast then invest in proper monitoring. Server can fail without disk failures.
traceroute66 5mo ago
> Because with a single-server setup like this, I'd imagine that hardware ... Yeah. This blog post reads like it was written by someone who didn't think things through and just focused on hyper-agressive cost-cutting. I bet their DigitalOcean vm did live migrations and supported snapshots. You can get that at Hetzner but only in their cloud product. You absolutely will not get that in Hetzner bare-metal. If your HD or other component dies, it dies. Hetzner will replace the HD, but its up to you to restore from scratch. Hetzner are very clear about this in multiple places.
- treesknees 5mo ago
  For the price, they could buy an exact replica bare metal server and still save money.
  - traceroute66 5mo ago
    > they could They could, but they didn't and instead they wrote that blog post which, even being generous is still kinda hard to avoid describing as misleading. I would not have written the post I did if they had presented a multi-node bare-metal cluster or whatever more realistic config.
    - locknitpicker 5mo ago
      > They could, but they didn't and instead they wrote that blog post which, even being generous is still kinda hard to avoid describing as misleading. What do you feel was misleading?
      - traceroute66 5mo ago
        > What do you feel was misleading? Erm. I already spelt it out in my original post ? I'm not going to re-write it, the TL;DR is they are making an Apples and Oranges comparison. Yes they "saved money" but in no way, shape or form are the two comparable. The polite way to put is is .... they saved as much money as they did because they made very heavy handed "architectural decisions". "Decisions" that they appear to be unaware of having made.
hnthrow0287345 5mo ago
It's possible no one will care much if it's down even for that long. I couldn't care less if my HOA mobile app was down even for a week for example. We don't need constant uptime for everything.
- wat10000 5mo ago
  I agree with the overall sentiment, but having an HOA app go down around the time when dues need to be paid could be a serious issue.
- acdha 5mo ago
  Don’t forget that integrity matters as much as availability in many applications. You might not mind if your HOA takes time to bring a server back up but you’d care a lot more if they lost the financial records or weren’t able to recover from a ransomware attack.
  - izacus 5mo ago
    Hetzner provides backups for VPS and machines across all tiers, which are very easy to set up.
kijin 5mo ago
If that's the tradeoff they're willing to make, who are you to say that they're doing it wrong? Not every app needs 24/7 availability. The vast majority of websites out there will not suffer any serious consequences from a few hours of downtime (scheduled or otherwise) every now and then. If the cost savings outweigh the risk, it can be a perfectly reasonable business decision. A more interesting question would be what kind of backup and recovery strategy they have, and which aspects of it (if any) they had to change when they moved to Hetzner.
faangguyindia 5mo ago
The easiest I’ve done is in MongoDB replication, sharding, failover, and all that is super easy. Recently, I did it in PostgreSQL using pg_auto_failover. I have 1 monitor node, 1 primary, and 1 replica. Surprisingly, once you get the hang of PostgreSQL configuration and its gotchas, it’s also very easy to replicate. I’m guessing MySQL is even easier than PostgreSQL for this. I also achieved zero downtime migration.
- acdha 5mo ago
  Replication is not a backup. It helps for migrations or clean single node failures but not human error, corruption, or an attack.

kro 5mo ago

Hetzner normally advertises their hardware servers as 2x 1 TB SSD, because it's strongly recommended to run them in SWraid1 for net 1TB. (Their image installer will default to that) Once the first SSD fails after some years, and your monitoring catches that, you can either migrate to a new box, find another intermediate solution/replica, or let them hotswap it while the other drive takes on. Of course though, going to physical servers loses redundency of the cloud, but that's something you need to price in when looking at the savings and deciding your risk model. And yes, running this without also at least daily snapshotting/backup to remote storage is insane - that applies to cloud aswell, albeit easier to setup there.

linsomniac 5mo ago
For over a decade I ran a small scale dedicated and virtual hosting business (hundreds of machines) and the sort of setup you describe works very well. Software RAID across 2 devices, redundant power supplies, backups. We never had a significant data loss event that I recall (significant = beyond user accidentally removing files). For quite a while we ran single power supplies because they were pretty high quality, but then Supermicro went through a ~6 month period where basically every power supply in machines we got during that time failed within a year, and replacements were hard to come by (because of high demand, because of failures), and we switched to redundant. This was all cost savings trade-offs. When running single power supplies, we had in-rack Auto Transfer Switches, so that the single power supplies could survive A or B side power failure. But, and this is important, we were monitoring the systems for drive failures and replacing them within 24 hours. Ditto for power supplies. If you don't monitor your hardware for failure, redundancy doesn't mean anything.
- iririririr 5mo ago
  [dead]
- Dylan16807 5mo ago
  > But, and this is important, we were monitoring the systems for drive failures and replacing them within 24 hours. Ditto for power supplies. If you don't monitor your hardware for failure, redundancy doesn't mean anything. It does still mean something. If you have a 5% annual chance of failure and no redundancy, your five year failure chance is 23%. If you have redundancy and literally never check for five years, your five year failure chance is 5%. That's already a huge improvement. If you do an inventory of broken parts twice a year, still no proper monitoring, it goes down to 0.6% For 2% the numbers are: 10% 1% 0.1% For 10% the numbers are: 41% 17% 2.6% (The approximations for small percents are x*5, x²*25, and x²*2.5)
faangguyindia 5mo ago
Do not rely on raid alone. Have 2x servers atleast then invest in proper monitoring. Server can fail without disk failures.

traceroute66 5mo ago

> Because with a single-server setup like this, I'd imagine that hardware ... Yeah. This blog post reads like it was written by someone who didn't think things through and just focused on hyper-agressive cost-cutting. I bet their DigitalOcean vm did live migrations and supported snapshots. You can get that at Hetzner but only in their cloud product. You absolutely will not get that in Hetzner bare-metal. If your HD or other component dies, it dies. Hetzner will replace the HD, but its up to you to restore from scratch. Hetzner are very clear about this in multiple places.

treesknees 5mo ago
For the price, they could buy an exact replica bare metal server and still save money.
- traceroute66 5mo ago
  > they could They could, but they didn't and instead they wrote that blog post which, even being generous is still kinda hard to avoid describing as misleading. I would not have written the post I did if they had presented a multi-node bare-metal cluster or whatever more realistic config.
  - locknitpicker 5mo ago
    > They could, but they didn't and instead they wrote that blog post which, even being generous is still kinda hard to avoid describing as misleading. What do you feel was misleading?
    - traceroute66 5mo ago
      > What do you feel was misleading? Erm. I already spelt it out in my original post ? I'm not going to re-write it, the TL;DR is they are making an Apples and Oranges comparison. Yes they "saved money" but in no way, shape or form are the two comparable. The polite way to put is is .... they saved as much money as they did because they made very heavy handed "architectural decisions". "Decisions" that they appear to be unaware of having made.

hnthrow0287345 5mo ago

It's possible no one will care much if it's down even for that long. I couldn't care less if my HOA mobile app was down even for a week for example. We don't need constant uptime for everything.

wat10000 5mo ago
I agree with the overall sentiment, but having an HOA app go down around the time when dues need to be paid could be a serious issue.
acdha 5mo ago
Don’t forget that integrity matters as much as availability in many applications. You might not mind if your HOA takes time to bring a server back up but you’d care a lot more if they lost the financial records or weren’t able to recover from a ransomware attack.
- izacus 5mo ago
  Hetzner provides backups for VPS and machines across all tiers, which are very easy to set up.

kijin 5mo ago

If that's the tradeoff they're willing to make, who are you to say that they're doing it wrong? Not every app needs 24/7 availability. The vast majority of websites out there will not suffer any serious consequences from a few hours of downtime (scheduled or otherwise) every now and then. If the cost savings outweigh the risk, it can be a perfectly reasonable business decision. A more interesting question would be what kind of backup and recovery strategy they have, and which aspects of it (if any) they had to change when they moved to Hetzner.

faangguyindia 5mo ago

The easiest I’ve done is in MongoDB replication, sharding, failover, and all that is super easy. Recently, I did it in PostgreSQL using pg_auto_failover. I have 1 monitor node, 1 primary, and 1 replica. Surprisingly, once you get the hang of PostgreSQL configuration and its gotchas, it’s also very easy to replicate. I’m guessing MySQL is even easier than PostgreSQL for this. I also achieved zero downtime migration.

acdha 5mo ago
Replication is not a backup. It helps for migrations or clean single node failures but not human error, corruption, or an attack.