10 ms·

Interesting to see this article again since it was the trigger of starting a homelab for me. After realizing cloud services are putting a major dent in my pocke

Interesting to see this article again since it was the trigger of starting a homelab for me. After realizing cloud services are putting a major dent in my pocket to get a lousy startup idea off the ground, I started to wonder if there's "The Cheapest" way? (I'm not cheap but I'm very frugal) Nowadays internet speed is great to do self hosting. I have a business line internet at home with ~1gb up&down! Bought couple of 6-7 year old enterprise Dell servers (2x12core xeon, 128gb ram each) and no longer pay any cloud provider ... i'm also hosting 2 backend solutions for mobile apps with decent traffic for friends' startups! The learning experience has been tremendous! It has actually gotten a lot better and easier with new solutions coming out for homelabs. Get started with Proxmox clusters and go from there...

runlaszlorun 3y ago
was the business line much more expensive than home rates? and were you given a static ipv4 IP?
- sokoloff 3y ago
  Not GP, but related: I've had the same DHCP IP from Xfinity (residential internet service) for well over a decade (and I think for 16 years). At what point does a dynamic-but-unchanging IP become functionally static?
  - kccqzy 3y ago
    In my case Xfinity kept the same IP for me for two years, then an outage happened where everyone in the neighborhood lost connectivity. When connectivity was restored I got a different IP. I feel like the biggest difference is the fact that there's no guarantee that the dynamic IP won't change, so all systems need to be prepared for that, or you need to be mentally prepared for that day.
- bamazizi 3y ago
  I am with Bell in canada. My business fiber internet is $100/m and I also pay $20/b for dedicated IP
seydor 3y ago
this is the most undersung hero. Internet speeds are reaching the equivalent of a datacenter. This will open up so many possibities, that it makes it seem like the 2000s again. And finally we might actually need ipv6
- klabb3 3y ago
  I just moved out of the consumer internet dystopia that is California (or most of US I guess) to Spain and am just about to get 10 Gbit symmetric for €25/mo. Even if I get half of that I’m ecstatic. This kind of infrastructure is so conducive to all kinds of interesting and “decentralized” innovations.
  - dzonga 3y ago
    I wish I could get that in london :(. uk internet infra is backwards.
    - graemep 3y ago
      Bad regulation. Ofcom is allowing Openreach to keep asymmetric pricing for FTTP. Unjustified market segmentation. The govt has a focus on download speeds (which are useful for high quality video) and does not care about upload or latency which filters down. On reflection, even the the terms "upload" and "download" are based on the assumption connections are for consuming media.
  - lionkor 3y ago
    In Germany you pay 60+€/month for 100mbit down.
- dekhn 3y ago
  datacenter network speeds are ~400+Gbit (between data centers) and in the Tbit range within data centers. A typical server now has a 10Gbit NIC.
- bamazizi 3y ago
  Yes indeed. My Bell.ca ~1gb fiber has a monthly cost of $100 + $20 for dedicated IP. Since it's business line fiber internet it comes with monitored service quality meaning it's prioritized over others using regular/residential fiber internet (claimed!) BTW, I can get multiple lines if i'd ever need it
p1necone 3y ago
Yeah the delta in cost/CPU/memory/storage between self hosting on second hand stuff and paying cloud services is insane. It's a no brainer if your use case needs beefier hardware than the typical $5/month VPS host and you don't have enterprise level liability/uptime expectations. With stuff like proxmox you get a pretty similar level of ease of use to managed VM services too.
- Karrot_Kream 3y ago
  I generally find these discussions unproductive on HN because of how binary they become, but definitely once you go off the beaten path it seems a lot cheaper (sans power though) to run your own infra. I've been thinking of doing some data ETL for something that generates 1 GB / day and has retention for 30 days and it's much cheaper to host a Postgres myself at that scale than run it in the cloud. I could store in something like S3, but then I need to deal with querying S3. I'd like to combine some of this with cloud infra but I suspect cloud bandwidth costs would kill me. Colo-ing 2-3 servers is probably the best bet here.
  - calvinmorrison 3y ago
    I was an ops engineer at Fastmail. We ran our own hardware. A mix of Debian on one stack and SmartOS (Illumos from Joyent) and there were plenty of physical problems and costs. Now putting petabytes of data with replication and syncing and all that would have been a cluster F on the cloud, we missed out on a lot of the awesome newer deployment other tooling because we had written our own. Before I left we swapped out a good chunk of these snowflake software but it was impressive actually how well it worked. Good multi data center multi master mysql support in 2008? Killer feature. Maintaining that in 2020? Horrible. Also there were plenty of upstream routing issues where solving that became a headache. The #1 thing we wanted was uptime and the #1 outage was our upstream providers having trouble routing to other upstream providers. The number one reason and tradeoff for cloud is uptime and availability and the cost of not having it
    - Karrot_Kream 3y ago
jedberg 3y ago
What will you do if your startup suddenly takes off? That's the main advantage of the cloud early on -- flexibility.
- flukus 3y ago
  Depends on what "taking off" means for the start up too. Taking off at a mass consumer scale might need that flexibility, taking off and even getting to market saturation in a specific B2B might be achievable on a raspberry pi level hardware. There are many more of the later.
  - xur17 3y ago
    Agreed - 1gbps is a surprising amount of bandwidth, you could easily host a fairly popular mobile app, saas, etc with plenty of breathing room. And in a lot of cases, you can just move your static file hosting behind Cloudflare, or onto something like s3, and give yourself even more room to grow.
    - bauruine 3y ago
      And even if you only use 500Mbit/s it's more than 3k$ in cloud egress fees saved.
  - deleted 3y ago
    [deleted]
  - jedberg 3y ago
    Any business that you can run from a home server with a residential business line is not the kind of business we are talking about here. Yes, you can potentially serve a lot of customers with that setup, but your reliability story is terrible so you better have very forgiving customers. What if your internet goes out? Even with a business line, I've had to wait five days for them to replace a fiber line that a squirrel chewed through. What if the power goes out? I just had a five hour power outage. Even if you have a battery backup, when the neighborhood power is out for a while, the ISP equipment will die when its batteries go out. What if your hardware dies and you aren't home to switch it out, assuming you even have spare hardware? What if your A/C goes out and your server overheats and has to get shut down? All of these are things you usually don't have to deal with when using the cloud or even a $5 VPS, because they design for all of these failure cases from the start. If you're running a business from your house, it is by definition a lifestyle business, and that's not really what we are talking about here.
nonrandomstring 3y ago
After this talk [0] I had several most interesting conversations with media folks about the real cost and advantages of "cloud". One thing that came up is development. Modern devops culture is quite a good thing, and what's lovely about "cloud" - as in the ability to quickly buy compute and storage capability - is that ideas you would have tinkered with in on-prem labs (or across private sites) for months can be imagined and prototyped in hours. I'm a big advocate of rapid prototyping as a _huge_ business lever, because the ability to try out ideas quickly, to easily reconfigure things, is the key for time to market. You can quickly see if something is going to fly or not. And that's where the advantage ends. After that, it's all downhill. Asymmetry. Lock-in and portability. Trust and privacy issues. Security perimeters. Unpredictable costs.... So the way forward is to render unto Caesar only the things that are Caesar's.... in other words, take the advantages of "cloud" when it suits you, and then get the hell out of Dodge. What is ongoing from that conversation is media companies being interested in strategic planning to build, and even share, their own distributed computing resources to pull back to once a technology is off the ground. Someone even mentioned that it's time for a European Cloud initiative, [0] https://www.youtube.com/watch?v=6OL2XmlgpdA https://www.youtube.com/watch?v=6OL2XmlgpdA
- plonk 3y ago
  What do you mean by asymmetry?
  - nonrandomstring 3y ago
    Maybe the most demonstrable is egress/ingress bandwidth. But since there's a power asymmetry when dealing with mega-corporations I had other asymmetries in mind too.
  - Spooky23 3y ago
    The cloud providers are so large that they don’t really need you. It’s all about churn management at the macro level. With hardware and software, there’s always an end of quarter leverage point. I spent most of my career in large enterprises. The leverage you have against AWS or Microsoft is 0 compared to the old days. They are probably landing more infrastructure every month than my global company had in datacenters 15 years ago.
Karrot_Kream 3y ago
I'm always curious about the power cost here. Everyone I've talked to who runs servers at home either lives in a low COL area where electricity is cheap (and electricity is dirty, but I don't really want to get into that) or just pays the power as a sort of "cost of playing around" which is completely fair (it's not like I need to use my table saw at home.) Most folks I know who run servers use a lot more power than I do at home. Of course I don't know how much of that is just that the folks that build expensive homelabs being attracted to beefy, power hog servers as opposed to an actual incentive to cut power costs.
- radlad 3y ago
  I have a Plex server with 6 spinning disks and a GPU which costs me around $20/month. The most expensive part is when the GPU is running (based on my Kill-A-Watt.) I host around 15 Docker services on it. I think even with the cost of electricity, you can easily beat cloud hosting on a per-month basis. But, factoring in the initial cost of hardware and electricity, it's probably a wash. But then, if you're running a hypervisor, and would otherwise have a LOT of VMs in the cloud, maybe it swings back the other way?
- aftbit 3y ago
  My rack draws around 750W at mid-idle (not totally idle but not doing anything big either). I have about $0.14/kWh delivered electricity cost in southwest Ohio. This means the rack costs me approximately $77 per month or approximately $920 per year. That's not a particular small cost, but it is far less than I would pay to host the same things at a professional data center, or shivers the cloud. If I really wanted to optimize for power efficiency, I could do much better. I've seen decent homelab setups (with NAS, router, switch, and some slow compute nodes) that run under 100W, which would cost me only $10 per month in power, and would be far more powerful than a small DigitalOcean droplet.
  - doubled112 3y ago
    I looked into cloud, but it's just not really feasible. It really is possible to do a lot with a little at home. I'm using about 60-100W for my home-prod, and a lot of it is "older". I'm running about 15 small VMs at any given time these days, and probably 20 containers. I think my biggest single draw is the Mikrotik CCR1036 in the garage, but it saved me from buying new gear. Sure there's a break even point with hydro, but that's years in the future when the device is free. It's also pretty fun to watch VPN connections testing at 700Mbps from home. I don't really care about uptime, and I've got gigabit fibre to the house, so bandwidth isn't a huge problem. It worked fine on 300/60Mb cable too. Ryzen 3 2200G, 32GB RAM, 1T NVMe, 10TB HDD. This one runs services. Orange Pi 5 16GB with another 500GB NVMe. This runs redundant services and monitoring.
ilyt 3y ago
Renting dedicated server off company like OVH, or even co-locating your own is also far cheaper than cloud, few times over, without fuss of turning your house into datacenter.
- graemep 3y ago
  There is a time cost, but most of it is a one off, and not that difficult, and there are tools to make it easier.
rakkhi 3y ago
I basically wrote the same thing a few months ago. TL;DR for most systems public cloud is not cost effective not secure. Another interesting Apple exploit using Spector on Safari on the front page today. Most systems better off self hosted with cloud tools to manage: https://open.substack.com/pub/rakkhi/p/why-you-should-move-back-to-datacenters?r=1afqp&utm_campaign=post&utm_medium=web https://open.substack.com/pub/rakkhi/p/why-you-should-move-b...
sgarland 3y ago
This is precisely what I plan on doing when I eventually launch my app. I have 3x Dell R620s in a Proxmox cluster running K8s nodes, Ceph on NVMe with CSI-RBD, and a separate Supermicro 2U exposing ZFS for spinning storage. The only missing link is 10G networking, because 10G switches aren’t super cheap.

runlaszlorun 3y ago

was the business line much more expensive than home rates? and were you given a static ipv4 IP?

sokoloff 3y ago
Not GP, but related: I've had the same DHCP IP from Xfinity (residential internet service) for well over a decade (and I think for 16 years). At what point does a dynamic-but-unchanging IP become functionally static?
- kccqzy 3y ago
  In my case Xfinity kept the same IP for me for two years, then an outage happened where everyone in the neighborhood lost connectivity. When connectivity was restored I got a different IP. I feel like the biggest difference is the fact that there's no guarantee that the dynamic IP won't change, so all systems need to be prepared for that, or you need to be mentally prepared for that day.
bamazizi 3y ago
I am with Bell in canada. My business fiber internet is $100/m and I also pay $20/b for dedicated IP

seydor 3y ago

this is the most undersung hero. Internet speeds are reaching the equivalent of a datacenter. This will open up so many possibities, that it makes it seem like the 2000s again. And finally we might actually need ipv6

klabb3 3y ago
I just moved out of the consumer internet dystopia that is California (or most of US I guess) to Spain and am just about to get 10 Gbit symmetric for €25/mo. Even if I get half of that I’m ecstatic. This kind of infrastructure is so conducive to all kinds of interesting and “decentralized” innovations.
- dzonga 3y ago
  I wish I could get that in london :(. uk internet infra is backwards.
  - graemep 3y ago
    Bad regulation. Ofcom is allowing Openreach to keep asymmetric pricing for FTTP. Unjustified market segmentation. The govt has a focus on download speeds (which are useful for high quality video) and does not care about upload or latency which filters down. On reflection, even the the terms "upload" and "download" are based on the assumption connections are for consuming media.
- lionkor 3y ago
  In Germany you pay 60+€/month for 100mbit down.
dekhn 3y ago
datacenter network speeds are ~400+Gbit (between data centers) and in the Tbit range within data centers. A typical server now has a 10Gbit NIC.
bamazizi 3y ago
Yes indeed. My Bell.ca ~1gb fiber has a monthly cost of $100 + $20 for dedicated IP. Since it's business line fiber internet it comes with monitored service quality meaning it's prioritized over others using regular/residential fiber internet (claimed!) BTW, I can get multiple lines if i'd ever need it

p1necone 3y ago

Yeah the delta in cost/CPU/memory/storage between self hosting on second hand stuff and paying cloud services is insane. It's a no brainer if your use case needs beefier hardware than the typical $5/month VPS host and you don't have enterprise level liability/uptime expectations. With stuff like proxmox you get a pretty similar level of ease of use to managed VM services too.

Karrot_Kream 3y ago
I generally find these discussions unproductive on HN because of how binary they become, but definitely once you go off the beaten path it seems a lot cheaper (sans power though) to run your own infra. I've been thinking of doing some data ETL for something that generates 1 GB / day and has retention for 30 days and it's much cheaper to host a Postgres myself at that scale than run it in the cloud. I could store in something like S3, but then I need to deal with querying S3. I'd like to combine some of this with cloud infra but I suspect cloud bandwidth costs would kill me. Colo-ing 2-3 servers is probably the best bet here.
- calvinmorrison 3y ago
  I was an ops engineer at Fastmail. We ran our own hardware. A mix of Debian on one stack and SmartOS (Illumos from Joyent) and there were plenty of physical problems and costs. Now putting petabytes of data with replication and syncing and all that would have been a cluster F on the cloud, we missed out on a lot of the awesome newer deployment other tooling because we had written our own. Before I left we swapped out a good chunk of these snowflake software but it was impressive actually how well it worked. Good multi data center multi master mysql support in 2008? Killer feature. Maintaining that in 2020? Horrible. Also there were plenty of upstream routing issues where solving that became a headache. The #1 thing we wanted was uptime and the #1 outage was our upstream providers having trouble routing to other upstream providers. The number one reason and tradeoff for cloud is uptime and availability and the cost of not having it
  - Karrot_Kream 3y ago

jedberg 3y ago

What will you do if your startup suddenly takes off? That's the main advantage of the cloud early on -- flexibility.

flukus 3y ago
Depends on what "taking off" means for the start up too. Taking off at a mass consumer scale might need that flexibility, taking off and even getting to market saturation in a specific B2B might be achievable on a raspberry pi level hardware. There are many more of the later.
- xur17 3y ago
  Agreed - 1gbps is a surprising amount of bandwidth, you could easily host a fairly popular mobile app, saas, etc with plenty of breathing room. And in a lot of cases, you can just move your static file hosting behind Cloudflare, or onto something like s3, and give yourself even more room to grow.
  - bauruine 3y ago
    And even if you only use 500Mbit/s it's more than 3k$ in cloud egress fees saved.
- deleted 3y ago
  [deleted]
- jedberg 3y ago
  Any business that you can run from a home server with a residential business line is not the kind of business we are talking about here. Yes, you can potentially serve a lot of customers with that setup, but your reliability story is terrible so you better have very forgiving customers. What if your internet goes out? Even with a business line, I've had to wait five days for them to replace a fiber line that a squirrel chewed through. What if the power goes out? I just had a five hour power outage. Even if you have a battery backup, when the neighborhood power is out for a while, the ISP equipment will die when its batteries go out. What if your hardware dies and you aren't home to switch it out, assuming you even have spare hardware? What if your A/C goes out and your server overheats and has to get shut down? All of these are things you usually don't have to deal with when using the cloud or even a $5 VPS, because they design for all of these failure cases from the start. If you're running a business from your house, it is by definition a lifestyle business, and that's not really what we are talking about here.

nonrandomstring 3y ago

After this talk [0] I had several most interesting conversations with media folks about the real cost and advantages of "cloud". One thing that came up is development. Modern devops culture is quite a good thing, and what's lovely about "cloud" - as in the ability to quickly buy compute and storage capability - is that ideas you would have tinkered with in on-prem labs (or across private sites) for months can be imagined and prototyped in hours. I'm a big advocate of rapid prototyping as a _huge_ business lever, because the ability to try out ideas quickly, to easily reconfigure things, is the key for time to market. You can quickly see if something is going to fly or not. And that's where the advantage ends. After that, it's all downhill. Asymmetry. Lock-in and portability. Trust and privacy issues. Security perimeters. Unpredictable costs.... So the way forward is to render unto Caesar only the things that are Caesar's.... in other words, take the advantages of "cloud" when it suits you, and then get the hell out of Dodge. What is ongoing from that conversation is media companies being interested in strategic planning to build, and even share, their own distributed computing resources to pull back to once a technology is off the ground. Someone even mentioned that it's time for a European Cloud initiative, [0] https://www.youtube.com/watch?v=6OL2XmlgpdA https://www.youtube.com/watch?v=6OL2XmlgpdA

plonk 3y ago
What do you mean by asymmetry?
- nonrandomstring 3y ago
  Maybe the most demonstrable is egress/ingress bandwidth. But since there's a power asymmetry when dealing with mega-corporations I had other asymmetries in mind too.
- Spooky23 3y ago
  The cloud providers are so large that they don’t really need you. It’s all about churn management at the macro level. With hardware and software, there’s always an end of quarter leverage point. I spent most of my career in large enterprises. The leverage you have against AWS or Microsoft is 0 compared to the old days. They are probably landing more infrastructure every month than my global company had in datacenters 15 years ago.

Karrot_Kream 3y ago

I'm always curious about the power cost here. Everyone I've talked to who runs servers at home either lives in a low COL area where electricity is cheap (and electricity is dirty, but I don't really want to get into that) or just pays the power as a sort of "cost of playing around" which is completely fair (it's not like I need to use my table saw at home.) Most folks I know who run servers use a lot more power than I do at home. Of course I don't know how much of that is just that the folks that build expensive homelabs being attracted to beefy, power hog servers as opposed to an actual incentive to cut power costs.

radlad 3y ago
I have a Plex server with 6 spinning disks and a GPU which costs me around $20/month. The most expensive part is when the GPU is running (based on my Kill-A-Watt.) I host around 15 Docker services on it. I think even with the cost of electricity, you can easily beat cloud hosting on a per-month basis. But, factoring in the initial cost of hardware and electricity, it's probably a wash. But then, if you're running a hypervisor, and would otherwise have a LOT of VMs in the cloud, maybe it swings back the other way?
aftbit 3y ago
My rack draws around 750W at mid-idle (not totally idle but not doing anything big either). I have about $0.14/kWh delivered electricity cost in southwest Ohio. This means the rack costs me approximately $77 per month or approximately $920 per year. That's not a particular small cost, but it is far less than I would pay to host the same things at a professional data center, or shivers the cloud. If I really wanted to optimize for power efficiency, I could do much better. I've seen decent homelab setups (with NAS, router, switch, and some slow compute nodes) that run under 100W, which would cost me only $10 per month in power, and would be far more powerful than a small DigitalOcean droplet.
- doubled112 3y ago
  I looked into cloud, but it's just not really feasible. It really is possible to do a lot with a little at home. I'm using about 60-100W for my home-prod, and a lot of it is "older". I'm running about 15 small VMs at any given time these days, and probably 20 containers. I think my biggest single draw is the Mikrotik CCR1036 in the garage, but it saved me from buying new gear. Sure there's a break even point with hydro, but that's years in the future when the device is free. It's also pretty fun to watch VPN connections testing at 700Mbps from home. I don't really care about uptime, and I've got gigabit fibre to the house, so bandwidth isn't a huge problem. It worked fine on 300/60Mb cable too. Ryzen 3 2200G, 32GB RAM, 1T NVMe, 10TB HDD. This one runs services. Orange Pi 5 16GB with another 500GB NVMe. This runs redundant services and monitoring.

ilyt 3y ago

Renting dedicated server off company like OVH, or even co-locating your own is also far cheaper than cloud, few times over, without fuss of turning your house into datacenter.

graemep 3y ago
There is a time cost, but most of it is a one off, and not that difficult, and there are tools to make it easier.

rakkhi 3y ago

I basically wrote the same thing a few months ago. TL;DR for most systems public cloud is not cost effective not secure. Another interesting Apple exploit using Spector on Safari on the front page today. Most systems better off self hosted with cloud tools to manage: https://open.substack.com/pub/rakkhi/p/why-you-should-move-back-to-datacenters?r=1afqp&utm_campaign=post&utm_medium=web https://open.substack.com/pub/rakkhi/p/why-you-should-move-b...

sgarland 3y ago

This is precisely what I plan on doing when I eventually launch my app. I have 3x Dell R620s in a Proxmox cluster running K8s nodes, Ceph on NVMe with CSI-RBD, and a separate Supermicro 2U exposing ZFS for spinning storage. The only missing link is 10G networking, because 10G switches aren’t super cheap.