19 ms·
Why we use our own hardware
- edithpixie 2y ago[flagged]
- oldpersonintx 2y agolongtime FM user here good on them, understanding infrastructure and cost/benefit is essential in any business you hope to run for the long haul
- bartvk 2y agoSuch an awesome article. I like how they didn't just go with the Cloud wave but kept sysadmin'ing, like ol' Unix graybeards. Two interesting things they wrote about their SSDs: 1) "At this rate, we’ll replace these [SSD] drives due to increased drive sizes, or entirely new physical drive formats (such E3.S which appears to finally be gaining traction) long before they get close to their rated write capacity." and 2) "We’ve also anecdotally found SSDs just to be much more reliable compared to HDDs (..) easily less than one tenth the failure rate we used to have with HDDs."
- tgv 2y agoTo avoid sysadmin tasks, and keep costs down, you've got to go so deep in the cloud, that it becomes just another arcane skill set. I run most of my stuff on virtual Linux servers, but some on AWS, and that's hard to learn, and doesn't transfer to GCP or Azure. Unless your needs are extreme, I think sysadmin'ing is the easier route in most cases.
- baxtr 2y agoI predict a slow but unstoppable comeback of the sysadmin job over the next 5-10 years.
- homebrewer 2y agoIt never disappeared in some places. In my region there's been zero interest in "the cloud" because of physical remoteness from all major GCP/AWS/Azure datacenters (resulting in high latency), for compliance reasons, and because it's easier and faster to solve problems by dealing with a local company than pleading with a global giant that gives zero shits about you because you're less than a rounding error in its books.
- wongarsu 2y agoFor so many things the cloud isn't really easier or cheaper, and most cloud providers stopped advertising it as such. My assumption is that cloud adoption is mainly driven by 3 forces: - for small companies: free credits - for large companies: moving prices as far away as possible from the deploy button, allowing dev and it to just deploy stuff without purchase orders - self-perpetuating due to hype, cv-driven development, and ease of hiring All of these are decent reasons, but none of them may apply to a company like fastmail
- graemep 2y agoAlso CYA. If you run your own servers and something goes wrong its your fault. if its an outage at AWS its their fault. Also a huge element of follow the crowd, branding non-technical management are familiar with, and so on. I have also found some developers (front end devs, or back end devs who do not have sysadmin skills) feel cloud is the safe choice. This is very common for small companies as they may have limited sysadmin skills (people who know how to keep windows desktops running are not likely to be who you want to deploy servers) and a web GUI looks a lot easier to learn.
- dietr1ch 2y ago> If its an outage at AWS its their fault. Well, still your fault, but easy to judo the risk into clients saying supporting multi-cloud is expensive and not a priority.
- graemep 2y agoManagement in many places will not even know what multi-cloud is (or even multi-region). As Cloudstrike showed, if you follow the crowd and tick the right boxes you will not be blamed.
- bobnamob 2y agonit: Crowdstrike Unless the incident is now being referred to as “Cloudstrike”, in which case, eww
- graemep 2y ago> it becomes just another arcane skill set Its an arcane skill set with a GUI. It makes it look much easier to learn.
- edward28 2y agoThe power of Moore's law.
- jeffbee 2y agoI don't see how point 2 could have come as a surprise to anyone.
- kwillets 2y agoSSD's are also a bit of an achilles heel for AWS -- they have their own Nitro firmware for wear levelling and key rotations, due to the hazards of multitenant. It's possible for one EC2 tenant to use up all the write cycles and then pass it to another, and encryption with key rotation is required to keep data from leaking across tenant changes. It's also slower. We had one outage where key rotation had been enabled on reboot, so data partitions were lost after what should have been a routine crash. Overall, for data warehousing, our failure rate on on-prem (DC-hosted) hardware was lower IME.
- brongondwana 2y agoMy beard isn't entirely grey yet! The new NVMe drives we've only had for a few years, but so far there's only been a single failure across the whole fleet, and we keep spares in stock. It's been very reliable, not like the weeks back in (hmm, 2006? 2007?) the ancient past, when we were losing 15kRPM velociraptors every other day. They had a firmware fault and we eventually got an update which made them reliable, but it was a wild few months.
- AndrewDavis 2y agoA few more than one, but it has been a lot less than when we were dealing with spinner. I think I requested about one or two replacements a year, a far cry from the one a week I was doing before.
- brongondwana 2y agoIf I didn't see it, it didn't happen... OK, I stand corrected. We lose one or two NVMe drives per year :)
- karolzlot 2y agoCan I ask which brands / models of SSD are you using?
- AndrewDavis 2y agoWe didn’t replace all servers at once, it was progressive, therefore due to availability the models we used changed over time. Our first batch of all nvme machines had SSDPE2KX080T8, they became harder to source and we moved to SSDPF2KX076T1. With Intel no longer in the ssd business I believe we have some Micron MTFDKCC7T6TGH and MTFDKCC30T7TGR. And as mentioned in the blog post we've recently purchased some Solidigm D5-P5336 which are 61TB monsters. Here's a fun related story. Our supplier had so much trouble finding SSDPE2KX080T8 that when we had exhausted our spares, I had to sync everything off a machine, tear it down and pull its drives for spares and rebuild it with the smaller SSDPF2KX076T1. Then we had lots of spares
- louwrentius 2y agoI like this writeup, informative and to-the-point. Today, the cloud isn’t about other people’s hardware. It’s about infrastructure being an API call away. Not just virtual machines but also databases, load-balancers, storage, and so on. The cost isn’t the DC or the hardware, but the hours spend on operations. And you can abuse developers to do operations on the side :-)
- zelphirkalt 2y agoAnd then come the weird aspects of bad cloud service providers, like IONOS, who have broken OS images, a provisioning API, that is a bottleneck, where what other people do and how much they do can slow down your own provisioning and creating network interfaces can take minutes via their API and their customer services says "That's how it is, cannot change it.", and you get a very shitty web user interface, that desperately tries to be a single page app, yet has all the default browser functionality like the back button broken. Yet they still cost literally 10x what Hetzner cloud costs, while Hetzner basically does everything better. And then it is still also about other people's hardware in addition to that.
- goldeneye13_ 2y agoDidn’t see this in the article, do they have multi az redundancy? I.e. if the entire raid goes up in flames what’s the recovery process?
- comboy 2y agoYeah, that makes me feel uneasy as a long time fastmail user.
- cyrnel 2y agoLooks like they do mention that elsewhere: https://www.fastmail.com/features/reliability/ https://www.fastmail.com/features/reliability/ > Fastmail has some of the best uptime in the business, plus a comprehensive multi data center backup system. It starts with real-time replication to geographically dispersed data centers, with additional daily backups and checksummed copies of everything. Redundant mirrors allow us to failover a server or even entire rack in the case of hardware failure, keeping your mail running.
- Amfy 2y agoI believe they replicate from NJ to WA (Seattle). At least that's something they spoke about many years ago.
- brongondwana 2y agoPHL to STL these days, but same design: https://www.fastmail.com/blog/moving-house-new-datacentre/ https://www.fastmail.com/blog/moving-house-new-datacentre/
- sufehmi 2y agohttps://www.fastmail.com/blog/throwback-security-confidentiality-integrity-availability/#:~:text=has%20paid%20off.-,Availability,-We%20also%20wrote https://www.fastmail.com/blog/throwback-security-confidentia...
- jmakov 2y agoWould be interesting to know how files get stored. They don't mention any distributed FS solutions like SeaweedFS so once a drive is full, does the file get sent to another one via some service? Also ZFS seems an odd choice since deletions (esp of small files) at +80% full drive are crazy slow.
- shrubble 2y agoThe open-source Cyrus IMAP server which they mention using, has replication built-in. ZFS also has built-in replication available. Deletion of files depends on how they have configured the message store - they may be storing a lot of data into a database, for example.
- mastax 2y agoZFS replication is quite unreliable when used with ZFS native encryption, in my experience. Didn't lose data but constant bugs.
- brongondwana 2y agoYeah, we're only using ZFS replication for logs; we're using the Cyrus replication for emails because it has other sanity checks and data model consistency enforcement which is really valuable. (And both are async. We'd need something like drbd for real synchronous replication)
- ackshi 2y agoKeeping enough free space should be much less of a problem with SSDs. They can tune it so the array needs to be 95% full before the slower best-fit allocator kicks in. https://openzfs.readthedocs.io/en/latest/performance-tuning.html#free-space https://openzfs.readthedocs.io/en/latest/performance-tuning.... I think that 80% figure is from when drives were much smaller and finding free space over that threshold with the first-fit allocator was harder.
- ryao 2y agoUnlike ext4 that locks the directory when unlinking, ZFS is able to scale on parallel unlinking. In specific, ZFS has range locks that permit directory entries to be removed in parallel from the extendible hash trees that store them. While this is relatively slow for sequential workloads, it is fast on parallel workloads. If you want to delete a large directory subtree fast on ZFS, do the rm operations in parallel. For example, this will run faster on ZFS than a naive rm operation: find /path/to/subtree -name -type f | parallel -j250 rm -- rm -r /path/to/subtree A friend had this issue on spinning disks the other day. I suggested he do this and the remaining files were gone in seconds when at the rate his naive rm was running, it should have taken minutes. It is a shame that rm does not implement a parallel unlink option internally (e.g. -j), which would be even faster, since it would eliminate the execve overhead and likely would eliminate some directory lookup overhead too, versus using find and parallel to run many rm processes. For something like fast mail that has many users, unlinking should be parallel already, so unlinking on ZFS will not be slow for them. By the way, that 80% figure has not been true for more than a decade. You are referring to the best fit allocator being used to minimize external fragmentation under low space conditions. The new figure is 96%. It is controlled by metaslab_df_free_pct in metaslab.c: https://github.com/openzfs/zfs/blob/zfs-2.2.0/module/zfs/metaslab.c#L177 https://github.com/openzfs/zfs/blob/zfs-2.2.0/module/zfs/met... Modification operations become slow when you are at/above 96% space filled, but that is to prevent even worse problems from happening. Note that my friend’s pool was below the 96% threshold when he was suffering from a slow rm -r. He just had a directory subtree with a large amount of directory entries he wanted to remove. For what it is worth, I am the ryao listed here and I was around when the 80% to 96% change was made: https://github.com/openzfs/zfs/graphs/contributors https://github.com/openzfs/zfs/graphs/contributors
- amelius 2y ago[flagged]
- pavlov 2y agoThere is a "Report Spam" function which is two clicks away (it's in the "More" menu).
- ramon156 2y agoIf you get 12 spam mails everyday and after 3 months of clicking "report spam" it still doesn't filter it, then it's not en par with Gmail.
- mcny 2y agoIf you meet someone new at a social event and give them your email address, where do you want your email provider to put the message that this person sent?
- crossroadsguy 2y agoThat's like Spotify telling me "keep disliking" when I complained to them why songs from a certain language (which I never liked or listened to and I certainly don't speak) keeps filling the home after I told them in the first complaint that I have been doing that since months.
- pavlov 2y agoWhat can I say, "Report Spam" seems to work for me. I'm just a customer of Fastmail.
- amelius 2y agoI don't want to report everything as spam. For example, promotional emails from businesses that I bought something from. I don't want to punish those businesses; and those emails might contain vouchers that I could use later. But I want those emails moved out of the way without any action from my side.
- caidan 2y agoI absolutely love Fastmail. I moved off of Gmail years ago with zero regrets. Better UI, better apps, better company, and need I say better service? I still maintain and fetch from a Gmail account so it all just works seamlessly for receiving and sending Gmail, so you don’t have to give anything up either.
- pawelduda 2y agoTheir android app has always been much snappier than Gmail, it's the little things that drew me to it years ago
- jb1991 2y agoTheir UI is definitely faster but I do prefer the gmail UI, for example how new messages are displayed in threads is quite useless in fastmail.
- petesergeant 2y agoI use Fastmail for my personal mail, and I don’t regret it, but I’m not quite as sold as you are, I guess maybe because I still have a few Google work accounts I need to use. Spam filtering in Fastmail is a little worse, and the search is _terrible_. The iOS app is usable but buggy. The easy masked emails are a big win though, and setting up new domains feels like less of a hassle with FM. I don’t regret using Fastmail, and I’d use them again for my personal email, but it doesn’t feel like a slam dunk.
- mlfreeman 2y agoI moved from my own colocated 1U running Mailcow to Fastmail and don't regret it one bit. This was an interesting read, glad to see they think things through nice and carefully. The only things I wish FM had are all software: 1. A takeout-style API to let me grab a complete snapshot once a week with one call 2. The ability to be an IdP for Tailscale.
- brongondwana 2y ago1. hoping to have a JMAP archive format at some point which should cover that. I'd hope that normally you'd be fetching a delta update rather than the whole thing. We've got enough bandwidth for a few people do to it, but I wouldn't want every customer pulling their entire archive every week of 99% the same immutable data; that would be kinda sucky. 2. yeah, I'd love that too - we're keen to integrate with everything else that people are using. We have a basic in-house IdP thing for our own staff to authenticate against our hosted services, but haven't scaled it out. This will happen eventually, though I've been burned enough times I don't want to promise a timeframe.
- tucnak 2y agoYeah, Cloud is a bit of a scam innit? Oxide is looking more and more attractive every day as the industry corrects itself from overspending on capabilities they would never need.
- klysm 2y agoIt’s trading time for money
- rob_c 2y agoYes, welcome to business. But frankly an email provider needs to have their own metal, if they don't they're not worth doing business with
- jgb1984 2y agoFake news. I've got my bare metal server deployed and installed with my ansible playbook even before you manage to log into the bazillion layers of abstraction that is AWS.
- mgaunard 2y agoWhy is it surprising? It's well known cloud is 3 times the price.
- diggan 2y agoBecause the default for companies today is cloud, even though it almost never makes sense. Sure, if you have really spikey load, need to dynamically scale at any point and don't care about your spend, it might make sense. Ive even worked in companies where the engineering team spent effort and time on building "scalable infrastructure" before the product itself even found product-market fit...
- dewey 2y agoNobody said it's surprising though, they are well aware of it having done it for more than two decades. Many newcomers are not aware of it though, as their default is "cloud" and they never even shopped for servers, colocation or looked around on the dedicated server market.
- aimanbenbaha 2y agoI don't think they're not just aware. But purely from scaling and distribution perspective it'd be wiser to start on cloud while you're still on the product-market fit phase. Also 'bare metal' requires more on the capex end and with how our corporate tax system is set it's just discouraging to go on this lane first and it'd be better off to spend on acquiring clients. Also I'd guess a lot of technical founders are more familiar with cloud/server-side than with dealing or delegating sysadmin taks that might require adding members to the team.
- dewey 2y agoI agree, the cloud definitely has a lot of use cases and when you are building more complicated systems it makes sense to just have to do a few clicks to get a new stack setup vs. having someone evaluate solutions and getting familiar with operating them on a deep level (backups etc.).
- rrgok 2y agoI would like to know the tech stack behind it.
- brongondwana 2y agoThere's various articles on our blog about our stack!
- antihero 2y agoI’ve started to host my own sites and stuff on an old MacBook in a cupboard with a shit old external hardware Ava microk8s and it’s great!
- theoreticalmal 2y agoAnother homelabber joins the ranks!!
- antihero 2y agoJust implemented a dyndns system using K8s CronJobs + GitOps + CloudFlare Terraform, however next stage will be moving that over to CloudFlare tunnels which should be more reliable and nicer, fully within the Terraform and not relying on polling a random JSON IP service (which a terrifying SPOF)
- tndibona 2y agoBut what about the cost and complexity of a room with the racks and the cooling needs of running these machines? And the uninterrupted power setup? The wiring mess behind the racks.
- jonatron 2y agoEven for cloud providers, these are mostly other people's problems, eg: Equinix
- hyhconito 2y agoI'm not fastmail but this is not rocket science. Has everyone forgotten how datacentre services work in 2024?
- rob_c 2y agoYes they have and they feel they deserve credit for discovering a WiFi cable is more reliable to the new shiny kit that was sold to them by a vendor...
- 7952 2y agoDo colocation facilities solve that?
- bradfa 2y agoThere is a very competitive market for colo providers in basically every major metropolitan area in the US, Europe, and Asia. The racks, power, cooling, and network to your machines is generally very robust and clearly documented on how to connect. Deploying servers in house or in a colo is a well understood process with many experts who can help if you don’t have these skills.
- rob_c 2y agoColo offers the ability to ship and deploy and keep latencies down if you're global, but if you're local yes you should just get someone on site and the modern equivalent of a T1 line setup to your premises if you're running "online" services.
- lokimedes 2y agoA mail-cloud provider uses its own hardware? Well, that’s to be expected, it would be a refreshing article if it was written by one of their customers.
- rob_c 2y agoNo they deserve me praise for simply running their stuff on metal... Like a thousand unix sysadmins before and after
- tuananh 2y agogmail does spam filtering very well for me. fastmail on the other hands, puts lots of legit emails into spam folder. manually marking "not spam" doesn't help other than that, i'm happy with fastmail.
- jacobdejean 2y agoiCloud is just as bad, sends important things to spam constantly and marking as “not spam” has never done anything perceivable.
- ghaff 2y agoIf I look at my Gmail SPAM folder, there is very rarely something genuinely important in it. What there is a fair bit of though is random newsletters and announcements that I may have signed up for in some way shape or form that I don't really care about or generally look at. I assume they've been reported as SPAM by enough people rather than simply unsubscribed to that Google now labels them as such.
- TMWNN 2y ago>fastmail on the other hands, puts lots of legit emails into spam folder. manually marking "not spam" doesn't help Fastmail explicitly says that moving mail to/from a spam folder via a mail client does not automaticallyl retrain. <https://www.fastmail.help/hc/en-us/articles/1500000278142-Improving-spam-protection#spamfolders https://www.fastmail.help/hc/en-us/articles/1500000278142-Im...> (I never did figure out if Gmail acts the same way or not.)
- xsc 2y agoAre those backups geographically distributed?
- christophilus 2y agoYes.
- _bare_metal 2y agoPlugging https://BareMetalSavings.com https://BareMetalSavings.com in case you want to ballpark-estimate your move off of the cloud Bonus points: I'm a Fastmail customer, so it tangentially tracks ---- Quick note about the article: ZFS encryption can be flaky, be sure you know what you're doing before deploying for your infrastructure. Relevant Reddit discussion: https://www.reddit.com/r/zfs/comments/1f59zp6/is_zfs_encryption_bug_still_a_thing/ https://www.reddit.com/r/zfs/comments/1f59zp6/is_zfs_encrypt... A spreadsheet of related issues that I can't remember who made: https://docs.google.com/spreadsheets/d/1OfRSXibZ2nIE9DGK6swwBZXgXwdCPKgp4SbPZwTexCg/htmlview https://docs.google.com/spreadsheets/d/1OfRSXibZ2nIE9DGK6sww...
- deleted 2y ago[deleted]
- brongondwana 2y agoYeah, we know about the ZFS encryption with send/receive bug, it's frustrating our attempts to get really nice HA support on our logging system... but so far it appears that just deleting the offsending snapshot and creating a new one works, and we're funding some research into the issue as well. This is the current script - it runs every minute for each pool synced between the two log servers: https://gist.github.com/brong/6a23fee1480f2d62b8a18ade5aea667a https://gist.github.com/brong/6a23fee1480f2d62b8a18ade5aea66...
- _bare_metal 2y agoThanks for sharing!
- sneak 2y agoMy main issue with ZFS encryption is that it only supports one key. LUKS2 has something like 9 key slots. I run ZoL over LUKS2 and it works great.
- ackshi 2y agoI'm a little surprised it seems they didn't have some existing compression solution before moving to zfs. With so much repetitive text across emails I would think there would be a LOT to gain, such as from dictionaries, compressing many emails into bigger blobs, and fine-tuning compression options.
- silvestrov 2y agoThey use ZFS with zstd which likely compresses well enough. Custom compression code can introduce bugs that can kill Fastmail's reputation of reliability. It's better to use a well tested solution that cost a bit more.
- rob_c 2y agoKeen to move certain tasks to ZFS but not the ones that matter... Frankly given emails are normally ~4kB objects I suspect the compression overheads are probably not that worth it unless it's for attachments only. Not attacking ZFS it's compression and checksumming are among best in class, but the compression would work better if it weren't limited to small files. Here ZFS has made a lot of wins I've not had a problem with many files on ZFS due to L1/L2 ARC but the cost is metadata ops can be painful on many small files. The evidence they IOPS limited it that they went for SSD or better when they could store the same capacity on rust for much cheaper now. Yeah I think moving the compression or file access up to abstract what is being written to disk ala protonmail (I don't like their offerings, but like their tech) means you can have compression over 4MB not 4kB blocks which matters when you recall data from disks for , I don't know... Backups or search? also remember RAID!=backups ;)
- rob_c 2y agoHosts online service seems to think deserving of medal for discovering that S3 buckets from a cloud provider are crap and cost a fortune. The heading in this space makes your think they're running custom FPGAs such as with Gmail, not just running on metal... As for drive failures, welcome to storage at scale. Build your solution so it's a weekly task to replace 10disks at a time not critical at 2am when a single disk dies... Storing/Accessing tonnes of <4kB files is difficult, but other providers are doing this on their own metal with CEPH at the PB scale. I love ZFS, it's great with per-disk redundancy but CEPH is really the only game in town for inter-rack/DC resilience which I would hope my email provider has.
- brongondwana 2y agoCeph is most certainly not the only game in town. It's good and stuff, but it's just tech. We're using protocol level replication for each of our data stores.
- rob_c 2y agoNo, let's be honest. CEPH is the only solution for data management at this scale (sub to few PB). The solution which is independent of application or workload. The market share, fact IBM is moving people off other projects internally for this, and the massive backing shows this. Yes you can have all or a bunch of these features like failure domains via other routes/products but none have all of the stuff together in one place like CEPH. There's a reason people call it the "Linux of storage". The only alternatives are manage this at a higher level in your stack (reinventing the wheel) or buying PB level solutions from corporate which is like saying I'm buying Oracle and MS over Linux. Protocol replication means you've reimplemented something which is storage related elsewhere in your stack. It's not incorrect to do so, but there exist better solutions and alternatives now.
- brongondwana 2y agoI mean, I'm happy to have this argument. CEPH is content agnostic and that's fantastic most of the time. Cyrus replication is data aware, so it's not just replicating the data, it's doing integrity checking and data model consistency handling. Most of all, it's doing split brain recovery; which - if we wanted CP rather than AP then we wouldn't need, but that wasn't the original design. If I was redoing this from scratch, I'd maybe do Ceph or similar and update Cyrus to work well with it, but that would be a big change from the current design. Anyway, I'm happy to stipulate that Ceph is great tech, without going and telling other people that it's the only choice.
- johnklos 2y agoThe whole push to the cloud has always fascinated me. I get it - most people aren't interested in babysitting their own hardware. On the other hand, a business of just about any size that has any reasonable amount of hosting is better off with their own systems when it comes purely to cost. All the pro-cloud talking points are just that - talking points that don't persuade anyone with any real technical understanding, but serve to introduce doubt to non-technical people and to trick people who don't examine what they're told. What's particularly fascinating to me, though, is how some people are so pro-cloud that they'd argue with a writeup like this with silly cloud talking points. They don't seem to care much about data or facts, just that they love cloud and want everyone else to be in cloud, too. This happens much more often on sites like Reddit (r/sysadmin, even), but I wouldn't be surprised to see a little of it here. It makes me wonder: how do people get so sold on a thing that they'll go online and fight about it, even when they lack facts or often even basic understanding? I can clearly state why I advocate for avoiding cloud: cost, privacy, security, a desire to not centralize the Internet. The reason people advocate for cloud for others? It puzzles me. "You'll save money," "you can't secure your own machines," "it's simpler" all have worlds of assumptions that those people can't possibly know are correct. So when I read something like this from Fastmail which was written without taking an emotional stance, I respect it. If I didn't already self-host email, I'd consider using Fastmail. There used to be so much push for cloud everything that an article like this would get fanatical responses. I hope that it's a sign of progress that that fanaticism is waning and people aren't afraid to openly discuss how cloud isn't right for many things.
- anotherhue 2y agoThey spent time and career points learning cloud things and dammit it's going to matter! You can't even blame them too much, the amount of cash poured into cloud marketing is astonishing.
- sgarland 2y agoThe thing that frustrates me is it’s possible to know how to do both. I have worked with multiple people who are quite proficient in both areas. Cloud has definite advantages in some circumstances, but so does self-hosting; moreover, understanding the latter makes the former much, much easier to reason about. It’s silly to limit your career options.
- tiffanyh 2y agoFYI - Fastmail web client has Offline support in beta right now. https://www.fastmail.com/blog/offline-in-beta/ https://www.fastmail.com/blog/offline-in-beta/
- ForHackernews 2y agoVery confused by this. What is in beta? I've had "offline" email access for 25 years. It's called an IMAP client.
- renewiltord 2y ago[flagged]
- ternnoburn 2y agoHey, this response makes you look like an adolescent asshole. Parent poster was clearly asking about prioritization.
- renewiltord 2y ago[flagged]
- ternnoburn 2y agoBeing an asshole is being a moron.
- renewiltord 2y agoObviously untrue.
- mdaniel 2y agoAnd if anyone is curious, I actually live on their https://betaapp.fastmail.com https://betaapp.fastmail.com release and find it just as stable as the "mainline" one but with the advantage of getting to play with all the cool toys earlier. Bonus points (for me) in that they will periodically conduct surveys to see how you like things
- DarkCrusader2 2y agoI have seen a common sentiment that self hosting is almost always better than cloud. What these discussions does not mention is how to effectively run your business applications on this infrastructure. Things like identity management (AAD/IAM), provisioning and running VMs, deployments. Network side of things like VNet, DNS, securely opening ports etc. Monitoring setup across the stack. There is so much functionalities that will be required to safely expose an application externally that I can't even coherently list them out here. Are people just using Saas for everything (which I think will defeat the purpose of on-prem infra) or a competent Sys admin can handle all this to give a cloud like experience for end developers? Can someone share their experience or share any write ups on this topic? For more context, I worked at a very large hedge fund briefly which had a small DC worth of VERY beefy machines but absolutely no platform on top of it. Hosting application was done by copying the binaries on a particular well known machine and running npm commands and restarting nginx. Log a ticket with sys admin to create a DNS entry to point a reserve and point a internal DNS to this machine (no load balancer). Deployment was a shell script which rcp new binaries and restarts nginx. No monitoring or observability stack. There was a script which will log you into a random machine for you to run your workloads (be ready to get angry IMs from more senior quants running their workload in that random machine if your development build takes up enough resources to effect their work). I can go on and on but I think you get the idea.
- noprocrasted 2y ago> identity management (AAD/IAM) Do you mean for administrative access to the machines (over SSH, etc) or for "normal" access to the hosted applications? Admin access: Ansible-managed set of UNIX users & associated SSH public keys, combined with remote logging so every access is audited and a malicious operator wiping the machine can't cover their tracks will generally get you pretty far. Beyond that, there are commercial solutions like Teleport which provide integration with an IdP, management web UI, session logging & replay, etc. Normal line-of-business access: this would be managed by whatever application you're running, not much different to the cloud. But if your application isn't auth-aware or is unsafe to expose to the wider internet, you can stick it behind various auth proxies such as Pomerium - it will effectively handle auth against an IdP and only pass through traffic to the underlying app once the user is authenticated. This is also useful for isolating potentially vulnerable apps. > provisioning and running VMs Provisioning: once a VM (or even a physical server) is up and running enough to be SSH'd into, you should have a configuration management tool (Ansible, etc) apply whatever configuration you want. This would generally involve provisioning users, disabling some stupid defaults (SSH password authentication, etc), installing required packages, etc. To get a VM to an SSH'able state in the first place, you can configure your hypervisor to pass through "user data" which will be picked up by something like cloud-init (integrated by most distros) and interpreted at first boot - this allows you to do things like include an initial SSH key, create a user, etc. To run VMs on self-managed hardware: libvirt, proxmox in the Linux world. bhyve in the BSD world. Unfortunately most of these have rough edges, so commercial solutions there are worth exploring. Alternatively, consider if you actually need VMs or if things like containers (which have much nicer tooling and a better performance profile) would fit your use-case. > deployments Depends on your application. But let's assume it can fit in a container - there's nothing wrong with a systemd service that just reads a container image reference in /etc/... and uses `docker run` to run it. Your deployment task can just SSH into the server, update that reference in /etc/ and bounce the service. Evaluate Kamal which is a slightly fancier version of the above. Need more? Explore cluster managers like Hashicorp Nomad or even Kubernetes. > Network side of things like VNet Wireguard tunnels set up (by your config management tool) between your machines, which will appear as standard network interfaces with their own (typically non-publicly-routable) IP addresses, and anything sent over them will transparently be encrypted. > DNS Generally very little reason not to outsource that to a cloud provider or even your (reputable!) domain registrar. DNS is mostly static data though, which also means if you do need to do it in-house for whatever reason, it's just a matter of getting a CoreDNS/etc container running on multiple machines (maybe even distributed across the world). But really, there's no reason not to outsource that and hosted offerings are super cheap - so go open an AWS account and configure Route53. > securely opening ports To begin with, you shouldn't have anything listening that you don't want to be accessible. Then it's not a matter of "opening" or closing ports - the only ports that actually listen are the ones you want open by definition because it's your application listening for outside traffic. But you can configure iptables/nftables as a second layer of defense, in case you accidentally start something that unexpectedly exposes some control socket you're not aware of. > Monitoring setup across the stack collectd running on each machine (deployed by your configuration management tool) sending metrics to a central machine. That machine runs Grafana/etc. You can also explore "modern" stuff that the cool kids play with nowadays like VictoriaMetrics, etc, but metrics is mostly a solved problem so there's nothing wrong with using old tools if they work and fit your needs. For logs, configure rsyslogd to log to a central machine - on that one, you can have log rotation. Or look into an ELK stack. Or use a hosted service - again nothing prevents you from picking the best of cloud and bare-metal, it's not one or the other. > safely expose an application externally There's a lot of snake oil and fear-mongering around this. First off, you need to differentiate between vulnerabilities of your application and vulnerabilities of the underlying infrastructure/host system/etc. App vulnerabilities, in your code or dependencies: cloud won't save you. It runs your application just like it's been told. If your app has an SQL injection vuln or one of your dependencies has an RCE, you're screwed either way. To manage this you'd do the same as you do in cloud - code reviews, pentesting, monitoring & keeping dependencies up to date, etc. Infrastructure-level vulnerabilities: cloud providers are responsible for keeping the host OS and their provided services (load balancers, etc) up to date and secure. You can do the same. Some distros provide unattended updates (which your config management tool) can enable. Stuff that doesn't need to be reachable from the internet shouldn't be (bind internal stuff to your Wireguard interfaces). Put admin stuff behind some strong auth - TLS client certificates are the gold standard but have management overheads. Otherwise, use an IdP-aware proxy (like mentioned above). Don't always trust app-level auth. Beyond that, it's the usual - common sense, monitoring for "spooky action at a distance", and luck. Not too much different from your cloud provider, because they won't compensate you either if they do get hacked. > For more context, I worked at a very large hedge fund briefly which had a small DC worth of VERY beefy machines but absolutely no platform on top of it... Nomad or Kubernetes.
- briHass 2y agoThe biggest win with running your own infra is disk/IO speeds, as noted here and in DHH's series on leaving cloud (https://world.hey.com/dhh/we-have-left-the-cloud-251760fb https://world.hey.com/dhh/we-have-left-the-cloud-251760fb) The cloud providers really kill you on IO for your VMs. Even if 'remote' SSDs are available with configurable ($$) IOPs/bandwidth limits, the size of your VM usually dictates a pitiful max IO/BW limit. In Azure, something like a 4-core 16GB RAM VM will be limited to 150MB/s across all attached disks. For most hosting tasks, you're going to hit that limit far before you max out '4 cores' of a modern CPU or 16GB of RAM. On the other hand, if you buy a server from Dell and run your own hypervisor, you get a massive reserve of IO, especially with modern SSDs. Sure, you have to share it between your VMs, but you own all of the IO of the hardware, not some pathetic slice of it like in the cloud. As is always said in these discussions, unless you're able to move your workload to PaaS offerings in the cloud (serverless), you're not taking advantage of what large public clouds are good at.
- noprocrasted 2y agoBiggest issue isn't even sequential speed but latency. In the cloud all persistent storage is networked and has significantly more latency than direct-attached disks. This is a physical (speed of light) limit, you can't pay your way out of it, or throw more CPU at it. This has a huge impact for certain workloads like relational databases.
- sgarland 2y agoYep. This is why my 12-year old Dell R620s with Ceph on NVMe via Infiniband outperform the newest RDS and Aurora instances: the disk latency is measured in microseconds. Locally attached is of course even faster.
- briHass 2y agoI ran into this directly trying to use Azure's SMB as a service offering (Azure Files) for a file-based DB. It currently runs on a network share on-prem, but moving it to an Azure VM using that service killed performance. SMB is chatty as it is, and the latency of tons of small file IO was horrendous. Interestingly, creating a file share VM deployed in the same proximity group has acceptable latency.
- Axsuul 2y agoAnyone know what are some good data centers or providers to host your bare metal servers?
- klysm 2y agoYou’re probably looking for the term “colo”
- nisa 2y agoLove this article and I'm also running some stuff on old enterprise servers in some racks somehwere. Now over the last year I've had to dive into Azure Cloud as we have customers using this (b2b company) and I finally understood why everyone is doing cloud despite the price: Global permissions, seamless organization and IaC. If you are Fastmail or a small startup - go buy some used dell poweredge with epycs in some Colo rack with 10Gbe transit and save tons of money. If you are a company with tons of customers, ton's of requirements it's powerful to put each concern into a landing zone, run some bicep/terraform - have a ressource group to control costs and get savings on overall core-count and be done with it. Assign permissions into a namespace for your employe or customer - have some back and forth about requirements and it's done. No need to sysadmin across servers. No need to check for broken disks. I'm also blaming the hell of vmware and virtual machines for everything that is a PITA to maintain as a sysadmin but is loved because it's common knowledge. I would only do k8s on bare-metal today and skip the whole virtualization thing completly. I guess it's also these pains that are softened in the cloud.
- akpa1 2y agoThe fact that Fastmail work like this, are transparent about what they're up to and how they're storing my email and the fact that they're making logical decisions and have been doing so for quite a long time is exactly the reason I practically trip over myself to pay them for my email. Big fan of Fastmail.
- pammf 2y agoCost isn’t always the most important metric. If that was the case, people would always buy the cheapest option of everything.
- veidr 2y ago"WHY we use our own hardware..." The why is is the interesting part of this article.
- veidr 2y agoI take that back; this is (to me)t he most interesting part: "Although we’ve only ever used datacenter class SSDs and HDDs failures and replacements every few weeks were a regular occurrence on the old fleet of servers. Over the last 3+ years, we’ve only seen a couple of SSD failures in total across the entire upgraded fleet of servers. This is easily less than one tenth the failure rate we used to have with HDDs."
- veidr 2y agoI wanted to revisit this after checking my own anecdata. (But based on logfiles not just like recollections.) I've had a ZFS system or some sort for about 10 years, and before that I had proprietary RAID chassis like Pegasus2 and Synology etc. I can't quite say how many drives I have used, because my records are not that good. But maybe its like 100 drives since 2008. Maybe 150. Less than 200. I had over 10 HDD devices fail (probably 13, confidence of like 90%). I've only ever had 1 SSD fail. I've also used the absolute cheapest shite SSDs. I suspect the failure modes tend to be - hard disks fail whenever the fuck, who knows - SSDs fail in the beginning or end of their reasonable service life P.S. With ZFS though, you don't really care if/when they fail. I've so far (knock on wood) never lost any data with a ZFS config with >1 disk redundancy and reasonable backups.
- indulona 2y agoI am working on a personal project(some would call it startup, but i have no intention of getting external financing and other americanisms) where i have set up my own cdn and video encoding, among other things. These days, whenever you have a problem, everyone answers "just use cloud" and that results in people really knowing nothing any more. It is saddening. But on the other hand it ensures all my decades of knowledge will be very well paid in the future, if i'd need to get a job.
- Beijinger 2y agoI was told Fastmail is excellent, and I am not a big fan of gmail. Once locked out for good in gmail, your email and apps associated with it, are gone forever. Source? Personal experience. "A private inbox $60 for 12 months". I assume it is USD, not AU$ (AFAIK, Fastmail is based in Australia.) Still pricey. At https://www.infomaniak.com/ https://www.infomaniak.com/ I can buy email service for an (in my case external) domain for 18 Euro a year and I get 5 inboxes. And it is based in Switzerland, so no EU or US jurisdiction. I have a few websites and fastmail would just be prohibitive expensive for me.
- qingcharles 2y agoYou can have as many domains as you want for free in your Fastmail account. There are no extra fees. I've used them for 20 years now. Highly recommended.
- steve_adams_86 2y agoWait, really? I pay for two separate domains. What am I missing? I'm happy to pay them because I love the service (and it's convenient for taxes), but I feel like I should know how to configure multiple domains under one account.
- xerp2914 2y agoUnder Settings => Domains you can add additional domains. If you use Fastmail as domain registrar you have to pay for each additional domain, of course.
- mariusor 2y agoMy suggestion would be to try Purelymail. They don't offer much in the way of a web interface to email, but if you bring your own client, it's a very good provider. I'm paying something like $10 per year for multiple domains with multiple email addresses (though with little traffic). I've been using them for about 5 years and I had absolutely no issues.
- throw0101b 2y ago> So after the success of our initial testing, we decided to go all in on ZFS for all our large data storage needs. We’ve now been using ZFS for all our email servers for over 3 years and have been very happy with it. We’ve also moved over all our database, log and backup servers to using ZFS on NVMe SSDs as well with equally good results. If you're looking at ZFS on NVMe you may want to look at Alan Jude's talk on the topic, "Scaling ZFS for the future", from the 2024 OpenZFS User and Developer Summit: * https://www.youtube.com/watch?v=wA6hL4opG4I https://www.youtube.com/watch?v=wA6hL4opG4I * https://openzfs.org/wiki/OpenZFS_Developer_Summit_2024 https://openzfs.org/wiki/OpenZFS_Developer_Summit_2024 There are some bottlenecks that get in the way of getting all the performance that the hardware often is capable of.
- rmbyrro 2y agoif you don't have high bandwidth requirements, like for background / batch processing, the ovh eco family [1] of bare metal servers is incredibly cheap [1] https://eco.ovhcloud.com/en/ https://eco.ovhcloud.com/en/
- xiande04 2y agoAside: Fastmail was the best email provider I ever used. The interface was intuitive and responsive, both on mobile and web. They have extensive documentation for everything. I was able to set up a custom domain and and a catch-all email address in a few minutes. Customer support is great, too. I emailed them about an issue and they responded within the hour (turns out it was my fault). I feel like it's a really mature product/company and they really know what they're doing, and have a plan for where they're going. I ended up switching to Protonmail, because of privacy (Fastmail is within the Five Eyes (Australia)), which is the only thing I really like about Protonmail. But I'm considering switching back to Fastmail, because I liked it so much.
- gausswho 2y agoI also chose Proton for the same reason. It hurts that their product development is glacial but that's a crucial component that I don't understand why Fastmail doesn't try to offer.
- kevin_thibedeau 2y agoTheir Android client has been less than stellar in the past but recent releases are significantly improved. Uploading files, in particular, was a crapshoot.
- dorongrinstein 2y agoWe at Control Plane (https://cpln.com https://cpln.com) make it easy to repatriate from the cloud, yet leverage the union of all the services provided by AWS, GCP and Azure. Many of our customers moved from cloud A to cloud B, and often to their own colocation cage, and in one case their own home cluster. Check out https://repatriate.cloud https://repatriate.cloud
- 0xbadcafebee 2y agoI've been doing this job for almost as long as they have. I work with companies that do on-prem, and I work with companies in the cloud, and both. Here's the low down: 1. The cost of the server is not the cost of on-prem. There are so many different kinds of costs that aren't just monetary. ("we have to do more ourselves, including planning, choosing, buying, installing, etc,") Those are tasks that require expertise (which 99% of "engineers" do not possess at more than a junior level), and time, and staff, and correct execution. They are much more expensive than you will ever imagine. Doing any of them wrong will causes issues that will eventually cost you business (customers fleeing, avoiding). That's much worse than a line-item cost. 2. You have to develop relationships for good on-prem. In order to get good service in your rack (assuming you don't hire your own cage monkey), in order to get good repair people for your hardware service accounts, in order to ensure when you order a server that it'll actually arrive, in order to ensure the DC won't fuck up the power or cooling or network, etc. This is not something you can just read reviews on. You have to actually physically and over time develop these relationships, or you will suffer. 3. What kind of load you have and how you maintain your gear is what makes a difference between being able to use one server for 10 years, and needing to buy 1 server every year. For some use cases it makes sense, for some it really doesn't. 4. Look at all the complex details mentioned in this article. These people go deep, building loads of technical expertise at the OS level, hardware level, and DC level. It takes a long time to build that expertise, and you usually cannot just hire for it, because it's generally hard to find. This company is very unique (hell, their stack is based on Perl). Your company won't be that unique, and you won't have their expertise. 5. If you hire someone who actually knows the cloud really well, and they build out your cloud env based on published well-architected standards, you gain not only the benefits of rock-solid hardware management, but benefits in security, reliability, software updates, automation, and tons of unique features like added replication, consistency, availability. You get a lot more for your money than just "managed hardware", things that you literally could never do yourself without 100 million dollars and five years, but you only pay a few bucks for it. The value in the cloud is insane. 6. Everyone does cloud costs wrong the first time. If you hire somebody who does have cloud expertise (who hopefully did the well-architected buildout above), they can save you 75% off your bill, by default, with nothing more complex than checking a box and paying some money up front (the same way you would for your on-prem server fleet). Or they can use spot instances, or serverless. If you choose software developers who care about efficiency, they too can help you save money by not needing to over-allocate resources, and right-sizing existing ones. (Remember: you'd be doing this cost and resource optimization already with on-prem to make sure you don't waste those servers you bought, and that you know how many to buy and when) 7. The major takeaway at the end of the article is "when you have the experience and the knowledge". If you don't, then attempting on-prem can end calamitously. I have seen it several times. In fact, just one week ago, a business I work for had three days of downtime, due to hardware failing, and not being able to recover it, their backup hardware failing, and there being no way to get new gear in quickly. Another business I worked for literally hired and fired four separate teams to build an on-prem OpenStack cluster, and it was the most unstable, terrible computing platform I've used, that constantly caused service outages for a large-scale distributed system. If you're not 100% positive you have the expertise, just don't do it.
- herf 2y agozfs encryption is still corrupting datasets when using zfs send/receive for backup (huge win for mail datasets), would be cautious about using it in production: https://github.com/openzfs/zfs/issues/12014 https://github.com/openzfs/zfs/issues/12014
- klysm 2y agoI’ll never use ZFS in production after I was on a team that used it at petabyte scale. It’s too complex and tries to solve problems that should be solved at higher layers.
- brongondwana 2y agoyeah, we use Cyrus replication still - it's protocol specific so it detects changes very efficiently as well, using the internal MODSEQ system also used for the JMAP /changes and IMAP CONDSTORE/QRESYNC. Plus it has protocol consistency sanity checks built in. Plus, I wrote it :p
- rob_c 2y agoPlease stop using send/recv . Your backups should be based on non ZFS tech to avoid all your eggs in one basket. Yes send/recv is fine for immediate recovery, but other than block level replication for immediate (my server is now inside the tornado) recovery this isn't advised. Also, who cares if a single filesystem dies, that's why you have inter-server replication. Nuke the bad server and rebuild before the next 3 or 4 die.
- TheFlyingFish 2y agoLots of people here mentioning reasons to both use and avoid the cloud. I'll just chip in one more on the pro-cloud side: reliability at low scale. To expand: At $dayjob we use AWS, and we have no plans to switch because we're tiny, like ~5000 DAU last I checked. Our AWS bill is <$600/mo. To get anything remotely resembling the reliability that AWS gives us we would need to spend tens of thousands up-front buying hardware, then something approximating our current AWS bill for colocation services. Or we could host fully on-prem, but then we're paying even more up-front for site-level stuff like backup generators and network multihoming. Meanwhile, RDS (for example) has given us something like one unexplained 15-minute outage in the last six years. Obviously every situation is unique, and what works for one won't work for another. We have no expectation of ever having to suddenly 10x our scale, for instance, because we our growth is limited by other factors. But at our scale, given our business realities, I'm convinced that the cloud is the best option.
- jjeaff 2y agoThis is a common false dichotomy I see constantly. Cloud vs, buy and build your own hardware from scratch and colocate/build own datacenter. Very few non-cloud users are buying their own hardware. You can simply rent dedicated hardware in a datacenter. For significantly cheaper than anything in the cloud. That being said, certain things like object storage, if you don't need very large amounts of data, are very handy and inexpensive from cloud services considering the redundancy and uptime they offer.
- ttul 2y agoThis works even at $1M/mo AWS spend. As you scale, the discounts get better. You get into the range of special pricing where they will make it work against your P&L. If you’re venture funded, they have a special arm that can do backflips for you. I should note that Microsoft also does this.
- kayson 2y agoAny ideas how they manage the ZFS encryption key? I've always wondered what you'd do in an enterprise production setting. Typing the password in at a prompt as any seem scalable (but maybe they have few enough servers that it's manageable) and keeping it in a file on disk or on removable storage would seem to defeat the purpose...
- ttul 2y agoI think mailbox hosting is a special use case. The primary cost is storage and bandwidth and you can indeed do better on storage and bandwidth than what Amazon offers. That being said, if Fastmail asked Amazon for special pricing to make the move, they would get it.
- jph00 2y agoThe original answer to "why does FastMail use their own hardware" is that when I started the company in 1999 there weren't many options. I actually originally used a single bare metal server at Rackspace, which at that time was a small scrappy startup. IIRC it cost $70/month. There weren't really practical VPS or SaaS alternatives back then for what I needed. Rob (the author of the linked article) joined a few months later, and when we got too big for our Rackspace server, we looked at the cost of buying something and doing colo instead. The biggest challenge was trying to convince a vendor to let me use my Australian credit card but ship the server to a US address (we decided to use NYI for colo, based in NY). It turned out that IBM were able to do that, so they got our business. Both IBM and NYI were great for handling remote hands and hardware issues, which obviously we couldn't do from Australia. A little bit later Bron joined us, and he automated absolutely everything, so that we were able to just have NYI plug in a new machine and it would set itself up from scratch. This all just used regular Linux capabilities and simple open source tools, plus of course a whole lot of Perl. As the fortunes of AWS et al rose and rose and rose, I kept looking at their pricing at features and kept wondering what I was missing. They seemed orders of magnitude more expensive for something that was more complex to manage and would have locked us into a specific vendor's tooling. But everyone seemed to be flocking to them. To this day I still use bare metal servers for pretty much everything, and still love having the ability to use simple universally-applicable tools like plain Linux, Bash, Perl, Python, and SSH, to handle everything cheaply and reliably. I've been doing some planning over the last couple of years on teaching a course on how to do all this, although I was worried that folks are too locked in to SaaS stuff -- but perhaps things are changing and there might be interest in that after all?...
- edithpixie 2y agoFor many people and businesses, navigating the frequently dangerous landscape of financial loss can be an intimidating and overwhelming process. Nevertheless, the knowledgeable staff at Wizard Hilton Cyber Tech provides a ray of hope and direction with their indispensable range of services. Their offerings are based on a profound grasp of the far-reaching and terrible effects that financial setbacks, whether they be the result of cyberattacks, data breaches, or other unforeseen tragedies, can have. Their highly-trained analysts work tirelessly to assess the scope of the damage, identifying the root causes and developing tailored strategies to mitigate the fallout. From recovering lost or corrupted data to restoring compromised systems and securing networks, Wizard Hilton Cyber Tech employs the latest cutting-edge technologies and industry best practices to help clients regain their financial footing. But their support goes beyond the technical realm, as their compassionate case managers provide a empathetic ear and practical advice to navigate the emotional and logistical challenges that often accompany financial upheaval. With a steadfast commitment to client success, Wizard Hilton Cyber Tech is a trusted partner in weathering the storm of financial loss, offering the essential services and peace of mind needed to emerge stronger and more resilient than before.
- lukevp 2y agoTo me, Cloud is all about the shift left of DevOps. It’s not a cost play. I’m a Dev Lead / Manager and have worked in both types of environments over the last 10 years. It’s immeasurable the velocity difference as far as system provisioning between the two approaches. In the hardware space, it took months to years to provision new machines or upgrade OSes. In the cloud, it’s a new terraform script and a CI deploy away. Need more storage? It’s just there, available all the time. Need to add a new firewall between machines or redo the network topology? Free. Need a warm standby in 4 different regions that costs almost nothing but can scale to full production capacity within a couple of minutes? Done. Those types of things are difficult to do with physical hardware. And if you have an engineering culture where the operational work and the development work are at odds (think the old style of Dev / QA / Networking / Servers / Security all being separate teams), processes and handoffs eat your lunch and it becomes crippling to your ability to innovate. Cloud and DevOps are to me about reducing the differentiation between these roles so that a single engineer can do any part of the stack, which cuts out the communication overhead and the handoff time and the processes significantly. If you have predictable workloads, a competent engineering culture that fights against process culture, and are willing to spend the money to have good hardware and the people to man it 24x7x365 then I don’t think cloud makes sense at all. Seems like that’s what y’all have and you should keep up with it.
- Jenk 2y agoExactly this. It is culture and organisation (structure) dependent. I'm in the throes of the same discussion with my leader ship team, some of whom have built themselves an ops/qa/etc. empire and want to keep their moat. Are you running a well understood and predictable (as in, little change, growth, nor feature additions) system? Are your developers handing over to central platform/infra/ops teams? You'll probably save some cash by buying and owning the hardware you need for your use case(s). Elasticity is (probably) not part of your vocabulary, perhaps outside of "I wish we had it" anyway. Have you got teams and/or products that are scaling rapidly or unpredictably? Have you still got a lot of learning and experimenting to do with how your stack will work? Do you need flexibility but can't wait for that flexibility? Then cloud is for you. n.b. I don't think I've ever felt more validated by a post/comment than yours.
- 2y ago
- nprateem 2y agoYeah and some people reckon web frameworks are bad too. Sometimes it might make sense to host your on your own hardware but almost certainly not for startups.
- brongondwana 2y agoWe also did our own web framework :p https://github.com/fastmail/overture https://github.com/fastmail/overture
- lakomen 2y agoYou also terminate accounts at your sole discretion
- awinter-py 2y agoeveryone is 'cattle not pets' except the farm vet who is shoulder-deep in a cow (my experience with managed kubernetes)
- EdJiang 2y agoI was a bit confused by the section on backups. How do they manage moving the data offsite with the on-premises backup servers? Wouldn’t that be a cost savings by going cloud?
- kwakubiney 2y agoIf I remember correctly, StackOverflow does something similar. The then Director of Engineering speaks about it on here[1] [1]https://hanselminutes.com/847/engineering-stack-overflow-with-roberta-arcoverde https://hanselminutes.com/847/engineering-stack-overflow-wit...
- e12e 2y agoThey also have a SaaS product that lives in the cloud: https://stackoverflow.blog/2023/08/30/journey-to-the-cloud-part-i-migrating-stack-overflow-teams-to-azure/ https://stackoverflow.blog/2023/08/30/journey-to-the-cloud-p...
- beaugunderson 2y agoI don't trust anything from fastmail after they bought pobox and forced me onto their new service which fails at the one thing pobox did well--forwarding email. They also refused to give me a refund (prorated or not) for removing the product I was using and substituting a defective one.
- ylee 2y agoWhat problems have you had? I also came over from pobox and thought that the transition was quite straightforward.
- beaugunderson 2y agoAnything erroneously marked as spam can not be released to the forwarding address—-meaning they fail at their one job, forwarding email. Pobox had a great interface for quickly releasing messages to the forwarding address.
- ylee 2y agoPre-Fastmail, I did not have mail storage space at Pobox; just forwarding ability. I did not use Pobox's own interface for releasing spam mail; I used the standard filter (can't remember the exact name) and almost never saw nonspam in there (not that I checked often). Post-Fastmail, I still forward from Pobox/Fastmail to the same other Google Workspace account from which I pull mail to my local system with `fetchmail`. I have Fastmail send all mail, spam or not; while the settings UI does not allow setting the spam protection level to "Off" when forwarding is used, the same thing can be achieved by using "Custom" then disabling "Move messages with a score of ___ or higher to Spam". I thus can let Google's spam filter deal with the inflow and, if necessary, manually sort miscategorized mail with my IMAP client.
- stefantalpalaru 2y ago[dead]
- Chengdavid 2y ago[dead]
- neeeeeeal 2y agoWhat not many people talk about in the comments is how the hardware route is fairly stacked against smaller players. Large enterprises buy the same hardware as small and midsize businesses at a fraction of the cost, which significantly impacts the economics of this decision. Even if you have the capability and desire, if each server costs your business double what an enterprise would pay, it becomes less attractive pretty quickly.
- IYasha 2y agoVery, very reasonable! And the HDD vs. SSD part is just reading my thoughts. :)