11 ms·

So you want to build your own data center

596 points by dban 2y ago

nextworddev 2y ago
First time checking out railway product- it seems like a “low code” and visual way to define and operate infrastructure? Like, if Terraform had a nice UI?
- justjake 2y ago
  Kinda. It's like if you had everything from an infra stack but didn't need to manage it (Kubernetes for resilience, Argo for rollouts, Terraform for safely evolving infrastructure, DataDog for observability) If you've heard of serverless, this is one step farther; infraless Give us your code, we will spin it up, keep it up, automate rollouts service discovery, cluster scaling, monitoring, etc
  - thornewolf 2y ago
    for additional social proof I've been using railway since 2022 and it's been great. I host all my personal projects there and I can go from code to a url by copy-pasting my single dockerfile around.
  - nextworddev 2y ago
    Ok so you guys are serverless-ifying backend components. Like Vercel but not just for front end
teleforce 2y ago
>despite multi-million dollar annual spend, we get about as much support from them as you would spending $100 Is it a good or a bad thing to have the same customer support across the board?
dban 2y ago
This is our first post about building out data centers. If you have any questions, we're happy to answer them here :)
- gschier 2y ago
  How do you deal with drive failures? How often does a Railway team member need to visit a DC? What's it like inside?
  - justjake 2y ago
    Everything is dual redundancy. We run RAID so if a drive fails it's fine; alerting will page oncall which will trigger remote hands onsite, where we have spares for everything in each datacenter
    - gschier 2y ago
      How much additional overhead is there for managing the bare-metal vs cloud? Is it mostly fine after the big effort for initial setup?
      - ca508 2y ago
        We built some internal tooling to help manage the hosts. Once a host is onboarded onto it, it's a few button clicks on an internal dashboard to provision a QEMU VM. We made a custom ansible inventory plugin so we can manage these VMs the same as we do machines on GCP. The host runs a custom daemon that programs FRR (an OSS routing stack), so that it advertises addresses assigned to a VM to the rest of the cluster via BGP. So zero config of network switches, etc... required after initial setup. We'll blog about this system at some point in the coming months.
- coder543 2y ago
  I thought it was an interesting post, so I tried to add Railway's blog to my RSS reader... but it didn't work. I tried searching the page source for RSS and also found nothing. Eventually, I noticed the RSS icon in the top right, but it's some kind of special button that I can't right click and copy the link from, and Safari prevents me from knowing what the URL is... so I had to open that from Firefox to find it. Could be worth adding a <meta> tag to the <head> so that RSS readers can autodiscover the feed. A random link I found on Google: https://www.petefreitag.com/blog/rss-autodiscovery/ https://www.petefreitag.com/blog/rss-autodiscovery/
exabrial 2y ago
I'm surprised you guys are building new! Tons of Colocation available nearly everywhere in the US, and in the KCMO area, there are even a few dark datacenters available for sale! cool project none-the-less. Bit jealous actually :P
- gschier 2y ago
  More info on the cost comparison between all the options would be interesting
  - dban 2y ago
    We pulled some cost stuff out of the post in final review because we weren't sure it was interesting ... we'll bring it back for a future post
- justjake 2y ago
  The requirements end up being pretty specific, based on workloads/power draw/supply chain So, while we could have bought something off the shelf, that would have been suboptimal from a specs perspective. Plus then we'd have to source supply chain etc. By owning not just the servers but the whole supply chain, we have redundancy at every layer, from the machine, to the parts on site (for failures), to the supply chain (refilling those spare parts/expanding capacity/etc)
- CMCDragonkai 2y ago
  Can you share a list of dark datacenters that are for sale. They sound interesting as a business.
- idlewords 2y ago
  They're not building new, though—the post is about renting a cage in a datacenter.
ramon156 2y ago
weird to think my final internship was running on one of these things. thanks for all the free minutes! it was a nice experience
__fst__ 2y ago
Can anyone recommend some engineering reading for building and running DC infrastructure?
- ca508 2y ago
  We didn't find many good up-to-date resources online on the hardware side of things - kinda why we wanted to write about it. The networking aspect was the most mystical - I highly recommend "BGP in the datacenter" by Dinesh Dutt on that (I think it's available for free via NVidia). Our design is heavily influenced by the ideas discussed there.
  - chatmasta 2y ago
    What was the background of your team going into this project? Did you hire specialists for it (whether full time or consultants)?
    - ca508 2y ago
      We talked to a few, I think they're called MSPs? We weren't super impressed. We decided to YOLO it. There are probably great outfits out there, but it's hard to find them through the noise. We're mostly software and systems folks, but Railway is a infrastructure company so we need to own stuff down to the cage-nut - we owe it to our users. All engineering, project management and procurement is in-house. We're lucky to have a few great distributors/manufacturers who help us pick the right gear. But we learnt a lot. We've found a lot of value in getting a broker in to source our transit though. My personal (and potentially misguided) hot take is that most of the baremetal world is stuck in the early 2000's, and the only companies doing anything interesting here the likes of AWS,Google and Meta. So the only way to innovate is to stumble around, escape the norms and experiment.
      - chatmasta 2y ago
        Did your investors give you any pushback or were they mostly supportive?
jonatron 2y ago
Why would you call colocation "building your own data center"? You could call it "colocation" or "renting space in a data center". What are you building? You're racking. Can you say what you mean?
- macintux 2y ago
  Dealing with power at that scale, arranging your own ISPs, seems a bit beyond your normal colocation project, but I haven’t bee in the data center space in a very long time.
  - redeux 2y ago
    I worked for a colo provider for a long time. Many tenants arranged for their own ISPs, especially the ones large enough to use a cage.
  - latchkey 2y ago
    One of the many reasons we went with Switch for our DC is because they have a service to handle all of that for you. Having stumbled on doing this ourselves before, it can be pretty tricky to negotiate everything. We had one provider give us a great price and then bait and switch at the last moment to tell us that there is some other massive installation charge that they didn't realize we had to pay. Switch Connect/Core is based off the old Enron business that Rob (CEO) bought... https://www.switch.com/switch-connect/ https://www.switch.com/switch-connect/ https://www.switch.com/the-core-cooperative/ https://www.switch.com/the-core-cooperative/
- xiconfjs 2y ago
  I have to second this. While it takes mich effort and in-depth knowledge do build up from an “empty” cage it’s still far from dealing with everything from building permits, to plan and realize a data center to code including redundant power lines, AC and fibre. Still kudos going this path in the cloud-centric time we live in.
  - j45 2y ago
j-b 2y ago
Love these kinds of posts. Tried railway for the first time a few days ago. It was a delightful experience. Great work!
- justjake 2y ago
  Thank you! Anything you think we can do better?
coolkil 2y ago
Awesome!! Hope to see more companies go this route. I had the pleasure to do something similar for a company(lot smaller scale though) It was my first job out of university. I will never forget the awesome experience of walking into the datacenter and start plugging cables and stuff
sitkack 2y ago
It would be nice to have a lot more detail. The WTF sections are the best part. Sounds like your gear needs "this side towards enemy" sign and/or the right affordances so it only goes in one way. Did you standardize on layout at the rack level? What poke-yoke processes did you put into place to prevent mistakes? What does your metal->boot stack look like? Having worked for two different cloud providers and built my own internal clouds with PXE booted hosts, I too find this stuff fascinating. Also take utmost advantage of a new DC when you are booting it to try out all the failure scenarios you can think of and the ones you can't through randomized fault injection.
- ca508 2y ago
  > It would be nice to have a lot more detail I'm going to save this for when I'm asked to cut the three paras on power circuit types. Re: standardising layout at the rack level; we do now! we only figured this out after site #2. It makes everything so much easier to verify. And yeah, validation is hard - manually doing it thus far; want to play around with scraping LLDP data but our switch software stack has a bug :/. It's an evolving process, the more we work with different contractors, the more edge cases we unearth and account for. The biggest improvement is that we have built a internal DCIM that templates a rack design and exports a interactive "cabling explorer" for the site techs - including detailed annotated diagrams of equipment showing port names, etc... The screenshot of the elevation is a screenshot of part of that tool. > What does your metal->boot stack look like? We've hacked together something on top of https://github.com/danderson/netboot/tree/main/pixiecore https://github.com/danderson/netboot/tree/main/pixiecore that serves a debian netboot + preseed file. We have some custom temporal workers to connect to Redfish APIs on the BMCs to puppeteer the contraption. Then a custom host agent to provision QEMU VMs and advertise assigned IPs via BGP (using FRR) from the host. Re: new DCs for failure scenarios, yeah we've already blown breakers etc... testing stuff (that's how we figured out our phase balancing was off). Went in with a thermal camera on another. A site in AMS is coming up next week and the goal for that is to see how far we can push a fully loaded switch fabric.
  - sitkack 2y ago
    Wonderful! The edge cases are the gold btw, collect the whole set and keep them in a human and machine readable format. I'd also go through and using a color coded set of cables, insert bad cables (one at a time at first) while the system is doing an aggressive all to all workload and see how quickly you can identify faults. It is the gray failures that will bring the system down, often multiple as a single failure will go undetected for months and then finally tip over an inflection point at a later time. Are you workloads ephemeral and/or do they live migrate? Or will physical hosts have long uptimes? It is nice to be able to rebaseline the hardware before and after host kernel upgrades so you can detect any anomalies. You would be surprised about how larger of a systemic performance degradation that major cloud providers have been able to see over months because "all machines are the same", high precision but low absolute accuracy. It is nice to run the same benchmarks on bare metal and then again under virtualization. I am sure you know, but you are running a multivariate longitudinal experiment, science the shit out of it.
deleted 2y ago
[deleted]
aetherspawn 2y ago
What brand of servers was used?
- neilv 2y ago
  Looks like Supermicro.
  - justjake 2y ago
    Winner winner chicken dinner!
  - aetherspawn 2y ago
    Where do you buy this, direct from Supermicro? Asking as a Dell customer… our servers are $$$
    - ca508 2y ago
      We have a distributor we work with - just because it makes import/export a lot easier. But we get to interface directly with Supermicro for the technical/design stuff, and they're super awesome. If you're looking in the US, reach out to their eStore - really great fuss-free turnaround and all direct.
- emmelaich 2y ago
  Yes, considering the importance of the power draw, I wondered if ARM servers were used.
  - ca508 2y ago
    oh yes we want to; I even priced a couple out. Most of the SKUs I found were pretty old, and we couldn't find anything compelling to risk deploying at the scale we wanted. It's on the wishlist, and if the right hardware comes along; we'll rack it up even as a bet. We maintain Nixpacks (https://nixpacks.com/docs/getting-started https://nixpacks.com/docs/getting-started), so for most of our users we could rebuild most their apps for ARM seamlessly - infact we mostly develop our build systems on ARM (because macbooks). One day.
sometalk 2y ago
I remember talking to Jake a couple of years ago when they were looking for someone with a storage background. Cool dude, and cool set of people. Really chuffed to see them doing what they believe in.
- justjake 2y ago
  Thanks dude <3. We are indeed doing the thing :D
mirshko 2y ago
y’all really need to open source that racking modeling tool, that would save sooooo many people so much time
- devenson 2y ago
  Not OSS but I have developed a tool for modeling racks and much more. https://schematix.com/video/racks https://schematix.com/video/racks
jdoss 2y ago
This is a pretty decent write up. One thing that comes to mind is why would you write your own internal tooling for managing a rack when Netbox exists? Netbox is fantastic and I wish I had this back in the mid 2000s when I was managing 50+ racks. https://github.com/netbox-community/netbox https://github.com/netbox-community/netbox
- ca508 2y ago
  we evaluated a lot of commercial and oss offerings before we decided do go build it ourselves - we still have a deploy of netbox somewhere. But our custom tool (Railyard) works so well because it integrates deeply into the our full software, hardware and orchestration stack. The problem with the OSS stuff is that it's almost too generic - you shape the problem to fit its data model vs. solve the problem. We're likely going to fold our tool into Railway itself eventually - want to go on-prem; button click hardware design, commission, deploy and devex. Sorta like what Oxide is doing, but approaching the problem from the opposite side.
- matt-p 2y ago
  It is not that difficult to build it into your app, if you're already storing information about hosts, networking etc. All you're really doing is expanding the scope, netbox is a fine starting point if you're willing to start there and build your systems around it, but if you've already got a system (or you need to do anything that doesn't fit netbox logic) you're probably better off just extending it. In this case railway will need to care about a lot of extra information beyond just racks, IP addresses and physical servers.
  - ca508 2y ago
    correct; I think the first version of our tool sprung up in the space of a couple of weekends. It wasn't planned, my colleague Pierre who wrote it just had a lot of fun building it.
    - walterbell 2y ago
      Were there any promising OSS alternatives to Netbox?
robertclaus 2y ago
I would be super interested to know how this stuff scales physically - how much hardware ended up in that cage (maybe in Cloud-equivalent terms), and how much does it cost to run now that it's set up?
renewiltord 2y ago
More to learn from the failures than the blog haha. It tells you what the risks are with a colocation facility. There really isn't any text on how to do this stuff. The last time I wanted to build out a rack there aren't even any instructions on how to do cable management well. It's sort of learned by apprenticeship and practice.
toddmorey 2y ago
Reminds me of the old Rackspace days! Boy we had some war stories: - Some EMC guys came to install a storage device for us to test... and tripped over each other and knocked out an entire Rack of servers like a comedy skit. (They uh... didn't win the contract.) - Some poor guy driving a truck had a heart attack and the crash took our DFW datecenter offline. (There were ballards to prevent this sort of scenario, but the cement hadn't been poured in them yet.) - At one point we temporarily laser-beamed bandwidth across the street to another building - There was one day we knocked out windows and purchased box fans because servers were literally catching on fire. Data center science has... well improved since the earlier days. We worked with Facebook on the OpenCompute Project that had some very forward looking infra concepts at the time.
- lostlogin 2y ago
  > There was one day we knocked out windows and purchased box fans because servers were literally catching on fire. Pointing the fans in or out?
  - sitkack 2y ago
    You want to point them in.
    - gr3ml1n 2y ago
      The datacenters I've been in with emergency cooling fans in the walls all exhaust out, not in. Easier to get portable CRACs inside and get a good draft going.
- wgx 2y ago
  I recall getting a DC tour of LON3 and being totally blown away by it all as a 20-something web dev. Good times.
  - chasd00 2y ago
    When I was in college I’d call up datacenters pretending to be a prospective customer and schedule a tour. I was totally fascinated by them and knew enough to sound legit, it was like going to an amusement park for me.
linsomniac 2y ago
Was really hoping this was was actually about building your own data center. Our town doesn't have a data center, we need to go an hour south or an hour north. The building that a past failed data center was in (which doesn't bode well for a data center in town, eh?), is up for lease and I'm tempted. But, I'd need to start off small, probably per-cabinet UPSes and transfer switches, smaller generators. I've built up cabinets and cages before, but never built up the exterior infrastructure.
- hinkley 2y ago
  Why did it fail would be my question. If it turns out to be any of “location, location, location” then getting a partially kitted out building may not help you. Did they get independent data into the building via different routes? How’s the power? Could be the data was coming in through a route that sees frequent construction. I knew a guy who ran the IT dept for a university and he discovered that the excavation crews found it was cheaper to maybe have to pay a fine for cutting data lines than it was to wait for them to be marked accurately. He spent a lot of time being stressed out.
  - linsomniac 2y ago
    I agree that one of the first steps would be to take someone from the previous facility out for a meal, which I can probably arrange fairly easily. I don't exactly know why it failed, it was run by the biggest local ISP. I can speculate about why they failed (DSL speeds are severely limited in our town, so really Xfinity was it, they tried providing fiber in some locations, but found it hard to keep up with fiber locate calls). The Colocation side of the business was never very big, but it's not clear if that is because there's not demand or that they just never really pushed it. Location is fairly good, as far as data centers go. It's got relatively good network connectivity, I believe, but I don't have specifics about entrances and diversity. It is close to one of the big fiber rings around the city, I believe the ring is pulled into the facility. I don't know if they had telco fiber in, or backhauled it via the fiber ring. Power is probably good, but not great -- I'd doubt it's fed from multiple substations. There was, at one point, some generator bays. While I could use data center space in town, it'd be hard to convince my work to move, partly as we just signed a 3 year agreement for hosting 60 miles away, partly just because of the cost of a move. It probably should remain a pipe dream.
pixelesque 2y ago
The date and time durations given seem a bit confusing to me... "we kicked off a Railway Metal project last year. Nine months later we were live with the first site in California". seems inconsistent with: "From kicking off the Railway Metal project in October last-year, it took us five long months to get the first servers plugged in" The article was posted today (Jan 2025), was it maybe originally written last year and the project has been going on for more than a year, and they mean that the Railway Metal project actually started in 2023?
- ca508 2y ago
  ah that's my bad - I wrote this in Dec, we only published in Jan. Obv. missed updating that. Timeline wise; - we decided to go for it and spend the $$$ in Oct '23 - Convos/planning started ~ Jan '24 - Picked the vendors we wanted by ~ Feb/Mar '24 - Lead-times, etc... meant everything was ready for us to go fit the first gear by mostly ourselves at the start of May (that's the 5mo) - We did the "proper" re-install around June, followed closely by the second site in ~ Sep, around when we started letting our users on it as a open beta - Sep-Dec we just doubled down on refining software/automation and process while building out successive installs Lead times can be mind numbing. We have certain switches from Arista that have a 3-6 mo leadtime. Servers are build to order, so again 2+ months depending on stock. And obv. holidays mean a lot of stuff shuts down around December. Sometimes you can swap stuff around to get better lead-times, but then the operational complexity explodes because you have this slightly different component at this one site. I used to be a EEE, and I thought supply chain there was bad. But with DCs I think it's sometimes worse because you don't directly control some parts of your BoM/supply chain (especially with build-to-order servers).
  - Twirrim 2y ago
    From working at a cloud, and speaking with capacity folks regularly when I was in certain roles, the supply chain strikes me as one of the biggest nightmares. Even at scale when vendors really, really want (or want to keep) your business. At times it almost seems like someone sneezes somewhere and whoops, there goes your hardware delivery timelines. The advantage at cloud scale is a lot of constant signal around capacity delivery, demand etc. so you can build mathematical models to best work out when to start placing orders, and for what.
matt-p 2y ago
If you’re using 7280-SR3 switches, they’re certainly a fine choice. However, have you considered the 7280-CR3(K) range? They're much better $/Gbps and more relevant edge interfaces. At this scale, why did you opt for a spine-and-leaf design with 25G switches and a dedicated 32×100G spine? Did you explore just collapsing it and using 1-2 32×100G switches per rack, then employing 100G>4×25G AOC breakout cables and direct 100G links for inter-switch connections and storage servers? Have you also thought about creating a record on PeeringDB?https://www.peeringdb.com/net/400940 https://www.peeringdb.com/net/400940. By the way, I’m not convinced I’d recommend a UniFi Pro for anything, even for out-of-band management.
- ca508 2y ago
  All valid points - and our ideas for Gen 2 sound directionally similar - but those are at crayon drawing stage. When we started, we didn't have much of an idea about what the rack needs to look like. So we chose a combination of things we thought we could pull this off. We're mostly software and systems folks, and there's a dearth of information out there on what to do. Vendors tend to gravitate towards selling BGP+EVPN+VXLAN or whatever "enterprise" reference designs; so we kinda YOLO'ed the Gen 1. We decided to spend extra money if we could get to a working setup sooner. When the clock is in cloud spend, there's uh... lots of opportunity cost :D. A lot of the chipset and switch choices were bets and we had to pick and choose what we gambled on - and what we could get our hands on. The main bets this round were eBGP to the hosts with BGP unnumbered, SONiC switches - this lets us do a lot of networking with our existing IPv6/Wireguard/eBPF overlay and a debian based switch OS + FRR (so fewer things to learn). And ofc. figuring out how to operationalise the install process and get stuff running on the hardware as soon as possible. Now we've got a working design, we'll start iterating a bit more on the hardware choice and network design. I'd love for us to write about it when we get through it. Plus I think we owe the internet a rant on networking in general. Edit: Also we don't use UniFi Pro / Uniquity gear anywhere?
chatmasta 2y ago
This is how you build a dominant company. Good for you ignoring the whiny conventional wisdom that keeps people stuck in the hyperscalers. You’re an infrastructure company. You gotta own the metal that you sell or you’re just a middleman for the cloud, and always at risk of being undercut by a competitor on bare metal with $0 egress fees. Colocation and peering for $0 egress is why Cloudflare has a free tier, and why new entrants could never compete with them by reselling cloud services. In fact, for hyperscalers, bandwidth price gouging isn’t just a profit center; it’s a moat. It ensures you can’t build the next AWS on AWS, and creates an entirely new (and strategically weaker) market segment of “PaaS” on top of “IaaS.”
- justjake 2y ago
  Yup. Bingo. We've had to pass the cloud egress costs onto our customers, which sucks. With this, it'll mean we can slash that in half, lower storage costs, remove "per seat" pricing, etc Super exciting
  - MasterScrat 2y ago
    How do bandwidth costs work now? do you pay the ISPs a flat fee, or is it still usage-based? how much cheaper is it compared to cloud providers?
    - kazen44 2y ago
      in my experience (Eu, datacenter/ISP space) connectivity is either sold based on 99percentile commitments (aka, you pay for sending us this amount of traffic, anything over this gets you billed extra) or based on a minimum commitment for the traffic you send. (atleast X amount of bandwith) or it is based on a flat free principle where you pay an upfront cost for the setup and the rest is a base price for X amount of bits per seconds. It depends a lot on what kind of connection you require, and things like oversubscription and congestion control also come into play. Peering ports with IXP's are usually flat rate, while ports in datacenters to end customers usually have more complex constructs. Hyperscaler bandwith is notoriously expensive. for instance, a 100Gbps port on the AMS-IX is 2500$[1]. Now, you need to account for extra costs to actually use this port (some IP space, ASN number etc) but even with all that added up i think you will not get much more expensive then 400$ per month averaged in total over a year. Now what makes comparing difficult is that hyperscalers are not transparant when it comes to connectivity costs. Looking at egress fees for example: AWS seems to charge 1 cent per Gigabyte transferred for egress fees. If we send data at line rate across our 100Gbps for an entire month we get the following: 100gbps = 12.5Gigabytes per second. 12.5 * 2 629 743 83 (number of seconds in a month) = 32871797875 Gigabytes 32871797875 / 0,001$ = 3.287.179,7875 thats 3,2 million dollars... compared to roughly 4000! AWS also seems to offer "dedicated connections". at roughly 22 dollars per hour [3] (no clue if this is even comparble to an IXP port, but the comparison would still be fun to make). 22$ x 720 (hours per month = 15.840$, or roughly 3times the IXP port price. In both cases, you are getting absolutely shafted by egress prices at cloud providers compared to doing it yourself. [1] https://www.ams-ix.net/ams/pricing-us https://www.ams-ix.net/ams/pricing-us [2] https://aws.amazon.com/ec2/pricing/on-demand/ https://aws.amazon.com/ec2/pricing/on-demand/ [3] https://aws.amazon.com/directconnect/pricing/ https://aws.amazon.com/directconnect/pricing/
dylan604 2y ago
My first colo box came courtesy of a friend of a friend that worked for one of the companies that did that (leaving out names to protect the innocent). It was a true frankenputer built out of whatever spare parts he had laying around. He let me come visit it, and it was an art project as much as a webserver. The mainboard was hung on the wall with some zip ties, the PSU was on the desk top, the hard drive was suspended as well. Eventually, the system was upgraded to newer hardware, put in an actual case, and then racked with an upgraded 100base-t connection. We were screaming in 1999.
cyberax 2y ago
It looked interesting, until I got to the egress cost. Ouch. $100 per TB is way too much if you're using bandwidth-intensive apps. Meta-comment: it's getting really hard to find hosting services that provide true unlimited bandwidth. I want to do video upload/download in our app, and I'm struggling to find providers of managed servers that would be willing to provide me with fixed price for 10/100GB ports.
- justjake 2y ago
  FWIW, we just pass the costs on from the current cloud providers. Doing this work will let us lower those egress prices!
  - cyberax 2y ago
    Yeah. Cloud providers are the worst. Their egress costs moved from "expensive but not unreasonable" circa 2010, to "what the fuck" territory now. A 10G port should be in the range of $2k per month, I believe? I don't mind paying that much.
    - kazen44 2y ago
      a 100g port is in the realm of 2K per month at IXP's.
Havoc 2y ago
Surprised to see pxe. Didn’t realise that was in common use in racks
- jpleger 2y ago
  Are there any alternatives these days? Or just that you weren't expecting to have systems boot off the network?
  - Havoc 2y ago
    The later. I was expecting local boot because pxe introduces a rather big dependency for potentially many machines. Issues with network or issues with pxe server and nothing boots
- louwrentius 2y ago
  Booting through the IPMI with virtual media isos over http is dog slow in my experience. Using PXE to bootstrap an installer kernel (only few MB) over TFTP that fetches the rest of the OS over HTTP is quick and you can pressed/kickstart a machine in minutes.
enahs-sf 2y ago
Curious why California when the kwh is so high here vs Oregon or Washington
physhster 2y ago
Having done data center builds for years, mostly on the network side but realistically with all the trades, this is a really cool article.
jpleger 2y ago
Makes me remember some of the days I had in my career. There were a couple really interesting datacenter things I learned by having to deploy tens of thousands of servers in the 2003-2010 timeframe. Cable management and standardization was extremely important (like you couldn't get by with shitty practices). At one place where we were deploying hundreds of servers per week, we had a menu of what ops people could choose if the server was different than one of the major clusters. We essentially had 2 chassis options, big disk servers which were 2u or 1u pizza boxes. You then could select 9/36/146gb SCSI drives. Everything was dual processor with the same processors and we basically had the bottom of the rack with about 10x 2u boxes and then the rest was filled with 20 or more 1u boxes. If I remember correctly we had gotten such an awesome deal on the price for power, because we used facility racks in the cage or something, since I think they threw in the first 2x 30 amp (240v) circuits for free when you used their racks. IIRC we had a 10 year deal on that and there was no metering on them, so we just packed each rack as much as we could. We would put 2x 30s on one side and 2x 20s on another side. I have to think that the DC was barely breaking even because of how much heat we put out and power consumption. Maybe they were making up for it in connection / peering fees. I can't remember the details, will have to check with one of my friends that worked there around that time.
Over2Chars 2y ago
I guess we can always try to re-hire all those "Sys Admins" we thought we could live without. LOL?
whalesalad 2y ago
Cliffhanger! Was mostly excited about the networking/hypervisor setup. Curious to see the next post about the software defined networking. Had not heard of FRR or SONIC previously.
- ca508 2y ago
  the good news on this is that we've got a tonne of deep-dive material on networking and whitebox switches we cut from this post. We'll definitely be talking more about this soon (also cos' BGP is cool).
ChuckMcM 2y ago
From the post: "...but also despite multi-million dollar annual spend, we get about as much support from them as you would spending $100." -- Ouch! That is a pretty huge problem for Google. I really enjoyed this post, mostly because we had similar adventures when setting up the infrastructure for Blekko. For Blekko, a company that had a lot of "east west" network traffic (that is traffic that goes between racks vs to/from the Internet at large) having physically colocated services without competing with other servers for bandwidth was both essential and much more cost effective than paying for this special case at SoftLayer (IBM's captive cloud). There are some really cool companies that will build an enclosure for your cold isle, basically it ensures all the air coming out of the floor goes into the back of your servers and not anywhere else. It also keeps not cold air from being entrained from the sides into your servers. The calculations for HVAC 'CRAC' capacity in a data center are interesting too. In the first CoLo facility we had a 'ROFOR' (right of first refusal) on expanding into the floor area next to our cage, but when it came time to expand the facility had no more cooling capacity left so it was meaningless. Once you've done this exercise, looking at the 0xide solution will make a lot more sense to you.
sour-taste 2y ago
I used to work on machine repair automation at a big tech company. IMO repairs are one of the overlooked and harder things to deal with. When you run on AWS you don't really think about broken hardware it mostly just repairs itself. When you do it yourself you don't have that luxury. You need to have spare parts, technician to do repairs, a process for draining/undraining jobs off hosts, testing suites, hardware monitoring tools and 1001 more things to get this right. At smaller scales you can cut corners on some of these things but they will eventually bite you. And this is just machines! Networking gear has it's own fun set of problems, and when it fails it can take down your whole rack. How much do you trust your colos not to lose power during peak load? I hope you run disaster recovery drills to prep for these situations! Wishing all the best to this team, seems like fun!
nyrikki 2y ago
> This will likely involve installing some overhead infrastructure and trays that let you route fiber cables from the edge of your cage to each of your racks, and to route cables between racks Perhaps I am reading this wrong, as you appear to be fiber heavy and do have space on the ladder rack for copper, but if you are commingling the two, be careful. A possible future iteration, would consider a smaller panduit fiberunner setup + a wire rack. Co-mingling copper and fiber, especially through the large spill-overs works until it doesn't. Depending on how adaptive you need to be with technology changes, you may run into this in a few years. 4x6 encourages a lot of people putting extra cable up in those runners, and sharing a spout with cat-6, cx-#, PDU serial, etc... will almost always end badly for some chunk of fiber. After those outages it also encourages people to 'upgrade in place'. When you are walking to your cage look at older cages, notice the loops sticking out of the tops of the trays and some switches that look like porcupines because someone caused an outage and old cables are left in place. Congrats on your new cage.
hintymad 2y ago
Per my experience with cloud, the most powerful Infra abstraction that AWS offers is actually EC2. The simplicity of getting a cluster of machines up and running with all the metadata readily available via APIs is just liberating. And it just works: the network is easy to configure, the ASGs are flexible enough to customize, and the autoscaling offers strong primitives for advanced scaling. Amazingly, few companies who run their own DCs could build anything comparable to EC2, even at a smaller scale. When I worked in those companies, I sorely missed EC2. I was wondering if there's any robust enough open-source alternatives to EC2's control-plane software to manage baremetals and offer VMs on top them. That'll be awesome for companies that build their own DCs.
- mdaniel 2y ago
  I believe eBay still runs on OpenStack, which as far as I know even has as ec2-compatible emulation layer https://docs.openstack.org/api-ref/ec2-api/#supported-features-and-limitations https://docs.openstack.org/api-ref/ec2-api/#supported-featur...
winash83 2y ago
We went down this path over the last year, lots of our devs need local and dev/test environments and AWS was costing us a bomb, With about 7 Bare metals(Colocation) we are running about 200+ VMs and could double that number with some capacity to spare. For management, we built a simple wrapper over libvirt. I am setting up another rack in the US and will end up costing around $75Kper year for a similar capacity. Our prod is on AWS but we plan to move everything else and it's expected to save at least a quarter of a million dollars per year
- cbg0 2y ago
  Sounds like a good chunk of money saved, but are you getting the same level of redundancy as you did on the cloud?
  - winash83 2y ago
    For most dev/test workflows redundancy is not a huge concern, because we can just recreate the environments, in practice things are quite stable, most HW vendors like Hp Dell etc let you rent the servers instead of buying them, in case of serious HW issues they take care of the fixes, and usually there is someone at the Colocation site to take care of the day to day
motoboi 2y ago
I’m my experience and based on writeups like this: Google hates having customers. Someone decided they have to have a public cloud, so they did it, but they want to keep clients away with a 3 meter pole. My AWS account manager is someone I am 100% certain would roll in the mud with me if necessary. Would sleep in the floor with us if we asked in a crisis. Our Google cloud representatives make me sad because I can see that they are even less loved and supported by Google than us. It’s sad seeing someone trying to convince their company to sell and actually do a good job providing service. It’s like they are setup to fail. Microsoft guys are just bulletproof and excel in selling, providing a good service and squeezing all your money out of your pockets and you are mortally convinced it’s for your own good. Also have a very strange cloud… thing. As for the railway company going metal, well, I have some 15 years of experience with it. I’ll never, NEVER, EVER return to it. It’s just not worth it. But I guess you’ll have to discover it by yourselves. This is the way. You soon discover what in freaking world is Google having so much trouble with. Just make sure you really really love and really really want to sell service to people, instead of building borgs and artificial brains and you’ll do 100x better.
- motorest 2y ago
  > Microsoft (...) have a very strange cloud… thing. Risking a going off on a tangent, this is something I rarely see discussed but is perhaps one of the main problems with Azure. The whole cloud service feels like something someone oblivious to cloud computing would design if all they knew was renting bare metal servers. It's cloud computing in a way that completely defeats the whole concept of cloud computing.
  - yeahwhatever10 2y ago
    Can you be more specific?
    - motorest 2y ago
      > Can you be more specific? To start off, read up on the ass-backwards concept of an app service plan. That nonsense is the ultimate anti-cloud computing approach to cloud computing. https://learn.microsoft.com/en-us/azure/app-service/overview-hosting-plans https://learn.microsoft.com/en-us/azure/app-service/overview...
scarab92 2y ago
Interesting that they call out the extortionate egress fees from the majors as a motivation, but are nevertheless also charging customers $0.10 per GB themselves.
- wmf 2y ago
  Bezos: Your margin is my opportunity. Railway: No, your margin is my opportunity.
  - ca508 2y ago
    We currently pass on our cloud egress costs to users via the current pricing. We'll be publishing a pricing update soon as part of our migration - and egress [and some other things] will be coming down.
- justjake 2y ago
  That $0.10 per GB is direct pass along for the cloud ingress fees We can lower that once we’re fully on metal
ThinkBeat 2y ago
1. Is the impression they decided to use a non datacenter location to put their datacenter, If so that is not a good idea. 2. Geographical distanced backups, if the primary fails. Without this you are already in trouble. What happens if the buildings burns down? 3. Hooking up with "local" ISPs That seems ok. As long as ISP failing is easily and autoamically dealt with. 4. I am a bit confused about what happens at the edge. On the one head it seems like you have 1 datacenter, and ISPs doing routing, other places I get the impression you have compute close to the edge. Which is it?
- wmf 2y ago
  1. No, they're using a cage inside a real data center in Ashburn VA which is basically data center city. 2. In the diagram you can see site 1 and site 2. 3. Yes, routers automatically deal with ISP failures.
technick 2y ago
I've spent more time than I care working in data centers and can tell you that your job req is asking for one person to perform 3 different roles, maybe 4. I guarantee you're going to find a "jack of all trades" and a master of none unless you break them out into these jobs. Application Developer DevOps Engineer Site Reliability Engineer Storage Engineer Good luck, hope you pay them well.
- concerndc1tizen 2y ago
  There's no reason to attack generalists.
random_savv 2y ago
I guess there's another in-between step buying your own hardware, even when merely "leasing individual racks", and EC2 instances: dedicated bare metal providers like Hetzner. This lets one get closer to the metal (e.g. all your data is on your specific disk, rather than an abstracted block storage, such as EBS, not shared with other users, cheaper, etc) without having to worry about the staff that installs the hardware or where/how it fits in a rack. For us, this was a way to get 6x performance for 1/6 of the cost. (Excluding, of course our time, but we enjoyed it!)
- t0mas88 2y ago
  Hetzner is very good for low cost high bandwidth things that don't need a serious SLA. But if you're selling a platform like Railway.com the stability and flexibility of Hetzner aren't going to be good enough.
- dboreham 2y ago
  Agreed. We run our own bare metal in a rack, but also rent machines from Hivelocity where the use case suits.
throwaway2037 2y ago
I promise my comment is not intended to troll. Why didn't you use Oxide pre-built racks? Just the power efficiency seems like a huge win.
- ca508 2y ago
  It's a fair question. What Oxide are building is cool, but it's too custom/monolithic for us to risk. We're more likely to look at OCP racks/chassis down the road.
solarkraft 2y ago
Cool post and cool to see Railway talked about more on here. I‘ve used their postgres offering for a small project (crucially it was accessible from the outside) and not only was setting it up a breeze, cost was also minimal (I believe staying within the free tier). I haven’t used the rest of the platform, but my interaction with them would suggest it would probably be pretty nice.
concerndc1tizen 2y ago
@railway What would you say are your biggest threats? Power seems to the big one, especially when the AI power and electric vehicle demand will drive up kWh prices. Networking seems another one. I'm out of the loop, but it seems to me like the internet is still stuck at 2010 network capacity concepts like "10Gb". If networking had progressed as compute power has (e.g. NVMe disks can provide 25GB/s), 100Gb would be the default server interface? And the ISP uplink would be measured in terabits? How is the diversity in datacenter providers? In my area, several datacenters were acquired and my instinct would be that: the "move to cloud" has lost smaller providers a lot of customers, and the industry consolidation has given suppliers more power in both controlling the offering and the pricing. Is it a free market with plenty of competitive pricing, or is it edging towards enshittification?
- kazen44 2y ago
  > Networking seems another one. I'm out of the loop, but it seems to me like the internet is still stuck at 2010 network capacity concepts like "10Gb". If networking had progressed as compute power has (e.g. NVMe disks can provide 25GB/s), 100Gb would be the default server interface? And the ISP uplink would be measured in terabits? High end network interfaces are entering the 800Gbps interface era right now. also, in 2010 10Gbps network connectivity to end hosts was NOT common. (it was common for router uplinks and interconnects though. Network interfaces have not scaled as nicely because getting fast enough lasers to handle higher then 100Gbps has been a challenge, and getting to higher speeds basically means doing wave division multiplex over multiple channels across a single fiber. Also, density of connections per fiber has increased massivly because the cost of DWDM equipment has come down significantly.
yread 2y ago
Useful article. I was almost planning to rent a rack somewhere but it seems there's just too much work and too many things to go wrong and it's better to rent cheap dedicated servers and make it somebody elses problem
ksec 2y ago
I am just fascinated by the need of Datacenter. The scale is beyond comprehension. 10 years ago, before the word HyperScaler was even invented or popularised, I would have thought DC market to be on the decline or levelled off now or around this time. One reason being HyperScaler, AWS, Google, Microsoft, Meta, Apple, Tencent, Alibaba, to smaller ones like Oracle and IBM. They would all have their own DC, taking on much of the compute for themselves and others. While left over space would be occupied by third parties. Another reason being the compute, memory and storage density continue to increase, which means for the same amount of floor space we are offering 5 - 20x of the previous CPU / RAM / Storage. Turns out we are building like mad and we are still not building enough.
- kazen44 2y ago
  just look at the insane amount of computers that currently exist and need to communicate to do data processing. I remember 30 years ago most people used one single computer for the family at home, half of the people i knew didn't have proper internet access. (and this is from a western perspective, the rest of the world was even far less digitized). Now look at how much networked computers are around you. - your phone - one or multiple TV's - laptops/desktops etc - smart home appliances. and this is just looking at the very small sample size of a normal household, add up to that things like the digitizalisation of factories, the digitalization of the rest of the world. (internet access has grown massively in the past 15 years in the developing world). We have far more computers then a decade ago, and far more people have them aswell, and it shows very little signs of stopping. IPv6 for instance, supports an absolutely infanthomable IP address space. (which people often seem to think is overkill), but looking at the past growth, i think having suchs a large address space is a wise choice. Another thing which people seem to not notice is that a lot of older DC's are being phased out, mainly because these facilities are repurposed telephone exchanges and far less suitable for more power hungry computing power.
  - ksec 2y ago
    >We have far more computers then a decade ago, Yes that was already taken into account. We rushed pass active usage of Smartphone of 4B in 2020. With additional 1B user who have access to 4G / Smartphone but not using any Data. 1B people who cant afford it or are using feature phone. And around 1B people who are outside the age of using smartphone, child, baby etc. That is a total Around 7B people already. And anything after is a long tail of new generation outpacing older generation. Tablet usage has levelled off. PC market hasn't had new growth outside China and India. COVID managed to fasten a lot these digitalisation. I wrote about how the Growth of AWS in 2023 was roughly equal to doubling of itself in 2016. i.e 2023 AWS was building out the size of AWS 2016. That is insane. And yet we are still building more. >Another thing which people seem to not notice is that a lot of older DC's are being phased out, That is something I am not aware. But certainly will keep a look out. This got me thinking if building out a new DC is easier and cheaper than repurposing older DC not designed for high computing power density usage? While I have said we could increase Compute, RAM , Storage density in Rack by 10-20x, we have also increase power usage by 4 - 5x. Not only electricity / power usage but cooling and design also requires additional thoughts.
blmt 2y ago
I am really thankful for this article as I finally get where my coworkers get "wrong" notions about three-phase power use in DC: >The calculations aren’t as simple as summing watts though, especially with 3-phase feeds — Cloudflare has a great blogpost covering this topic. What's written in the Cloudflare blogpost linked in the article holds true only of you can use a Delta config (as done in the US to obtain 208V) as opposed to the Wye config used in Europe. The latter does not give a substantial advantage: no sqrt(3) boost to power distribution efficiency and you end up adding Watts for three independent single phase circuits (cfr. https://en.m.wikipedia.org/wiki/Three-phase_electric_power https://en.m.wikipedia.org/wiki/Three-phase_electric_power).
Bluecobra 2y ago
Good writeup! Google really screws you when you are looking for 100G speeds, it's almost insulting. For example redundant 100G dedicated interconnects are about $35K per month and that doesn't include VLAN attachments, colo x-connect fees, transit, etc. Not only that, they max out on 50G for VLAN attachments. To put this cost into perspective, you can buy two brand new 32 port 100G switches from Arista for the same amount of money. In North America, you can get 100G WAN circuits (managed Wavelength) for less than $5K/month. If it's a local metro you can also get dark fiber for less and run whatever speed you want.
- kazen44 2y ago
  also, buy some DWDM equipment and you can easily scale those darkfibers to offer multiple 100GBPS connections for very little cost.
a1o 2y ago
Excellent write up! This is not the first blog post I see in recent times on going to owning infrastructure direction, but it is certainly well written and I liked the use of Excel in it, a good use, although visually daunting!
lifeinthevoid 2y ago
Man, I get an anxiety attack just thinking about making this stuff work. Kudos to all the people doing this.
- mdaniel 2y ago
  My experience has been that with most things, making it work is often simple, keeping it working is where people start to get mega bucks for having the required experience
arschfick 2y ago
[dead]
kolanos 2y ago
As someone who lost his shirt building a data center in the early 2000s, Railway is absolutely going about this the right way with colocation.
maxclark 2y ago
There's places where it makes sense to be on the cloud, and places where it doesn't. The two best examples I can give are for high bandwidth, or heavy disk intensive applications. Take Netflix. While almost everything is in the cloud the actual delivery of video is via their own hardware. Even at their size I doubt this business would be economically feasible if they were paying someone else for this. Something I've seen often (some numbers changed because...) 20 PB Egress at $0.02/GB = $400,000/month 20 PB is roughly 67 Gbps 95th Percentile It's not hard to find 100 Gbps flat rate for $5,000/month Yes this is overly simplistic, and yes there's a ton more that goes into it than this. But the delta is significant. For some companies $4,680,000/year doesn't move the needle, for others this could mean survival.
praveen9920 2y ago
Reliability stats aside, would have loved to see cost differences between on-prem and cloud.
superq 2y ago
"So you want to build OUT your own data center" is a better title.
Melatonic 2y ago
We're back to the cycle of Mainframe/Terminal --> Personal Computer
Agingcoder 2y ago
They’re not building their own data center - they’re doing what lots of companies have been doing for years ( including where I work , and I specialize in hpc so this is all fairly standard ), which is buying space and power in a dc, and installing boxes in there. Yes, it’s possible to get it wrong. It is however not the same as building a DC …
esher 2y ago
I can relate. We provide a small PaaS-like hosting service, kinda similar to Railway (but more niche). We have recently re-elaborated our choice for AWS (since $$$) as infra provider, but will now stick to it [1]. We started with colocation 20 years ago. For a tiny provider it was quite a hustle (but also an experience). We just had too many single point of failures and we found ourselves dealing with physical servers way too often. We also struggled to fade out and replace hardware. Without reading all the comments thoroughly: For me, being on infra that runs on green energy is important. I think it's also a trend with customers, there even service for this [2]. I don't see it mentioned here. [1] https://blog.fortrabbit.com/infra-research-2024 https://blog.fortrabbit.com/infra-research-2024 [2] https://www.thegreenwebfoundation.org/ https://www.thegreenwebfoundation.org/

596 points by dban 2y ago

First time checking out railway product- it seems like a “low code” and visual way to define and operate infrastructure? Like, if Terraform had a nice UI?

justjake 2y ago
Kinda. It's like if you had everything from an infra stack but didn't need to manage it (Kubernetes for resilience, Argo for rollouts, Terraform for safely evolving infrastructure, DataDog for observability) If you've heard of serverless, this is one step farther; infraless Give us your code, we will spin it up, keep it up, automate rollouts service discovery, cluster scaling, monitoring, etc
- thornewolf 2y ago
  for additional social proof I've been using railway since 2022 and it's been great. I host all my personal projects there and I can go from code to a url by copy-pasting my single dockerfile around.
- nextworddev 2y ago
  Ok so you guys are serverless-ifying backend components. Like Vercel but not just for front end

teleforce 2y ago

>despite multi-million dollar annual spend, we get about as much support from them as you would spending $100 Is it a good or a bad thing to have the same customer support across the board?

dban 2y ago

This is our first post about building out data centers. If you have any questions, we're happy to answer them here :)

gschier 2y ago
How do you deal with drive failures? How often does a Railway team member need to visit a DC? What's it like inside?
- justjake 2y ago
  Everything is dual redundancy. We run RAID so if a drive fails it's fine; alerting will page oncall which will trigger remote hands onsite, where we have spares for everything in each datacenter
  - gschier 2y ago
    How much additional overhead is there for managing the bare-metal vs cloud? Is it mostly fine after the big effort for initial setup?
    - ca508 2y ago
      We built some internal tooling to help manage the hosts. Once a host is onboarded onto it, it's a few button clicks on an internal dashboard to provision a QEMU VM. We made a custom ansible inventory plugin so we can manage these VMs the same as we do machines on GCP. The host runs a custom daemon that programs FRR (an OSS routing stack), so that it advertises addresses assigned to a VM to the rest of the cluster via BGP. So zero config of network switches, etc... required after initial setup. We'll blog about this system at some point in the coming months.
coder543 2y ago
I thought it was an interesting post, so I tried to add Railway's blog to my RSS reader... but it didn't work. I tried searching the page source for RSS and also found nothing. Eventually, I noticed the RSS icon in the top right, but it's some kind of special button that I can't right click and copy the link from, and Safari prevents me from knowing what the URL is... so I had to open that from Firefox to find it. Could be worth adding a <meta> tag to the <head> so that RSS readers can autodiscover the feed. A random link I found on Google: https://www.petefreitag.com/blog/rss-autodiscovery/ https://www.petefreitag.com/blog/rss-autodiscovery/

exabrial 2y ago

I'm surprised you guys are building new! Tons of Colocation available nearly everywhere in the US, and in the KCMO area, there are even a few dark datacenters available for sale! cool project none-the-less. Bit jealous actually :P

gschier 2y ago
More info on the cost comparison between all the options would be interesting
- dban 2y ago
  We pulled some cost stuff out of the post in final review because we weren't sure it was interesting ... we'll bring it back for a future post
justjake 2y ago
The requirements end up being pretty specific, based on workloads/power draw/supply chain So, while we could have bought something off the shelf, that would have been suboptimal from a specs perspective. Plus then we'd have to source supply chain etc. By owning not just the servers but the whole supply chain, we have redundancy at every layer, from the machine, to the parts on site (for failures), to the supply chain (refilling those spare parts/expanding capacity/etc)
CMCDragonkai 2y ago
Can you share a list of dark datacenters that are for sale. They sound interesting as a business.
idlewords 2y ago
They're not building new, though—the post is about renting a cage in a datacenter.

ramon156 2y ago

weird to think my final internship was running on one of these things. thanks for all the free minutes! it was a nice experience

__fst__ 2y ago

Can anyone recommend some engineering reading for building and running DC infrastructure?

ca508 2y ago
We didn't find many good up-to-date resources online on the hardware side of things - kinda why we wanted to write about it. The networking aspect was the most mystical - I highly recommend "BGP in the datacenter" by Dinesh Dutt on that (I think it's available for free via NVidia). Our design is heavily influenced by the ideas discussed there.
- chatmasta 2y ago
  What was the background of your team going into this project? Did you hire specialists for it (whether full time or consultants)?
  - ca508 2y ago
    We talked to a few, I think they're called MSPs? We weren't super impressed. We decided to YOLO it. There are probably great outfits out there, but it's hard to find them through the noise. We're mostly software and systems folks, but Railway is a infrastructure company so we need to own stuff down to the cage-nut - we owe it to our users. All engineering, project management and procurement is in-house. We're lucky to have a few great distributors/manufacturers who help us pick the right gear. But we learnt a lot. We've found a lot of value in getting a broker in to source our transit though. My personal (and potentially misguided) hot take is that most of the baremetal world is stuck in the early 2000's, and the only companies doing anything interesting here the likes of AWS,Google and Meta. So the only way to innovate is to stumble around, escape the norms and experiment.
    - chatmasta 2y ago
      Did your investors give you any pushback or were they mostly supportive?

jonatron 2y ago

Why would you call colocation "building your own data center"? You could call it "colocation" or "renting space in a data center". What are you building? You're racking. Can you say what you mean?

macintux 2y ago
Dealing with power at that scale, arranging your own ISPs, seems a bit beyond your normal colocation project, but I haven’t bee in the data center space in a very long time.
- redeux 2y ago
  I worked for a colo provider for a long time. Many tenants arranged for their own ISPs, especially the ones large enough to use a cage.
- latchkey 2y ago
  One of the many reasons we went with Switch for our DC is because they have a service to handle all of that for you. Having stumbled on doing this ourselves before, it can be pretty tricky to negotiate everything. We had one provider give us a great price and then bait and switch at the last moment to tell us that there is some other massive installation charge that they didn't realize we had to pay. Switch Connect/Core is based off the old Enron business that Rob (CEO) bought... https://www.switch.com/switch-connect/ https://www.switch.com/switch-connect/ https://www.switch.com/the-core-cooperative/ https://www.switch.com/the-core-cooperative/
xiconfjs 2y ago
I have to second this. While it takes mich effort and in-depth knowledge do build up from an “empty” cage it’s still far from dealing with everything from building permits, to plan and realize a data center to code including redundant power lines, AC and fibre. Still kudos going this path in the cloud-centric time we live in.
- j45 2y ago

j-b 2y ago

Love these kinds of posts. Tried railway for the first time a few days ago. It was a delightful experience. Great work!

justjake 2y ago
Thank you! Anything you think we can do better?

coolkil 2y ago

Awesome!! Hope to see more companies go this route. I had the pleasure to do something similar for a company(lot smaller scale though) It was my first job out of university. I will never forget the awesome experience of walking into the datacenter and start plugging cables and stuff

sitkack 2y ago

It would be nice to have a lot more detail. The WTF sections are the best part. Sounds like your gear needs "this side towards enemy" sign and/or the right affordances so it only goes in one way. Did you standardize on layout at the rack level? What poke-yoke processes did you put into place to prevent mistakes? What does your metal->boot stack look like? Having worked for two different cloud providers and built my own internal clouds with PXE booted hosts, I too find this stuff fascinating. Also take utmost advantage of a new DC when you are booting it to try out all the failure scenarios you can think of and the ones you can't through randomized fault injection.

ca508 2y ago
> It would be nice to have a lot more detail I'm going to save this for when I'm asked to cut the three paras on power circuit types. Re: standardising layout at the rack level; we do now! we only figured this out after site #2. It makes everything so much easier to verify. And yeah, validation is hard - manually doing it thus far; want to play around with scraping LLDP data but our switch software stack has a bug :/. It's an evolving process, the more we work with different contractors, the more edge cases we unearth and account for. The biggest improvement is that we have built a internal DCIM that templates a rack design and exports a interactive "cabling explorer" for the site techs - including detailed annotated diagrams of equipment showing port names, etc... The screenshot of the elevation is a screenshot of part of that tool. > What does your metal->boot stack look like? We've hacked together something on top of https://github.com/danderson/netboot/tree/main/pixiecore https://github.com/danderson/netboot/tree/main/pixiecore that serves a debian netboot + preseed file. We have some custom temporal workers to connect to Redfish APIs on the BMCs to puppeteer the contraption. Then a custom host agent to provision QEMU VMs and advertise assigned IPs via BGP (using FRR) from the host. Re: new DCs for failure scenarios, yeah we've already blown breakers etc... testing stuff (that's how we figured out our phase balancing was off). Went in with a thermal camera on another. A site in AMS is coming up next week and the goal for that is to see how far we can push a fully loaded switch fabric.
- sitkack 2y ago
  Wonderful! The edge cases are the gold btw, collect the whole set and keep them in a human and machine readable format. I'd also go through and using a color coded set of cables, insert bad cables (one at a time at first) while the system is doing an aggressive all to all workload and see how quickly you can identify faults. It is the gray failures that will bring the system down, often multiple as a single failure will go undetected for months and then finally tip over an inflection point at a later time. Are you workloads ephemeral and/or do they live migrate? Or will physical hosts have long uptimes? It is nice to be able to rebaseline the hardware before and after host kernel upgrades so you can detect any anomalies. You would be surprised about how larger of a systemic performance degradation that major cloud providers have been able to see over months because "all machines are the same", high precision but low absolute accuracy. It is nice to run the same benchmarks on bare metal and then again under virtualization. I am sure you know, but you are running a multivariate longitudinal experiment, science the shit out of it.

deleted 2y ago

[deleted]

aetherspawn 2y ago

What brand of servers was used?

neilv 2y ago
Looks like Supermicro.
- justjake 2y ago
  Winner winner chicken dinner!
- aetherspawn 2y ago
  Where do you buy this, direct from Supermicro? Asking as a Dell customer… our servers are $$$
  - ca508 2y ago
    We have a distributor we work with - just because it makes import/export a lot easier. But we get to interface directly with Supermicro for the technical/design stuff, and they're super awesome. If you're looking in the US, reach out to their eStore - really great fuss-free turnaround and all direct.
emmelaich 2y ago
Yes, considering the importance of the power draw, I wondered if ARM servers were used.
- ca508 2y ago
  oh yes we want to; I even priced a couple out. Most of the SKUs I found were pretty old, and we couldn't find anything compelling to risk deploying at the scale we wanted. It's on the wishlist, and if the right hardware comes along; we'll rack it up even as a bet. We maintain Nixpacks (https://nixpacks.com/docs/getting-started https://nixpacks.com/docs/getting-started), so for most of our users we could rebuild most their apps for ARM seamlessly - infact we mostly develop our build systems on ARM (because macbooks). One day.

sometalk 2y ago

I remember talking to Jake a couple of years ago when they were looking for someone with a storage background. Cool dude, and cool set of people. Really chuffed to see them doing what they believe in.

justjake 2y ago
Thanks dude <3. We are indeed doing the thing :D

mirshko 2y ago

y’all really need to open source that racking modeling tool, that would save sooooo many people so much time

devenson 2y ago
Not OSS but I have developed a tool for modeling racks and much more. https://schematix.com/video/racks https://schematix.com/video/racks

jdoss 2y ago

This is a pretty decent write up. One thing that comes to mind is why would you write your own internal tooling for managing a rack when Netbox exists? Netbox is fantastic and I wish I had this back in the mid 2000s when I was managing 50+ racks. https://github.com/netbox-community/netbox https://github.com/netbox-community/netbox

ca508 2y ago
we evaluated a lot of commercial and oss offerings before we decided do go build it ourselves - we still have a deploy of netbox somewhere. But our custom tool (Railyard) works so well because it integrates deeply into the our full software, hardware and orchestration stack. The problem with the OSS stuff is that it's almost too generic - you shape the problem to fit its data model vs. solve the problem. We're likely going to fold our tool into Railway itself eventually - want to go on-prem; button click hardware design, commission, deploy and devex. Sorta like what Oxide is doing, but approaching the problem from the opposite side.
matt-p 2y ago
It is not that difficult to build it into your app, if you're already storing information about hosts, networking etc. All you're really doing is expanding the scope, netbox is a fine starting point if you're willing to start there and build your systems around it, but if you've already got a system (or you need to do anything that doesn't fit netbox logic) you're probably better off just extending it. In this case railway will need to care about a lot of extra information beyond just racks, IP addresses and physical servers.
- ca508 2y ago
  correct; I think the first version of our tool sprung up in the space of a couple of weekends. It wasn't planned, my colleague Pierre who wrote it just had a lot of fun building it.
  - walterbell 2y ago
    Were there any promising OSS alternatives to Netbox?

robertclaus 2y ago

I would be super interested to know how this stuff scales physically - how much hardware ended up in that cage (maybe in Cloud-equivalent terms), and how much does it cost to run now that it's set up?

renewiltord 2y ago

More to learn from the failures than the blog haha. It tells you what the risks are with a colocation facility. There really isn't any text on how to do this stuff. The last time I wanted to build out a rack there aren't even any instructions on how to do cable management well. It's sort of learned by apprenticeship and practice.

toddmorey 2y ago

Reminds me of the old Rackspace days! Boy we had some war stories: - Some EMC guys came to install a storage device for us to test... and tripped over each other and knocked out an entire Rack of servers like a comedy skit. (They uh... didn't win the contract.) - Some poor guy driving a truck had a heart attack and the crash took our DFW datecenter offline. (There were ballards to prevent this sort of scenario, but the cement hadn't been poured in them yet.) - At one point we temporarily laser-beamed bandwidth across the street to another building - There was one day we knocked out windows and purchased box fans because servers were literally catching on fire. Data center science has... well improved since the earlier days. We worked with Facebook on the OpenCompute Project that had some very forward looking infra concepts at the time.

lostlogin 2y ago
> There was one day we knocked out windows and purchased box fans because servers were literally catching on fire. Pointing the fans in or out?
- sitkack 2y ago
  You want to point them in.
  - gr3ml1n 2y ago
    The datacenters I've been in with emergency cooling fans in the walls all exhaust out, not in. Easier to get portable CRACs inside and get a good draft going.
wgx 2y ago
I recall getting a DC tour of LON3 and being totally blown away by it all as a 20-something web dev. Good times.
- chasd00 2y ago
  When I was in college I’d call up datacenters pretending to be a prospective customer and schedule a tour. I was totally fascinated by them and knew enough to sound legit, it was like going to an amusement park for me.

linsomniac 2y ago

Was really hoping this was was actually about building your own data center. Our town doesn't have a data center, we need to go an hour south or an hour north. The building that a past failed data center was in (which doesn't bode well for a data center in town, eh?), is up for lease and I'm tempted. But, I'd need to start off small, probably per-cabinet UPSes and transfer switches, smaller generators. I've built up cabinets and cages before, but never built up the exterior infrastructure.

hinkley 2y ago
Why did it fail would be my question. If it turns out to be any of “location, location, location” then getting a partially kitted out building may not help you. Did they get independent data into the building via different routes? How’s the power? Could be the data was coming in through a route that sees frequent construction. I knew a guy who ran the IT dept for a university and he discovered that the excavation crews found it was cheaper to maybe have to pay a fine for cutting data lines than it was to wait for them to be marked accurately. He spent a lot of time being stressed out.
- linsomniac 2y ago
  I agree that one of the first steps would be to take someone from the previous facility out for a meal, which I can probably arrange fairly easily. I don't exactly know why it failed, it was run by the biggest local ISP. I can speculate about why they failed (DSL speeds are severely limited in our town, so really Xfinity was it, they tried providing fiber in some locations, but found it hard to keep up with fiber locate calls). The Colocation side of the business was never very big, but it's not clear if that is because there's not demand or that they just never really pushed it. Location is fairly good, as far as data centers go. It's got relatively good network connectivity, I believe, but I don't have specifics about entrances and diversity. It is close to one of the big fiber rings around the city, I believe the ring is pulled into the facility. I don't know if they had telco fiber in, or backhauled it via the fiber ring. Power is probably good, but not great -- I'd doubt it's fed from multiple substations. There was, at one point, some generator bays. While I could use data center space in town, it'd be hard to convince my work to move, partly as we just signed a 3 year agreement for hosting 60 miles away, partly just because of the cost of a move. It probably should remain a pipe dream.

pixelesque 2y ago

The date and time durations given seem a bit confusing to me... "we kicked off a Railway Metal project last year. Nine months later we were live with the first site in California". seems inconsistent with: "From kicking off the Railway Metal project in October last-year, it took us five long months to get the first servers plugged in" The article was posted today (Jan 2025), was it maybe originally written last year and the project has been going on for more than a year, and they mean that the Railway Metal project actually started in 2023?

ca508 2y ago
ah that's my bad - I wrote this in Dec, we only published in Jan. Obv. missed updating that. Timeline wise; - we decided to go for it and spend the $$$ in Oct '23 - Convos/planning started ~ Jan '24 - Picked the vendors we wanted by ~ Feb/Mar '24 - Lead-times, etc... meant everything was ready for us to go fit the first gear by mostly ourselves at the start of May (that's the 5mo) - We did the "proper" re-install around June, followed closely by the second site in ~ Sep, around when we started letting our users on it as a open beta - Sep-Dec we just doubled down on refining software/automation and process while building out successive installs Lead times can be mind numbing. We have certain switches from Arista that have a 3-6 mo leadtime. Servers are build to order, so again 2+ months depending on stock. And obv. holidays mean a lot of stuff shuts down around December. Sometimes you can swap stuff around to get better lead-times, but then the operational complexity explodes because you have this slightly different component at this one site. I used to be a EEE, and I thought supply chain there was bad. But with DCs I think it's sometimes worse because you don't directly control some parts of your BoM/supply chain (especially with build-to-order servers).
- Twirrim 2y ago
  From working at a cloud, and speaking with capacity folks regularly when I was in certain roles, the supply chain strikes me as one of the biggest nightmares. Even at scale when vendors really, really want (or want to keep) your business. At times it almost seems like someone sneezes somewhere and whoops, there goes your hardware delivery timelines. The advantage at cloud scale is a lot of constant signal around capacity delivery, demand etc. so you can build mathematical models to best work out when to start placing orders, and for what.

matt-p 2y ago

If you’re using 7280-SR3 switches, they’re certainly a fine choice. However, have you considered the 7280-CR3(K) range? They're much better $/Gbps and more relevant edge interfaces. At this scale, why did you opt for a spine-and-leaf design with 25G switches and a dedicated 32×100G spine? Did you explore just collapsing it and using 1-2 32×100G switches per rack, then employing 100G>4×25G AOC breakout cables and direct 100G links for inter-switch connections and storage servers? Have you also thought about creating a record on PeeringDB?https://www.peeringdb.com/net/400940 https://www.peeringdb.com/net/400940. By the way, I’m not convinced I’d recommend a UniFi Pro for anything, even for out-of-band management.

ca508 2y ago
All valid points - and our ideas for Gen 2 sound directionally similar - but those are at crayon drawing stage. When we started, we didn't have much of an idea about what the rack needs to look like. So we chose a combination of things we thought we could pull this off. We're mostly software and systems folks, and there's a dearth of information out there on what to do. Vendors tend to gravitate towards selling BGP+EVPN+VXLAN or whatever "enterprise" reference designs; so we kinda YOLO'ed the Gen 1. We decided to spend extra money if we could get to a working setup sooner. When the clock is in cloud spend, there's uh... lots of opportunity cost :D. A lot of the chipset and switch choices were bets and we had to pick and choose what we gambled on - and what we could get our hands on. The main bets this round were eBGP to the hosts with BGP unnumbered, SONiC switches - this lets us do a lot of networking with our existing IPv6/Wireguard/eBPF overlay and a debian based switch OS + FRR (so fewer things to learn). And ofc. figuring out how to operationalise the install process and get stuff running on the hardware as soon as possible. Now we've got a working design, we'll start iterating a bit more on the hardware choice and network design. I'd love for us to write about it when we get through it. Plus I think we owe the internet a rant on networking in general. Edit: Also we don't use UniFi Pro / Uniquity gear anywhere?

chatmasta 2y ago

This is how you build a dominant company. Good for you ignoring the whiny conventional wisdom that keeps people stuck in the hyperscalers. You’re an infrastructure company. You gotta own the metal that you sell or you’re just a middleman for the cloud, and always at risk of being undercut by a competitor on bare metal with $0 egress fees. Colocation and peering for $0 egress is why Cloudflare has a free tier, and why new entrants could never compete with them by reselling cloud services. In fact, for hyperscalers, bandwidth price gouging isn’t just a profit center; it’s a moat. It ensures you can’t build the next AWS on AWS, and creates an entirely new (and strategically weaker) market segment of “PaaS” on top of “IaaS.”

justjake 2y ago
Yup. Bingo. We've had to pass the cloud egress costs onto our customers, which sucks. With this, it'll mean we can slash that in half, lower storage costs, remove "per seat" pricing, etc Super exciting
- MasterScrat 2y ago
  How do bandwidth costs work now? do you pay the ISPs a flat fee, or is it still usage-based? how much cheaper is it compared to cloud providers?
  - kazen44 2y ago
    in my experience (Eu, datacenter/ISP space) connectivity is either sold based on 99percentile commitments (aka, you pay for sending us this amount of traffic, anything over this gets you billed extra) or based on a minimum commitment for the traffic you send. (atleast X amount of bandwith) or it is based on a flat free principle where you pay an upfront cost for the setup and the rest is a base price for X amount of bits per seconds. It depends a lot on what kind of connection you require, and things like oversubscription and congestion control also come into play. Peering ports with IXP's are usually flat rate, while ports in datacenters to end customers usually have more complex constructs. Hyperscaler bandwith is notoriously expensive. for instance, a 100Gbps port on the AMS-IX is 2500$[1]. Now, you need to account for extra costs to actually use this port (some IP space, ASN number etc) but even with all that added up i think you will not get much more expensive then 400$ per month averaged in total over a year. Now what makes comparing difficult is that hyperscalers are not transparant when it comes to connectivity costs. Looking at egress fees for example: AWS seems to charge 1 cent per Gigabyte transferred for egress fees. If we send data at line rate across our 100Gbps for an entire month we get the following: 100gbps = 12.5Gigabytes per second. 12.5 * 2 629 743 83 (number of seconds in a month) = 32871797875 Gigabytes 32871797875 / 0,001$ = 3.287.179,7875 thats 3,2 million dollars... compared to roughly 4000! AWS also seems to offer "dedicated connections". at roughly 22 dollars per hour [3] (no clue if this is even comparble to an IXP port, but the comparison would still be fun to make). 22$ x 720 (hours per month = 15.840$, or roughly 3times the IXP port price. In both cases, you are getting absolutely shafted by egress prices at cloud providers compared to doing it yourself. [1] https://www.ams-ix.net/ams/pricing-us https://www.ams-ix.net/ams/pricing-us [2] https://aws.amazon.com/ec2/pricing/on-demand/ https://aws.amazon.com/ec2/pricing/on-demand/ [3] https://aws.amazon.com/directconnect/pricing/ https://aws.amazon.com/directconnect/pricing/

dylan604 2y ago

My first colo box came courtesy of a friend of a friend that worked for one of the companies that did that (leaving out names to protect the innocent). It was a true frankenputer built out of whatever spare parts he had laying around. He let me come visit it, and it was an art project as much as a webserver. The mainboard was hung on the wall with some zip ties, the PSU was on the desk top, the hard drive was suspended as well. Eventually, the system was upgraded to newer hardware, put in an actual case, and then racked with an upgraded 100base-t connection. We were screaming in 1999.

cyberax 2y ago

It looked interesting, until I got to the egress cost. Ouch. $100 per TB is way too much if you're using bandwidth-intensive apps. Meta-comment: it's getting really hard to find hosting services that provide true unlimited bandwidth. I want to do video upload/download in our app, and I'm struggling to find providers of managed servers that would be willing to provide me with fixed price for 10/100GB ports.

justjake 2y ago
FWIW, we just pass the costs on from the current cloud providers. Doing this work will let us lower those egress prices!
- cyberax 2y ago
  Yeah. Cloud providers are the worst. Their egress costs moved from "expensive but not unreasonable" circa 2010, to "what the fuck" territory now. A 10G port should be in the range of $2k per month, I believe? I don't mind paying that much.
  - kazen44 2y ago
    a 100g port is in the realm of 2K per month at IXP's.

Havoc 2y ago

Surprised to see pxe. Didn’t realise that was in common use in racks

jpleger 2y ago
Are there any alternatives these days? Or just that you weren't expecting to have systems boot off the network?
- Havoc 2y ago
  The later. I was expecting local boot because pxe introduces a rather big dependency for potentially many machines. Issues with network or issues with pxe server and nothing boots
louwrentius 2y ago
Booting through the IPMI with virtual media isos over http is dog slow in my experience. Using PXE to bootstrap an installer kernel (only few MB) over TFTP that fetches the rest of the OS over HTTP is quick and you can pressed/kickstart a machine in minutes.

enahs-sf 2y ago

Curious why California when the kwh is so high here vs Oregon or Washington

physhster 2y ago

Having done data center builds for years, mostly on the network side but realistically with all the trades, this is a really cool article.

jpleger 2y ago

Makes me remember some of the days I had in my career. There were a couple really interesting datacenter things I learned by having to deploy tens of thousands of servers in the 2003-2010 timeframe. Cable management and standardization was extremely important (like you couldn't get by with shitty practices). At one place where we were deploying hundreds of servers per week, we had a menu of what ops people could choose if the server was different than one of the major clusters. We essentially had 2 chassis options, big disk servers which were 2u or 1u pizza boxes. You then could select 9/36/146gb SCSI drives. Everything was dual processor with the same processors and we basically had the bottom of the rack with about 10x 2u boxes and then the rest was filled with 20 or more 1u boxes. If I remember correctly we had gotten such an awesome deal on the price for power, because we used facility racks in the cage or something, since I think they threw in the first 2x 30 amp (240v) circuits for free when you used their racks. IIRC we had a 10 year deal on that and there was no metering on them, so we just packed each rack as much as we could. We would put 2x 30s on one side and 2x 20s on another side. I have to think that the DC was barely breaking even because of how much heat we put out and power consumption. Maybe they were making up for it in connection / peering fees. I can't remember the details, will have to check with one of my friends that worked there around that time.

Over2Chars 2y ago

I guess we can always try to re-hire all those "Sys Admins" we thought we could live without. LOL?

whalesalad 2y ago

Cliffhanger! Was mostly excited about the networking/hypervisor setup. Curious to see the next post about the software defined networking. Had not heard of FRR or SONIC previously.

ca508 2y ago
the good news on this is that we've got a tonne of deep-dive material on networking and whitebox switches we cut from this post. We'll definitely be talking more about this soon (also cos' BGP is cool).

ChuckMcM 2y ago

From the post: "...but also despite multi-million dollar annual spend, we get about as much support from them as you would spending $100." -- Ouch! That is a pretty huge problem for Google. I really enjoyed this post, mostly because we had similar adventures when setting up the infrastructure for Blekko. For Blekko, a company that had a lot of "east west" network traffic (that is traffic that goes between racks vs to/from the Internet at large) having physically colocated services without competing with other servers for bandwidth was both essential and much more cost effective than paying for this special case at SoftLayer (IBM's captive cloud). There are some really cool companies that will build an enclosure for your cold isle, basically it ensures all the air coming out of the floor goes into the back of your servers and not anywhere else. It also keeps not cold air from being entrained from the sides into your servers. The calculations for HVAC 'CRAC' capacity in a data center are interesting too. In the first CoLo facility we had a 'ROFOR' (right of first refusal) on expanding into the floor area next to our cage, but when it came time to expand the facility had no more cooling capacity left so it was meaningless. Once you've done this exercise, looking at the 0xide solution will make a lot more sense to you.

sour-taste 2y ago

I used to work on machine repair automation at a big tech company. IMO repairs are one of the overlooked and harder things to deal with. When you run on AWS you don't really think about broken hardware it mostly just repairs itself. When you do it yourself you don't have that luxury. You need to have spare parts, technician to do repairs, a process for draining/undraining jobs off hosts, testing suites, hardware monitoring tools and 1001 more things to get this right. At smaller scales you can cut corners on some of these things but they will eventually bite you. And this is just machines! Networking gear has it's own fun set of problems, and when it fails it can take down your whole rack. How much do you trust your colos not to lose power during peak load? I hope you run disaster recovery drills to prep for these situations! Wishing all the best to this team, seems like fun!

nyrikki 2y ago

> This will likely involve installing some overhead infrastructure and trays that let you route fiber cables from the edge of your cage to each of your racks, and to route cables between racks Perhaps I am reading this wrong, as you appear to be fiber heavy and do have space on the ladder rack for copper, but if you are commingling the two, be careful. A possible future iteration, would consider a smaller panduit fiberunner setup + a wire rack. Co-mingling copper and fiber, especially through the large spill-overs works until it doesn't. Depending on how adaptive you need to be with technology changes, you may run into this in a few years. 4x6 encourages a lot of people putting extra cable up in those runners, and sharing a spout with cat-6, cx-#, PDU serial, etc... will almost always end badly for some chunk of fiber. After those outages it also encourages people to 'upgrade in place'. When you are walking to your cage look at older cages, notice the loops sticking out of the tops of the trays and some switches that look like porcupines because someone caused an outage and old cables are left in place. Congrats on your new cage.

hintymad 2y ago

Per my experience with cloud, the most powerful Infra abstraction that AWS offers is actually EC2. The simplicity of getting a cluster of machines up and running with all the metadata readily available via APIs is just liberating. And it just works: the network is easy to configure, the ASGs are flexible enough to customize, and the autoscaling offers strong primitives for advanced scaling. Amazingly, few companies who run their own DCs could build anything comparable to EC2, even at a smaller scale. When I worked in those companies, I sorely missed EC2. I was wondering if there's any robust enough open-source alternatives to EC2's control-plane software to manage baremetals and offer VMs on top them. That'll be awesome for companies that build their own DCs.

mdaniel 2y ago
I believe eBay still runs on OpenStack, which as far as I know even has as ec2-compatible emulation layer https://docs.openstack.org/api-ref/ec2-api/#supported-features-and-limitations https://docs.openstack.org/api-ref/ec2-api/#supported-featur...

winash83 2y ago

We went down this path over the last year, lots of our devs need local and dev/test environments and AWS was costing us a bomb, With about 7 Bare metals(Colocation) we are running about 200+ VMs and could double that number with some capacity to spare. For management, we built a simple wrapper over libvirt. I am setting up another rack in the US and will end up costing around $75Kper year for a similar capacity. Our prod is on AWS but we plan to move everything else and it's expected to save at least a quarter of a million dollars per year

cbg0 2y ago
Sounds like a good chunk of money saved, but are you getting the same level of redundancy as you did on the cloud?
- winash83 2y ago
  For most dev/test workflows redundancy is not a huge concern, because we can just recreate the environments, in practice things are quite stable, most HW vendors like Hp Dell etc let you rent the servers instead of buying them, in case of serious HW issues they take care of the fixes, and usually there is someone at the Colocation site to take care of the day to day

motoboi 2y ago

I’m my experience and based on writeups like this: Google hates having customers. Someone decided they have to have a public cloud, so they did it, but they want to keep clients away with a 3 meter pole. My AWS account manager is someone I am 100% certain would roll in the mud with me if necessary. Would sleep in the floor with us if we asked in a crisis. Our Google cloud representatives make me sad because I can see that they are even less loved and supported by Google than us. It’s sad seeing someone trying to convince their company to sell and actually do a good job providing service. It’s like they are setup to fail. Microsoft guys are just bulletproof and excel in selling, providing a good service and squeezing all your money out of your pockets and you are mortally convinced it’s for your own good. Also have a very strange cloud… thing. As for the railway company going metal, well, I have some 15 years of experience with it. I’ll never, NEVER, EVER return to it. It’s just not worth it. But I guess you’ll have to discover it by yourselves. This is the way. You soon discover what in freaking world is Google having so much trouble with. Just make sure you really really love and really really want to sell service to people, instead of building borgs and artificial brains and you’ll do 100x better.

motorest 2y ago
> Microsoft (...) have a very strange cloud… thing. Risking a going off on a tangent, this is something I rarely see discussed but is perhaps one of the main problems with Azure. The whole cloud service feels like something someone oblivious to cloud computing would design if all they knew was renting bare metal servers. It's cloud computing in a way that completely defeats the whole concept of cloud computing.
- yeahwhatever10 2y ago
  Can you be more specific?
  - motorest 2y ago
    > Can you be more specific? To start off, read up on the ass-backwards concept of an app service plan. That nonsense is the ultimate anti-cloud computing approach to cloud computing. https://learn.microsoft.com/en-us/azure/app-service/overview-hosting-plans https://learn.microsoft.com/en-us/azure/app-service/overview...

scarab92 2y ago

Interesting that they call out the extortionate egress fees from the majors as a motivation, but are nevertheless also charging customers $0.10 per GB themselves.

wmf 2y ago
Bezos: Your margin is my opportunity. Railway: No, your margin is my opportunity.
- ca508 2y ago
  We currently pass on our cloud egress costs to users via the current pricing. We'll be publishing a pricing update soon as part of our migration - and egress [and some other things] will be coming down.
justjake 2y ago
That $0.10 per GB is direct pass along for the cloud ingress fees We can lower that once we’re fully on metal

ThinkBeat 2y ago

1. Is the impression they decided to use a non datacenter location to put their datacenter, If so that is not a good idea. 2. Geographical distanced backups, if the primary fails. Without this you are already in trouble. What happens if the buildings burns down? 3. Hooking up with "local" ISPs That seems ok. As long as ISP failing is easily and autoamically dealt with. 4. I am a bit confused about what happens at the edge. On the one head it seems like you have 1 datacenter, and ISPs doing routing, other places I get the impression you have compute close to the edge. Which is it?

wmf 2y ago
1. No, they're using a cage inside a real data center in Ashburn VA which is basically data center city. 2. In the diagram you can see site 1 and site 2. 3. Yes, routers automatically deal with ISP failures.

technick 2y ago

I've spent more time than I care working in data centers and can tell you that your job req is asking for one person to perform 3 different roles, maybe 4. I guarantee you're going to find a "jack of all trades" and a master of none unless you break them out into these jobs. Application Developer DevOps Engineer Site Reliability Engineer Storage Engineer Good luck, hope you pay them well.

concerndc1tizen 2y ago
There's no reason to attack generalists.

random_savv 2y ago

I guess there's another in-between step buying your own hardware, even when merely "leasing individual racks", and EC2 instances: dedicated bare metal providers like Hetzner. This lets one get closer to the metal (e.g. all your data is on your specific disk, rather than an abstracted block storage, such as EBS, not shared with other users, cheaper, etc) without having to worry about the staff that installs the hardware or where/how it fits in a rack. For us, this was a way to get 6x performance for 1/6 of the cost. (Excluding, of course our time, but we enjoyed it!)

t0mas88 2y ago
Hetzner is very good for low cost high bandwidth things that don't need a serious SLA. But if you're selling a platform like Railway.com the stability and flexibility of Hetzner aren't going to be good enough.
dboreham 2y ago
Agreed. We run our own bare metal in a rack, but also rent machines from Hivelocity where the use case suits.

throwaway2037 2y ago

I promise my comment is not intended to troll. Why didn't you use Oxide pre-built racks? Just the power efficiency seems like a huge win.

ca508 2y ago
It's a fair question. What Oxide are building is cool, but it's too custom/monolithic for us to risk. We're more likely to look at OCP racks/chassis down the road.

solarkraft 2y ago

Cool post and cool to see Railway talked about more on here. I‘ve used their postgres offering for a small project (crucially it was accessible from the outside) and not only was setting it up a breeze, cost was also minimal (I believe staying within the free tier). I haven’t used the rest of the platform, but my interaction with them would suggest it would probably be pretty nice.

concerndc1tizen 2y ago

@railway What would you say are your biggest threats? Power seems to the big one, especially when the AI power and electric vehicle demand will drive up kWh prices. Networking seems another one. I'm out of the loop, but it seems to me like the internet is still stuck at 2010 network capacity concepts like "10Gb". If networking had progressed as compute power has (e.g. NVMe disks can provide 25GB/s), 100Gb would be the default server interface? And the ISP uplink would be measured in terabits? How is the diversity in datacenter providers? In my area, several datacenters were acquired and my instinct would be that: the "move to cloud" has lost smaller providers a lot of customers, and the industry consolidation has given suppliers more power in both controlling the offering and the pricing. Is it a free market with plenty of competitive pricing, or is it edging towards enshittification?

kazen44 2y ago
> Networking seems another one. I'm out of the loop, but it seems to me like the internet is still stuck at 2010 network capacity concepts like "10Gb". If networking had progressed as compute power has (e.g. NVMe disks can provide 25GB/s), 100Gb would be the default server interface? And the ISP uplink would be measured in terabits? High end network interfaces are entering the 800Gbps interface era right now. also, in 2010 10Gbps network connectivity to end hosts was NOT common. (it was common for router uplinks and interconnects though. Network interfaces have not scaled as nicely because getting fast enough lasers to handle higher then 100Gbps has been a challenge, and getting to higher speeds basically means doing wave division multiplex over multiple channels across a single fiber. Also, density of connections per fiber has increased massivly because the cost of DWDM equipment has come down significantly.

yread 2y ago

Useful article. I was almost planning to rent a rack somewhere but it seems there's just too much work and too many things to go wrong and it's better to rent cheap dedicated servers and make it somebody elses problem

ksec 2y ago

I am just fascinated by the need of Datacenter. The scale is beyond comprehension. 10 years ago, before the word HyperScaler was even invented or popularised, I would have thought DC market to be on the decline or levelled off now or around this time. One reason being HyperScaler, AWS, Google, Microsoft, Meta, Apple, Tencent, Alibaba, to smaller ones like Oracle and IBM. They would all have their own DC, taking on much of the compute for themselves and others. While left over space would be occupied by third parties. Another reason being the compute, memory and storage density continue to increase, which means for the same amount of floor space we are offering 5 - 20x of the previous CPU / RAM / Storage. Turns out we are building like mad and we are still not building enough.

kazen44 2y ago
just look at the insane amount of computers that currently exist and need to communicate to do data processing. I remember 30 years ago most people used one single computer for the family at home, half of the people i knew didn't have proper internet access. (and this is from a western perspective, the rest of the world was even far less digitized). Now look at how much networked computers are around you. - your phone - one or multiple TV's - laptops/desktops etc - smart home appliances. and this is just looking at the very small sample size of a normal household, add up to that things like the digitizalisation of factories, the digitalization of the rest of the world. (internet access has grown massively in the past 15 years in the developing world). We have far more computers then a decade ago, and far more people have them aswell, and it shows very little signs of stopping. IPv6 for instance, supports an absolutely infanthomable IP address space. (which people often seem to think is overkill), but looking at the past growth, i think having suchs a large address space is a wise choice. Another thing which people seem to not notice is that a lot of older DC's are being phased out, mainly because these facilities are repurposed telephone exchanges and far less suitable for more power hungry computing power.
- ksec 2y ago
  >We have far more computers then a decade ago, Yes that was already taken into account. We rushed pass active usage of Smartphone of 4B in 2020. With additional 1B user who have access to 4G / Smartphone but not using any Data. 1B people who cant afford it or are using feature phone. And around 1B people who are outside the age of using smartphone, child, baby etc. That is a total Around 7B people already. And anything after is a long tail of new generation outpacing older generation. Tablet usage has levelled off. PC market hasn't had new growth outside China and India. COVID managed to fasten a lot these digitalisation. I wrote about how the Growth of AWS in 2023 was roughly equal to doubling of itself in 2016. i.e 2023 AWS was building out the size of AWS 2016. That is insane. And yet we are still building more. >Another thing which people seem to not notice is that a lot of older DC's are being phased out, That is something I am not aware. But certainly will keep a look out. This got me thinking if building out a new DC is easier and cheaper than repurposing older DC not designed for high computing power density usage? While I have said we could increase Compute, RAM , Storage density in Rack by 10-20x, we have also increase power usage by 4 - 5x. Not only electricity / power usage but cooling and design also requires additional thoughts.

blmt 2y ago

I am really thankful for this article as I finally get where my coworkers get "wrong" notions about three-phase power use in DC: >The calculations aren’t as simple as summing watts though, especially with 3-phase feeds — Cloudflare has a great blogpost covering this topic. What's written in the Cloudflare blogpost linked in the article holds true only of you can use a Delta config (as done in the US to obtain 208V) as opposed to the Wye config used in Europe. The latter does not give a substantial advantage: no sqrt(3) boost to power distribution efficiency and you end up adding Watts for three independent single phase circuits (cfr. https://en.m.wikipedia.org/wiki/Three-phase_electric_power https://en.m.wikipedia.org/wiki/Three-phase_electric_power).

Bluecobra 2y ago

Good writeup! Google really screws you when you are looking for 100G speeds, it's almost insulting. For example redundant 100G dedicated interconnects are about $35K per month and that doesn't include VLAN attachments, colo x-connect fees, transit, etc. Not only that, they max out on 50G for VLAN attachments. To put this cost into perspective, you can buy two brand new 32 port 100G switches from Arista for the same amount of money. In North America, you can get 100G WAN circuits (managed Wavelength) for less than $5K/month. If it's a local metro you can also get dark fiber for less and run whatever speed you want.

kazen44 2y ago
also, buy some DWDM equipment and you can easily scale those darkfibers to offer multiple 100GBPS connections for very little cost.

a1o 2y ago

Excellent write up! This is not the first blog post I see in recent times on going to owning infrastructure direction, but it is certainly well written and I liked the use of Excel in it, a good use, although visually daunting!

lifeinthevoid 2y ago

Man, I get an anxiety attack just thinking about making this stuff work. Kudos to all the people doing this.

mdaniel 2y ago
My experience has been that with most things, making it work is often simple, keeping it working is where people start to get mega bucks for having the required experience

arschfick 2y ago

[dead]

kolanos 2y ago

As someone who lost his shirt building a data center in the early 2000s, Railway is absolutely going about this the right way with colocation.

maxclark 2y ago

There's places where it makes sense to be on the cloud, and places where it doesn't. The two best examples I can give are for high bandwidth, or heavy disk intensive applications. Take Netflix. While almost everything is in the cloud the actual delivery of video is via their own hardware. Even at their size I doubt this business would be economically feasible if they were paying someone else for this. Something I've seen often (some numbers changed because...) 20 PB Egress at $0.02/GB = $400,000/month 20 PB is roughly 67 Gbps 95th Percentile It's not hard to find 100 Gbps flat rate for $5,000/month Yes this is overly simplistic, and yes there's a ton more that goes into it than this. But the delta is significant. For some companies $4,680,000/year doesn't move the needle, for others this could mean survival.

praveen9920 2y ago

Reliability stats aside, would have loved to see cost differences between on-prem and cloud.

superq 2y ago

"So you want to build OUT your own data center" is a better title.

Melatonic 2y ago

We're back to the cycle of Mainframe/Terminal --> Personal Computer

Agingcoder 2y ago

They’re not building their own data center - they’re doing what lots of companies have been doing for years ( including where I work , and I specialize in hpc so this is all fairly standard ), which is buying space and power in a dc, and installing boxes in there. Yes, it’s possible to get it wrong. It is however not the same as building a DC …

esher 2y ago

I can relate. We provide a small PaaS-like hosting service, kinda similar to Railway (but more niche). We have recently re-elaborated our choice for AWS (since $$$) as infra provider, but will now stick to it [1]. We started with colocation 20 years ago. For a tiny provider it was quite a hustle (but also an experience). We just had too many single point of failures and we found ourselves dealing with physical servers way too often. We also struggled to fade out and replace hardware. Without reading all the comments thoroughly: For me, being on infra that runs on green energy is important. I think it's also a trend with customers, there even service for this [2]. I don't see it mentioned here. [1] https://blog.fortrabbit.com/infra-research-2024 https://blog.fortrabbit.com/infra-research-2024 [2] https://www.thegreenwebfoundation.org/ https://www.thegreenwebfoundation.org/