6 ms·
Facebook open sources its servers and data centers
- flyt 15y agoGet all the CAD files and other specifications here: http://opencompute.org http://opencompute.org
- swah 15y agoAny idea on how much would it cost to produce a small quantity of those boards? At which scale does it starts to make sense to use a custom made motherboard instead of a off-the-shelf PC?
- flyt 15y agoHP, Dell and other vendors are already showing off hardware built to the new specifications: http://instagr.am/p/C66FU/ http://instagr.am/p/C66FU/
- hollerith 15y agoYour link does not support your assertion. It's just a picture without a caption.
- wmf 15y agohttp://twitter.com/#!/Scobleizer/status/56062315665178624 http://twitter.com/#!/Scobleizer/status/56062315665178624 "Process journalism" at work.
- jedsmith 15y agoWith a source fairly prominently displayed at the bottom. Follow along if you absolutely must know more: 1. Navigate to twitter.com/scobleizer. 2. Observe first tweet. http://twitter.com/Scobleizer/status/56062315665178624 http://twitter.com/Scobleizer/status/56062315665178624 That said, I lost all interest in the image due to the dumb focus effect.
- smoody 15y agoditto on the focus effect. reduces the value of the photo.
- wmf 15y agoOne thing to keep in mind is that one rack holds 90 servers and each battery rack feeds 180 servers, so (at least in Facebook's case) that's the basic unit. If this stuff takes off I would expect someone to produce a smaller 30-slot rack, though.
- ewams 15y agoNo. Each rack holds 30 servers and each triplet holds 90. The battery cabinet can support 2 triplets. http://opencompute.org/datacenters/ http://opencompute.org/datacenters/
- deleted 15y ago[deleted]
- sundars 15y agoOne of the most important fact form this article which is likely to be overlooked is : "the Prineville facility runs at 85°F with a 65 percent relative humidity". Running at such a high humidity + Temp combination is impressive esp when you look at the money saved in environmental costs.
- chopsueyar 15y agoThat quote made it seem like the incoming air was at 65% relative humidity, but... "which in turns lets Facebook rely on evaporative cooling instead of air conditioning." So it runs at 65% relative humidity after the swamp coolers. I doubt this would be possible in datacenters in more tropical environments (south Florida). Thoughts?
- yanw 15y agoNot sure how many startups built their own servers anymore, this event seems like a response to the Greenpeace accusations of Facebook's environmental responsibility or wherever. Edit: I agree it's a good thing, it's just that hosting a press event rather than just making the announcement through a blog post suggests other motives as well.
- nroman 15y agoThis is true. However, the services that startups use to host their servers (Amazon Web Services, Rackspace, etc.) could adopt these practices in new datacenters because the specs are open. This could lower costs for the startups that use these services, and be good for the environment. Seems like everyone wins.
- deleted 15y ago[deleted]
- vl 15y agoAny large scale service already adopted such practices, may be not to the same degree. All large players buy custom hardware with specific tweaks to be more energy (read cost) efficient.
- toddmorey 15y ago"Graham Weston, the chairman of Rackspace, said that his company would use the new Open Compute servers in its own designs, and Zynga's chief technical officer said that his company would take a serious look at adding the new technology to its own cloud." http://www.pcmag.com/article2/0,2817,2383257,00.asp http://www.pcmag.com/article2/0,2817,2383257,00.asp
- flyt 15y agoThis isn't a response to Greenpeace, this is a response to the real costs that it takes to run compute infrastructure at the scale of Facebook, Amazon, Google, etc. If a datacenter can run at 1.07 PUE (as Facebook's new datacenter does), then it directly translates to massive power cost savings. Environmental efficiency just comes along with it.
- amitraman1 15y agoWow.
- whakojacko 15y agoLink to an article in James Hamilton's blog with actual pdfs of various designs: http://perspectives.mvdirona.com/2011/04/07/OpenComputeProject.aspx http://perspectives.mvdirona.com/2011/04/07/OpenComputeProje...
- wmf 15y agoAs much as I like James and his commentary, I have this thing about primary sources: http://opencompute.org/ http://opencompute.org/
- beagledude 15y agosay what you want about Facebook but I give them props for open sourcing so much code to the community. Cassandra, Thrift, Scribe,Hive, etc...
- ZoFreX 15y agoA list of Facebook's open-sourced applications and their major contributions to other open-source projects can be found here: http://developers.facebook.com/opensource/ http://developers.facebook.com/opensource/
- lrcg 15y ago"Please take a look, tell us what we did wrong and join us in working together to make every data center more efficient." Mad Props.
- skeletonjelly 15y agoIndeed. This is how I want to see society. Sharing knowledge for the greater good. Probably the reason I'm not a fan of detriments of this such as software patents and Apple in general.
- farlington 15y agoApple has open sourced a lot of their projects. Check it out: http://opensource.apple.com/ http://opensource.apple.com/ http://developer.apple.com/opensource/ http://developer.apple.com/opensource/
- tallanvor 15y agoI'm sure they've done a lot of tests, but running servers in an environment with 65% humidity just doesn't sound good for them.
- ceejayoz 15y ago> In August 2008, Intel conducted a 10-month study to assess the effectiveness of using only outside air to cool a data center. The temperature range was 64°F to 92°F. Humidity varied from 4% to over 90% and changed rapidly at times. No increase in server failure was observed. http://www.energystar.gov/index.cfm?c=power_mgt.datacenter_efficiency_inlet_temp http://www.energystar.gov/index.cfm?c=power_mgt.datacenter_e...
- ams6110 15y agoI don't know whether 65% humidity is going to hurt a server, but evaporative cooling doesn't work well if the relative humidity is high. So maybe it's a good idea in dry climates, but I can't see it being very effective in the midwest of the US in July/August.
- delinka 15y agoA certain amount of humidity helps to prevent issues with static electricity. And unless you're pouring water directly across electrical components, there's really not a problem.
- joblessjunkie 15y agoBut it's great for all those plants they have hidden behind the false rack fronts.
- michaelbuckbee 15y agoI'd be very interested in hearing from someone with more experience in running their own hardware what portions (if any) of what Facebook has announced today is applicable at the small scale of say having a couple of co-located racks in some datacenter. Maybe the base server designs?
- rhizome 15y agoI'd say it's more of a refinement than something new. I haven't looked at much more than what they're saying in the video, but it boils down to: - new featureset mobos: stripped down, no parallel ports among other things I'm sure. there looked to be some SATA/Mini-USB type sockets. - fancy new power supplies - the triple racks don't seem to be anything special at face value, convenient maybe, but with the power features above two of these triple racks are serviced by one battery rack. So, you'd need, oh, 400+ servers to take full advantage at this level. NOW HOW MUCH WOULD YOU PAY? - Fancy power distribution and cabling and stuff, mostly part of the racks, possibly involving wacky connectors on the compute side. For smaller operations this would be like having a 60-outlet power strip. - 95% efficient from plug to party-time. This helps with the bills. - Using the room/building as the cooling mechanism. While this has likely been done before in some manner (at least an intentional use of stack-effect in HVAC), that weird slot that one of the boxes gets pushed into makes me think they have some kind of sealed thingy from floor to ceiling that interfaces with the racks, basically using the building cooling to push in and suck out air forced through the racks. at a basic level it's all about CFMs, after all. This could also be done with forced current from front to back, or vice versa. whatever is going on here, you first have to get to the "your own server room" part of business. this can be had at smaller companies, too, but for the room-cool ducting you'd need to cut into walls and stuff in the server room to pipe that stuff in. Spendy. So, this leaves the smalltimer to save $15 on their monthly power bill by using new rack servers that don't need or have an NVidia GeForce 9000+ and 10-drive RAID. Stripped down BIOS and maybe no more IDE support, that kind of thing. Hot Rod rack servers of the skeleton/pure-compute variety, not the AlienWare one.
- cft 15y agoI can give you an interesting data point: our power bill is only about $600/mo (2.5KWh), our bandwidth bill is about $3,500/mo (1.2Gbps). So power is a smaller consideration.
- budwin 15y agoThis is a great talent acquisition play.
- iloveponies 15y agoOne thing I've found interesting: The decision to have batteries not per server like Google, or per data centre, but per group of racks.
- Swannie 15y agoPretty standard in telecoms. Never quite understood why Google went per server.
- iloveponies 15y agoTheir claims were that the efficiency was that great, and that it's acceptable to pay the extra few dollars in motherboard manufacture to build it in.
- Swannie 15y agoI guess in their instance the lifetime of the server was never likely to be eclipsed by that of the battery. In telco, 15-20yrs of operation is not uncommon, so a single set of replaceable/easily service batteries is important.
- corin_ 15y agoSorry ARM. Have ARM actually done more than announce that they will be moving into servers? If they have then I missed the announcement. Either way, seems like a fairly stupid dig at ARM, can't really expect companies like Facebook to have moved onto ARM servers this quickly, even if it is the direction they intend to go in.
- steevdave 15y agoNo they haven't. The closest you would come would be a tegra2 which is dual core and I believe the omap4 is dual core as well. Though iirc, you can't currently use both cores in Linux. Freescale have announced the imx6 which should have quad, dual and single cores but availability won't be until q4 (though they are pushing hard to get it before then I think.) You still are currently limited with ram on an ARM machine which I think is the biggest hinderance to going in to servers. Presently almost all server software works on an ARM machine with a few exceptions like spidermonkey (so no running Launchpad) but plenty of people use them as personal servers.
- wmf 15y agoThis "sorry ARM" stuff is mostly just the media's need to turn every story into a horse race. But there is some technical detail here: Facebook said they're not ever willing to use 32-bit and ARM won't have 64-bit for years. Also notice that Facebook's servers are "conventional" 2-socket with lots of DIMM slots, not 1-socket "microservers".
- steevdave 15y agoFwiw, Freescale has a 64bit ppc processor now... Either way though, until you're able to just drop an ARM processor into a "commodity" motherboard, it won't really see widespread use, aside from cell phones, and other mobile devices. There is also the fact that ARM processors all seem to have the graphics card embedded in the chip.
- wmf 15y agoQorIQ is pretty cool, but for whatever reason there's massive hype about ARM servers and zero about PowerPC. Perhaps this is evidence of an information cascade. If Facebook is already getting custom Intel and AMD motherboards, ARM motherboards should be no problem. (Note that a major undercurrent of Facebook's announcement is that "custom" is actually cheaper than "commodity" at "Web scale".) ARM server chips (Armada XP and Calxeda) don't have GPUs.
- lrcg 15y agoEthernet-powered LED lighting :D
- oasisbob 15y agoI found that interesting too, thought it was a joke at first. The only details I was able to dig up are in the Data Center specs [1], and they're pretty brief: > Energy-efficient LED lighting is used throughout the data center interior. / Innovative power over Ethernet LED lighting system. / Each fixture has an occupancy sensor with manual override. / Programmable alerts via flashing LEDs. I wondered what the justification for PoE lighting could possibly be, sounds like all the lighting is also functional as instrumentation. Anyone know more? [1] http://opencompute.org/specs/Open_Compute_Project_Data_Center_v1.0.pdf http://opencompute.org/specs/Open_Compute_Project_Data_Cente...
- chopsueyar 15y agoA couple of guesses. They probably have a lot more cat5/cat6 around than 3-conductor 12 gauge copper wire (not sure if it is cheaper by the foot). Also, the network switches can output the PoE, so maybe it is easier to wire into/from the racks or overhead than a seperate AC line with conduit. Labor installation and material costs may be less. Also, since it is DC, and not an AC lighting source, interference may be less (just a guess). And as you mentioned, the lighting as instrumentation.
- bhousel 15y agoAll true. Right now 14/3 Romex costs about 3x more than Cat5. Copper is expensive! LED lamps use DC power. The LED lamps that you can buy that screw into a standard Edison socket contain electronics to convert the AC line voltage into DC. This is inefficient -- generates heat and wastes power (though still not nearly as much as a traditional incandescent bulb). By using power over Ethernet for their lighting, the datacenter can use cheaper, cooler, more efficient bulbs, and save a lot of money on the wiring too. I can definitely foresee a future where new construction includes wiring for both line voltage AC and also low voltage DC. It could eliminate all the bulky transformers scattered around a typical house and save energy and money.
- davidpitkin 15y agoFacebooks attempts at openness make me wonder what they are trying to hide... misdirection?
- rbanffy 15y agoI don't get the AC PSUs on the servers (http://opencompute.org/servers/ http://opencompute.org/servers/). Any reason why those are being used when each 6-rack group has a UPS (and batteries) connected to them? Going DC would get rid of the inverter on the UPS out and the 200 or so PSUs on each server.
- lrcg 15y agoThe UPS doesn't need an inverter - the PSUs have a DC in.
- rbanffy 15y agoBut they are supposed to use the AC input when not running off the UPS. I wonder why that would be a good idea.
- dhess 15y agoI'm also puzzled why they don't go all-DC. This thread doesn't appear to be getting any traction here. Is there a HN for datacenter geeks?
- daveman692 15y agoAmir Michael who worked on the hardware team said, "The utility provides us with AC voltage which we then convert to DC very close to the motherboard in the power supply. Our goal was to bring the high voltage as close as possible to the load to minimize IR losses. We could do an efficient AC –DC conversion in the rack and then distribute DC to the individual servers but that would mean several feet of low voltage conductors which would be made from large copper bars and a would have higher IR losses."
- skorgu 15y agoWhat are the chances of these being commercially available anytime soon? Is a single consumer (even if it's facebook-sized) enough to jumpstart a B2C supply chain? It's hard to see this catching on with anyone not building green field unless it can come in at least on-par with a Dell/HP/Supermicro quote.
- deleted 15y ago[deleted]
- ignifero 15y agoShall we expect Google, Amazon, Microsoft et. al. to start posting their energy efficiency stats from tomorrow?
- wmf 15y agoWay ahead of you: http://www.google.com/corporate/datacenter/efficiency-measurements.html http://www.google.com/corporate/datacenter/efficiency-measur...
- Flenser 15y agoLooked through Google's numbers but they don't seem to be directly comparable to Facebook's. Anyone know how far apart they are?
- codex 15y agoThis is a strategic attack on Google. A proliferation of scalable data centers hurts Google a lot more than Facebook by enabling Google's competitors. Cheap computation matters much more to search engines than social networking sites.
- db42 15y agoThat is a good point, that you have raised. But I wonder, why well established companies like microsoft, apple etc. are not doing this.
- diegob 15y agoA proliferation of scalable data centers could also enable the next generation of social networking sites ... I don't see why it's limited to helping Google's competitors and not Facebook's.
- whatusername 15y agoBecuase Search is ultimately about technology, but Social is about the users. If I was the only user of Google (and they had the same technology stack) then +1 aside - my experience would be the same as it is now. If I was the only user of facebook - what would be the point.
- diegob 15y agoI'm sorry, but it seems like you're implying that search doesn't need users. Some of Google's ranking depends on analyzing search histories, so if you were the only user, the experience would be worse. That means that search is also about the users. Data center technology can benefit everybody, that's all I'm trying to say.
- jacques_chester 15y agoAh, economies of scale, is there anything you can't improve?