5 ms·

Are there industry options or methods of wiring to allow for a UPS room separate from the actual rooms the racks are stored in? It's almost tradition to have a

Are there industry options or methods of wiring to allow for a UPS room separate from the actual rooms the racks are stored in? It's almost tradition to have a rack with UPS's in the bottom and then the rest of the space filled with servers or drive arrays. We wouldn't ever think of putting a tiny backup generator in the bottom of every rack, so why do we put a battery storage system there? Also, with the advances in battery chemistry technology that improve reliability and density, it's only a matter of time until Lithium chemistry batteries are available and that also increases the risk of fire. Is there any reason not to move backup power to another room, or even to a separate structure like how they put backup generators on a pad outside of the building?

tyingq 6y ago
There are plenty of datacenters with a separate battery room, sure.
- sschueller 6y ago
  So far I have never been in one where this was not the case. Also they always have fire suppression.
mikepurvis 6y ago
For the extreme opposite of that, Google famously trolled everyone 10 years ago by announcing that every one of their servers had its own in-chassis 12V battery: https://www.cnet.com/news/google-uncloaks-once-secret-server-10209580/ https://www.cnet.com/news/google-uncloaks-once-secret-server...
- jeffbee 6y ago
  Yes but Google also later moved to a 48VDC power architecture with the batteries at the bottom of the rack. See http://apec.dev.itswebs.com/Portals/0/APEC%202017%20Files/Plenary/APEC%20Plenary%20Google.pdf http://apec.dev.itswebs.com/Portals/0/APEC%202017%20Files/Pl... page 6.
  - tyingq 6y ago
    That presentation is pretty interesting, including Google inventing their own "Switched Tank" DC-DC converters because the existing ones weren't efficient or reliable enough.
    - jeffbee 6y ago
      There's a whole separate paper on that: https://storage.googleapis.com/pub-tools-public-publication-data/pdf/55746647b67067298c9455ece6d405872c9101c8.pdf https://storage.googleapis.com/pub-tools-public-publication-...
- LinuxBender 6y ago
  And Facebook had small UPS/ATS units at the end of each row. Not sure if they still do that today it was like that when I walked through their datacenter. They did that for the purpose of power efficiency. They lost far less power by having many smaller units.
  - lclarkmichalek 6y ago
    What you saw likely wasn't a UPS, but an RPP > A data center typically spans four rooms, called suites, where racks of servers are arranged in rows. Up to four MSBs provide power to each suite. In turn, each MSB supplies up to four 1.25 MW Switch Boards (SBs). From each SB, power is fed to the 190 KW Reactive Power Panels (RPPs) stationed at the end of each row of racks. https://research.fb.com/wp-content/uploads/2016/11/dynamo_facebook_s_data_center-wide_power_management_system.pdf https://research.fb.com/wp-content/uploads/2016/11/dynamo_fa... The UPS role is taken more by BBUs (Battery Backup Units). https://www.youtube.com/watch?v=KNsposM0sJE https://www.youtube.com/watch?v=KNsposM0sJE has lots of info about them. (I work for FB, on entirely unrelated things)
    - LinuxBender 6y ago
      They specifically called em out as UPS and talked about the efficiency improvements over having a dedicated UPS room. I don't remember the percentage numbers though. It was orders of magnitude less power loss. It could be they ditched all of that hardware by now. This was some time ago. You might ask some of the old timers in the DC. They probably have pictures of the hardware.
- dilyevsky 6y ago
  It’s true! I wrote software that upgraded firmware on every one of those batteries without frying them up (most of the time). There was a public paper/talk on that few years back
- WrtCdEvrydy 6y ago
  That was actually true, not an actual April Fool's day.
- tinus_hn 6y ago
  It sounds rather wasteful to have one device generate mains voltage AC power, only to have another device transform it to lower voltage power and rectify it.
  - xvf22 6y ago
    I mean chances are they are doing DC distribution. https://datacenterfrontier.com/google-unveils-48-volt-data-center-rack-joins-open-compute/ https://datacenterfrontier.com/google-unveils-48-volt-data-c...
kazen44 6y ago
in most datacenters, the UPS's are stored in the same collum like structure as their generators. A Datacenter is divided into vertical columns, with eacht collum being powered by redundant power feeds, generator(s) and UPS's.
bayindirh 6y ago
Our current DC has generation and UPSes in different rooms, in isolated places from each other. Both are pretty far from the actual DC itself.
walshemj 6y ago
Depends for a DC / Telco your normally feeding 48v DC from the UPS which is normally separate. The lack of fire suppression is also very worrying.
- bcrl 6y ago
  s/UPS/batteries/. DC makes uninterruptible power much more simpler than an AC UPS.
  - walshemj 6y ago
    Oh yes I sort of skipped over that part :-)
deleted 6y ago
[deleted]
throw0101a 6y ago
> Are there industry options or methods of wiring to allow for a UPS room separate from the actual rooms the racks are stored in? Yes: longer cables. See Figure 1 in the Schneider-APC white paper, where they have "Electrical Space", "Mechanical Space" (HVAC), and IT Space: * https://download.schneider-electric.com/files?p_File_Name=VAVR-8W4MEX_R2_EN.pdf https://download.schneider-electric.com/files?p_File_Name=VA... Power is generated hundreds of kilometres from where it is used, so having your UPS room a few dozen metres from your actual DC room isn't a big deal. I-squared-R losses aren't going to be that huge. Europe uses 400Y/230 for nominal low-voltage distribution (see Table 1 in above), so stringing some 400V extra copper to the PDUs, which then have 230V at the plugs, isn't a big deal.
hinkley 6y ago
The farther your UPS is away from the server the fewer causes of power loss you can prevent. I've seen people use UPSes to allow them to rearrange wiring. I've seen them fail by relying on the UPSes as well, of course. If you wire the entire room with 2-3 separate electrical systems all powered off of separate remote UPSes, you can do whatever you want, but it's harder to change your mind or build out incrementally if you do.
Reason077 6y ago
> "it's only a matter of time until Lithium chemistry batteries are available and that also increases the risk of fire." Not all Li-ion chemistries are equal. In particular, the increasingly popular LiFePO4 (LFP) technology has much higher energy density, longer lifespan, improved environmental characteristics, and similar if not better safety characteristics compared to lead-acid. (Besides sharing lead-acid's very low risk of fire, LiFePO4 also contains no corrosive acids which could damage and short equipment if a leak were to occur.)
- branko_d 6y ago
  > lead-acid's very low risk of fire Isn't lead-acid prone to releasing hydrogen gas? When jump-starting a car, it is commonly recommended to connect the ground (black) cable to the chassis, not to the battery's black terminal, to avoid a spark igniting the hydrogen and causing an explosion.
  - tempay 6y ago
    The lead-acid batteries used in data centres are normally "sealed lead-acid batteries" and release significantly less hydrogen. See: https://en.wikipedia.org/wiki/VRLA_battery https://en.wikipedia.org/wiki/VRLA_battery
    - doggodaddo78 6y ago
      Wikipedia is wrong. VRLAs aren't SLAs, and vice-versa. Actual SLAs are very, very expensive. All VRLAs can vent, by definition. VRLAs can overcharge and catch fire just like flooded batteries. They just don't go kaboom as much or spray acid everywhere. When having thousands of batteries in a room, one of them catching fire is inevitable. This is why batteries are in separate fire containment rooms, on nonflammable shelving, in redundant strings, and protected by FM200.
  - Nextgrid 6y ago
    Isn't hydrogen produced only while the battery is charging, or at worst, when discharging? If so, this still makes them much safer as they're otherwise inert while not in operation, compared to lithium-polymer which contains materials flammable at all times.
bzzzt 6y ago
I've been led around a DC in the Netherlands where they used a separate room for the UPS batteries. Not only because of safety but because the reliability of small rack-mount UPSes really sucks (and you have to match them to the power draw of the servers in the rack or pay for a lot of unused capacity) All racks and servers were connected to dual power feeds, so even when one of the feeds goes down the servers should run fine.
- jve 6y ago
  We do exactly the same you describe here in Latvia.
doggodaddo78 6y ago
Batteries burn up, especially NMC Li ion. Battery rooms were traditionally separate, used lead acid batteries, were surrounded by thicker walls, and equipped with FM200, just like main datacenter floors. They were typically placed near the transfer and PDU switchgear. I wouldn't put anything more flammable than LiFePO4 in a battery room, much less anywhere near a server. It's people who decide to throw away conventions, common sense, and building codes because "they know better" who get into trouble. I suspect this datacenter company could be sued into oblivion.

tyingq 6y ago

There are plenty of datacenters with a separate battery room, sure.

sschueller 6y ago
So far I have never been in one where this was not the case. Also they always have fire suppression.

mikepurvis 6y ago

For the extreme opposite of that, Google famously trolled everyone 10 years ago by announcing that every one of their servers had its own in-chassis 12V battery: https://www.cnet.com/news/google-uncloaks-once-secret-server-10209580/ https://www.cnet.com/news/google-uncloaks-once-secret-server...

jeffbee 6y ago
Yes but Google also later moved to a 48VDC power architecture with the batteries at the bottom of the rack. See http://apec.dev.itswebs.com/Portals/0/APEC%202017%20Files/Plenary/APEC%20Plenary%20Google.pdf http://apec.dev.itswebs.com/Portals/0/APEC%202017%20Files/Pl... page 6.
- tyingq 6y ago
  That presentation is pretty interesting, including Google inventing their own "Switched Tank" DC-DC converters because the existing ones weren't efficient or reliable enough.
  - jeffbee 6y ago
    There's a whole separate paper on that: https://storage.googleapis.com/pub-tools-public-publication-data/pdf/55746647b67067298c9455ece6d405872c9101c8.pdf https://storage.googleapis.com/pub-tools-public-publication-...
LinuxBender 6y ago
And Facebook had small UPS/ATS units at the end of each row. Not sure if they still do that today it was like that when I walked through their datacenter. They did that for the purpose of power efficiency. They lost far less power by having many smaller units.
- lclarkmichalek 6y ago
  What you saw likely wasn't a UPS, but an RPP > A data center typically spans four rooms, called suites, where racks of servers are arranged in rows. Up to four MSBs provide power to each suite. In turn, each MSB supplies up to four 1.25 MW Switch Boards (SBs). From each SB, power is fed to the 190 KW Reactive Power Panels (RPPs) stationed at the end of each row of racks. https://research.fb.com/wp-content/uploads/2016/11/dynamo_facebook_s_data_center-wide_power_management_system.pdf https://research.fb.com/wp-content/uploads/2016/11/dynamo_fa... The UPS role is taken more by BBUs (Battery Backup Units). https://www.youtube.com/watch?v=KNsposM0sJE https://www.youtube.com/watch?v=KNsposM0sJE has lots of info about them. (I work for FB, on entirely unrelated things)
  - LinuxBender 6y ago
    They specifically called em out as UPS and talked about the efficiency improvements over having a dedicated UPS room. I don't remember the percentage numbers though. It was orders of magnitude less power loss. It could be they ditched all of that hardware by now. This was some time ago. You might ask some of the old timers in the DC. They probably have pictures of the hardware.
dilyevsky 6y ago
It’s true! I wrote software that upgraded firmware on every one of those batteries without frying them up (most of the time). There was a public paper/talk on that few years back
WrtCdEvrydy 6y ago
That was actually true, not an actual April Fool's day.
tinus_hn 6y ago
It sounds rather wasteful to have one device generate mains voltage AC power, only to have another device transform it to lower voltage power and rectify it.
- xvf22 6y ago
  I mean chances are they are doing DC distribution. https://datacenterfrontier.com/google-unveils-48-volt-data-center-rack-joins-open-compute/ https://datacenterfrontier.com/google-unveils-48-volt-data-c...

kazen44 6y ago

in most datacenters, the UPS's are stored in the same collum like structure as their generators. A Datacenter is divided into vertical columns, with eacht collum being powered by redundant power feeds, generator(s) and UPS's.

bayindirh 6y ago

Our current DC has generation and UPSes in different rooms, in isolated places from each other. Both are pretty far from the actual DC itself.

walshemj 6y ago

Depends for a DC / Telco your normally feeding 48v DC from the UPS which is normally separate. The lack of fire suppression is also very worrying.

bcrl 6y ago
s/UPS/batteries/. DC makes uninterruptible power much more simpler than an AC UPS.
- walshemj 6y ago
  Oh yes I sort of skipped over that part :-)

deleted 6y ago

[deleted]

throw0101a 6y ago

> Are there industry options or methods of wiring to allow for a UPS room separate from the actual rooms the racks are stored in? Yes: longer cables. See Figure 1 in the Schneider-APC white paper, where they have "Electrical Space", "Mechanical Space" (HVAC), and IT Space: * https://download.schneider-electric.com/files?p_File_Name=VAVR-8W4MEX_R2_EN.pdf https://download.schneider-electric.com/files?p_File_Name=VA... Power is generated hundreds of kilometres from where it is used, so having your UPS room a few dozen metres from your actual DC room isn't a big deal. I-squared-R losses aren't going to be that huge. Europe uses 400Y/230 for nominal low-voltage distribution (see Table 1 in above), so stringing some 400V extra copper to the PDUs, which then have 230V at the plugs, isn't a big deal.

hinkley 6y ago

The farther your UPS is away from the server the fewer causes of power loss you can prevent. I've seen people use UPSes to allow them to rearrange wiring. I've seen them fail by relying on the UPSes as well, of course. If you wire the entire room with 2-3 separate electrical systems all powered off of separate remote UPSes, you can do whatever you want, but it's harder to change your mind or build out incrementally if you do.

Reason077 6y ago

> "it's only a matter of time until Lithium chemistry batteries are available and that also increases the risk of fire." Not all Li-ion chemistries are equal. In particular, the increasingly popular LiFePO4 (LFP) technology has much higher energy density, longer lifespan, improved environmental characteristics, and similar if not better safety characteristics compared to lead-acid. (Besides sharing lead-acid's very low risk of fire, LiFePO4 also contains no corrosive acids which could damage and short equipment if a leak were to occur.)

branko_d 6y ago
> lead-acid's very low risk of fire Isn't lead-acid prone to releasing hydrogen gas? When jump-starting a car, it is commonly recommended to connect the ground (black) cable to the chassis, not to the battery's black terminal, to avoid a spark igniting the hydrogen and causing an explosion.
- tempay 6y ago
  The lead-acid batteries used in data centres are normally "sealed lead-acid batteries" and release significantly less hydrogen. See: https://en.wikipedia.org/wiki/VRLA_battery https://en.wikipedia.org/wiki/VRLA_battery
  - doggodaddo78 6y ago
    Wikipedia is wrong. VRLAs aren't SLAs, and vice-versa. Actual SLAs are very, very expensive. All VRLAs can vent, by definition. VRLAs can overcharge and catch fire just like flooded batteries. They just don't go kaboom as much or spray acid everywhere. When having thousands of batteries in a room, one of them catching fire is inevitable. This is why batteries are in separate fire containment rooms, on nonflammable shelving, in redundant strings, and protected by FM200.
- Nextgrid 6y ago
  Isn't hydrogen produced only while the battery is charging, or at worst, when discharging? If so, this still makes them much safer as they're otherwise inert while not in operation, compared to lithium-polymer which contains materials flammable at all times.

bzzzt 6y ago

I've been led around a DC in the Netherlands where they used a separate room for the UPS batteries. Not only because of safety but because the reliability of small rack-mount UPSes really sucks (and you have to match them to the power draw of the servers in the rack or pay for a lot of unused capacity) All racks and servers were connected to dual power feeds, so even when one of the feeds goes down the servers should run fine.

jve 6y ago
We do exactly the same you describe here in Latvia.

doggodaddo78 6y ago

Batteries burn up, especially NMC Li ion. Battery rooms were traditionally separate, used lead acid batteries, were surrounded by thicker walls, and equipped with FM200, just like main datacenter floors. They were typically placed near the transfer and PDU switchgear. I wouldn't put anything more flammable than LiFePO4 in a battery room, much less anywhere near a server. It's people who decide to throw away conventions, common sense, and building codes because "they know better" who get into trouble. I suspect this datacenter company could be sued into oblivion.