5 ms·

> I don't see how this is any different from a regular server rack well, the difference is that most rack cannot fit 10 TB of RAM and 140x5GHz cores: http://ww

by kkuduk 11y ago

> I don't see how this is any different from a regular server rack well, the difference is that most rack cannot fit 10 TB of RAM and 140x5GHz cores: http://www-03.ibm.com/systems/z/hardware/z13_specs.html http://www-03.ibm.com/systems/z/hardware/z13_specs.html but surely those are super-expensive and I doubt all that effective when it comes to performance per watt. I'm actually a bit more excited about Power8 servers, but are there any non-IBM available yet?

ajdlinux 11y ago
> I'm actually a bit more excited about Power8 servers, but are there any non-IBM available yet? Yep, OpenPOWER machines are starting to come out - Tyan has the Habanero server as well as the Palmetto reference/development platform. http://www.tyan.com/solutions/tyan_openpower_system.html http://www.tyan.com/solutions/tyan_openpower_system.html (disclosure: I'm an IBMer working on Power)
- A010 11y ago
  > High-performance computing (HPC) > Data centers > Big data Can you give me some specific roles for this range of servers? What makes it's worth compared to x86 system?
  - rwmj 11y ago
    The high-end POWER 7/8 hardware has incredible single core performance, beating the pants off Xeon. It uses huge amounts of power to do that, so it's not appropriate for all roles. Low end POWER is pretty niche. Freescale uses the architecture for telecoms applications. In general Linux upstream these days "just works" on ppc64 & ppc64le. There's RHEL for POWER already, and IBM have loaned hardware to the CentOS project so we'll get CentOS on POWER pretty soon. The licensing of (Open-)POWER is more open than x86 (but not as open as stuff like RISC-V), and there are several second sources for chips, in situations where that matters.
  - Spooky23 11y ago
    You get a pretty decent multiple of performance on power vs. intel. Probably something like 4-7x for common use cases. Generally speaking, you buy the hardware because your workload needs the single core performance, or you're arbitraging vendor licensing costs for software. Also, IBM's bread and butter is "peaky" financial services and gov't business, so they have business models that makes it work from a $ pov. You can buy a box with 100 cores, pay for 20, and lease 30 more for a few days to meet your peak demands for tax/christmas/billing season.
rodgerd 11y ago
> well, the difference is that most rack cannot fit 10 TB of RAM and 140x5GHz cores: http://www-03.ibm.com/systems/z/hardware/z13_specs.html http://www-03.ibm.com/systems/z/hardware/z13_specs.html I can fit 4 C7000 chassis in a single rack, which top out at an aggregate 32 TB of memory and over 2000 cores. 5 GHz cores running s390x microcode are not, FYI, twice as fast as Xeons. It would be great if people who weren't familiar with the state of the art in x86 kit didn't blindly assume IBM et al's advertising was reality.
- jamestnz 11y ago
  I'm no expert, but my understanding is that the IBM "z" platform has a number of interesting features perhaps not found in more typical server platforms/configurations, beyond just scaling out to loads of cores and RAM. As with all IBM mainframes going right back to the 60s, the "z" systems are designed for continuous uptime, on the order of decades, and this is evident in many of the design decisions. For example, various subsystems and components have hot-spares available, so that even outright component failures will not cause downtime. Many components are then hot-swappable, even those that might not ordinarily be in other architectures, such as processors and main memory. No interruption to OS or application-level services is expected by hot-swapping such hardware. Across the useful life of the mainframe, most repairs and maintenance would be carried out without ever shutting it down. I understand the platform also has extensive internal integrity checking built in, a potentially important factor for various types of jobs. Its auditing service is capable of detecting unusual conditions in various subsystems or jobs ("I've just picked up a fault in the AE-35 unit"), automatically retrying instructions on the processor if they executed anomalously. If the fault continues, the suspect processor is routed-around with no interruption to OS or applications, the job is resumed from last checkpoint on another processor, and the system phones home to IBM to log a service call. This monitoring is not being performed by processes running in userland or by the kernel, but is in fact baked into the hardware/firmware platform. Furthermore, the systems can be configured with a variety of specialty offload processors or subsystems for tasks like encryption, key management, compression, and even logging -- which again might not be so commonly found on-board of some commodity servers. (And, of course, even if you could put together an analogous solution with commodity kit, it's IBM! For the sorts of companies looking at a mainframe in 2015, having the IBM name on the SLA has got to be a pretty big part of the equation, right?) Moreover, if you proposed to build and manage these sorts of capabilities from commodity x86 kit, I imagine IBM would claim that they'd have the lower TCO.
vidarh 11y ago
> most rack cannot fit 10 TB of RAM and 140x5GHz cores Any rack can fit 10TB of RAM and the equivalent of 140x5GHz POWER cores. I can easily get ~1TB RAM and 64 2GHz+ x86 cores in 1U. A standard rack is at least 42U. In terms of density it's nothing special. What makes these different is that you get it in a box that someone else will be maintaining, and where all the dirty work of designing and building a high availability redundant system has been done for you. Most people never build anything remotely as redundant as these things tend to be.
- vegabook 11y ago
  I'm guessing part of the appeal will also be super-wide bandwidth between the cores. Interconnect throughput and/or shared memory could make this significantly more interesting than a modular rack-based system for some workloads.
  - rodgerd 11y ago
    > I'm guessing part of the appeal will also be super-wide bandwidth between the cores. If it's anything like previous-generations, that will be infiniband between books. The benefits on Z-class systems tend to be very big per-core, per-socket caches and, as the person you're replying to said, a lot of internal redundancy and automated failover (multiple backplanes with failover, spare memory and cores with failover, and so on and so forth).
jdboyd 11y ago
There is the TYAN GN70-BP010, although they state it isn't for production use.

by kkuduk 11y ago

ajdlinux 11y ago

> I'm actually a bit more excited about Power8 servers, but are there any non-IBM available yet? Yep, OpenPOWER machines are starting to come out - Tyan has the Habanero server as well as the Palmetto reference/development platform. http://www.tyan.com/solutions/tyan_openpower_system.html http://www.tyan.com/solutions/tyan_openpower_system.html (disclosure: I'm an IBMer working on Power)

A010 11y ago
> High-performance computing (HPC) > Data centers > Big data Can you give me some specific roles for this range of servers? What makes it's worth compared to x86 system?
- rwmj 11y ago
  The high-end POWER 7/8 hardware has incredible single core performance, beating the pants off Xeon. It uses huge amounts of power to do that, so it's not appropriate for all roles. Low end POWER is pretty niche. Freescale uses the architecture for telecoms applications. In general Linux upstream these days "just works" on ppc64 & ppc64le. There's RHEL for POWER already, and IBM have loaned hardware to the CentOS project so we'll get CentOS on POWER pretty soon. The licensing of (Open-)POWER is more open than x86 (but not as open as stuff like RISC-V), and there are several second sources for chips, in situations where that matters.
- Spooky23 11y ago
  You get a pretty decent multiple of performance on power vs. intel. Probably something like 4-7x for common use cases. Generally speaking, you buy the hardware because your workload needs the single core performance, or you're arbitraging vendor licensing costs for software. Also, IBM's bread and butter is "peaky" financial services and gov't business, so they have business models that makes it work from a $ pov. You can buy a box with 100 cores, pay for 20, and lease 30 more for a few days to meet your peak demands for tax/christmas/billing season.

rodgerd 11y ago

> well, the difference is that most rack cannot fit 10 TB of RAM and 140x5GHz cores: http://www-03.ibm.com/systems/z/hardware/z13_specs.html http://www-03.ibm.com/systems/z/hardware/z13_specs.html I can fit 4 C7000 chassis in a single rack, which top out at an aggregate 32 TB of memory and over 2000 cores. 5 GHz cores running s390x microcode are not, FYI, twice as fast as Xeons. It would be great if people who weren't familiar with the state of the art in x86 kit didn't blindly assume IBM et al's advertising was reality.

jamestnz 11y ago
I'm no expert, but my understanding is that the IBM "z" platform has a number of interesting features perhaps not found in more typical server platforms/configurations, beyond just scaling out to loads of cores and RAM. As with all IBM mainframes going right back to the 60s, the "z" systems are designed for continuous uptime, on the order of decades, and this is evident in many of the design decisions. For example, various subsystems and components have hot-spares available, so that even outright component failures will not cause downtime. Many components are then hot-swappable, even those that might not ordinarily be in other architectures, such as processors and main memory. No interruption to OS or application-level services is expected by hot-swapping such hardware. Across the useful life of the mainframe, most repairs and maintenance would be carried out without ever shutting it down. I understand the platform also has extensive internal integrity checking built in, a potentially important factor for various types of jobs. Its auditing service is capable of detecting unusual conditions in various subsystems or jobs ("I've just picked up a fault in the AE-35 unit"), automatically retrying instructions on the processor if they executed anomalously. If the fault continues, the suspect processor is routed-around with no interruption to OS or applications, the job is resumed from last checkpoint on another processor, and the system phones home to IBM to log a service call. This monitoring is not being performed by processes running in userland or by the kernel, but is in fact baked into the hardware/firmware platform. Furthermore, the systems can be configured with a variety of specialty offload processors or subsystems for tasks like encryption, key management, compression, and even logging -- which again might not be so commonly found on-board of some commodity servers. (And, of course, even if you could put together an analogous solution with commodity kit, it's IBM! For the sorts of companies looking at a mainframe in 2015, having the IBM name on the SLA has got to be a pretty big part of the equation, right?) Moreover, if you proposed to build and manage these sorts of capabilities from commodity x86 kit, I imagine IBM would claim that they'd have the lower TCO.

vidarh 11y ago

> most rack cannot fit 10 TB of RAM and 140x5GHz cores Any rack can fit 10TB of RAM and the equivalent of 140x5GHz POWER cores. I can easily get ~1TB RAM and 64 2GHz+ x86 cores in 1U. A standard rack is at least 42U. In terms of density it's nothing special. What makes these different is that you get it in a box that someone else will be maintaining, and where all the dirty work of designing and building a high availability redundant system has been done for you. Most people never build anything remotely as redundant as these things tend to be.

vegabook 11y ago
I'm guessing part of the appeal will also be super-wide bandwidth between the cores. Interconnect throughput and/or shared memory could make this significantly more interesting than a modular rack-based system for some workloads.
- rodgerd 11y ago
  > I'm guessing part of the appeal will also be super-wide bandwidth between the cores. If it's anything like previous-generations, that will be infiniband between books. The benefits on Z-class systems tend to be very big per-core, per-socket caches and, as the person you're replying to said, a lot of internal redundancy and automated failover (multiple backplanes with failover, spare memory and cores with failover, and so on and so forth).

jdboyd 11y ago

There is the TYAN GN70-BP010, although they state it isn't for production use.