The IBM POWER8 Review: Challenging the Intel Xeon
by Johan De Gelas on November 6, 2015 8:00 AM EST- Posted in
- IT Computing
- CPUs
- Enterprise
- Enterprise CPUs
- IBM
- POWER
- POWER8
Software Issues
So although the S824 is IBM's benchmark flagship for the scale-out range, the S812L and S822L are the servers that have the best chance at converting the kinds of users currently opt for x86 Xeons:
- Support for Little Endian data
- Best Linux support (Suse, Redhat & Ubuntu)
- (Somewhat) lower power
- 2U form factor which offers decent performance per U
- and probably the most important reason of all: Affordable! ($10k-25k instead of $30-60k)
So yes, the S822L looks like the first worthy alternative since 2010 for the dual Xeon servers. But the S822L did not inherit all the strong points of the typical "Big Blue" servers. The clockspeeds are a bit lower to keep the power consumption in check, and more importantly the LE Linux support is still very young. Sure, POWERLinux has been around for ages, but the software ecosystem was mostly supporting a few Big Endian applications like heavy duty Java servers and SAP.
Let's make the issue at hand a bit more tangible. IBM offers a migration advisor that helps developers to port their applications. That is definitely a good thing, but it also clearly illustrates that building a software ecosystem is a lot more cumbersome than the POWERPoint slides let you believe. In case of IBM's LE Linux, porting the rich x86 Linux software ecosystem to OpenPOWER is not that straightforward:
- Some code has inline x86 assembly such as thread resource locking code.
- Some code has x86 specific APIs
- No support for POWER in the make files which makes recompiling not straight forward
- POWER is 64 bit only.
We have experienced ourselves that this was more than just theory.
Case in point: for X86-64 we simply installed well tuned, ready to run, pre compiled binaries. Benchmarking is pretty easy here with a minor scripting effort.
The story was very different on the IBM S822L. We installed Ubuntu 15.04 (3.19.0-15 - ppc64le). To satisfy our curiosity we did a quick benchmark run with Linux-Bench, an automated benchmarking tool that Ian also likes to use. The benchmark did almost nothing on our POWER system despite the fact that most of the software had some form of support for POWER based systems.
The same was true for most software out there: We had to port most of the software by delving deep in all kinds of config, Readme, and make files. In many cases, we had to search around for alternative libraries that did support OpenPOWER.
Although a lot of software had an entry for "IBM POWER" in the make files, we encountered a lot of trouble. The server nor IBM is to blame: it is simply a fact that most developers - especially those with HPC software - have put a lot more effort in optimizing and validating their Intel x86 version of their software than the more "exotic" platforms.
Linux Ecosystem Not at Full Throttle.. Yet
It is clear to us that the OpenPOWER Linux ecosytem is still young and as a result does not offer the same performance as the older PowerVM and AIX platforms. There is still quite a bit of performance headroom.
A good example is the crypto acceleration. The IBM POWER8 has a dedicated cryptographic unit supporting new POWER ISA instructions to accelerate AES (Encryption), SHA (Hashing), and CRC (Cyclic Redundancy Check) codes. A similar encryption unit was already available in the POWER7+ . We found out that an nx-crypto driver was available and part of the Linux 3.5 kernel. However, even though Ubuntu 15.04 LE for OpenPOWER is based upon the Linux kernel 3.19, the nx-crypto driver was nowhere to be found. You could argue that the same is true for Intel as they introduce new instructions, but as far as we could see, there was no encryption acceleration whatsoever possible, not even based upon the older POWER7+.
A few days after we have finished testing, we found out the vmx-crypto driver will be available in distributions using the Kernel 4.1 and later and will be enabled in OpenSSL 1.0.2 (currently 1.0.1f in the standard repositories). The slide below - found in a presentation given this month - show how fast the ecosystem is expanding but also that it is still in flux.
OpenPOWER gained traction in 2014, the POWER8 is the first POWER chip with LE support and the number of Linux servers on top of OpenPOWER systems is still very small compared to x86. It is pretty simple: it is a much smaller community than the x86 linux server community. According to "the platform", IBM claims that "scale-out POWER8 machines have seen double digit revenue growth in the first half of 2015" but those growth numbers are "against a very small base". That tells us a lot: it is indeed a very small community, but a quickly growing one.
Facebook
Twitter
RSS
146 Comments
View All Comments
JohanAnandtech - Saturday, November 7, 2015 - link
suggestions on how to to do this? OpenSSL 1.02 will support the build in crypto accelerator, but I am not sure on how I would be able to see if the crypto code uses VMX.SarahKerrigan - Monday, November 9, 2015 - link
Compile with -qreport in XL C/C++.Oxford Guy - Saturday, November 7, 2015 - link
Typo on page 2:The resuls are that Google is supporting the efforts and Rackspace has even build their own OpenPOWER server called "Barreleye".
Ryan Smith - Saturday, November 7, 2015 - link
Thanks.iwod - Saturday, November 7, 2015 - link
In terms of 100, POWER Software Ecosystem manage to scale from 10 to 20, so that is a 100% increase but still very very low. Will we see POWER CPU / Server that is cheap enough to compete with Xeon E3 / E5, where most of the volume are? Compared to E7 is like comparing Server CPU for the 10% of the market.Intel will be moving to 14nm E7, I don't see anyone making POWER CPU at 14nm anytime soon.
Intel DC business are growing, and it desperately need a competitor, such as POWER to combat E7 and AMD Zen from the bottom.
Frenetic Pony - Saturday, November 7, 2015 - link
Nice review! It just confirms my question however of "What does IBM do?" Seriously, what do they do anymore? All I see are headlines for things that never come out as actual products. Their servers suck up too much power per watt, they don't have their own semi conductor foundries, their semi conductor research seems like a bunch of useless paper tiger stuff, their much vaunted AI is better at playing Jeapordy than seemingly any real world use.Countdown to complete IBM bankruptcy/spinoff/selloff is closer than ever.
ws3 - Saturday, November 7, 2015 - link
Since the dawn of computing, IBM has been in the business of providing solutions, rather than merely hardware. When you buy IBM you pay a huge amount of money, and what you get for that is support, with some hardware thrown in.Obviously this only appeals to wealthy customers who don't have or don't want to have an internal support organization that can duplicate what IBM offers. It seems to me that the number of such customers is decreasing over time, but as long as the US government is around, IBM will have at least one customer.
xype - Sunday, November 8, 2015 - link
They make 2-5 Billion dollars of profit per quarter. "Countdown to complete IBM bankruptcy/spinoff/selloff is closer than ever." my ass.PowerTrumps - Sunday, November 8, 2015 - link
Pretty fair and even handed review; don't agree with it all and definitely feel there is room to learn and improve. Btw, full disclosure, I am a System Architect focusing on Power technology for a Business Partner.With regard to compilers I would suggest IBM's SDK for Linux on Power & Advanced Tool Chain (ATC) provide development tools and open source optimized dev stack (ie gcc) for POWER8. Details at: https://www-304.ibm.com/webapp/set2/sas/f/lopdiags... and https://www.ibm.com/developerworks/community/wikis...
MySQL is definitely relevant but with the new Linux distro's packaging MariaDB in place of MySQL I would have liked to see an Intel vs Power comparison with this MySQL alternative. MariaDB just announced v10.1 is delivering over 1M queries per second on POWER8. https://blog.mariadb.org/10-1-mio-qps/
A commenter asked about Spark with POWER8. This blog discusses how it performs vs Intel. https://www.ibm.com/developerworks/community/blogs...
In addition to the commercial benchmarks often quoted such as SPEC, SAP and TPC like this SAP HANA result with SUSE on POWER8 ; SAP BW-EML (ie HANA) shows tremendous scaling with POWER8. http://www.smartercomputingblog.com/power-systems/... many of the ISV's have produced their own. I have seen results for PostgreSQL, STAC (http://financial.mcobject.com/press-release-novemb... Redis Labs, etc.
Benchmarks are great, all vendors do them and most people realize you should take them with a grain of salt. One benefit of Power servers when using PowerVM, its native firmware based hypervisor is that it delivers tremendous compute efficiency to VM's. On paper things like TDP seem higher for Power vs Intel (especially E5_v3 chips) but when Power servers deliver consolidation ratio's with 2-4X (and greater) more VM's per core the TCA & TCO get real interesting. One person commented how SAP on Power would blow out a budget. It does just the opposite because how you can run in a Tier-2 architecture obtaining intra-server VM to VM efficiencies, compute efficiencies with fewer cores & servers which impacts everything in the datacenter. Add in increased reliability & serviceability features and you touch the servers less which means your business is running longer.
And for more details on the open platform or those based on the OpenPOWER derivative using the "LC" designator such as S822LC in contrast to the S822L used as the focus in this article. http://www.smartercomputingblog.com/power-systems/... and http://businesspartnervoices.com/ibm-power-systems...
JohanAnandtech - Sunday, November 8, 2015 - link
Great feedback. We hope to get access to another POWER8(+) server and build further upon our existing knowledge. We have real world experience with Spark, so it is definitely on the list. The blog you linked seems to have used specific SPARK optimization for POWER, but the x86 reference system looks a bit "neglected". A real independent test would be very valuable there. The interesting part of Spark is that a good benchmark would be also very relevant for the real world as peak performance is one of the most important aspects of Spark, in contrast with databases where maximum performance is only a very small part of the experience.About MySQL, people have pointed out that the 5.7 version seems to scale a lot better, so that is together with MariaDB also on my "to test" list. Redis does not seem relevant for this kind of machine, it is single-threaded, almost impossible to test 160 instances.
The virtualization part is indeed one of the most interesting parts, but it is a benchmarking nightmare. You got to keep response times at more or less the same levels while loading the machine with more and more VMs. We did that kind of testing until 2 years ago on x86, but it was very time consuming and we had a deep understanding on how vSphere worked. Building that kind of knowledge on PowerVM might be beyond our manpower and time :-).