8 ms·

What Linux can learn from Solaris performance and vice-versa

151 points by deirdres 13y ago

bingaling 13y ago
Looking forward to the video!
sharms 13y ago
This is a topic that I would love to see brought up more, and I think there is a lot Linux can learn from Solaris. As someone who was only exposed to Solaris in the last 4 years (using Linux since '96), I have become increasingly interested in the seemingly unimportant features which have been incredibly useful. Zones have a distinct maturity and robustness over LxC (to be expected), ZFS is the filesystem I wish we could have, dtrace isn't a bolt-on etc. It would be nice to see it fully open-sourced again as I think the one thing it has working against it is a smaller community and it lacks the same size army of driver authors.
- rasur 13y ago
  You'll be wanting to have a look at IllumOS then, methinks
- freiheit 13y ago
  There are a couple different ZFS ports for linux available. Native and FUSE. There are some linux distributions which include ZFS and can even do a ZFS root fs.
  - bashinator 13y ago
    I've run into severe problems with using the NFS provided by native ZFS on linux - it tends to completely die under load.
- bcantrill 13y ago
  As rasur mentions, illumos has inherited the Solaris heritage -- and added quite a bit to it besides.[1][2] There are now several vibrant variants of illumos, including SmartOS[3][4], OmniOS[5][6] and DelphixOS[7] -- all of which are open source, and all of which feature advancements in the core differentiators of the system including ZFS, DTrace and zones. Because we collectively have been entirely separate from our Solaris legacy for over three years now (and because we are entirely open source where Solaris is entirely closed source), the differences are actually substantial -- and I don't really think of our system as "Solaris" any more than I think of it as "System V" or "7th Edition" or anything else in our long and complicated ancestry... [1] http://www.youtube.com/watch?v=-zRN7XLCRhc http://www.youtube.com/watch?v=-zRN7XLCRhc [2] http://www.youtube.com/watch?v=7YN6_eRIWWc http://www.youtube.com/watch?v=7YN6_eRIWWc [3] http://smartos.org/ http://smartos.org/ [4] https://github.com/joyent/illumos-joyent https://github.com/joyent/illumos-joyent [5] http://omnios.omniti.com/ http://omnios.omniti.com/ [6] http://omnios.omniti.com/browse.php http://omnios.omniti.com/browse.php [7] https://github.com/delphix/delphix-os https://github.com/delphix/delphix-os
  - lmm 13y ago
    Have any of the open-source variants implemented ZFS encryption yet?
    - ketralnis 13y ago
      Not directly (Sun open sourced v28, but encryption shows up in v30), but there are other solutions to that problem. e.g. FreeBSD has geom_eli which can happily underly ZFS or any other filesystem (http://www5.us.freebsd.org/doc/handbook/disks-encrypting.html http://www5.us.freebsd.org/doc/handbook/disks-encrypting.htm...)
  - angersock 13y ago
    So, to a newcomer, what are the things that these flavors offer over, say, stock Debian?
    - nosequel 13y ago
      Try reading the linked slide deck for a start. ZFS, Zones, DTrace, Crossbow, etc., etc.
    - thelambentonion 13y ago
      While I haven't had a chance to use illumos in any of my projects, the Service Management Facility (SMF)[0] and Fault Manager Daemon (FMD)[1] are two big reasons why I'm looking at switching from FreeBSD to an illumos variant for my home server. It's been a little while since I've used Debian, but I don't believe any systems comparable to SMF or FMD have been implemented[2]. [0] https://www.illumos.org/man/5/smf https://www.illumos.org/man/5/smf [1] https://www.illumos.org/man/1M/fmd https://www.illumos.org/man/1M/fmd [2] I'm not sure how well systemd compares to SMF, and I'm mostly interested in FMD/SMF due to how well it can handle ZFS fault management.
- pjmlp 13y ago
  Yeah, many GNU/Linux users tend to think UNIX == GNU/Linux, which couldn't be further from the truth. Beyond the basic POSIX compatibility, there are lots of differences across UNIX systems, each has their plus and minus points.
nailer 13y ago
'Linux might get DTrace-like capabilities via SystemTap' Might? SystemTap's been stable & out of the box on RHEL 5 for years. The extensive list of default tap points are supported on production by RH. When I was doing systems stuff 5 years ago we used it for double checking TCP socket options that weren't normally exposed to userspace tools. Dtrace can do some awesome stuff stap can't, like trace something from userspace down to kernel space. But systemtap does provide live instrumentation on running kernels and has for years. PS: Just saw p93. Sun got in trouble for the obvious "Setting up SystemTap is difficult on distros that don't maintain, support, use or like SystemTap" years ago. This is repeated on p 93. Stop it, this makes Joyent look stupid, you're better than this. And p100. Re: 'several million dollar E10K' performance? Red Hat were replacing them with Xeon's by 2003 (with Sun's old sales staff) and crushing them. Though that was SPARC's fault not Solaris's.
- bcantrill 13y ago
  (Disclosure: I work at Joyent.) Are you accusing Brendan of misrepresenting SystemTap on slide 93? If so, could you be specific as to what that misrepresentation is? Brendan has a ton of experience earnestly using SystemTap (or rather, trying to use it), and it's hard to see how sharing that experience makes either Brendan or his employer "look stupid." For more details on Brendan's explorations of SystemTap: http://dtrace.org/blogs/brendan/2011/10/15/using-systemtap/ That's several years old at this point, but the architectural limitations highlighted by Brendan haven't changed.
  - nailer 13y ago
    Yes, I am accusing the author of deliberately misrepresenting facts. I'm not sure what's vague, let me recap: 1. 'might' implies the future. As you can tell from the post you're replying to, Systemtap is present thing. I think I actually put most of the tapsets shown on the Wikipedia page around 2009, and I was using them then. 2. Also, as the author partially acknowledges in the next slide, Ubuntu is the dumbest place to illustrate a typical Systemtap experience. I suspect that's deliberate, rather than ignorance as Sun tried that a few years back and I don't think anyone with any interest in the matter is ignorant enough not to remember the shitstorm. As I've written elsewhere: if your customers mostly run Ubuntu, then test dtrace on Ubuntu too. It'd give you just as accurate impression about what the dtrace 'experience' is as installing SystemTap on Ubuntu. Edit: actually looks like Joyent wrote the shitstorm post for a few years ago, you've still got it up, you still don't see anything wrong, and you just linked to it. christ read the HN comments from 2011. https://news.ycombinator.com/item?id=3118416 https://news.ycombinator.com/item?id=3118416 Edit 2: you're Bryan Cantrill. You didn't seem to care much about performance a while ago: http://cryptnet.net/mirrors/texts/kissedagirl.html http://cryptnet.net/mirrors/texts/kissedagirl.html Edit 3: more Cantrill classiness http://www.quora.com/Node-js/Why-did-Ben-Noordhuis-decide-to-leave-Node-js-development http://www.quora.com/Node-js/Why-did-Ben-Noordhuis-decide-to... Sorry to anyone who thinks I've gotten personal, but this guy is a well known... unsavoury character.
lazyloop 13y ago
Just yesterday Perl has once again been declared dead on Hacker News, and here we have it happily strutting along in a keynote about a bleeding edge topic. :)
the_mitsuhiko 13y ago
The sad part is that both dtrace and ZFS originally did not end up in the kernel purely because of licensing reasons :-(
- throwaway2048 13y ago
  There is more to it than that, the CDDL was intentionally created by Sun to be GPL incompatable, in a bid to create their own linux-like ecosystem with OpenSolaris. The incompatable nature of the license was so linux could not merge things like ZFS and dtrace. it was yet more Sun licensing games that only hurt adoption in the end, like their Java shenanagins. Summing it up as "aww gee shucks licensing problems have struck again" ignores the fact that such crap was an intentional power-play from sun.
  - lucian1900 13y ago
    After an intentional power play from GNU. Let's not absolve them of blame. A clash of politics had a sad effect on actual technology.
    - deleted 13y ago
      [deleted]
apaprocki 13y ago
What surprised me most is that SPARC is only ~2x slower after all these years (3.6ghz T5 / Sol11) ;) $ time perl -e 'for ($i = 0; $i < 100_000_000; $i++) { $s = "SCaLE12x" }' real 0m33.59s user 0m33.57s sys 0m0.00s
- twoodfin 13y ago
  I know multithread is the T5 sweet spot (to put it mildly), but my 2012 Retina MacBook Pro hits 10s and cost ~$2,800. The cheapest T5 I can find on Oracle's site starts at around $28,000. I'm only jealous because I'd really like my employer to buy a T5 for compiling and running our software on Solaris SPARC, but unfortunately I don't think it's justified given the state of the platform.
deleted 13y ago
[deleted]
jackgavigan 13y ago
How can you talk about Solaris performance without mentioning dispadmin?
- brendangregg 13y ago
  Linux has CPU scheduler tunables via CFS/cgroups. dispadmin allows fine tuning of the dispatcher table, but I've never seen that provide significant or consistent wins for production (and I've had use cases where it seemed possible in theory, and dtrace'd dispatcher queue latency, but dispadmin didn't really help.) I might mention dispadmin in a 5 day talk. :)
binarycrusader 13y ago
It seems like futexes are really just an alternative interface made to build pthread mutexes on top of. In addition, they may only be faster on Linux because of implementation issues. It doesn't seem worth including for comparison. Also, I'm confused by the claim that Solaris doesn't have the vfsstat utility; yes, it doesn't have that utility, but it does have "fsstat" which provides VFS-level statistics. So it'd be helpful to see a more detailed comparison.
patrickg_zill 13y ago
One thing I noticed about Solaris - it handles low-memory situations (like 15.5G out of 16GB being used) and swapping, very very well. If Linux has enough RAM, it will outperform Solaris, but, it is quicker to get bogged down if there is swapping. (These observations are from about Centos 5.x , so, perhaps they are no longer valid, however.)
- GFK_of_xmaspast 13y ago
  I find myself having to run with ulimit -v #### in a term on linux if I think I might start eating memory because I'm sick of having to craw under my desk to power cycle.
  - ithkuil 13y ago
    why not turn off swap entirely?
GFK_of_xmaspast 13y ago
The last bullet in slide 83 isn't exactly a selling point.
angry_octet 13y ago
I detest slideshare. The way they try to monetize and track me by making me log in to download the PDF. The dumb way they show the slides on a third of the screen and fill the rest with crud. So if you really don't want your slides read, put them on slideshare.
- brendangregg 13y ago
  There is a direct PDF link here: http://www.brendangregg.com/linuxperf.html http://www.brendangregg.com/linuxperf.html
vidoc 13y ago
Sure yeah, Solaris is full of great technologies, it is undeniable, and DTrace is an awesome and unique one. Now there's one thing those guys can learn from Linux: modesty.
deirdres 13y ago
Video now available: http://www.joyent.com/blog/what-linux-can-learn-from-solaris-performance-and-vice-versa http://www.joyent.com/blog/what-linux-can-learn-from-solaris...

151 points by deirdres 13y ago

Looking forward to the video!

This is a topic that I would love to see brought up more, and I think there is a lot Linux can learn from Solaris. As someone who was only exposed to Solaris in the last 4 years (using Linux since '96), I have become increasingly interested in the seemingly unimportant features which have been incredibly useful. Zones have a distinct maturity and robustness over LxC (to be expected), ZFS is the filesystem I wish we could have, dtrace isn't a bolt-on etc. It would be nice to see it fully open-sourced again as I think the one thing it has working against it is a smaller community and it lacks the same size army of driver authors.

rasur 13y ago
You'll be wanting to have a look at IllumOS then, methinks
freiheit 13y ago
There are a couple different ZFS ports for linux available. Native and FUSE. There are some linux distributions which include ZFS and can even do a ZFS root fs.
- bashinator 13y ago
  I've run into severe problems with using the NFS provided by native ZFS on linux - it tends to completely die under load.
bcantrill 13y ago
As rasur mentions, illumos has inherited the Solaris heritage -- and added quite a bit to it besides.[1][2] There are now several vibrant variants of illumos, including SmartOS[3][4], OmniOS[5][6] and DelphixOS[7] -- all of which are open source, and all of which feature advancements in the core differentiators of the system including ZFS, DTrace and zones. Because we collectively have been entirely separate from our Solaris legacy for over three years now (and because we are entirely open source where Solaris is entirely closed source), the differences are actually substantial -- and I don't really think of our system as "Solaris" any more than I think of it as "System V" or "7th Edition" or anything else in our long and complicated ancestry... [1] http://www.youtube.com/watch?v=-zRN7XLCRhc http://www.youtube.com/watch?v=-zRN7XLCRhc [2] http://www.youtube.com/watch?v=7YN6_eRIWWc http://www.youtube.com/watch?v=7YN6_eRIWWc [3] http://smartos.org/ http://smartos.org/ [4] https://github.com/joyent/illumos-joyent https://github.com/joyent/illumos-joyent [5] http://omnios.omniti.com/ http://omnios.omniti.com/ [6] http://omnios.omniti.com/browse.php http://omnios.omniti.com/browse.php [7] https://github.com/delphix/delphix-os https://github.com/delphix/delphix-os
- lmm 13y ago
  Have any of the open-source variants implemented ZFS encryption yet?
  - ketralnis 13y ago
    Not directly (Sun open sourced v28, but encryption shows up in v30), but there are other solutions to that problem. e.g. FreeBSD has geom_eli which can happily underly ZFS or any other filesystem (http://www5.us.freebsd.org/doc/handbook/disks-encrypting.html http://www5.us.freebsd.org/doc/handbook/disks-encrypting.htm...)
- angersock 13y ago
  So, to a newcomer, what are the things that these flavors offer over, say, stock Debian?
  - nosequel 13y ago
    Try reading the linked slide deck for a start. ZFS, Zones, DTrace, Crossbow, etc., etc.
  - thelambentonion 13y ago
    While I haven't had a chance to use illumos in any of my projects, the Service Management Facility (SMF)[0] and Fault Manager Daemon (FMD)[1] are two big reasons why I'm looking at switching from FreeBSD to an illumos variant for my home server. It's been a little while since I've used Debian, but I don't believe any systems comparable to SMF or FMD have been implemented[2]. [0] https://www.illumos.org/man/5/smf https://www.illumos.org/man/5/smf [1] https://www.illumos.org/man/1M/fmd https://www.illumos.org/man/1M/fmd [2] I'm not sure how well systemd compares to SMF, and I'm mostly interested in FMD/SMF due to how well it can handle ZFS fault management.
pjmlp 13y ago
Yeah, many GNU/Linux users tend to think UNIX == GNU/Linux, which couldn't be further from the truth. Beyond the basic POSIX compatibility, there are lots of differences across UNIX systems, each has their plus and minus points.

nailer 13y ago

'Linux might get DTrace-like capabilities via SystemTap' Might? SystemTap's been stable & out of the box on RHEL 5 for years. The extensive list of default tap points are supported on production by RH. When I was doing systems stuff 5 years ago we used it for double checking TCP socket options that weren't normally exposed to userspace tools. Dtrace can do some awesome stuff stap can't, like trace something from userspace down to kernel space. But systemtap does provide live instrumentation on running kernels and has for years. PS: Just saw p93. Sun got in trouble for the obvious "Setting up SystemTap is difficult on distros that don't maintain, support, use or like SystemTap" years ago. This is repeated on p 93. Stop it, this makes Joyent look stupid, you're better than this. And p100. Re: 'several million dollar E10K' performance? Red Hat were replacing them with Xeon's by 2003 (with Sun's old sales staff) and crushing them. Though that was SPARC's fault not Solaris's.

bcantrill 13y ago
(Disclosure: I work at Joyent.) Are you accusing Brendan of misrepresenting SystemTap on slide 93? If so, could you be specific as to what that misrepresentation is? Brendan has a ton of experience earnestly using SystemTap (or rather, trying to use it), and it's hard to see how sharing that experience makes either Brendan or his employer "look stupid." For more details on Brendan's explorations of SystemTap: http://dtrace.org/blogs/brendan/2011/10/15/using-systemtap/ That's several years old at this point, but the architectural limitations highlighted by Brendan haven't changed.
- nailer 13y ago
  Yes, I am accusing the author of deliberately misrepresenting facts. I'm not sure what's vague, let me recap: 1. 'might' implies the future. As you can tell from the post you're replying to, Systemtap is present thing. I think I actually put most of the tapsets shown on the Wikipedia page around 2009, and I was using them then. 2. Also, as the author partially acknowledges in the next slide, Ubuntu is the dumbest place to illustrate a typical Systemtap experience. I suspect that's deliberate, rather than ignorance as Sun tried that a few years back and I don't think anyone with any interest in the matter is ignorant enough not to remember the shitstorm. As I've written elsewhere: if your customers mostly run Ubuntu, then test dtrace on Ubuntu too. It'd give you just as accurate impression about what the dtrace 'experience' is as installing SystemTap on Ubuntu. Edit: actually looks like Joyent wrote the shitstorm post for a few years ago, you've still got it up, you still don't see anything wrong, and you just linked to it. christ read the HN comments from 2011. https://news.ycombinator.com/item?id=3118416 https://news.ycombinator.com/item?id=3118416 Edit 2: you're Bryan Cantrill. You didn't seem to care much about performance a while ago: http://cryptnet.net/mirrors/texts/kissedagirl.html http://cryptnet.net/mirrors/texts/kissedagirl.html Edit 3: more Cantrill classiness http://www.quora.com/Node-js/Why-did-Ben-Noordhuis-decide-to-leave-Node-js-development http://www.quora.com/Node-js/Why-did-Ben-Noordhuis-decide-to... Sorry to anyone who thinks I've gotten personal, but this guy is a well known... unsavoury character.

lazyloop 13y ago

Just yesterday Perl has once again been declared dead on Hacker News, and here we have it happily strutting along in a keynote about a bleeding edge topic. :)

the_mitsuhiko 13y ago

The sad part is that both dtrace and ZFS originally did not end up in the kernel purely because of licensing reasons :-(

throwaway2048 13y ago
There is more to it than that, the CDDL was intentionally created by Sun to be GPL incompatable, in a bid to create their own linux-like ecosystem with OpenSolaris. The incompatable nature of the license was so linux could not merge things like ZFS and dtrace. it was yet more Sun licensing games that only hurt adoption in the end, like their Java shenanagins. Summing it up as "aww gee shucks licensing problems have struck again" ignores the fact that such crap was an intentional power-play from sun.
- lucian1900 13y ago
  After an intentional power play from GNU. Let's not absolve them of blame. A clash of politics had a sad effect on actual technology.
  - deleted 13y ago
    [deleted]

apaprocki 13y ago

What surprised me most is that SPARC is only ~2x slower after all these years (3.6ghz T5 / Sol11) ;) $ time perl -e 'for ($i = 0; $i < 100_000_000; $i++) { $s = "SCaLE12x" }' real 0m33.59s user 0m33.57s sys 0m0.00s

twoodfin 13y ago
I know multithread is the T5 sweet spot (to put it mildly), but my 2012 Retina MacBook Pro hits 10s and cost ~$2,800. The cheapest T5 I can find on Oracle's site starts at around $28,000. I'm only jealous because I'd really like my employer to buy a T5 for compiling and running our software on Solaris SPARC, but unfortunately I don't think it's justified given the state of the platform.

deleted 13y ago

[deleted]

jackgavigan 13y ago

How can you talk about Solaris performance without mentioning dispadmin?

brendangregg 13y ago
Linux has CPU scheduler tunables via CFS/cgroups. dispadmin allows fine tuning of the dispatcher table, but I've never seen that provide significant or consistent wins for production (and I've had use cases where it seemed possible in theory, and dtrace'd dispatcher queue latency, but dispadmin didn't really help.) I might mention dispadmin in a 5 day talk. :)

binarycrusader 13y ago

It seems like futexes are really just an alternative interface made to build pthread mutexes on top of. In addition, they may only be faster on Linux because of implementation issues. It doesn't seem worth including for comparison. Also, I'm confused by the claim that Solaris doesn't have the vfsstat utility; yes, it doesn't have that utility, but it does have "fsstat" which provides VFS-level statistics. So it'd be helpful to see a more detailed comparison.

patrickg_zill 13y ago

One thing I noticed about Solaris - it handles low-memory situations (like 15.5G out of 16GB being used) and swapping, very very well. If Linux has enough RAM, it will outperform Solaris, but, it is quicker to get bogged down if there is swapping. (These observations are from about Centos 5.x , so, perhaps they are no longer valid, however.)

GFK_of_xmaspast 13y ago
I find myself having to run with ulimit -v #### in a term on linux if I think I might start eating memory because I'm sick of having to craw under my desk to power cycle.
- ithkuil 13y ago
  why not turn off swap entirely?

GFK_of_xmaspast 13y ago

The last bullet in slide 83 isn't exactly a selling point.

angry_octet 13y ago

I detest slideshare. The way they try to monetize and track me by making me log in to download the PDF. The dumb way they show the slides on a third of the screen and fill the rest with crud. So if you really don't want your slides read, put them on slideshare.

brendangregg 13y ago
There is a direct PDF link here: http://www.brendangregg.com/linuxperf.html http://www.brendangregg.com/linuxperf.html

vidoc 13y ago

Sure yeah, Solaris is full of great technologies, it is undeniable, and DTrace is an awesome and unique one. Now there's one thing those guys can learn from Linux: modesty.

deirdres 13y ago

Video now available: http://www.joyent.com/blog/what-linux-can-learn-from-solaris-performance-and-vice-versa http://www.joyent.com/blog/what-linux-can-learn-from-solaris...