27 ms·

cpuinfo or don't trust your cores to rackspace part i

9 points by inaka 17y ago

jacquesm 17y ago
If you never run top '1' then you shouldn't be operating servers for customers.
- inaka 17y ago
  attempting to ignore snarkyness, but failing - and that says what about the dozens of rackspace engineers configuring and monitoring and supporting the box over the past two years?
  - jacquesm 17y ago
    There once was a really nice quote here on HN: "you can't outsource responsibility". If in two years time you've never ever had a look at what kernel you are running, especially while tuning a system for performance you only have yourself to blame. Don't tell me you're running a 'stock' kernel and never bothered tuning it for your application, or considered upgrading it. Also, in your resources list you should have the exact machine configuration, there are tools to retrieve that sort of info automatically. Then, when you're done, store it in http://inventory.sf.net/ http://inventory.sf.net/ or something like that. It's typical that the people at rackspace would simply drop in the requested hardware, and that you yourself deal with the configuration. The smart money is on running some tests after they've done that to make sure it went ok. Asking for a CPU upgrade and not checking if they're operational is just plain stupid. I figure you literally asked rackspace to upgrade the CPU, and that's what they did. Did you explicitly ask them to install an SMP kernel with a specific version and they didn't do it ? Or did you expect them to do it but you didn't check if they actually did until today ? Two full years of trying to tune a box for performance and not noticing this, then publicly blaming rackspace is simply cheap, an attempt at pinning the blame on rackspace, for something that you should have noticed long ago yourself. Kudos for writing about it but the title should be "How I messed up". That's taking responsibility and then make sure it never ever happens again.
    - sailormoon 17y ago
      I really do not dig this tone. The guy is obviously not a system admin. He paid top dollar for rackspace managed hosting precisely so he wouldn't have to do the kinds of things you mention. "You can't outsource responsibility" is utter nonsense. It is completely impossible to "own" responsibility for everything important in a complex society. Meaningless platitudes should not distract from the fact - Rackspace did not do their job. Yes, he messed up. He messed up by making assumptions and not checking Rackspace's work more closely. That's not the same as messing up in your own work. His post is a reminder to be more careful checking on the work of your "upstream". There's no need to pile on with the "if you didn't know 'top 1' you shouldn't be running a startup!" etc.
      - jacquesm 17y ago
        I didn't say he shouldn't be running a startup, I said he should not be managing the servers their customers stuff runs on. As for the tone, you may disagree with that but that does not distract from the fact that if you operate a business, that you should know your stuff. And if you outsource something you should at least know how to check up on the bits that you've outsourced. Outsourcing does not mean that your responsibility disappears, it simply changes from 'doing' to 'monitoring'. Maybe rackspace did not do their job, I have no insight in the communications that went on between the party involved and rackspace. All we get here is a pointing finger without any responsibility taken, that is not a realistic picture. It could be the difference in the wording of the upgrade request ("please install another CPU in our machine" vs "please install and configure another CPU in our machine"). Even then, rackspace probably should get part of the blame, but really not all of it. The fact that the situation persisted for two years is completely on the OPs account, in two years you have many more opportunities than your hosting provider to find this out, after all they will leave your machine alone unless it malfunctions and there is no indication that they ever were requested to look in to this, and when they were they actually found the problem. I quote from the article "In investigating an unrelated issue, we followed up with Rackspace on a Kernel patch that couldn’t be applied to our server. One of the technicians immediately realized why – we were not running the SMP kernel." How come someone is trying to patch a kernel, can't apply the patch and then still doesn't clue in to the situation ? Also, we do not know if the SMP kernel was installed or not, it might have been, and then on the final reboot the wrong kernel was brought up. And that's a very easy mistake to make. But dmesg would tell you in a heartbeat, as would 'top '1'', which you would be using plenty of times while debugging performance issues to make sure all your cores are doing the right amount of work.
- aminuit 17y ago
  If you're bragging about 300ms average response times... If you're using "spidey-sense" to determine the underperforming elements in your software stack... There are a number of uncomfortable ideas in this article.
  - thaumaturgy 17y ago
    If you're swapping out your web server software without doing some more careful analysis first...
ericwaller 17y ago
After the conversion from Apache [+passenger] to Nginx [+unicorn] overall throughput went up by a factor of 3. I noticed a similar gain (10-12 req/s to 25-30req/s) for action cached pages after switching out nginx+passenger for nginx+thin. This is on a 256mb slice, and seems totally counter to the general opinion that passenger is great for VPSes.
- jcapote 17y ago
  Have you tried nginx/passenger? I ask because I wonder how much of that gain is part of switching to nginx rather than switching to unicorn...
  - inaka 17y ago
    on a separate application and server, yes, and it seems faster than apache/passenger, but it still suffers from the 'passenger choke' where touching tmp/restart.txt seems to pause all web traffic for 10-15 seconds, so we never considered it here...
jbyers 17y ago
Don't trust any servers to anyone. When we get a new server we check its stats against reality (have had upside and downside surprises on CPUs), run bonnie++ to make sure IO is as expected (it hasn't been due to exotic RAID problems), and run memtester to see if we have bad RAM (had that too). Takes more time, sure, but no surprises later.
- jacquesm 17y ago
  That's the right attitude. For extra points do serious burn-ins, especially on network hardware, keep a good eye on those error counters as well as mcelog in case you got a faulty ram in there. It's all part of commissioning a server, especially if you host at a cheap outlet like rackspace.
  - cperciva 17y ago
    a cheap outlet like rackspace. I've heard rackspace called lots of things, but this is the first time I've heard someone call them "cheap". Have their prices gone down lately?
    - jacquesm 17y ago
      On a relative scale they're cheap, what they call 'managed hosting' though is not what I'd call managed hosting. I think they call it managed hosting because they will do backups for you or something like that :) The Planet/EV1, which was my choice when hosting in the US earlier was quite a bit cheaper, but service there was absolutely terrible. It got to the point where I reprogrammed the DRAC cards to lock out their sys admins. After The Planet took over we had all kinds of issues, then finally they had an explosion in a transformer in one of their datacenters taking down all of our stuff for days on end. After that we moved out. They said it had nothing to do with them and we would be credited for the downtime if we stayed for at least another 6 months, but we had by then already signed up elsewhere and restored from backups. I figure if you're willing to run your operation that close to the red line then we should be taking our business elsewhere. They were lucky nobody got hurt. Right now we're hosting in three places, leaseweb, mojohost and virtual acccess. VXS is by far the best but expensive, leaseweb is somewhere in the middle and for high volume mojohost is absolutely unbeatable. btw, you have me curious what else you've heard rackspace called :)

9 points by inaka 17y ago

jacquesm 17y ago

If you never run top '1' then you shouldn't be operating servers for customers.

inaka 17y ago
attempting to ignore snarkyness, but failing - and that says what about the dozens of rackspace engineers configuring and monitoring and supporting the box over the past two years?
- jacquesm 17y ago
  There once was a really nice quote here on HN: "you can't outsource responsibility". If in two years time you've never ever had a look at what kernel you are running, especially while tuning a system for performance you only have yourself to blame. Don't tell me you're running a 'stock' kernel and never bothered tuning it for your application, or considered upgrading it. Also, in your resources list you should have the exact machine configuration, there are tools to retrieve that sort of info automatically. Then, when you're done, store it in http://inventory.sf.net/ http://inventory.sf.net/ or something like that. It's typical that the people at rackspace would simply drop in the requested hardware, and that you yourself deal with the configuration. The smart money is on running some tests after they've done that to make sure it went ok. Asking for a CPU upgrade and not checking if they're operational is just plain stupid. I figure you literally asked rackspace to upgrade the CPU, and that's what they did. Did you explicitly ask them to install an SMP kernel with a specific version and they didn't do it ? Or did you expect them to do it but you didn't check if they actually did until today ? Two full years of trying to tune a box for performance and not noticing this, then publicly blaming rackspace is simply cheap, an attempt at pinning the blame on rackspace, for something that you should have noticed long ago yourself. Kudos for writing about it but the title should be "How I messed up". That's taking responsibility and then make sure it never ever happens again.
  - sailormoon 17y ago
    I really do not dig this tone. The guy is obviously not a system admin. He paid top dollar for rackspace managed hosting precisely so he wouldn't have to do the kinds of things you mention. "You can't outsource responsibility" is utter nonsense. It is completely impossible to "own" responsibility for everything important in a complex society. Meaningless platitudes should not distract from the fact - Rackspace did not do their job. Yes, he messed up. He messed up by making assumptions and not checking Rackspace's work more closely. That's not the same as messing up in your own work. His post is a reminder to be more careful checking on the work of your "upstream". There's no need to pile on with the "if you didn't know 'top 1' you shouldn't be running a startup!" etc.
    - jacquesm 17y ago
      I didn't say he shouldn't be running a startup, I said he should not be managing the servers their customers stuff runs on. As for the tone, you may disagree with that but that does not distract from the fact that if you operate a business, that you should know your stuff. And if you outsource something you should at least know how to check up on the bits that you've outsourced. Outsourcing does not mean that your responsibility disappears, it simply changes from 'doing' to 'monitoring'. Maybe rackspace did not do their job, I have no insight in the communications that went on between the party involved and rackspace. All we get here is a pointing finger without any responsibility taken, that is not a realistic picture. It could be the difference in the wording of the upgrade request ("please install another CPU in our machine" vs "please install and configure another CPU in our machine"). Even then, rackspace probably should get part of the blame, but really not all of it. The fact that the situation persisted for two years is completely on the OPs account, in two years you have many more opportunities than your hosting provider to find this out, after all they will leave your machine alone unless it malfunctions and there is no indication that they ever were requested to look in to this, and when they were they actually found the problem. I quote from the article "In investigating an unrelated issue, we followed up with Rackspace on a Kernel patch that couldn’t be applied to our server. One of the technicians immediately realized why – we were not running the SMP kernel." How come someone is trying to patch a kernel, can't apply the patch and then still doesn't clue in to the situation ? Also, we do not know if the SMP kernel was installed or not, it might have been, and then on the final reboot the wrong kernel was brought up. And that's a very easy mistake to make. But dmesg would tell you in a heartbeat, as would 'top '1'', which you would be using plenty of times while debugging performance issues to make sure all your cores are doing the right amount of work.
aminuit 17y ago
If you're bragging about 300ms average response times... If you're using "spidey-sense" to determine the underperforming elements in your software stack... There are a number of uncomfortable ideas in this article.
- thaumaturgy 17y ago
  If you're swapping out your web server software without doing some more careful analysis first...

ericwaller 17y ago

After the conversion from Apache [+passenger] to Nginx [+unicorn] overall throughput went up by a factor of 3. I noticed a similar gain (10-12 req/s to 25-30req/s) for action cached pages after switching out nginx+passenger for nginx+thin. This is on a 256mb slice, and seems totally counter to the general opinion that passenger is great for VPSes.

jcapote 17y ago
Have you tried nginx/passenger? I ask because I wonder how much of that gain is part of switching to nginx rather than switching to unicorn...
- inaka 17y ago
  on a separate application and server, yes, and it seems faster than apache/passenger, but it still suffers from the 'passenger choke' where touching tmp/restart.txt seems to pause all web traffic for 10-15 seconds, so we never considered it here...

jbyers 17y ago

Don't trust any servers to anyone. When we get a new server we check its stats against reality (have had upside and downside surprises on CPUs), run bonnie++ to make sure IO is as expected (it hasn't been due to exotic RAID problems), and run memtester to see if we have bad RAM (had that too). Takes more time, sure, but no surprises later.

jacquesm 17y ago
That's the right attitude. For extra points do serious burn-ins, especially on network hardware, keep a good eye on those error counters as well as mcelog in case you got a faulty ram in there. It's all part of commissioning a server, especially if you host at a cheap outlet like rackspace.
- cperciva 17y ago
  a cheap outlet like rackspace. I've heard rackspace called lots of things, but this is the first time I've heard someone call them "cheap". Have their prices gone down lately?
  - jacquesm 17y ago
    On a relative scale they're cheap, what they call 'managed hosting' though is not what I'd call managed hosting. I think they call it managed hosting because they will do backups for you or something like that :) The Planet/EV1, which was my choice when hosting in the US earlier was quite a bit cheaper, but service there was absolutely terrible. It got to the point where I reprogrammed the DRAC cards to lock out their sys admins. After The Planet took over we had all kinds of issues, then finally they had an explosion in a transformer in one of their datacenters taking down all of our stuff for days on end. After that we moved out. They said it had nothing to do with them and we would be credited for the downtime if we stayed for at least another 6 months, but we had by then already signed up elsewhere and restored from backups. I figure if you're willing to run your operation that close to the red line then we should be taking our business elsewhere. They were lucky nobody got hurt. Right now we're hosting in three places, leaseweb, mojohost and virtual acccess. VXS is by far the best but expensive, leaseweb is somewhere in the middle and for high volume mojohost is absolutely unbeatable. btw, you have me curious what else you've heard rackspace called :)