8 ms·

attempting to ignore snarkyness, but failing - and that says what about the dozens of rackspace engineers configuring and monitoring and supporting the box over

by inaka 17y ago

attempting to ignore snarkyness, but failing - and that says what about the dozens of rackspace engineers configuring and monitoring and supporting the box over the past two years?

jacquesm 17y ago
There once was a really nice quote here on HN: "you can't outsource responsibility". If in two years time you've never ever had a look at what kernel you are running, especially while tuning a system for performance you only have yourself to blame. Don't tell me you're running a 'stock' kernel and never bothered tuning it for your application, or considered upgrading it. Also, in your resources list you should have the exact machine configuration, there are tools to retrieve that sort of info automatically. Then, when you're done, store it in http://inventory.sf.net/ http://inventory.sf.net/ or something like that. It's typical that the people at rackspace would simply drop in the requested hardware, and that you yourself deal with the configuration. The smart money is on running some tests after they've done that to make sure it went ok. Asking for a CPU upgrade and not checking if they're operational is just plain stupid. I figure you literally asked rackspace to upgrade the CPU, and that's what they did. Did you explicitly ask them to install an SMP kernel with a specific version and they didn't do it ? Or did you expect them to do it but you didn't check if they actually did until today ? Two full years of trying to tune a box for performance and not noticing this, then publicly blaming rackspace is simply cheap, an attempt at pinning the blame on rackspace, for something that you should have noticed long ago yourself. Kudos for writing about it but the title should be "How I messed up". That's taking responsibility and then make sure it never ever happens again.
- sailormoon 17y ago
  I really do not dig this tone. The guy is obviously not a system admin. He paid top dollar for rackspace managed hosting precisely so he wouldn't have to do the kinds of things you mention. "You can't outsource responsibility" is utter nonsense. It is completely impossible to "own" responsibility for everything important in a complex society. Meaningless platitudes should not distract from the fact - Rackspace did not do their job. Yes, he messed up. He messed up by making assumptions and not checking Rackspace's work more closely. That's not the same as messing up in your own work. His post is a reminder to be more careful checking on the work of your "upstream". There's no need to pile on with the "if you didn't know 'top 1' you shouldn't be running a startup!" etc.
  - jacquesm 17y ago
    I didn't say he shouldn't be running a startup, I said he should not be managing the servers their customers stuff runs on. As for the tone, you may disagree with that but that does not distract from the fact that if you operate a business, that you should know your stuff. And if you outsource something you should at least know how to check up on the bits that you've outsourced. Outsourcing does not mean that your responsibility disappears, it simply changes from 'doing' to 'monitoring'. Maybe rackspace did not do their job, I have no insight in the communications that went on between the party involved and rackspace. All we get here is a pointing finger without any responsibility taken, that is not a realistic picture. It could be the difference in the wording of the upgrade request ("please install another CPU in our machine" vs "please install and configure another CPU in our machine"). Even then, rackspace probably should get part of the blame, but really not all of it. The fact that the situation persisted for two years is completely on the OPs account, in two years you have many more opportunities than your hosting provider to find this out, after all they will leave your machine alone unless it malfunctions and there is no indication that they ever were requested to look in to this, and when they were they actually found the problem. I quote from the article "In investigating an unrelated issue, we followed up with Rackspace on a Kernel patch that couldn’t be applied to our server. One of the technicians immediately realized why – we were not running the SMP kernel." How come someone is trying to patch a kernel, can't apply the patch and then still doesn't clue in to the situation ? Also, we do not know if the SMP kernel was installed or not, it might have been, and then on the final reboot the wrong kernel was brought up. And that's a very easy mistake to make. But dmesg would tell you in a heartbeat, as would 'top '1'', which you would be using plenty of times while debugging performance issues to make sure all your cores are doing the right amount of work.
    - sailormoon 17y ago
      "I said he should not be managing the servers their customers stuff runs on." And what if it's only him? No go then huh? You've been saying a lot of this kind of thing lately. That guy before with the App Store payment problem? You came down on him like a ton of bricks. And now this. Just because people haven't dotted every i and crossed every t. It's not exactly the hacker mentality is it?
- nailer 17y ago
  'Don't tell me you're running a 'stock' kernel and never bothered tuning it for your application, or considered upgrading it.' Not sure why you've got 'stock' in quotes. Vendor kernels are used by hundres of thousands of servers, each sharing the same bug reports and security updates. There's a massive benefit unless you think you can do those bug reports and security updates better than your OS vendor. Most custom compiles are by people who don't understand loadable modules or read somethign written before they existed.
  - jacquesm 17y ago
    Ok, point taken, but there are definite advantages to 'rolling your own'.
    - forkqueue 17y ago
      Such as? Using a vendor-supplied kernel means that there are extremely likely to be other people using most of the same stack as you, many of them on the same hardware. If there are problems, it's likely that other people have noticed the issue, even if the bug hasn't been found, so it's much more likely to get fixed. If you compile your own kernel (and/or copy of Apache, MySQL etc etc) you're running something unique to you. If you have problems, you're on your own. If you're paying for Red Hat Enterprise, use the Red Hat Enterprise packages unless there's a good reason not to. If something goes wrong, you can call Red Hat support and have at least a steer in the right direction. Custom-compiling everything just for the sake of it, just to have new 'shiny' stuff is crazy.
      - jacquesm 17y ago
        It's not for the 'new shiny' at all, it's got to do with optimizing your kernel to match your hardware and getting rid of loadable module support in favor of a kernel that has on board exactly that which is needed to operate your system. A 'stock' kernel has a whole pile of things in it that might be the next remote exploit, by removing such stuff you marginally increase security. Other things you might need: - kernel support for booting from raid filesystems without trickery - processor family optimizations - maximum number of cores (stock = 8, we run 16 on quite a few machines) As for compiling, I do that anyway, it's a small job compared to the number of times that you need to do it. And you're just as much 'on your own' to solve problems, the chances of having them are less though (because the system you are running is considerably leaner). Second your redhat enterprise solution, that's not what I'm using though on most of our machines (either centos or debian), but that's a good solution too.
- inaka 17y ago
  Honestly your tone stings a bit, jacquesm. I guess i shouldn't have titled it 'don't trust rackspace' but my larger point is exactly the opposite of what you wrote - take responsibility for your servers - don't trust anyone, even the most expensive hosting provider, to do it for you.
  - jacquesm 17y ago
    That's a whole lot better, if that was the message then it somehow got lost to me. Again, apologies for the tone, but it really seems to be a trend to make a mistake, 'blame someone', then blog about it. What I would suggest you do, and this is meant very seriously, is find a cheaper hosting provider (EV1/The Planet is about half of what you pay right now) and spend the rest on getting a part-time sysadmin that really knows his stuff. The difference in $ should be minimal, then look over the guys shoulder at how it is done, but keep doing what you know is your 'level' anyway. That way you get the best of both worlds, excellent care and you don't break the bank, at the same time you'll learn a huge amount. And if your startup grows you just might have found an employee for the future. Spend some time looking around, your best bet would be a guy or girl that does sysadmin duties for a larger company using UNIX that wants to make some extra $ in their spare time.

by inaka 17y ago

attempting to ignore snarkyness, but failing - and that says what about the dozens of rackspace engineers configuring and monitoring and supporting the box over the past two years?

jacquesm 17y ago

There once was a really nice quote here on HN: "you can't outsource responsibility". If in two years time you've never ever had a look at what kernel you are running, especially while tuning a system for performance you only have yourself to blame. Don't tell me you're running a 'stock' kernel and never bothered tuning it for your application, or considered upgrading it. Also, in your resources list you should have the exact machine configuration, there are tools to retrieve that sort of info automatically. Then, when you're done, store it in http://inventory.sf.net/ http://inventory.sf.net/ or something like that. It's typical that the people at rackspace would simply drop in the requested hardware, and that you yourself deal with the configuration. The smart money is on running some tests after they've done that to make sure it went ok. Asking for a CPU upgrade and not checking if they're operational is just plain stupid. I figure you literally asked rackspace to upgrade the CPU, and that's what they did. Did you explicitly ask them to install an SMP kernel with a specific version and they didn't do it ? Or did you expect them to do it but you didn't check if they actually did until today ? Two full years of trying to tune a box for performance and not noticing this, then publicly blaming rackspace is simply cheap, an attempt at pinning the blame on rackspace, for something that you should have noticed long ago yourself. Kudos for writing about it but the title should be "How I messed up". That's taking responsibility and then make sure it never ever happens again.

sailormoon 17y ago
I really do not dig this tone. The guy is obviously not a system admin. He paid top dollar for rackspace managed hosting precisely so he wouldn't have to do the kinds of things you mention. "You can't outsource responsibility" is utter nonsense. It is completely impossible to "own" responsibility for everything important in a complex society. Meaningless platitudes should not distract from the fact - Rackspace did not do their job. Yes, he messed up. He messed up by making assumptions and not checking Rackspace's work more closely. That's not the same as messing up in your own work. His post is a reminder to be more careful checking on the work of your "upstream". There's no need to pile on with the "if you didn't know 'top 1' you shouldn't be running a startup!" etc.
- jacquesm 17y ago
  I didn't say he shouldn't be running a startup, I said he should not be managing the servers their customers stuff runs on. As for the tone, you may disagree with that but that does not distract from the fact that if you operate a business, that you should know your stuff. And if you outsource something you should at least know how to check up on the bits that you've outsourced. Outsourcing does not mean that your responsibility disappears, it simply changes from 'doing' to 'monitoring'. Maybe rackspace did not do their job, I have no insight in the communications that went on between the party involved and rackspace. All we get here is a pointing finger without any responsibility taken, that is not a realistic picture. It could be the difference in the wording of the upgrade request ("please install another CPU in our machine" vs "please install and configure another CPU in our machine"). Even then, rackspace probably should get part of the blame, but really not all of it. The fact that the situation persisted for two years is completely on the OPs account, in two years you have many more opportunities than your hosting provider to find this out, after all they will leave your machine alone unless it malfunctions and there is no indication that they ever were requested to look in to this, and when they were they actually found the problem. I quote from the article "In investigating an unrelated issue, we followed up with Rackspace on a Kernel patch that couldn’t be applied to our server. One of the technicians immediately realized why – we were not running the SMP kernel." How come someone is trying to patch a kernel, can't apply the patch and then still doesn't clue in to the situation ? Also, we do not know if the SMP kernel was installed or not, it might have been, and then on the final reboot the wrong kernel was brought up. And that's a very easy mistake to make. But dmesg would tell you in a heartbeat, as would 'top '1'', which you would be using plenty of times while debugging performance issues to make sure all your cores are doing the right amount of work.
  - sailormoon 17y ago
    "I said he should not be managing the servers their customers stuff runs on." And what if it's only him? No go then huh? You've been saying a lot of this kind of thing lately. That guy before with the App Store payment problem? You came down on him like a ton of bricks. And now this. Just because people haven't dotted every i and crossed every t. It's not exactly the hacker mentality is it?
nailer 17y ago
'Don't tell me you're running a 'stock' kernel and never bothered tuning it for your application, or considered upgrading it.' Not sure why you've got 'stock' in quotes. Vendor kernels are used by hundres of thousands of servers, each sharing the same bug reports and security updates. There's a massive benefit unless you think you can do those bug reports and security updates better than your OS vendor. Most custom compiles are by people who don't understand loadable modules or read somethign written before they existed.
- jacquesm 17y ago
  Ok, point taken, but there are definite advantages to 'rolling your own'.
  - forkqueue 17y ago
    Such as? Using a vendor-supplied kernel means that there are extremely likely to be other people using most of the same stack as you, many of them on the same hardware. If there are problems, it's likely that other people have noticed the issue, even if the bug hasn't been found, so it's much more likely to get fixed. If you compile your own kernel (and/or copy of Apache, MySQL etc etc) you're running something unique to you. If you have problems, you're on your own. If you're paying for Red Hat Enterprise, use the Red Hat Enterprise packages unless there's a good reason not to. If something goes wrong, you can call Red Hat support and have at least a steer in the right direction. Custom-compiling everything just for the sake of it, just to have new 'shiny' stuff is crazy.
    - jacquesm 17y ago
      It's not for the 'new shiny' at all, it's got to do with optimizing your kernel to match your hardware and getting rid of loadable module support in favor of a kernel that has on board exactly that which is needed to operate your system. A 'stock' kernel has a whole pile of things in it that might be the next remote exploit, by removing such stuff you marginally increase security. Other things you might need: - kernel support for booting from raid filesystems without trickery - processor family optimizations - maximum number of cores (stock = 8, we run 16 on quite a few machines) As for compiling, I do that anyway, it's a small job compared to the number of times that you need to do it. And you're just as much 'on your own' to solve problems, the chances of having them are less though (because the system you are running is considerably leaner). Second your redhat enterprise solution, that's not what I'm using though on most of our machines (either centos or debian), but that's a good solution too.
inaka 17y ago
Honestly your tone stings a bit, jacquesm. I guess i shouldn't have titled it 'don't trust rackspace' but my larger point is exactly the opposite of what you wrote - take responsibility for your servers - don't trust anyone, even the most expensive hosting provider, to do it for you.
- jacquesm 17y ago
  That's a whole lot better, if that was the message then it somehow got lost to me. Again, apologies for the tone, but it really seems to be a trend to make a mistake, 'blame someone', then blog about it. What I would suggest you do, and this is meant very seriously, is find a cheaper hosting provider (EV1/The Planet is about half of what you pay right now) and spend the rest on getting a part-time sysadmin that really knows his stuff. The difference in $ should be minimal, then look over the guys shoulder at how it is done, but keep doing what you know is your 'level' anyway. That way you get the best of both worlds, excellent care and you don't break the bank, at the same time you'll learn a huge amount. And if your startup grows you just might have found an employee for the future. Spend some time looking around, your best bet would be a guy or girl that does sysadmin duties for a larger company using UNIX that wants to make some extra $ in their spare time.