7 ms·

This is definitely cool but the real hard problem isn't in these simple easily scripted cases. The real hard to solve problem is managing all the complexity of

This is definitely cool but the real hard problem isn't in these simple easily scripted cases. The real hard to solve problem is managing all the complexity of similar-but-different hosts. This article could just as easily have taken the complete opposite view of Ansible by saying things like "parallel ssh sessions don't scale, strong encryption costs too much CPU time; push can never work reliably therefore pull is the only viable model; etc. etc." I feel one of Ansible's strongest points to champion is the low barrier to entry. It takes minimal understanding to get going, compare that with at least 1 month hands on with cfengine or perhaps 2 weeks puppet before you would consider yourself proficient enough. With Ansible it's 20 mins or so.

hunvreus 13y ago
I keep on hearing these arguments of scale. I haven't ever experienced such issues. Now yes, if you're running Facebook infrastructure you may run into issues. Though honestly, who here manages more than a few hundreds hosts? I can't find the post that was making the point of encryption cost and parallel SSH sessions, but it'd be helpful to get actual hard numbers backing these claims, and how they compare to Puppet's or Chef's approach. I agree with the complexity of similar-but-different hosts, and that's actually something we're set to solve with devo.ps (we'll see if we pull if off).
- 3amOpsGuy 13y ago
  To some extent i agree; most will never scale so why worry about it? X times out of Y you'll come out ahead by just ploughing on and not considering the future. Even if we are thinking up front, we can rationalise that it's just a gamble, often one worth taking! It definitely doesn't take Facebook sized infra to outgrow a technology though. What if the gamble doesn't pay off? What if you planned to scale the central NFS server dependency by just adding an extra NFS server but have now found there's no rack space left / no budget / a purchasing delay / insufficient network capacity / cooling capacity / power capacity / a.n. other unplanned problem. For the SSH question, i couldn't reliably get more than 250 concurrent connections outbound from one circa 2008 blade server. From memory that would have had a spec of dual core CPU, maybe around 2.4Ghz with 8Gb ram using PAE as it would have been a 32bit kernel (our spec, the cores will have been 64bit). They were multiply-connected at chassis level on myrinet fabric in one DC and infiniband in another and the resource being exhausted was CPU. These days all the blade servers are gone but we see an absolute explosion of virtual machines so it's a similar and still relevant problem in many ways.
  - mpdehaan2 13y ago
    It is worth pointing out that Ansible does not need concurrent connections to each server to manage each machine, you can address as large of groups as you want and control parallelism with the --forks parameter.
  - hunvreus 13y ago
    It's not a matter of taking the gamble you'll stay small. It's that there are different kind of big, only 1 times out of 1 billion there is a single player (Google, Facebook, Amazon) who reach an insane scale. By then, you'll have the resources to solve whatever issue you may hit. As Michael (creator of the project) said here; it scales just fine even with serious players (thousands of instances). He has actual hard data and concrete use cases to back it up. So far, all I've heard from detractors is the "OMG Chef is so hardcore Facebook uses it". Well, they use some. They'd probably do just fine with Ansible instead, provided they were putting half the brain power they put into Chef to make it work for their infrastructure. You may get very big, just likely not Facebook big.
mpdehaan2 13y ago
We've got quite a few users managing hundreds and thousands of hosts, so I'm not seeing these kinds of compliants. If I would, I'd feel it, but we don't :) One of the things many people want to do is rolling updates too, and Ansible is remarkably good at them, having a language that is really good for talking to load balancers and monitoring and saying, "of this 500 servers, update 50 at a time, and keep my uptime". Folks like AppDynamics are using this to update all of their infrastructure every 15 minutes, continuously, and it's pretty cool stuff. For those folks that do want to do the 'facebook scale' stuff, ansible-pull is a really good option. One of the features in our upcoming product is a nice callback that enables this while still preserving centralized reporting. Happy to have the conversation, but definitely I've never heard the CPU time compliant. I think the one thing we see is a lot of users are happy that Ansible is not running when it is not managing anything, rather than having daemons sucking CPU/RAM/etc, and folks are actually getting a little better performance from avoiding the thundering herd agent problems.
- MichaelSalib 13y ago
  I just did some consulting on helping another team improve their hadoop cluster performance and the first thing I noticed is that all 40 boxes in the cluster were burning a CPU core with a puppet agent process that was running at 100% CPU for months.
  - mpdehaan2 13y ago
    That's one of the nicer things about the no agent setup, when Ansible is not managing something, there is nothing eating CPU or RAM, and you don't have a problem with your agents falling over (SSHd is rock solid), so you get out of the 'managing the management' problem as well as the 'management is effecting my workload performance' problem. In particular with virtualization, running a heavy agent on every instance can add up. (reports of the ruby virtual machine eating 400MB occasionally occur).
- otterley 13y ago
  How does Ansible effectively scale to thousands of hosts using ssh? My experience is that you can only run a few hundred ssh sessions at a time with reasonable performance, and that's on beefy hardware to begin with.
eblume 13y ago
I really don't intend this as a brag, but I just wanted to point out that I learned Puppet yesterday. No, I mean, I really learned it yesterday. I started with my company's vagrant-managed VM and took their existing puppet architecture, and armed with that I learned (from no previous experience) how to write Puppet manifests, modules, 'define's, etc. My only point is that it doesn't take two weeks to learn Puppet. I'm not saying that Ansible is worse or anything like that, rather I just wanted to contribute another data point.
- mechanical_fish 13y ago
  You're tinkering with an existing, working architecture and a company-maintained Vagrant VM? Of course it's going well. But you should make sure to buy all the people who built all these well-engineered things a tasty drink at the next company get-together, because rest assured: Not every Puppet setup is easy to tinker with.
  - eblume 13y ago
    I absolutely do, and having a working environment is a huge help! However I found the puppet docs to be more than sufficient, and in fact I spent the day learning puppet not through our existing code (which actually needs a fair bit of work still) but rather using the learning VM that Puppet provides on their website.
    - gazarsgo 13y ago
      i, on the other hand, tried to learn puppet over the course of a weekend with a brand new laptop and Boxen just after it was released. There is a nightmare of broken dependencies that still aren't resolved months later (librarian-puppet won't install apache because apache's version has a -rc in it which isn't a ruby version blah blah blah...) now I know that using VMs / vagrant is critical to sane server orchestration development workflows though, so there's that.
      - eblume 13y ago

hunvreus 13y ago

I keep on hearing these arguments of scale. I haven't ever experienced such issues. Now yes, if you're running Facebook infrastructure you may run into issues. Though honestly, who here manages more than a few hundreds hosts? I can't find the post that was making the point of encryption cost and parallel SSH sessions, but it'd be helpful to get actual hard numbers backing these claims, and how they compare to Puppet's or Chef's approach. I agree with the complexity of similar-but-different hosts, and that's actually something we're set to solve with devo.ps (we'll see if we pull if off).

3amOpsGuy 13y ago
To some extent i agree; most will never scale so why worry about it? X times out of Y you'll come out ahead by just ploughing on and not considering the future. Even if we are thinking up front, we can rationalise that it's just a gamble, often one worth taking! It definitely doesn't take Facebook sized infra to outgrow a technology though. What if the gamble doesn't pay off? What if you planned to scale the central NFS server dependency by just adding an extra NFS server but have now found there's no rack space left / no budget / a purchasing delay / insufficient network capacity / cooling capacity / power capacity / a.n. other unplanned problem. For the SSH question, i couldn't reliably get more than 250 concurrent connections outbound from one circa 2008 blade server. From memory that would have had a spec of dual core CPU, maybe around 2.4Ghz with 8Gb ram using PAE as it would have been a 32bit kernel (our spec, the cores will have been 64bit). They were multiply-connected at chassis level on myrinet fabric in one DC and infiniband in another and the resource being exhausted was CPU. These days all the blade servers are gone but we see an absolute explosion of virtual machines so it's a similar and still relevant problem in many ways.
- mpdehaan2 13y ago
  It is worth pointing out that Ansible does not need concurrent connections to each server to manage each machine, you can address as large of groups as you want and control parallelism with the --forks parameter.
- hunvreus 13y ago
  It's not a matter of taking the gamble you'll stay small. It's that there are different kind of big, only 1 times out of 1 billion there is a single player (Google, Facebook, Amazon) who reach an insane scale. By then, you'll have the resources to solve whatever issue you may hit. As Michael (creator of the project) said here; it scales just fine even with serious players (thousands of instances). He has actual hard data and concrete use cases to back it up. So far, all I've heard from detractors is the "OMG Chef is so hardcore Facebook uses it". Well, they use some. They'd probably do just fine with Ansible instead, provided they were putting half the brain power they put into Chef to make it work for their infrastructure. You may get very big, just likely not Facebook big.

mpdehaan2 13y ago

We've got quite a few users managing hundreds and thousands of hosts, so I'm not seeing these kinds of compliants. If I would, I'd feel it, but we don't :) One of the things many people want to do is rolling updates too, and Ansible is remarkably good at them, having a language that is really good for talking to load balancers and monitoring and saying, "of this 500 servers, update 50 at a time, and keep my uptime". Folks like AppDynamics are using this to update all of their infrastructure every 15 minutes, continuously, and it's pretty cool stuff. For those folks that do want to do the 'facebook scale' stuff, ansible-pull is a really good option. One of the features in our upcoming product is a nice callback that enables this while still preserving centralized reporting. Happy to have the conversation, but definitely I've never heard the CPU time compliant. I think the one thing we see is a lot of users are happy that Ansible is not running when it is not managing anything, rather than having daemons sucking CPU/RAM/etc, and folks are actually getting a little better performance from avoiding the thundering herd agent problems.

MichaelSalib 13y ago
I just did some consulting on helping another team improve their hadoop cluster performance and the first thing I noticed is that all 40 boxes in the cluster were burning a CPU core with a puppet agent process that was running at 100% CPU for months.
- mpdehaan2 13y ago
  That's one of the nicer things about the no agent setup, when Ansible is not managing something, there is nothing eating CPU or RAM, and you don't have a problem with your agents falling over (SSHd is rock solid), so you get out of the 'managing the management' problem as well as the 'management is effecting my workload performance' problem. In particular with virtualization, running a heavy agent on every instance can add up. (reports of the ruby virtual machine eating 400MB occasionally occur).
otterley 13y ago
How does Ansible effectively scale to thousands of hosts using ssh? My experience is that you can only run a few hundred ssh sessions at a time with reasonable performance, and that's on beefy hardware to begin with.

eblume 13y ago

I really don't intend this as a brag, but I just wanted to point out that I learned Puppet yesterday. No, I mean, I really learned it yesterday. I started with my company's vagrant-managed VM and took their existing puppet architecture, and armed with that I learned (from no previous experience) how to write Puppet manifests, modules, 'define's, etc. My only point is that it doesn't take two weeks to learn Puppet. I'm not saying that Ansible is worse or anything like that, rather I just wanted to contribute another data point.

mechanical_fish 13y ago
You're tinkering with an existing, working architecture and a company-maintained Vagrant VM? Of course it's going well. But you should make sure to buy all the people who built all these well-engineered things a tasty drink at the next company get-together, because rest assured: Not every Puppet setup is easy to tinker with.
- eblume 13y ago
  I absolutely do, and having a working environment is a huge help! However I found the puppet docs to be more than sufficient, and in fact I spent the day learning puppet not through our existing code (which actually needs a fair bit of work still) but rather using the learning VM that Puppet provides on their website.
  - gazarsgo 13y ago
    i, on the other hand, tried to learn puppet over the course of a weekend with a brand new laptop and Boxen just after it was released. There is a nightmare of broken dependencies that still aren't resolved months later (librarian-puppet won't install apache because apache's version has a -rc in it which isn't a ruby version blah blah blah...) now I know that using VMs / vagrant is critical to sane server orchestration development workflows though, so there's that.
    - eblume 13y ago