5 ms·

Confessions of a Full Stack DevOp

115 points by brianshumate 12y ago

geerlingguy 12y ago
> When done correctly, automation tools are giving [developers] time back — and helping out of this problem of needing to wear many hats. I've found this to be the case both in my full-time job (enterprise/healthcare industry) and in my larger side projects. Using some simple tools to make sure both server environments and deployments are uniform/repeatable/known in a good state has given me much more time to focus on actually developing. Additionally, there are times when certain features (especially surrounding performance and scalability) requires a deeper knowledge of part of your stack. Being a good developer requires you to be a specialist (in your primary language/tools) and generalist (in almost all other areas) at the same time.
Thaxll 12y ago
"Full Stack DevOp", worst three words of 2013 in the same title.
- SkyMarshal 12y ago
  2013?
- rch 12y ago
  It's just a euphemism for 'engineer'.
  - chris_mahan 12y ago
    But you can't be an "engineer" without a Engineering degree from an accredited university, so DevOps.
    - nfoz 12y ago
      That's true in some countries, like Canada (where the term "engineer" is trademarked), but it's not true in the U.S. Here, everyone is an engineer.
    - rch 12y ago
      Unfortunately, the profession is leading the credentialing at this point. I'd expect that to change in the next decade or so, ideally resulting in a useful PE designation (in the US anyway).
- twic 12y ago
  But how can the sharing economy disrupt the cloud without full-stack agile devops?
KaiserPro 12y ago
I am a horseshoe crab. I am a sysadmin. I can code, I do code, but I'm a much better sysadmin, mainly because I've been doing it as a dayjob. You want networking? I'll do that. you want multi gigabyte a second storage arrays? I'll do that. You want monitoring? I'll do that. You want me to recompile python 3.3, naaa you can do that. I can, but you'll do better at it
- lewaldman 12y ago
  Wow mate, don't get me wrong (I'm a SysAdmin/DevOps guy) but... From where I came and from my background... RPM/DEB packaging is a SysAdmin job. And... In a properly built infrastructure, what is recompile python 3.3 if not just repackage it?
  - KaiserPro 12y ago
    because unless you work in a startup you have to support legacy, which involves recompiling python(or anything else for that matter) because someone previous did something stupid....
volume 12y ago
Here's my unsolicited opinion: Dehaan is pretty much spot on. Knupp's post seemed to first latch onto the idea that devops is about devs doing ops works. To add, I think Knupp should do a rewrite after listening to some devops cafe and food fight podcasts, if he really believes in what he wrote. It's somewhat an unfair comparison though. If you're the author/creator of ansible you surely have spent a lot of time thinking and understanding what devops means. To be frank, I cannot tell if Knupps post is just link bait to sell his book. The concept (and value) of the devops mindset & culture is less debatable in 2014 than in the past 5 years. Those who latch onto the totem pole model will simply get left behind while the rest of the industry moves on... or at least that's what they tell me!
simonw 12y ago
Pet peeve: this URL is incorrect. When you're linking to an article on a blog, it's important to use the permalink for the specific entry (in this case it's http://www.ansible.com/blog/confessions-of-a-full-stack-devop http://www.ansible.com/blog/confessions-of-a-full-stack-devo... which admittedly wasn't easy to find - you had to click the heading on the article which lacks any clear visual way to tell that it's a link). If you link to the blog homepage e.g. http://www.ansible.com/blog http://www.ansible.com/blog then in a few weeks time this post on Hacker News will no longer link to the correct content.
- dang 12y ago
  Thanks. We changed it. We'll soon have a new way for users to suggest better urls, like you did, that is more reliable than a moderator happening to see a comment. I'll announce it as a Tell HN.
- deleted 12y ago
  [deleted]
pradocchia 12y ago
The last time I looked for a senior sysadmin -- less than a year ago -- I didn't get anyone who was comfortable programming in Perl/Python/Ruby until I started using the term DevOps. [1] via TFA Ah, now I understand DevOps. Well, the sysadmin who can personally automate his job through programming--this is very productive person. [1] https://news.ycombinator.com/item?id=7593681 https://news.ycombinator.com/item?id=7593681
- p8952 12y ago
  In my personal opinion this is what DevOps is, in fact "OpsDev" might even be a better word for it in most cases. It's about carrying out what are traditionally "Operations" tasks (Provisioning, Configuration, Orchestration, Monitoring, ect) using tools and workflows with are traditionally used by Developers (Version Control, Automated Testing, ect) In large companies this work is often carried out by operations people who have expanded their skill set to include things previously limited to developer roles. In startups this work is often carried out by developers, as in the original rant, who are tasked with maintaining infrastructure as well as developing.
  - beat 12y ago
    In the traditional development/operations divide, developers often have no access to and little insight into the production environment where their code will be run. As such, they cobble together whatever makes it work, and then "throw it over the wall" to operations, where it has to run in a very different environment. This makes for hellish manual deployments, done on multi-day phone calls with ops staff who don't understand how the code works talking to developers who don't understand the rules and procedures of ops. If you see places where production code has to run in WebSphere or Weblogic because it's "enterprise", but want developers working in Tomcat to save money, run away. This leads to situations where operations starts providing "developer resources" in terms of servers, dba support, maven repositories, etc, that turn into a tar pit where development gets stuck in incomprehensible bureaucracy, and "hack" around the system in order to actually get anything done. And then security or someone else in ops finds some little skunkworks dev system and has a fit because developers don't understand what they're playing with - or worse, they find out about the skunkworks because they get pulled in to fix what developers screwed up, with their own managers howling about who's gonna pay for it... So to me, the beauty of DevOps is getting dev and ops working together in concert from the start to create a consistent development/production environment, where the same automation can be used to maintain both. This is a win for everyone.
peterbotond 12y ago
This is pretty much a good article on DevOps. In my words i see DevOps as a bridge spanning the kernel, systems, networks, qa, packaging, and deployment. for a highly optimal system all these need to be planned for with care and attention. DevOps is a tough and long learning to be good at.
beat 12y ago
This is a much better, more sensible article than yesterday. My good takeaway here is that DevOps means whatever you think it means - it's a lot of different things to a lot of different people. That's a much more positive spin on my frankly negative opinion that it's devolving into a meaningless buzzword to attach to resumes, recruiting, and PowerPoint presentations for management. I know I came in to DevOps (the word, not the job - I've been doing the job since the mid-1990s) with a strong opinion of what it ought to mean - cross-org cooperation between development and operations. This article has me rethinking that a bit.
- cwp 12y ago
  Yeah, the word is only meaningful when the speaker and listener have enough shared context to pin down the meaning. In that situation, there's probably a better word anyway. It's not quite the red flag that "ninja" and "rockstar" have become, but it definitely makes me wary when I hear it.
michaelochurch 12y ago
DevOps and "full stack" and Agile and the flamewars around them are just bikeshedding. None of this gnashing of teeth actually matters. You know what matters? Writing code and making it work. Solving problems. Helping people. If the right way to do that is for me to make a system more reliable, then I'll do my best even if I'm not interested in being "an ops guy". If the best thing for me to do is read machine learning papers, then I'll do that. The real problem: technology companies have terrible leadership. You know how small towns in the Midwest used to send their genetically unfortunate to San Francisco, where there'd be more social services and a warmer climate? Well, that's what the business world does with its rejects, too. Buses 'em to San Francisco. Except, in tech, they end up making $400k as VPs of HR at startups and get to boss nerds around. "Agile" won't fix that. Nor will "DevOps". These terms mean too many different things to different people, and ultimately fail to address the real, core issues with technology.
_superposition_ 12y ago
Knupps idea of the totem pole rubbed me the wrong way too. I've met great developers who didn't have a clue about how to scale or monitor their software outside of their framework. How to properly deploy across DEV, UAT and PROD without making code modifications. How to keep these environments clean and flexible to enable these deployments. Nor should they have to. Call me a Dev because I read and write code. Call me Ops because I do deployments and fix production issues. At the end of the day I solve problems and just plain make stuff work. With or without so called DevOps tools.
- billmalarky 12y ago
  You've met great developers who couldn't deploy code?
  - _superposition_ 12y ago
    Sure. I've met great ones who could too. I don't think deployment is part of the development job description. Or is it?
    - e12e 12y ago
      For me, the idea of devops was always to help developers gain enough insight into operations, to be more useful. I don't care if you can easily test your war-file from within eclipse -- make sure that it can be sensibly deployed and upgraded on a locked down headless box. Basically turning great developers into great system architects.
    - cmelbye 12y ago
      To me, it seems like it makes sense that a great developer should absolutely know how his code is running, how his changes interact with the production environment in which the code runs, etc.
      - marcosdumay 12y ago
        You'd take a hit if you visit any big company that's not centered around IT. It's quite common for the developers to be forbidden from knowing the details of the production environment, and usually they have no easy way to get that info. But don't mind, a DBA will think about the interaction between the several programs, and the ops people will think about performance.
rch 12y ago
Back when I thought I was going to be a Mechanical Engineer, I had a summer course that included surveying along with machine design and robotics. I felt the latter were more relevant to me, but had I gone on to work in oil fields in SE Asia I probably would have appreciated my time tromping around campus taking measurements. I see all of this as just 'engineering' work, but it's easier to delegate certain things than others. When I ask someone new to configure Postgres for secure remote administration, script or otherwise automate backups, and rough out a schema for some tool we're thinking about, they'll likely accomplish at least a subset those tasks. The remainder is an opportunity for discussion. Someone else might just ask me to build something, and we'll talk about usability when he's done with the physics side of things. But if I leave, what does that physicist look for in a new hire? The skillset of an 'engineer' is frustratingly broad and vague from a hiring point of view. Getting so-called DevOps out of the way might give another new hire the chance to develop as a programmer, and vice versa. Hopefully one or both of those two will eventually develop into an engineer.
nraynaud 12y ago
I would also add that you just don't automatize stuff. Because the more you add layers, the more you'll have to dig into at 3AM with the customer screaming at you on the phone. It's using the complete stack and using the least technology required to do the task at hand. While having access to the whole picture. I have heard too many times that something is complicated then we can automatize it. But automatization has to be debugged, so, if the task is not complex per se, try to find a simpler tool, try to limit your intervention in the config files, try to limit the number of tools etc. But you can make those tradeoff across the whole stack. And suddenly you're balancing between adding a library to the code or a program on the machine, or a function package in the DB. You have a wider view, and you have the pressure to do the right thing because your phone is connected to new relic.
- beat 12y ago
  The "Let Them Eat Cake" antipattern, that is. Maybe no one has written it and I just made it up...
  - nraynaud 12y ago
    I don't get it. Google turned that up: http://robmyers.org/2009/09/21/free_software_debate_antipatterns/ http://robmyers.org/2009/09/21/free_software_debate_antipatt... and I don't get it either.
    - beat 12y ago
      Basically, the idea is that if something is hard to automate, then automate it and then it won't be hard to automate. Peasants have no bread? Let them eat cake! I've encountered this a lot in the enterprise. Baby-step automations aren't allowed because they don't contribute to a Grand Unified Theory of Automation that will Solve All Our Problems. Worse, you wind up babysitting antiquated automation that should be replaced, and you're not allowed to replace it because it's already there and already works.
xorgar831 12y ago
DevOps is really the traditional Unix admin role, however the operator role has now be renamed sysadmin in most companies.
RA_Fisher 12y ago
Great post. While I agree with this, "Automation shouldn't be about spending all your time with a tool ..." I wonder if employers are pressured to push you into more and more Ops work by this paradox? http://en.wikipedia.org/wiki/Jevons_paradox http://en.wikipedia.org/wiki/Jevons_paradox This kind of situation, specifically (relative to other work): http://upload.wikimedia.org/wikipedia/commons/b/b8/JevonsParadoxA.png http://upload.wikimedia.org/wikipedia/commons/b/b8/JevonsPar...
ghalusa 12y ago
I can't read this article. The font and spacing is horrible. Makes my vision shimmer and shake. Perhaps I'm just getting old.
deleted 12y ago
[deleted]

115 points by brianshumate 12y ago

> When done correctly, automation tools are giving [developers] time back — and helping out of this problem of needing to wear many hats. I've found this to be the case both in my full-time job (enterprise/healthcare industry) and in my larger side projects. Using some simple tools to make sure both server environments and deployments are uniform/repeatable/known in a good state has given me much more time to focus on actually developing. Additionally, there are times when certain features (especially surrounding performance and scalability) requires a deeper knowledge of part of your stack. Being a good developer requires you to be a specialist (in your primary language/tools) and generalist (in almost all other areas) at the same time.

Thaxll 12y ago

"Full Stack DevOp", worst three words of 2013 in the same title.

SkyMarshal 12y ago
2013?
rch 12y ago
It's just a euphemism for 'engineer'.
- chris_mahan 12y ago
  But you can't be an "engineer" without a Engineering degree from an accredited university, so DevOps.
  - nfoz 12y ago
    That's true in some countries, like Canada (where the term "engineer" is trademarked), but it's not true in the U.S. Here, everyone is an engineer.
  - rch 12y ago
    Unfortunately, the profession is leading the credentialing at this point. I'd expect that to change in the next decade or so, ideally resulting in a useful PE designation (in the US anyway).
twic 12y ago
But how can the sharing economy disrupt the cloud without full-stack agile devops?

KaiserPro 12y ago

I am a horseshoe crab. I am a sysadmin. I can code, I do code, but I'm a much better sysadmin, mainly because I've been doing it as a dayjob. You want networking? I'll do that. you want multi gigabyte a second storage arrays? I'll do that. You want monitoring? I'll do that. You want me to recompile python 3.3, naaa you can do that. I can, but you'll do better at it

lewaldman 12y ago
Wow mate, don't get me wrong (I'm a SysAdmin/DevOps guy) but... From where I came and from my background... RPM/DEB packaging is a SysAdmin job. And... In a properly built infrastructure, what is recompile python 3.3 if not just repackage it?
- KaiserPro 12y ago
  because unless you work in a startup you have to support legacy, which involves recompiling python(or anything else for that matter) because someone previous did something stupid....

volume 12y ago

Here's my unsolicited opinion: Dehaan is pretty much spot on. Knupp's post seemed to first latch onto the idea that devops is about devs doing ops works. To add, I think Knupp should do a rewrite after listening to some devops cafe and food fight podcasts, if he really believes in what he wrote. It's somewhat an unfair comparison though. If you're the author/creator of ansible you surely have spent a lot of time thinking and understanding what devops means. To be frank, I cannot tell if Knupps post is just link bait to sell his book. The concept (and value) of the devops mindset & culture is less debatable in 2014 than in the past 5 years. Those who latch onto the totem pole model will simply get left behind while the rest of the industry moves on... or at least that's what they tell me!

simonw 12y ago

Pet peeve: this URL is incorrect. When you're linking to an article on a blog, it's important to use the permalink for the specific entry (in this case it's http://www.ansible.com/blog/confessions-of-a-full-stack-devop http://www.ansible.com/blog/confessions-of-a-full-stack-devo... which admittedly wasn't easy to find - you had to click the heading on the article which lacks any clear visual way to tell that it's a link). If you link to the blog homepage e.g. http://www.ansible.com/blog http://www.ansible.com/blog then in a few weeks time this post on Hacker News will no longer link to the correct content.

dang 12y ago
Thanks. We changed it. We'll soon have a new way for users to suggest better urls, like you did, that is more reliable than a moderator happening to see a comment. I'll announce it as a Tell HN.
deleted 12y ago
[deleted]

pradocchia 12y ago

The last time I looked for a senior sysadmin -- less than a year ago -- I didn't get anyone who was comfortable programming in Perl/Python/Ruby until I started using the term DevOps. [1] via TFA Ah, now I understand DevOps. Well, the sysadmin who can personally automate his job through programming--this is very productive person. [1] https://news.ycombinator.com/item?id=7593681 https://news.ycombinator.com/item?id=7593681

p8952 12y ago
In my personal opinion this is what DevOps is, in fact "OpsDev" might even be a better word for it in most cases. It's about carrying out what are traditionally "Operations" tasks (Provisioning, Configuration, Orchestration, Monitoring, ect) using tools and workflows with are traditionally used by Developers (Version Control, Automated Testing, ect) In large companies this work is often carried out by operations people who have expanded their skill set to include things previously limited to developer roles. In startups this work is often carried out by developers, as in the original rant, who are tasked with maintaining infrastructure as well as developing.
- beat 12y ago
  In the traditional development/operations divide, developers often have no access to and little insight into the production environment where their code will be run. As such, they cobble together whatever makes it work, and then "throw it over the wall" to operations, where it has to run in a very different environment. This makes for hellish manual deployments, done on multi-day phone calls with ops staff who don't understand how the code works talking to developers who don't understand the rules and procedures of ops. If you see places where production code has to run in WebSphere or Weblogic because it's "enterprise", but want developers working in Tomcat to save money, run away. This leads to situations where operations starts providing "developer resources" in terms of servers, dba support, maven repositories, etc, that turn into a tar pit where development gets stuck in incomprehensible bureaucracy, and "hack" around the system in order to actually get anything done. And then security or someone else in ops finds some little skunkworks dev system and has a fit because developers don't understand what they're playing with - or worse, they find out about the skunkworks because they get pulled in to fix what developers screwed up, with their own managers howling about who's gonna pay for it... So to me, the beauty of DevOps is getting dev and ops working together in concert from the start to create a consistent development/production environment, where the same automation can be used to maintain both. This is a win for everyone.

peterbotond 12y ago

This is pretty much a good article on DevOps. In my words i see DevOps as a bridge spanning the kernel, systems, networks, qa, packaging, and deployment. for a highly optimal system all these need to be planned for with care and attention. DevOps is a tough and long learning to be good at.

beat 12y ago

This is a much better, more sensible article than yesterday. My good takeaway here is that DevOps means whatever you think it means - it's a lot of different things to a lot of different people. That's a much more positive spin on my frankly negative opinion that it's devolving into a meaningless buzzword to attach to resumes, recruiting, and PowerPoint presentations for management. I know I came in to DevOps (the word, not the job - I've been doing the job since the mid-1990s) with a strong opinion of what it ought to mean - cross-org cooperation between development and operations. This article has me rethinking that a bit.

cwp 12y ago
Yeah, the word is only meaningful when the speaker and listener have enough shared context to pin down the meaning. In that situation, there's probably a better word anyway. It's not quite the red flag that "ninja" and "rockstar" have become, but it definitely makes me wary when I hear it.

michaelochurch 12y ago

DevOps and "full stack" and Agile and the flamewars around them are just bikeshedding. None of this gnashing of teeth actually matters. You know what matters? Writing code and making it work. Solving problems. Helping people. If the right way to do that is for me to make a system more reliable, then I'll do my best even if I'm not interested in being "an ops guy". If the best thing for me to do is read machine learning papers, then I'll do that. The real problem: technology companies have terrible leadership. You know how small towns in the Midwest used to send their genetically unfortunate to San Francisco, where there'd be more social services and a warmer climate? Well, that's what the business world does with its rejects, too. Buses 'em to San Francisco. Except, in tech, they end up making $400k as VPs of HR at startups and get to boss nerds around. "Agile" won't fix that. Nor will "DevOps". These terms mean too many different things to different people, and ultimately fail to address the real, core issues with technology.

_superposition_ 12y ago

Knupps idea of the totem pole rubbed me the wrong way too. I've met great developers who didn't have a clue about how to scale or monitor their software outside of their framework. How to properly deploy across DEV, UAT and PROD without making code modifications. How to keep these environments clean and flexible to enable these deployments. Nor should they have to. Call me a Dev because I read and write code. Call me Ops because I do deployments and fix production issues. At the end of the day I solve problems and just plain make stuff work. With or without so called DevOps tools.

billmalarky 12y ago
You've met great developers who couldn't deploy code?
- _superposition_ 12y ago
  Sure. I've met great ones who could too. I don't think deployment is part of the development job description. Or is it?
  - e12e 12y ago
    For me, the idea of devops was always to help developers gain enough insight into operations, to be more useful. I don't care if you can easily test your war-file from within eclipse -- make sure that it can be sensibly deployed and upgraded on a locked down headless box. Basically turning great developers into great system architects.
  - cmelbye 12y ago
    To me, it seems like it makes sense that a great developer should absolutely know how his code is running, how his changes interact with the production environment in which the code runs, etc.
    - marcosdumay 12y ago
      You'd take a hit if you visit any big company that's not centered around IT. It's quite common for the developers to be forbidden from knowing the details of the production environment, and usually they have no easy way to get that info. But don't mind, a DBA will think about the interaction between the several programs, and the ops people will think about performance.

rch 12y ago

Back when I thought I was going to be a Mechanical Engineer, I had a summer course that included surveying along with machine design and robotics. I felt the latter were more relevant to me, but had I gone on to work in oil fields in SE Asia I probably would have appreciated my time tromping around campus taking measurements. I see all of this as just 'engineering' work, but it's easier to delegate certain things than others. When I ask someone new to configure Postgres for secure remote administration, script or otherwise automate backups, and rough out a schema for some tool we're thinking about, they'll likely accomplish at least a subset those tasks. The remainder is an opportunity for discussion. Someone else might just ask me to build something, and we'll talk about usability when he's done with the physics side of things. But if I leave, what does that physicist look for in a new hire? The skillset of an 'engineer' is frustratingly broad and vague from a hiring point of view. Getting so-called DevOps out of the way might give another new hire the chance to develop as a programmer, and vice versa. Hopefully one or both of those two will eventually develop into an engineer.

nraynaud 12y ago

I would also add that you just don't automatize stuff. Because the more you add layers, the more you'll have to dig into at 3AM with the customer screaming at you on the phone. It's using the complete stack and using the least technology required to do the task at hand. While having access to the whole picture. I have heard too many times that something is complicated then we can automatize it. But automatization has to be debugged, so, if the task is not complex per se, try to find a simpler tool, try to limit your intervention in the config files, try to limit the number of tools etc. But you can make those tradeoff across the whole stack. And suddenly you're balancing between adding a library to the code or a program on the machine, or a function package in the DB. You have a wider view, and you have the pressure to do the right thing because your phone is connected to new relic.

beat 12y ago
The "Let Them Eat Cake" antipattern, that is. Maybe no one has written it and I just made it up...
- nraynaud 12y ago
  I don't get it. Google turned that up: http://robmyers.org/2009/09/21/free_software_debate_antipatterns/ http://robmyers.org/2009/09/21/free_software_debate_antipatt... and I don't get it either.
  - beat 12y ago
    Basically, the idea is that if something is hard to automate, then automate it and then it won't be hard to automate. Peasants have no bread? Let them eat cake! I've encountered this a lot in the enterprise. Baby-step automations aren't allowed because they don't contribute to a Grand Unified Theory of Automation that will Solve All Our Problems. Worse, you wind up babysitting antiquated automation that should be replaced, and you're not allowed to replace it because it's already there and already works.

xorgar831 12y ago

DevOps is really the traditional Unix admin role, however the operator role has now be renamed sysadmin in most companies.

RA_Fisher 12y ago

Great post. While I agree with this, "Automation shouldn't be about spending all your time with a tool ..." I wonder if employers are pressured to push you into more and more Ops work by this paradox? http://en.wikipedia.org/wiki/Jevons_paradox http://en.wikipedia.org/wiki/Jevons_paradox This kind of situation, specifically (relative to other work): http://upload.wikimedia.org/wikipedia/commons/b/b8/JevonsParadoxA.png http://upload.wikimedia.org/wikipedia/commons/b/b8/JevonsPar...

ghalusa 12y ago

I can't read this article. The font and spacing is horrible. Makes my vision shimmer and shake. Perhaps I'm just getting old.

deleted 12y ago

[deleted]