7 ms·

Remember, if you have a DevOps Team, then you are absolutely not doing DevOps. Developers should be involved in managing everything about their applications, in

Remember, if you have a DevOps Team, then you are absolutely not doing DevOps. Developers should be involved in managing everything about their applications, including infrastructure. Any sufficiently complicated infrastructure that has uptime requirements and significant revenue associated with it is going to have a DevOps Team (or the equivalent) ultimately responsible for ensuring that things are working. I guess it's possible to turn your entire dev team into part-time DevOps engineers, while still calling them Software Engineers, but I've usually found that doesn't work long-term and causes employee retention issues. It's like saying your company does 'No-Support' because you don't hire Support Engineers, while in fact you've enlisted your Software Engineering team to handle all support requests. Also, if you're working in a regulated field like Healthcare or Finance, or anything that touches PII, your developers often can't have access to deploy code directly to production. Again, you could maybe work around this in the short-term by turning all developers into developers+devops, but they're different skillsets.

heisenbit 9y ago
Remember, if someone claims something absolutely then there is absolutely the chance that this gentleman is partially wrong.
- tdb7893 9y ago
  Or as I like to quote at work "only a Sith deals in absolutes"
throwanem 9y ago
Yeah, I was doing "DevOps" back when it was still called "wow, good sysadmins are really expensive! I know..." The rebranding is cute and all, and the tooling's a fair bit better, but nothing I've seen suggests that combining these two full-time roles in the same people is any more a good idea today than it was fifteen years ago.
- eeZah7Ux 9y ago
  Most of the modern devops hipster tooling is far worse and more complex to use that the old bash scripts put together by grumpy sysadmins.
  - throwanem 9y ago
    I've used those scripts - I've written those scripts! - and I have no idea what you're talking about. Puppet is a marvel, and when it's too heavyweight for the task at hand, there's Ansible. Both are far, far preferable to me@homebox $ for host in foo bar baz; do cat random_stuff_who_knows_if_it_works.sh | ssh $host 'bash -'; done like we did in the old days you're talking about.
    - etatoby 9y ago
      You can have my cryptic Bash one-liners when you pry them from my cold, dead hands. But yeah, I agree that Ansible is the minimum that should be in place.
      - throwanem 9y ago
        Oh, I still perpetrate cryptic Bash one-liners myself, on a more or less daily basis. What's nice today, by comparison with the before time, is having a realistic option to do otherwise, and having that option not be Perl.
        marcosdumay 9y ago
        > and having that option not be Perl Oh, Python is quite old already :)
        throwanem 9y ago
        Not better, though. Just different.
    - 9y ago
pjc50 9y ago
Where's that quote about how software architecture evolves to reflect the organisational structure of the business? Amazon does it this way because it's made of teams that are not just loosely coupled but somewhat firewalled from each other, along with a brutal internally competitive process.
- devonkim 9y ago
  Conway's Law
- taneq 9y ago
  Don't they have explicit rules dictating that any software must work as a service under failure-prone conditions?
  - whateveracct 9y ago
    More than that, internal Amazon tools make it stupidly easy to do everything-as-a-service, so it's just how everyone does it.
- deleted 9y ago
  [deleted]
Lazare 9y ago
I think this is basically just an argument about terms. If you write code, you're a developer. If you run servers, your in ops. If you are doing both, it's devops. So when discussing the idea of having a total separate team for running the servers, OP says it's not devops and he's right, because the definition of devops is not having the teams be separate. And you say that it's often totally needed to have that separation and you're also right. > I guess it's possible to turn your entire dev team into part-time DevOps engineers, I think you mean, it's possible to turn your entire dev team into part-time ops engineers. Which is 1) the entire concept of "devops", and 2) often a horrible idea. :) (Sorry, but people who use "devops" to mean "ops" is a pet peeve of mine. If you're not a developer, it's just ops.)
- TheCoelacanth 9y ago
  Most devops teams that I see don't manage the servers. They provide tooling that makes it easier for dev teams to manage their own servers. If a devops teams is actually managing all the servers, I agree that they aren't doing devops, but at a large enough company it makes sense to have a team working just on things that make it easier for the company to do devops.
eeZah7Ux 9y ago
Dev + Ops == Devops, get over it.
- dang 9y ago
  Please post civilly and substantively, or not at all.
  - eeZah7Ux 9y ago
    Sorry, I'll contextualize. > is going to have a DevOps Team The parent is arguing that the concept of a "DevOps Team" that does Ops makes sense and is not an oxymoron. Yet "DevOps" literally means doing Dev and Ops together. Otherwise have a "Developer Team" and an "Operations|Engineering|SRE|SysAdmin Team".
gloverkcn 9y ago
The statement touches on a real issue. I would revise the statement to: If your DevOps people reside in a separate group outside of the development team, then you are not doing DevOps. Architecture and Infrastructure go hand in hand, and DevOps is the glue that merges them together. I've worked with both DevOps and Architecutre groups in larger companies where Development, DevOps, and Infrastructure have been separated and siloed. This invariably leads to waste that undercuts any advantage provided by modern development practices. The core of the issue is the designs will only consider trade offs within their skill set. Problems that are easily solved with a combination of dev/devOps/inf are instead solved using complex designs within a single skill set. The other end of the spectrum is that decisions get made in one group that will undercut the efforts of another group. Real life examples: The dev group wants to deploy backend service version 2 with fail back option in production. The API's are identical, the database is unchanged. Instead of using a load balancer and monitoring to automate role back the dev group builds another system with the job of routing traffic between the two systems, identifying failures, and then stop sending traffic to the new system. The dev group designs a system with "micro-services" in mind. The developers have all containerized the services and run them on their laptop. The infrastructure and/or devops group doesn't want to deal with containers and instead deploys one service per VM. (In these scenarios the dev group will get a bill for the extra services). The problem is that it's easier to grab the people sitting next to design something than scheduling a meeting with groups you rarely see. This is a key driver of Conway's law.
- Jare 9y ago
  > If your DevOps people reside in a separate group outside of the development team, then you are not doing DevOps Devops and swdevs sitting and working together, understanding each other's problems and needs, and keeping them in mind when designing solutions, is indeed a beautiful thing to watch.
dominotw 9y ago
I always thought DevOps was the guy who is writing Kubernetes, not the guy who is installing it. Not sure why a special title is needed though.
pacala 9y ago
+1. Nothing promotes the ability to produce quality code like being woken at 3am to troubleshot a problem 3 microservices away from your own. In my anecdotical experience, the ability to recover from such incidents degrades with age, so nowadays I regard "everybody [but management] does DevOps" as codeword for ageism.
solatic 9y ago
> Any sufficiently complicated infrastructure that has uptime requirements and significant revenue associated with it is going to have a DevOps Team (or the equivalent) ultimately responsible for ensuring that things are working. I don't think these are mutually exclusive; I think you can both have a dedicated applicative operations (rebranded DevOps or SRE) team and still allow developers to rapidly provision and deploy. The point is a change in responsibilities and workflow on the part of operations: whereas their task was originally to deploy, verify success, rollback if necessary, monitoring, etc., their new task is to embed those tasks in code, so that developers can safely and reliably perform those tasks by themselves. You can have an organization where developers can automatically provision infrastructure which is behind your enterprise firewalls, hooked up to up-to-date enterprise artifact repositories, has enterprise PKI management already set up, etc., because expecting developers to know the ins and outs of setting that up manually is moronic and much of that should be scripted for ease of management anyway.
dkersten 9y ago
Developers should be involved in managing everything about their applications, including infrastructure. As a software engineer, there is just no way I can stay up to date with my craft as well as everything there is to know about infrastructure AND the all important security. I don't know about all the latest kernel patches and firewall rules. I don't know about monitoring failed hard drives. Etc. I mean, I do know a lot about deploying and maintaining my running software, but Ops has a ton of other concerns that themselves are a full time job and I already have my time fully consumed by development tasks. I'm also ok with being pushed enough of the work so that I don't accidentally make software that's a PITA to deploy or manage and that I don't needlessly wake somebody else. That doesn't mean that I can take on the entire job (well, I can, but I wouldn't be doing as good a job, especially when security and such is concerned, and I wouldn't be doing as good a development job either). What next - DevOpsSalesMarkettingAccounting? Much better, IMO, is to have small tightly-knit cross-functional teams which include an ops engineer.
- 0xdeadbeefbabe 9y ago
  > well, I can, but I wouldn't be doing as good a job You do a better job communicating with yourself though?
  - dkersten 9y ago
    Sure, but any advantage from that is lost in the fact that I've lost a ton of development time on doing ops tasks, which I don't have time to do properly and don't have time to learn how to do properly. Even bad communication to a competent ops person who does have time to, you know, do their job, and does have time to stay up to date, will result in much better results than if I tried to fit the work into my schedule. For example, in my current work, I could try to figure all the ops stuff out, but I've already got a backlog of months of development tasks for a project that needs to be done asap.
    - 0xdeadbeefbabe 9y ago
      I'd like to argue the advantage is sometimes lost, but not often.
enknamel 9y ago
> Remember, if you have a DevOps Team, then you are absolutely not doing DevOps. Developers should be involved in managing everything about their applications, including infrastructure. I think the intent here is a good one. You as a developer should be involved with how your software is deployed and managed. It's just that there is a pretty grey boundary between where developer involvement ends and ops involvement begins. Another issue is that devops is extremely poorly defined. Some people insist this means only writing chef scripts. Some people it's diagnosing and fixing production issues. Some people it's just sysadmin work. Or other responsibilities and combinations.

heisenbit 9y ago

Remember, if someone claims something absolutely then there is absolutely the chance that this gentleman is partially wrong.

tdb7893 9y ago
Or as I like to quote at work "only a Sith deals in absolutes"

throwanem 9y ago

Yeah, I was doing "DevOps" back when it was still called "wow, good sysadmins are really expensive! I know..." The rebranding is cute and all, and the tooling's a fair bit better, but nothing I've seen suggests that combining these two full-time roles in the same people is any more a good idea today than it was fifteen years ago.

eeZah7Ux 9y ago
Most of the modern devops hipster tooling is far worse and more complex to use that the old bash scripts put together by grumpy sysadmins.
- throwanem 9y ago
  I've used those scripts - I've written those scripts! - and I have no idea what you're talking about. Puppet is a marvel, and when it's too heavyweight for the task at hand, there's Ansible. Both are far, far preferable to me@homebox $ for host in foo bar baz; do cat random_stuff_who_knows_if_it_works.sh | ssh $host 'bash -'; done like we did in the old days you're talking about.
  - etatoby 9y ago
    You can have my cryptic Bash one-liners when you pry them from my cold, dead hands. But yeah, I agree that Ansible is the minimum that should be in place.
    - throwanem 9y ago
      Oh, I still perpetrate cryptic Bash one-liners myself, on a more or less daily basis. What's nice today, by comparison with the before time, is having a realistic option to do otherwise, and having that option not be Perl.
      - marcosdumay 9y ago
        > and having that option not be Perl Oh, Python is quite old already :)
        throwanem 9y ago
        Not better, though. Just different.
  - 9y ago

pjc50 9y ago

Where's that quote about how software architecture evolves to reflect the organisational structure of the business? Amazon does it this way because it's made of teams that are not just loosely coupled but somewhat firewalled from each other, along with a brutal internally competitive process.

devonkim 9y ago
Conway's Law
taneq 9y ago
Don't they have explicit rules dictating that any software must work as a service under failure-prone conditions?
- whateveracct 9y ago
  More than that, internal Amazon tools make it stupidly easy to do everything-as-a-service, so it's just how everyone does it.
deleted 9y ago
[deleted]

Lazare 9y ago

I think this is basically just an argument about terms. If you write code, you're a developer. If you run servers, your in ops. If you are doing both, it's devops. So when discussing the idea of having a total separate team for running the servers, OP says it's not devops and he's right, because the definition of devops is not having the teams be separate. And you say that it's often totally needed to have that separation and you're also right. > I guess it's possible to turn your entire dev team into part-time DevOps engineers, I think you mean, it's possible to turn your entire dev team into part-time ops engineers. Which is 1) the entire concept of "devops", and 2) often a horrible idea. :) (Sorry, but people who use "devops" to mean "ops" is a pet peeve of mine. If you're not a developer, it's just ops.)

TheCoelacanth 9y ago
Most devops teams that I see don't manage the servers. They provide tooling that makes it easier for dev teams to manage their own servers. If a devops teams is actually managing all the servers, I agree that they aren't doing devops, but at a large enough company it makes sense to have a team working just on things that make it easier for the company to do devops.

eeZah7Ux 9y ago

Dev + Ops == Devops, get over it.

dang 9y ago
Please post civilly and substantively, or not at all.
- eeZah7Ux 9y ago
  Sorry, I'll contextualize. > is going to have a DevOps Team The parent is arguing that the concept of a "DevOps Team" that does Ops makes sense and is not an oxymoron. Yet "DevOps" literally means doing Dev and Ops together. Otherwise have a "Developer Team" and an "Operations|Engineering|SRE|SysAdmin Team".

gloverkcn 9y ago

The statement touches on a real issue. I would revise the statement to: If your DevOps people reside in a separate group outside of the development team, then you are not doing DevOps. Architecture and Infrastructure go hand in hand, and DevOps is the glue that merges them together. I've worked with both DevOps and Architecutre groups in larger companies where Development, DevOps, and Infrastructure have been separated and siloed. This invariably leads to waste that undercuts any advantage provided by modern development practices. The core of the issue is the designs will only consider trade offs within their skill set. Problems that are easily solved with a combination of dev/devOps/inf are instead solved using complex designs within a single skill set. The other end of the spectrum is that decisions get made in one group that will undercut the efforts of another group. Real life examples: The dev group wants to deploy backend service version 2 with fail back option in production. The API's are identical, the database is unchanged. Instead of using a load balancer and monitoring to automate role back the dev group builds another system with the job of routing traffic between the two systems, identifying failures, and then stop sending traffic to the new system. The dev group designs a system with "micro-services" in mind. The developers have all containerized the services and run them on their laptop. The infrastructure and/or devops group doesn't want to deal with containers and instead deploys one service per VM. (In these scenarios the dev group will get a bill for the extra services). The problem is that it's easier to grab the people sitting next to design something than scheduling a meeting with groups you rarely see. This is a key driver of Conway's law.

Jare 9y ago
> If your DevOps people reside in a separate group outside of the development team, then you are not doing DevOps Devops and swdevs sitting and working together, understanding each other's problems and needs, and keeping them in mind when designing solutions, is indeed a beautiful thing to watch.

dominotw 9y ago

I always thought DevOps was the guy who is writing Kubernetes, not the guy who is installing it. Not sure why a special title is needed though.

pacala 9y ago

+1. Nothing promotes the ability to produce quality code like being woken at 3am to troubleshot a problem 3 microservices away from your own. In my anecdotical experience, the ability to recover from such incidents degrades with age, so nowadays I regard "everybody [but management] does DevOps" as codeword for ageism.

solatic 9y ago

> Any sufficiently complicated infrastructure that has uptime requirements and significant revenue associated with it is going to have a DevOps Team (or the equivalent) ultimately responsible for ensuring that things are working. I don't think these are mutually exclusive; I think you can both have a dedicated applicative operations (rebranded DevOps or SRE) team and still allow developers to rapidly provision and deploy. The point is a change in responsibilities and workflow on the part of operations: whereas their task was originally to deploy, verify success, rollback if necessary, monitoring, etc., their new task is to embed those tasks in code, so that developers can safely and reliably perform those tasks by themselves. You can have an organization where developers can automatically provision infrastructure which is behind your enterprise firewalls, hooked up to up-to-date enterprise artifact repositories, has enterprise PKI management already set up, etc., because expecting developers to know the ins and outs of setting that up manually is moronic and much of that should be scripted for ease of management anyway.

dkersten 9y ago

Developers should be involved in managing everything about their applications, including infrastructure. As a software engineer, there is just no way I can stay up to date with my craft as well as everything there is to know about infrastructure AND the all important security. I don't know about all the latest kernel patches and firewall rules. I don't know about monitoring failed hard drives. Etc. I mean, I do know a lot about deploying and maintaining my running software, but Ops has a ton of other concerns that themselves are a full time job and I already have my time fully consumed by development tasks. I'm also ok with being pushed enough of the work so that I don't accidentally make software that's a PITA to deploy or manage and that I don't needlessly wake somebody else. That doesn't mean that I can take on the entire job (well, I can, but I wouldn't be doing as good a job, especially when security and such is concerned, and I wouldn't be doing as good a development job either). What next - DevOpsSalesMarkettingAccounting? Much better, IMO, is to have small tightly-knit cross-functional teams which include an ops engineer.

0xdeadbeefbabe 9y ago
> well, I can, but I wouldn't be doing as good a job You do a better job communicating with yourself though?
- dkersten 9y ago
  Sure, but any advantage from that is lost in the fact that I've lost a ton of development time on doing ops tasks, which I don't have time to do properly and don't have time to learn how to do properly. Even bad communication to a competent ops person who does have time to, you know, do their job, and does have time to stay up to date, will result in much better results than if I tried to fit the work into my schedule. For example, in my current work, I could try to figure all the ops stuff out, but I've already got a backlog of months of development tasks for a project that needs to be done asap.
  - 0xdeadbeefbabe 9y ago
    I'd like to argue the advantage is sometimes lost, but not often.

enknamel 9y ago

> Remember, if you have a DevOps Team, then you are absolutely not doing DevOps. Developers should be involved in managing everything about their applications, including infrastructure. I think the intent here is a good one. You as a developer should be involved with how your software is deployed and managed. It's just that there is a pretty grey boundary between where developer involvement ends and ops involvement begins. Another issue is that devops is extremely poorly defined. Some people insist this means only writing chef scripts. Some people it's diagnosing and fixing production issues. Some people it's just sysadmin work. Or other responsibilities and combinations.