6 ms·

SRE was born of putting software engineers to work building operational software and automation tailored to an organization and application. In contrast, no mat

SRE was born of putting software engineers to work building operational software and automation tailored to an organization and application. In contrast, no matter what anyone says, DevOps was objectively born of replacing the operations discipline and career track with a poorly-understood tool economy and ongoing opex to a cloud provider. As you say, typical JavaScript engineers can’t be bothered to understand network capacity planning yet feel they are more qualified to take their application to production by deferring all decisions to cloud providers. Who all employ SREs/PEs, not DevOps Engineers, by the way, and there is a big distinction. We have people who can handle the operational stuff. They’re called systems administrators, network engineers, yes, even SREs, and other folks who are really good at understanding how computers and the Internet actually work, and a webdev bootcamp gives zero context into exactly what they do. None. Then, your ten-head startup suddenly scales to needing a physical footprint because it will literally save 80% of opex, and all your DevOps Engineers say “but why? There’s AWS,” and you’re in Medium hell for weeks arguing about it. Apropos, if I interview you and find you have written a thoughtpiece on Medium about how “sysadmin is evolving” and it’s “time for the gray beards to learn code,” you do not get a call back. That has actually happened, and no, sysadmin is not evolving. I know staff-level “full stack engineers” who can’t tell me what an ASN is. The motions in the industry have merely made those people more in demand at a few companies and served to cement their position as “where computing gets done”. Expect serious, existential operational threats and breaches to rise dramatically as DevOps continues to “win,” and consider it a smart strategic move to avoid DevOps culture in the long term. If you write a req called “DevOps Engineer,” I don’t even know what to say to you.

delusional 6y ago
Has sysadmin not evolved? if I found some sysadmin logging into a production system and editing the config file in nano today, I'd be downright depressed.
- dielectrikboog 6y ago
  Sounds like you’re going to be depressed when you learn how the entire Internet plane, all software engineering outside of “SV”, all IT, all government, and basically everything except your GitHub CI/CD adventure works, then. Sorry.
  - aprdm 6y ago
    Even within SF. Having talked with a bunch of folks from Amazon and Netflix they're far from having most of their workflows running in containers... imagine is the same for google.
  - jmb12686 6y ago
    This isn't an accurate statement. I work on behalf of a federal government agency, and no one has write access in development, let alone production. Everything is required to run thru our ci/cd pipeline. Times are changing.
    - dielectrikboog 6y ago
      For the better? I’m not asking out of preference, I’m asking out of actual conclusion: is trading the operational overhead of running LDAP for a usually homegrown, usually wobbly automated scripting soufflé that turns Make into a distributed system objectively better? Has nobody stopped to ask, is DevOps and CI/CD the best framework we can achieve? Did nobody think to ask before they told your agency it was the ‘right’ methodology and the objectively best way to build industrial, business process software in the government sector? Did the changing times come from ideology and belief or identified process gaps? I ask because I think there’s something better. I don’t know what it is yet, but I want to find out. I’m worried about wastage in DevOps methodologies, a system where nobody is incentivized to care about the right things, going on to spook the policymakers on doing software before we find out if the DevOps and Cloud worlds, both, are objectively the best way to do software for their purposes. I strongly, strongly feel like the craft is on the wrong path, and persuasive successes in industry are getting to the right ears before we know if the discipline to efficiently handle agile infrastructure with today’s tooling is even possible. I’m not convinced DevOps will organically find the right calculus to spur the kind of systems research that took us to not only where we are, but that which will take us where we need to go. Speaking of, I’m lazily glancing at Agile here as well but I’m not prepared for a coherent argument there, beyond pointing out that we now have better tooling for managing specifications, particularly formal and mathematical ones, than the waterfall development experiences that prompted agile thinking. We need more systems research, tinkering, rethinking POSIX, all of it.
icedchai 6y ago
Most "DevOps" folks I know are actually former sysadmins who evolved to work more with cloud technologies. To say sysadmin hasn't evolved in a bit of an exaggeration. Titles follow the trend. What people actually do is often similar.
- dielectrikboog 6y ago
  No, it isn’t an exaggeration. They ceded one particular competency, systems administrator, and now pay cloud providers to do it instead. The job didn’t go anywhere. Capacity planning, change management, peering, supply chain management, all of that stuff is still happening, they just willingly tapped out of it and took another job (probably because the DevOps people came in with a slide deck and hand waved them out of a job at the SVP level). That is not evolution (nor an indictment of those people, importantly). The side effect, which literally nobody is paying attention to, is a future where computing as a concept is concentrating on a few major companies. Every every every every single person who says “why would I buy a rack? There’s AWS” is furthering that outcome.
  - icedchai 6y ago
    You still need to do capacity planning, change management, monitoring, etc. within your cloud environment. Those AWS instances and the software their running doesn't manage itself. For some subset of "cloud", such as PaaS providers like Heroku, etc., you are absolutely correct. For another subset of "cloud", you still need sysadmin / ops skills to manage it.
    - dielectrikboog 6y ago
      Yeah, you do. It’s a shame pretty much every single cloud-native shop in existence, you know, doesn’t bother, and pushes out the people arguing for bothering. I’ve been at this nearly two decades, and I have yet to find engineers even running an Excel notebook of inventory, much less capacity planning. You know, because describe-instances and monitoring and Ansible and Chef and blah blah. My role right now is telling a major government agency how much they’re wasting on Azure. You know, because describe-instances. It’s a lot, and I think there might be a business model in “let me at your API for a week and give me 10%.” I’d be retired by Labor Day. Reminder: They’re sending Congressionally appropriated funds. To Redmond. And they’re not entirely sure why, in $millions of cases. Line up fifty startups that have had a Demo Day and I’d bet you’d find the same thing in fifty-one of them. That’s the DevOps legacy: don’t mind the budget, because AWS, Azure, and GCP have our financial interests in mind and APIs are cheaper to staff than fiber. Parades of like-minded individuals came to D.C. and said “DevOps! Do it!” and the agencies are now increasingly beholden to organizations incentivized by profit and those contractors took their Innovation Award check and don’t return the “um, what now?” calls. That’s the mess I’m trying to help clean up, and it’s happening across every major governmental organ in the United States.
aprdm 6y ago
This is a great post but I want to say that I feel there's space for both. IMO a DevOps Engineer would sit between the sysadmin/network folks and the developers who wants to be users of a system. In my current gig we've moved from the DevOps department to the Platform department as it aligns more with what we are trying to provide. A Platform for developers. That said we essentially can speak sysadmin and can speak developers. We trust sysadmins with network, linux image and more specialist topics. We make tools for both sides and try to make them work together often sitting in the middle and negotiating.
- dielectrikboog 6y ago
  Call them SREs and cross-train SWEs into it. It’s not a toothless distinction even though it seems like one. You absolutely, positively will hire better staff with better deliverables if you frame the work as “a software engineer focused on operational integration,” which SRE understands more. SREs like to build platforms for exactly the same reasons you’re touching. You sound like you’re halfway there already. I strongly suggest the Google book, with “I am not Google scale” written in Sharpie on the cover for help digesting it.
  - aprdm 6y ago
    In my understanding SRE is more related to "keep the lights on and systems running", it might be just a different understanding of the nomenclature. E.g: In my current case the software teams own their ops, my team doesn't ops for them. We give them a platform of centralized logging, monitoring and etc. so that they can easily ops their services but is not my phone that rings and I am not on call. I am on call if some component of the platform itself fails. At least my perception of SRE is that they're on call for products. That said I would frame the work we do as “a software engineer focused on operational integration“. That does sound like a good book and I will add to my to read list.
    - dodobirdlord 6y ago
      > In my understanding SRE is more related to "keep the lights on and systems running", it might be just a different understanding of the nomenclature. SREs at Google own production in a very deep sense. They are decision makers on things like when teams can deploy, how frequently, what dependencies they can use, and possibly most significantly, who gets SRE support and who has to handle their own on call rotation. They also build monitoring and reliability services and tools. Google also employs traditional Ops people, but not as many as you might suspect. When SREs look at traditional Ops work, they see a threat to reliability and a target for automation. The mantra is that the "E" isn't for show, and that SREs are software engineers who specialize on the topic of running highly reliable services. One of the things the SRE book stresses is making sure that SRE teams aren't so bogged down in oncall responsibilities that they don't have time to work on automating their oncall responsibilities.
twic 6y ago
> no matter what anyone says, DevOps was objectively born of replacing the operations discipline and career track with a poorly-understood tool economy and ongoing opex to a cloud provider This isn't the origin of DevOps.
elbear 6y ago
Hello! I am a developer who wants to understand networks. Can you point me to some reading resources? For now, I've just been looking at the wikipedia pages for the different protocols. But I think it would help me to work with concrete scenarios in which you use knowledge of networks to better understand things. I would appreciate it if you pointed me to anything you think worthwhile.
- dielectrikboog 6y ago
  Take certifications. Not to get them, but because preparing for them will structure the learning better than anyone can in response to that question.
kqr 6y ago
I agree with your overall point, but I also think there's a bit of conflation going on with the term "DevOps". It means different things to different people. What I think is as close as we get to a canonical meaning is the meaning in which it is used by The State of DevOps reports, based on very good science and research by Nicole Forsgren et al. They characterise "DevOps" as a transition into faster deployments, shorter feedback cycles, less warehousing of unexecuted code, and having developers have generally more insight into what's going on in the production environment. This, of course, can (and arguably should) be done in cooperation with proper system administrators, network engineers, etc. In other words, DevOps is not in opposition of having the right people people operate the systems. In particular, it has nothing to do with cloudifying things. You can run a product with a DevOps approach right onto bare metal servers – in fact, there are a lot of companies doing that, for simple economical and reliability reasons. I'm all for ranting against the cloud and the little experience people have when trying to operate systems, but blaming "DevOps" for it seems like a mischaracterisation. There's a lot of value to be had by getting more feedback from production, whether production means bare metal or virtualised environments.
- dielectrikboog 6y ago
  As you say: DevOps means a million things to a million people. That’s why I ignored the person who tried to explain to me that I had the origin of DevOps wrong. Nobody alive or dead is qualified to make such a pronouncement, because nobody knows. It is an amorphous blob that usually manifests as a weapon for developers to beat the operations disciplines out of their company, which is why I speak about it as I do. Given the overwhelming evidence that the interpretation I’m going after is the popular one, arguing over the definition of the term is pointless. You’re conflating my argument with cloud ranting and assuming DevOps methodology is the only method to acquire more feedback from production by stating your last paragraph like that. I’m saying there are potentially others, but we are entrenching on this way of doing things, and people picked this particular way of doing things and started talking organizations outside “SV” into it. That conversation gets harder a second, third, and fourth time. The prevalence of COBOL reqs should warn you of this, and what DevOps will look like in about a hundred years.
jen20 6y ago
> no matter what anyone says, DevOps was objectively born of replacing the operations discipline and career track with a poorly-understood tool economy and ongoing opex to a cloud provider. This is abjectly untrue with regards to the origins of the term - though it is the current state of the world, and your assertion about job reqs for "DevOps Engineers" is spot on. "DevOps" as a term was coined by Patrick Debois and Kris Buytaert to succinctly refer to the concept of operations teams and development teams collaborating in a more appropriate manner than the "throw stuff over a wall" which is still common in many enterprises. It was unrelated to tooling. We must not let vendors co-opt terms in such a way as this.

delusional 6y ago

Has sysadmin not evolved? if I found some sysadmin logging into a production system and editing the config file in nano today, I'd be downright depressed.

dielectrikboog 6y ago
Sounds like you’re going to be depressed when you learn how the entire Internet plane, all software engineering outside of “SV”, all IT, all government, and basically everything except your GitHub CI/CD adventure works, then. Sorry.
- aprdm 6y ago
  Even within SF. Having talked with a bunch of folks from Amazon and Netflix they're far from having most of their workflows running in containers... imagine is the same for google.
- jmb12686 6y ago
  This isn't an accurate statement. I work on behalf of a federal government agency, and no one has write access in development, let alone production. Everything is required to run thru our ci/cd pipeline. Times are changing.
  - dielectrikboog 6y ago
    For the better? I’m not asking out of preference, I’m asking out of actual conclusion: is trading the operational overhead of running LDAP for a usually homegrown, usually wobbly automated scripting soufflé that turns Make into a distributed system objectively better? Has nobody stopped to ask, is DevOps and CI/CD the best framework we can achieve? Did nobody think to ask before they told your agency it was the ‘right’ methodology and the objectively best way to build industrial, business process software in the government sector? Did the changing times come from ideology and belief or identified process gaps? I ask because I think there’s something better. I don’t know what it is yet, but I want to find out. I’m worried about wastage in DevOps methodologies, a system where nobody is incentivized to care about the right things, going on to spook the policymakers on doing software before we find out if the DevOps and Cloud worlds, both, are objectively the best way to do software for their purposes. I strongly, strongly feel like the craft is on the wrong path, and persuasive successes in industry are getting to the right ears before we know if the discipline to efficiently handle agile infrastructure with today’s tooling is even possible. I’m not convinced DevOps will organically find the right calculus to spur the kind of systems research that took us to not only where we are, but that which will take us where we need to go. Speaking of, I’m lazily glancing at Agile here as well but I’m not prepared for a coherent argument there, beyond pointing out that we now have better tooling for managing specifications, particularly formal and mathematical ones, than the waterfall development experiences that prompted agile thinking. We need more systems research, tinkering, rethinking POSIX, all of it.

icedchai 6y ago

Most "DevOps" folks I know are actually former sysadmins who evolved to work more with cloud technologies. To say sysadmin hasn't evolved in a bit of an exaggeration. Titles follow the trend. What people actually do is often similar.

dielectrikboog 6y ago
No, it isn’t an exaggeration. They ceded one particular competency, systems administrator, and now pay cloud providers to do it instead. The job didn’t go anywhere. Capacity planning, change management, peering, supply chain management, all of that stuff is still happening, they just willingly tapped out of it and took another job (probably because the DevOps people came in with a slide deck and hand waved them out of a job at the SVP level). That is not evolution (nor an indictment of those people, importantly). The side effect, which literally nobody is paying attention to, is a future where computing as a concept is concentrating on a few major companies. Every every every every single person who says “why would I buy a rack? There’s AWS” is furthering that outcome.
- icedchai 6y ago
  You still need to do capacity planning, change management, monitoring, etc. within your cloud environment. Those AWS instances and the software their running doesn't manage itself. For some subset of "cloud", such as PaaS providers like Heroku, etc., you are absolutely correct. For another subset of "cloud", you still need sysadmin / ops skills to manage it.
  - dielectrikboog 6y ago
    Yeah, you do. It’s a shame pretty much every single cloud-native shop in existence, you know, doesn’t bother, and pushes out the people arguing for bothering. I’ve been at this nearly two decades, and I have yet to find engineers even running an Excel notebook of inventory, much less capacity planning. You know, because describe-instances and monitoring and Ansible and Chef and blah blah. My role right now is telling a major government agency how much they’re wasting on Azure. You know, because describe-instances. It’s a lot, and I think there might be a business model in “let me at your API for a week and give me 10%.” I’d be retired by Labor Day. Reminder: They’re sending Congressionally appropriated funds. To Redmond. And they’re not entirely sure why, in $millions of cases. Line up fifty startups that have had a Demo Day and I’d bet you’d find the same thing in fifty-one of them. That’s the DevOps legacy: don’t mind the budget, because AWS, Azure, and GCP have our financial interests in mind and APIs are cheaper to staff than fiber. Parades of like-minded individuals came to D.C. and said “DevOps! Do it!” and the agencies are now increasingly beholden to organizations incentivized by profit and those contractors took their Innovation Award check and don’t return the “um, what now?” calls. That’s the mess I’m trying to help clean up, and it’s happening across every major governmental organ in the United States.

aprdm 6y ago

This is a great post but I want to say that I feel there's space for both. IMO a DevOps Engineer would sit between the sysadmin/network folks and the developers who wants to be users of a system. In my current gig we've moved from the DevOps department to the Platform department as it aligns more with what we are trying to provide. A Platform for developers. That said we essentially can speak sysadmin and can speak developers. We trust sysadmins with network, linux image and more specialist topics. We make tools for both sides and try to make them work together often sitting in the middle and negotiating.

dielectrikboog 6y ago
Call them SREs and cross-train SWEs into it. It’s not a toothless distinction even though it seems like one. You absolutely, positively will hire better staff with better deliverables if you frame the work as “a software engineer focused on operational integration,” which SRE understands more. SREs like to build platforms for exactly the same reasons you’re touching. You sound like you’re halfway there already. I strongly suggest the Google book, with “I am not Google scale” written in Sharpie on the cover for help digesting it.
- aprdm 6y ago
  In my understanding SRE is more related to "keep the lights on and systems running", it might be just a different understanding of the nomenclature. E.g: In my current case the software teams own their ops, my team doesn't ops for them. We give them a platform of centralized logging, monitoring and etc. so that they can easily ops their services but is not my phone that rings and I am not on call. I am on call if some component of the platform itself fails. At least my perception of SRE is that they're on call for products. That said I would frame the work we do as “a software engineer focused on operational integration“. That does sound like a good book and I will add to my to read list.
  - dodobirdlord 6y ago
    > In my understanding SRE is more related to "keep the lights on and systems running", it might be just a different understanding of the nomenclature. SREs at Google own production in a very deep sense. They are decision makers on things like when teams can deploy, how frequently, what dependencies they can use, and possibly most significantly, who gets SRE support and who has to handle their own on call rotation. They also build monitoring and reliability services and tools. Google also employs traditional Ops people, but not as many as you might suspect. When SREs look at traditional Ops work, they see a threat to reliability and a target for automation. The mantra is that the "E" isn't for show, and that SREs are software engineers who specialize on the topic of running highly reliable services. One of the things the SRE book stresses is making sure that SRE teams aren't so bogged down in oncall responsibilities that they don't have time to work on automating their oncall responsibilities.

twic 6y ago

elbear 6y ago

Hello! I am a developer who wants to understand networks. Can you point me to some reading resources? For now, I've just been looking at the wikipedia pages for the different protocols. But I think it would help me to work with concrete scenarios in which you use knowledge of networks to better understand things. I would appreciate it if you pointed me to anything you think worthwhile.

dielectrikboog 6y ago
Take certifications. Not to get them, but because preparing for them will structure the learning better than anyone can in response to that question.

kqr 6y ago

I agree with your overall point, but I also think there's a bit of conflation going on with the term "DevOps". It means different things to different people. What I think is as close as we get to a canonical meaning is the meaning in which it is used by The State of DevOps reports, based on very good science and research by Nicole Forsgren et al. They characterise "DevOps" as a transition into faster deployments, shorter feedback cycles, less warehousing of unexecuted code, and having developers have generally more insight into what's going on in the production environment. This, of course, can (and arguably should) be done in cooperation with proper system administrators, network engineers, etc. In other words, DevOps is not in opposition of having the right people people operate the systems. In particular, it has nothing to do with cloudifying things. You can run a product with a DevOps approach right onto bare metal servers – in fact, there are a lot of companies doing that, for simple economical and reliability reasons. I'm all for ranting against the cloud and the little experience people have when trying to operate systems, but blaming "DevOps" for it seems like a mischaracterisation. There's a lot of value to be had by getting more feedback from production, whether production means bare metal or virtualised environments.

dielectrikboog 6y ago
As you say: DevOps means a million things to a million people. That’s why I ignored the person who tried to explain to me that I had the origin of DevOps wrong. Nobody alive or dead is qualified to make such a pronouncement, because nobody knows. It is an amorphous blob that usually manifests as a weapon for developers to beat the operations disciplines out of their company, which is why I speak about it as I do. Given the overwhelming evidence that the interpretation I’m going after is the popular one, arguing over the definition of the term is pointless. You’re conflating my argument with cloud ranting and assuming DevOps methodology is the only method to acquire more feedback from production by stating your last paragraph like that. I’m saying there are potentially others, but we are entrenching on this way of doing things, and people picked this particular way of doing things and started talking organizations outside “SV” into it. That conversation gets harder a second, third, and fourth time. The prevalence of COBOL reqs should warn you of this, and what DevOps will look like in about a hundred years.

jen20 6y ago

> no matter what anyone says, DevOps was objectively born of replacing the operations discipline and career track with a poorly-understood tool economy and ongoing opex to a cloud provider. This is abjectly untrue with regards to the origins of the term - though it is the current state of the world, and your assertion about job reqs for "DevOps Engineers" is spot on. "DevOps" as a term was coined by Patrick Debois and Kris Buytaert to succinctly refer to the concept of operations teams and development teams collaborating in a more appropriate manner than the "throw stuff over a wall" which is still common in many enterprises. It was unrelated to tooling. We must not let vendors co-opt terms in such a way as this.