11 ms·
What Is ‘Site Reliability Engineering’?
- deleted 9y ago[deleted]
- zeckalpha 9y agoIs this interview new or was it released as part of the book?
- pronoiac 9y agoI think it's new to the website. I don't see it in the table of contents for the rebook, and it's been online since August, according to https://web-beta.archive.org/web/20160804182333/https://landing.google.com/sre/interview/ben-treynor.html https://web-beta.archive.org/web/20160804182333/https://land...
- dogecoinbase 9y agoSREs are a tool to turn N ops engineers paid X each into 1 SRE paid 2X and N manual laborers paid X/4 each. This doesn't make the role bad. But it's important to remember that the role exists as a cost savings to the org, not because it's an inherently better way to run a technical infrastructure.
- pram 9y agoActually in most places I've seen 'ops' just turned into 'SRE'
- otoburb 9y agoAnd because other organizations are sold on the idea that SREs can realize cost savings benefits without compromising reliability of their technology assets, boards and c-suite teams definitively define this as a better way to run technical infrastructure.
- icebraining 9y agoManual laborers? At Google? What are they doing?
- tyingq 9y agoI would guess they are referring to the people racking and stacking equipment, running network cables, doing any needed physical reboots, swapping out old servers, etc.
- imesh 9y agoI work at a web host, and my SRE title means being being a developer who gets constantly interrupted by alerts and customer chats.
- a_imho 9y agoDo you earn a developer salary?
- poikniok 9y agoI would presume more than a developer salary.
- FLUX-YOU 9y agoIME, it is safer to assume they are NOT paying him for knowledge across multiple roles. It's a business win to find someone with multi-disciplinary skills without having to pay them the combined salary of those disciplines. It's even possible they are only paying him a developer's salary.
- nandemo 9y agoUnless you're working a startup that's intentionally "doing things that don't scale" (because e.g. you're still yet to get to product-market fit), I can't see how it makes sense for a developer or SRE to get interrupted by "customer chats". As for endless alerts, I've been learning that there are 3 types: • false alerts, which were set up "just in case". The "handling" tends to be "take a look, declare that nothing is wrong after all, mark it solved". Your job is to summarily eliminate these. • "good" alerts that indicate a temporary problem that could/should be handled automatically in an ideal world, but can't be handled automatically as things stand; your job is to move the status quo in the direction of the ideal world, gradually. • all the rest. These are the unavoidable alerts. Usually, the best you can do is to ensure the alert message and related logs provide all the relevant information necessary to handle it. PS: of course, TFA indicates Google has a much more systematic approach, but you probably won't be able to just copy them.
- 9y ago
- raz32dust 9y agoWith more automation and containerization, I see the SRE role and dev role coming together, eventually merging into "devops". Today, these roles are separate because they require slightly different skill sets. Maintaining production systems takes up about as much time as developing new features. As it becomes easier and easier, dev will be the ops, even in big companies.
- adrianN 9y agoMaintaining production systems won't become easier in the same way software engineering didn't become easier because of the introduction of high level languages. The systems only become more complex if it becomes easier to manage complex systems.
- tomtompl 9y agoPint for this guy. Related post: https://medium.com/outsystems-engineering/the-law-of-conservation-of-complexity-1-simple-rule-6578a2bbfdbf https://medium.com/outsystems-engineering/the-law-of-conserv...
- rconti 9y ago... and in the same way virtualization didn't give us time to kick back and relax in all of our 'free' time now that we're not racking boxes and cabling stuff all the time. Instead of creating dedicated application users and chroot jails and alternate port numbers to let applications coexist on a server, we're spinning a bazillion instances and building out storage backend to support it, and so on. And the lowest-level problems still exist, though we don't troubleshoot them as often; we just re-spin. Same reason nobody's repairing their RAID card with a soldering iron anymore.
- HeavenBanned 9y agoA "SRE" is what happens when you want to pronounce the word "SWE" but can't. For some reason you keep saying "SRE" over and over and over again. They were overcompensating for the fact that SREs aren't SWEs so hard. It's like "we get it, SREs are wannabe SWEs, stop trying to sugar coat it". 50% development? What a disaster. If half your job is the job that you want and the other half is administrative bullshit, why in the living fuck would you try to make a puff piece about that? It seems as though from what everyone has said in this thread, that SRE is basically a scam along with DevOps and that the real job people want is the SWE. I don't like internal memo propaganda pieces by big companies. It's not intellectually stimulating: it's hogwash. Let the truth reign always.
- rconti 9y agoAssuming SWE means software engineer. No. Thanks.
- HeavenBanned 9y agoHey - just sayin' ...I'm not even interested in becoming a SWE OR A SRE...just academia... Only academia. I have the pedigree/potential. I just need to unlock it.
- burntrelish1273 9y agoHere's a script to fetch an offline copy https://gist.github.com/steakknife/76214a4bb378592669655e3bbc30a1cc/ https://gist.github.com/steakknife/76214a4bb378592669655e3bb...
- sigi45 9y agoJepp thats how i always wanted to do software engineering: Understanding / controlling the full stack and taking responsibility for it.
- ZanyProgrammer 9y agoUnderstanding, sure. Taking responsibility? No way.
- SwellJoe 9y agoI have a theory that Google invented the SRE because they didn't know how to hire system administrators, but they knew how to hire software engineers. So, they just hired software engineers and told them to figure out the systems. I say this only partially in jest.
- lclarkmichalek 9y agoEh, I think they say something similar in the book. The other side of it was that they wanted a software engineering approach to systems administration (which completely depends on your opinions of the respective professions)
- dj_jorjinho 9y agoThe opposite also occurs: a SysAdmin gets hired to do DevOps, but they're not qualified to be Software Engineers. The results is that, if you're not looking, your ops code ends up a series of "scripts" instead of a series of well structured tools. But I guess this depends on the company culture to begin with.
- deleted 9y ago[deleted]
- SwellJoe 9y agoIt seems to be mostly working for them, though I think companies that haven't gone so far down that rabbit hole are also doing fine. One of the premises of SRE that I occasionally take issue with is "build your own tools". That's a fine idea, sometimes, but it also results in NIH syndrome, which has a real cost. Obviously, Google is operating at a scale that no one ever has before (and almost no one else does or ever will), and so, sometimes they have to invent the technologies for doing it. But, the somewhat dismissive tone of the SRE literature I've read that implies system administrators scale linearly while SREs scale drastically higher is kind of off-putting, and misrepresentative, I think. Sysadmins have been scaling systems for as long as there have been systems; sometimes just one person or a small team in the data center, running the whole show for quite large companies.
- deckardb26354 9y agoSRE? Apparently the only 'software' job Google has in Dublin. It doesn't matter if you have a PhD or wrote your own kernel, want to write code for Google, move to mountain view. Oh and the seven interviews. Complete waste of time.
- NickNameNick 9y agoIEEE software engineering radio did a good episode on Site Reliability engineering. http://www.se-radio.net/2016/12/se-radio-episode-276-bjorn-rabenstein-on-site-reliability-engineering/ http://www.se-radio.net/2016/12/se-radio-episode-276-bjorn-r...
- grabcocque 9y agoSRE: because DevOps isn't buzzwordy enough these days.
- StreamBright 9y agoSRE predates DevOps by 5 years at least.
- ZanyProgrammer 9y agoI always associate the word "site" in SRE with some sort of industrial/facilities engineering position when my mind first parses that word.
- nailer 9y ago> We care deeply about keeping SRE an engineering function, so our rule of thumb is that an SRE team must spend at least 50% of its time doing development. Prior to SRE any good system administrator was doing this: "if it's worth doing, it's worth automating". But there was another half who were cutting and pasting shit from Word files into Solaris boxes. sysadmin -> SRE seems to have cleaned out the chaff.
- traf68 9y agoHow many developers selected hardware, configured hardware, burnt it in, racked and cabled it, entered it into a company insurance roster, made and maintained the interfaces and documented it? This _was_ SA 101 and is the part you ignore. Hell even in 2007 if you didn't know hardware you were unemployable as an SA. What do devs writing json-rpc interfaces to monolith 1212 running oracle X as backend care about infra? This is the sea change and not everybody agrees with it. You have a 'cloudy' perspective.
- chrisp_dc 9y agoI think knowing the hardware is less challenging than previously. The trend is to make everything whitebox and then abstract as software. Hypervisors replace supporting many hardware configurations and software defined SANs replace dedicated appliances.
- nailer 9y agoConfiguration, asset management, interface config and documentation are included under things to automate and always have been.
- rconti 9y agoBoth your "good" and "bad" examples seem to be more operator-type roles. so much sysadmin work in many shops cannot readily be automated -- doubly so if all of the 'easy' stuff has already been automated!
- rodionos 9y agoIt's a euphemism for a system administrator with responsibilities to test, integrate, and automate systems with code.
- atsaloli 9y agohttps://www.usenix.org/conference/lisa16/conference-program/presentation/closing-plenary https://www.usenix.org/conference/lisa16/conference-program/... is a video of Niall Murphy's excellent presentation with Todd Underwood of how smaller organizations can implement SRE basics. Dec 2016. USENIX LISA in Boston. I had the privilege to attend it.
- zatkin 9y ago>We've held that hiring bar constant through the years, even at times when it's been very hard to find people, and there's been a lot of pressure to relax that bar in order to increase hiring volume. We've never changed our standards in this respect. That has, I think, been incredibly important for the group. Because what you end up with is, a team of people who fundamentally will not accept doing things over and over by hand, but also a team that has a lot of the same academic and intellectual background as the rest of the development organization. This ensures that mutual respect and mutual vocabulary pertains between SRE and SWE. It seems like changing their hiring process is a double edged sword. If they change it to allow more hiring volume, then other employees might become frustrated with how easy it becomes to work at Google. On the other hand, keeping an old hiring process where false negatives continue to occur seems very bad.
- pm90 9y agoGood point. I've had a very cautious opinion about hiring process at Google; I know that there are people who feel extremes either way. But something really seems to be wrong if the company is still so hugely dependent on search advertising for revenue after more than a decade of business. Maybe you need some (relatively) dumber people to discover new revenue streams.
- workerIbe 9y agoWe prefer the term "non-linear thinkers".
- awkbug 9y agoI recently attended interviews at LinkedIn, attlasian and my experience was very bad. First round is online exam and I answered all the questions. Attlasian rejected even having 100% right with all test cases. No response from LinkedIn. They told I can use any language to solve and I chose bash. I think they didn't like me using bash. The guy who interviewed me at LinkedIn is system administrator with sre title. Funny thing is he said he doesn't do programming. Companies are just misusing these titles. They need software engineering who can do system administration. The types who run apt-get on Centos :p
- StreamBright 9y agoOne thing that I was not aware prior to working as an SRE is how much they rely on statistics. This approach that you can use stats to determine what is a normal or abnormal level for a particular metric (like packet loss for example) became pretty useful through my career. A quick example, I was called into a meeting at a company where I worked in a non-SRE role, and the team explained to me that they are not able to identify what is wrong but their cluster is misbehaving and there is a node that gets kicked out of it regularly. I pulled up a console and started to compare OS level metrics across the cluster. The sysadmin team thought I am stupid because they explicitly told me which node was in trouble. After the third metrics I checked I found out that the node in question was doing 5000x more packet loss than the second worst in the cluster. It was a faulty NIC at that time. The sysadmin team was checking all of the metrics on the broken node but never compared the results to the healthy ones.
- graycat 9y agoYup, such things are some of why, when I did work in anomaly detection, I wanted it to be multi-dimensional. What you observed is the same song, second verse of: There was a cluster doing transaction processing. One of the computers in the cluster got a little sick and was throwing all its incoming transactions into the bit bucket. The load leveling for the cluster sent the next transaction to the least busy computer in the cluster, and the sick computer, not doing any real work, was usually the least busy so got nearly all the incoming transactions. So, really the one sick computer was throwing away nearly all the incoming transactions of the whole cluster, made the whole cluster look sick. So, sure, some anomaly detection that takes as its input data, say, the CPU busy of each of the computers in the cluster should see that the one sick computer was comparatively low on CPU busy, call that an anomaly, raise an alarm, and let diagnosis begin. Sure, if do have such an anomaly detector, very much want good control over false alarm rate. So, can see what I posted five days ago (gee, suddenly anomaly detection is getting popular on HN?) in https://news.ycombinator.com/item?id=14119753 https://news.ycombinator.com/item?id=14119753 and https://news.ycombinator.com/user?id=graycat https://news.ycombinator.com/user?id=graycat
- traf68 9y agoIt is stupidity, hubris and a disposition to chaos.
- NotQuantum 9y agoI've fallen in love with SRE field. I'm a Computer Engineering senior currently. I'm used two kinds of classes: CS ones where you learn a lot of theory and apply it on a test, then the CprE ones where you also learn, but then have to make it work in labs. I've always liked lab based classes where you have to take a concept to fruition. I've been interested in all aspects of CprE, and I taught myself how to run a Linux box along with DNS, VPN, and other services. Last year around this time, I was contacted by a recruiter for an SRE internship. At the time, I had no idea what SRE was and I thought it was just a glorified IT job. Boy was I wrong. I got through a few interviews and got the position for the summer. About a week or two into the internship I fell in love. This job was all about designing and implementing systems that have to be resilient and must scale. The idea of building automation to make my job easier was and is great. It was just like the labs I enjoyed in college. Fast forward to now, and I'm accepting a full time SRE position at the same company. I couldn't be happier with my choice in specification. The need for resilient, distributed systems will only grow in the coming years, and I'm looking forward to being an SRE.
- robhirschfeld 9y agoSRE is a job function. By design, it's intended to be equivalent in pay and status with developers (SWE) to overcome the bias against operators and sysadmins in organizations. This is an important recognition because cloud-first operations requires a lot of automation and coding expertise that previous operations roles did not demand. DevOps is really a process definition with Lean system thinking and code workflow priorities. Many people will tell you that it is NOT a job function but a culture or approach. DevOps for developers generally means CI/CD pipelines and owning code into production. DevOps for operators generally means building configuration automation and integrated monitoring tools. In this was, DevOps highly complementary of the SRE job function. I've been writing a lot about this on my personal (robhirschfeld.com) and company (rackn.com/sre) blogs. I'd be happy to discuss this in more detail here.