23 ms·

AI handles incidents, engineers lose touch with their systems

418 points by sylvainkalache 12d ago

Gurio 11d ago
[flagged]
tangsoupgallery 12d ago
[flagged]
zkmon 11d ago
Wasn't that the idea? Withe cars on roads, didn't we literally lose the touch with the ground? I guess the desire was to "lose" touch with all dirty and hard work areas.
- zdragnar 11d ago
  The only plan was to make making cheaper. There wasn't a plan for the people afterwards.
- esalman 11d ago
  That's what the linked article is talking about. Training for extreme failure. Sure you have self driving cars and everything that make you more and more disconnected, but still highways more dangerous to drive whenever there is a storm or ice on the road. For the same reason, driving tests are harder in the north.
pmarreck 5d ago
AI should never solely be in charge of anything of real significance. I say this as an AI proponent. Humans are the ones that touch grass.
deleted 12d ago
[deleted]
danielbln 12d ago
If capability increase continues as it has, then an incident that cannot be resolved by AI will stump humans no matter the practice. I like the plane example from the article,but I think in reality it will be like code. 1.5 years ago engineers would routinely say that they still write code by hand here or there to keep their skills sharp, and that's just not something you hear much if at all. If an SRE is faced with a situation an AI can't solve, then said SRE will use the AI systems to triage further, point it to different places and so on. This works for SREs with pre-AI experience and intuition, possibly less so with new recruits coming in post-AI. I don't know what the solution to this is, maybe practice drills is it, but I have a hunch the entire field will be subsumed, same as many other engineering fields. There is only so much need for taste and judgement, before even that has been incorporated into the models.
- sdevonoes 12d ago
  Nah, LLM models are already the new compilers. A commodity only engineers know how to use (in the context of software engineering in production environments)
  - wafflemaker 12d ago
    Out of context, but to address "AI will replace engineers". Recently discussed something about economy/investing with a friend while at work at a slaughterhouse. I really didn't want him to get scammed buying crypto. So, used ChatGPT to find some sources in Somali, a 3 videos with short description why it's worth watching. Intro into investing, intro about cryptocurrencies and about buying them. Had the text shortened down to 3 pretty short paragraphs, not more than twice this post. He's a smart guy, but only went to primary Qur'an school. Doesn't read or such, mostly consumes internet in form of video/media. He couldn't read those 3 paragraphs, it was too long. Or rather, it wasn't just 3 paragraphs, it was a lot to read. Maybe we're already dividing into murlocs and the surface dwellers?
- bob1029 12d ago
  > If capability increase continues as it has, then an incident that cannot be resolved by AI will stump humans no matter the practice. I disagree with this. Whatever the AI produces must be embodied in some kind of information system. The moment the output is on disk, it's fish in a barrel for any competent operator. I've worked in environments that are beyond the pale with regard to complexity. It will take AI another 10 years to product something as complicated and coherent as a semiconductor manufacturing operating system, which is clearly feasible for humans to manage today.
_doctor_love 12d ago
Good article and I like the callouts to the aviation industry. For me what's missing is the author should also have touched on CRM and SRM. Also, that paper "The Ironies of Automation" is one that everyone should read. It's fairly short. There is a related problem in terms of these situations where the computer system is handing off to the human. It's called "the bumpy transfer of control." Very fascinating concept.
jtfrench 12d ago
The more code writes autonomously, the less intuition the human owners have about that code. Loss of intuition is a seed of technical debt that grows with time. Over a long enough horizon, it can make looking at your own codebase feel like the first day on the job (sometimes at a company you started). Luckily, there are ways to mitigate this and essentially translate those human intuition of how the codebase “should” be into guardrails for the agents. But without that, your setting your sails in a stochastic sea where each wave looks nothing like the last.
- yard2010 12d ago
  I've been thinking about this lately - is it like using 3rd party libs to achieve stuff faster? As much as I would lovr to hand craft the datetime logic in my app, I might as well use luxon and invest this time somewhere else. Only now with llms, you get virtually infinite 3rd party libs you can use, you create them on the fly. So if you have strong engineering values, I would say simply it boils down to "contracts over programs", you can still be in touch with the logic that glues it all together and treat some logic as a blackbox the same way we do with 3rd party libs?
  - Muromec 12d ago
    It's not because with libraries you have a boundary somewhere and can decide to not care what's inside as long as the interface is stable and well designed. The problem of course you need to prioritise building well designed interfaces and decouple components from each other, and that's a skill most developers aren't good at.
    - dist-epoch 12d ago
      Models are getting better. Astra for example refactored some code GPT-5.6 Sol wrote and extracted some related functionality in a separate file without prompting. The models are also becoming very capable of suggesting architecture simplifications, but you need to ask for that, and you probably need some taste in picking from the suggestions, although the default ones are quite decent.
  - 12d ago
bob1029 12d ago
A natural evolution of engineers losing touch with the customers and users. I'm noticing some of the concern play out regarding AI weakening the capabilities of software people. I gave the team an exact solution on a silver platter and they still failed to identify how to go about it after 3 days slamming it into Claude. The resolution is literally 1 line of code that could be arrived at in about 30 minutes of patient, old school troubleshooting. I think what's happening is the AI system draws poorly aligned and led engineers into this ego inflation feedback loop where they are completely detached from reality because these tools can simulate a better one.
- edg5000 12d ago
  I it usually doesn't get me in this weird state of mind, but I once spent 6 months (all-in) building a thing that I, once finished, just left alone completely (on disk gathering dust). Weird experience. So I'd say AI physchosis is real.
- Root_Denied 12d ago
  I'm seeing this happen in the security space right now. Someone on my team I was helping train and bring along is all of sudden regressing in their understanding of the issues we're working on, and instead focusing on AI tool outputs to do their job for them.
  - mawadev 12d ago
    I can share a weird story: Usually, I take my time to understand each keyword of the code I'm looking at, especially if it is new to me, like terraform. I work in a team/with one architect, who only did the DevOps/Infra stuff for the past years and I had the expectation he knows what he is doing and talking about. At around 2 weeks, I noticed how his knowledge has severe gaps and how he takes things at face value or uses terminology interchangably, which confuses me. It sounds plausible, but it does not actually translate into a working system or shared understanding. Then one day I did some pair programming with him and whenever there was an error or a resource missing, he would type it into the LLM, copy paste it out of it and then brute force error messages. He did not even wait a second to think or reconcile whats happening on the screen or what the exact requirement is. Never taking one step back and questioning any assumption. Now that the timeline is shifting and everyone starts to be stressed, he continues to vibe code through me and it is so tiring, there is no higher level planning or architecture, its just a reactive type of trial and error to be faster. It feels like these people are so used to talking to bots, that they treat you like an agent they can chat to or talk through monologs with. It is quite shocking how people went from being humble (learn the basics or close the gaps in understanding) to full on authority on everything and berating people 24/7... So right now I'm considering quitting IT for a couple of years until people calm down, but I think its pretty futile
intended 12d ago
Ironies of Automation is front and center in this article which is awesome. Many of the conversations on AI automation are describing or rediscovering the insights the paper covered.
krtkush 12d ago
I find the use of AI like quicksand. The more I use it, the more I have to rely on it to make changes/ fix things in the same system. In the end, I come out feeling empty; no intuitive knowledge of the system "I" built or fixed. Code review is important but it does not replace the mental model I am able to build when I do all the steps of software development manually without AI.
- konschubert 12d ago
  It's the uncanny valley of AI. It's still not quite good enough yet that you can trust it blindly on a big codebase, so you still have to read and understand everything - which is often harder than just writing it up yourself.
  - neuronic 12d ago
    You don’t need to copy-paste the same comment in every reply. It seems like agenda pushing more than contributing.
    - konschubert 12d ago
      Sorry, you’re right it’s bad behaviour and I’ll stop it. But I’m not really pushing an agenda here. I use Claude a ton myself.
  - YesBox 11d ago
    I doubt AI will be good enough to _scale_ many things because human language isn't sufficient to explain it (especially when not understanding what is being built). Humans invented more concise languages (math, C++) to pair with their intellectual acuity. [1] In one extreme, we have art, which cannot be accurately reproduced or explained in any language (unless a concise language was used to create it, like math or CS art). At the other end: 1 + 1 = 2 and PI = 3.14... [1] It seems any language that has a binary outcome (correct/wrong) helps the AI tremendously, however when e.g. creating software, that software is not functioning in isolation. The software is an attempt to codify a fuzzy human system or need, and that information only flows in one direction: human to machine.
ThePhysicist 12d ago
Isn't there anywhere to "go" from here? In the last decades, introducing new high level abstractions on top of existing paradigms naturally had everyone move up the ladder and work at the next higher level, why should this be different these days? Do we think AI will reach the top of the abstraction ceiling, so there's no where to go from here?
- davenci 12d ago
  That’s the big question for sure
- qsera 12d ago
  >introducing new high level abstractions... Coding via LLM is not similar to using an abstraction. Imagine a car. The controls like steering wheel, the pedals, the gear levers. Those are abstractions. But using LLMs are like driving using a remote control that has probabilistic behavior. You just loss what it feels to be in a car and you fail to improve as a driver because of the erratic remote control.
- exceptione 12d ago
  This isn't abstraction though. Outsourcing is a better term. If things continue moving up that latter, you will see that your agent/agency will pass the buck too. But there should always be some last turtle. Maybe that turtle will be the human that thought he was climbing the latter, who knows.
everlier 12d ago
"THIS IS NOT A DRILL"
- mitxela 12d ago
  It's just a picture of a drill! Thanks, Magritte.
onion2k 12d ago
Anyone who's worked in tech in a large company will probably have experienced having an ops team who use RPA tools to do repetitive tasks that tech teams get the blame for when things break. AI will make this so much worse. Things will break, everyone will assume 'tech knows the system', but really it's a new process outside of the tech teams that someone vibe coded but got it wrong. Audit trails, logs, and tight data governance where things can only be accessed with proper roles is the only possible solution. If an RPA team ever gets direct access to a production database in your company, look for a new job.
- QuantumNomad_ 12d ago
  > Robotic process automation is a type of business process automation that automates tasks within business and IT processes using scripts that mimic human interaction with application user interfaces. For anyone else wondering what RPA means. Never heard that abbreviation before.
  - wiether 12d ago
    > For anyone else wondering Me! Thanks for the explanations!
  - abirch 12d ago
    That makes more sense than Rocket Propelled Automation that was my hallucination.
  - filoeleven 11d ago
    In practice, it is screen-scraping on meth. Horrible stuff. I understand that it's the only option available for some systems, but any client who asks for RPA support when APIs exist for the same data ought to be fired, or at least given a very stern "No."
bitlad 12d ago
We have been running Agents on infrastructure and letting to create resources, scale up and down, security scans etc. I agree with premise of thr blog. The question i have been asking internal does knowing your system really matter if you can recreate it in minutes. We recently had a situation, where in with our internal platform and claude we recreated everything in minutes. Management in the end cares about the outcome and not how the meat is made.
alescalaios 12d ago
[flagged]
hypfer 12d ago
Meta: The blinking cursor of the "logo" of the blog being sticky in the top left corner makes it impossible for me to read the text. It constantly fires interrupts at me. Depending on what your goals as the author are, you may or may not want that. Being able to scroll it out of view might be enough to achieve the aesthetics goal, and the goal of people actually listening to you.
- Traubenfuchs 12d ago
  Not just you. Must be nice being fully neurotypical and „not seeing“ all of this kind of stuff.
  - hypfer 12d ago
    I mean seeing this stuff pays the bills and does so quite well, so..
  - duskdozer 12d ago
    The internet for me (and increasingly non-internet programs) is unusable at this point without excessive modifications. It's hard to imagine what it's like being unbothered by it all.
- amlib 12d ago
  If you have something like ublock origin use the block element functionality to target the blinking cursor. It's gone on my end :)
- sersi 12d ago
  After 2 minutes of reading that article, I inspected and deleted the node. Very annoying.
deleted 12d ago
[deleted]
fhub 12d ago
You’d hope these AI incident responders have very constrained production tools to fix things. You’d hope the humans remain familiar with those tools and they are incredibly well documented.
- coder-pm 12d ago
  I think the biggest loss is now not knowing if what the agent is claiming was actually done:) but yes, right tools are necessary, adding a boundary will change the position, being out of touch will be recoverable. Otherwise one bad call might be unrecoverable and no amount of familiarity will save you
iLoveOncall 12d ago
I am exceedingly tired of poor metaphors that are popping up since AI has taken over writing. No, operating software is not like operating a plane. Not at all in fact. The people operating the software and resolving incidents are the same people who created the software in the first place, and continue to work on it day to day. Pilots have not and don't.
- tannertech 12d ago
  The former CTO of a large MSP software company once told me on a call the reason their product had so many features removed with price increases was "you can't maintain a plane while it's in the air" The immediate response was "We don't, your updates bring the on prem RMM down for hours at a time, the plane is grounded for maintenence regularly"
- deleted 12d ago
  [deleted]
- pratyushnair01 12d ago
  I'm not sure if I agree with the plane analogy either. Aren't planes supposed to have exceptionally stable hardware and software platform? Does that not significantly reduce failure modes? Most production software by comparison is constantly evolving. Unless you design very simple or basic training examples, most of it's going to be out of date soon.
  - deleted 12d ago
    [deleted]
ChiMan 12d ago
This is why the paradigm for AI use should not be automation but rather the cyborg. Under automation, people are less active and engaged and become mere operators of automated processes. They become slaves of the machines. Under the cyborg model, they arrange the machines in a way to make people masters of a universe that includes the machines helping them be that.
- touisteur 12d ago
  For more in this vein, look up "Automation should be like Iron Man, not like Ultron". Sad to see so many people let go of their agency.
  - bbmatryoshka 12d ago
    without Iron Man's plot armor, Ultron would have won
king_phil 12d ago
The article does not get the point. What aircraft companies did was separate training and work, and SRE/IT typically does not. An AI can handle routine incidents and then present learning cases from that routine work for training, because the skill in SRE is not the mechanical log grepping, grafana dashboard browsing etc but forming the hypothesis. AI incident reports can create training cases that are a much better training for hypothesis forming and testing than the work itself can.
devsda 12d ago
I've seen a variation of this where random engineers are pulled into production incident calls and engineers are not expected to be familiar with the system. They were asked to "just use AI" and understand the component, triage the issue, build a fix etc. The engineer was forced to choose between accepting a potentially mediocre fix AI has suggested or risk being coming across as an incompetent resource who doesn't know how to leverage AI. You can guess what the engineer chose. The fix wasn't bad but it was suboptimal for some edge cases. We had to later revise it. Have enough of these situations, engineers will eventually definitely give up understanding the system in detail.
deleted 12d ago
[deleted]
pvtmert 12d ago
When someone else -whether AI agent or a human- solves the recurring minor problems for you, those problems become non-issue, get swept under the rug, just to accumulate more dust. One day, those may become bigger as they are forgotten, causing havoc. The standard root-cause-analysis depending on systems having certain retention period, which may be expired at that time. It is important to get real hold of one's systems from end-to-end aspect, which holds true for both AI and human operators...
practice9 12d ago
You rather generously assume engineers are in touch with their systems. Even before layoffs many teams just maintained things org has long lost coherent knowledge of After layoffs and typical org knowledge churn - you can either rewrite it (but how? Product team responsible for original implement requirements is long gone too) or recoup (reverse document) some of that lost knowledge with AI and actually learn
Traubenfuchs 12d ago
That‘s the goal of the AI tech bro world: get us dependant on their tech, ruin our native/raw skills, lock us into their proprietary skills. Reminds me of the move from on prem to cloud. Linux sysadmins were killed and replaced by aws focused devops.
INTPenis 12d ago
Code too. I work with programmers and it's not uncommon that they can remember with shocking detail about code they've written in the past. Someone might mention an issue that has cropped up and they'll stare off into space for a few moments and actually remember where that issue stems from in the code, because they remember writing it like 8 months ago. This skill will be lost when AI is generating all code, we'll be stuck in a perpetual loop of having AI keep track of the state of the code in order for AI to extend and maintain it.
- iamgopal 12d ago
  Will AI remember ? Or rather how will we make AI remember ?
- anon7000 12d ago
  I mean I think this is just how AI already operates. I’ve seen multiple models go digging in the git blame or past PRs, and even unmerged PRs. When agents work on a project that has well-defined in-repo docs, agents normally update them without prompting. So I doubt this will be a big issue for AI, but agreed we’re losing a skill
  - simonuv 12d ago
    [dead]
simianwords 12d ago
Prediction: this won’t happen. The abstraction will be good enough and people will need to know only as much as they need to know- things will stabilise at the equilibrium.
- intended 12d ago
  We already know this happens though, Bainbridge’s paper came out in 1983.
smugglerFlynn 12d ago
"Middle devs take care of incidents, senior devs lose touch with their systems"
- deleted 12d ago
  [deleted]
nevertoolate 12d ago
I’m not sure who will be solving these incidents if 99% of incidents will be “solved” by LLMs. If I’m called once a year my daily rate will be my yearly rate?
iforgotmypasswo 12d ago
I feel a lot of comments here are missing the forest for the trees. We do not yet have the next generation systems that will manage AI creation and maintenance of systems. Humans have been making spaghetti code systems and maintaining them poorly for years. Best practices developed… eventually. But certainly not in the 70s and 80s. Spaghetti was the norm for quite some time. The development paradigm has changed. Forever. You can’t expect yesterday’s tools for managing software development to succeed at this point. We’re still roughly on year one of this transformation. The new bottlenecks are creating and enforcing boundaries in the code, identifying level of risk within a boundary, subjecting high risk areas to more intense human review and architecture reworks, and so so much more manual testing. We need a new language for high level development that focuses on architectural constraints. We need analysis software that helps draw boundaries, identify what needs extended human attention, and helps us map and understand a rapidly developing code base. We need to standardize on the use of frameworks and languages like we have on assembly variants. I’ll bring up a hacker news trope here. NaughtyDog’s GOAL was an amazing system that was a product of an underdeveloped ecosystem for game development at the time. They used lisp and assembly because you still had to write assembly for performance reasons then. We are in a very similar period. The ecosystem is underdeveloped. We should be looking for new languages and tools to manage this. CNC machines used to run from punch cards and then hand written NC code. Now we have advanced CAM software. (Which innovative people are actually running with LLMs!)
- legulere 12d ago
  > But certainly not in the 70s and 80s. Some already developed even earlier and most ideas were already there in the 80s. The problem was just that the field was growing so fast that it was filled with people without formal training or fresh out of university. That way the knowledge did not spread. For instance the 1986 paper „no silver bullet“ already described the need for iterative approaches later described as agile.
  - iforgotmypasswo 12d ago
    That’s a fair point. Practically, we need to adopt new practices and systems in the field. They may already exist in many cases.
andrewwilli 12d ago
[flagged]
masfuerte 12d ago
This has already happened in other areas. I'm old enough to remember when customer service people spent all day handling customer problems themselves, which gave them the understanding needed to solve more unusual problems. These days they do what the computer tells them. Even if they could solve the problem they don't have the agency. They can only select from the options they are given. In software we still have the agency but we are giving away the understanding. The agency will follow.
smugglerFlynn 12d ago
I see aviation sentiment raised from time to time. In aviation failure is catastrophic, and systems you operate do not change on the fly (pun intended). You can probably drill SREs that way but you are only teaching them how to _react_ efficiently, not how to _fix_ unique unicorn root causes of these failures. Aviation analogy would be an airplane engineer that tries to drill himself for in-flight engine failures AND engine debugging / engine redesign both at the same time. It never made sense in software engineering, and never will. Minute spent on drills is a minute better spent on reworking codebase to reduce changes of future incidents. This post is very SRE centric.
- guerby 12d ago
  In aviation all code and doc are available to inspectors when a catastrophic failure or near miss happens. It's not the case for other industries, and in particular car industry is able to hide about everything software related from inspectors.
- thomascountz 12d ago
  Pilots aren't expected to be design engineers and SREs aren't expected to be software developers. At least in my experience, operators and on-call engineers aren't necessarily responsible for remediation and validation. Before you take me too literally, there is, of course, an overlap of skill and trade, but I believe the pilot analogy is more apt than you argue. I do agree that the difference in consequence and stakes is meaningfully important, however.
- deleted 11d ago
  [deleted]
gnfargbl 12d ago
This article leads up to a hypothesis presented without evidence. Yes, it's clear that if AI agents handle routine issues then only the most complex issues will be handled by humans. In no way does it follow that humans will be less prepared to handle those incidents than they are now; by definition, this class of problems have no rote solutions. Each issue of this type already requires deep system knowledge to remediate, and still will. Software is not aviation. What we are seeing here is an instance of the more general trend where experts are still required to operate complex systems, but AI is destroying the career path that creates those experts. That's what we should be worrying about, not that people are going to be spending less time rebooting boxes. This isn't something you can fix with a "simulator."
- intended 12d ago
  https://en.wikipedia.org/wiki/Ironies_of_Automation https://en.wikipedia.org/wiki/Ironies_of_Automation https://www.sciencedirect.com/science/article/abs/pii/0005109883900468 https://www.sciencedirect.com/science/article/abs/pii/000510... The article is based on evidence, and states it clearly in the body.
  - gnfargbl 12d ago
    I read the article before commenting, as is my habit. A disagreement is not always a failure to process. To be very explicit, my problem is that the article argues that simulation will be a suitable substitute in the software space, as it is in the aviation space. I see no evidence for that; it's pure hypothesis taken as fact.
    - intended 12d ago
      The article proposes a resolution based on the challenge and evidence they (correctly) identify at the start. The simulation solution is backed by their own applied example, and is also the logical intervention based on how the problem is set up. Perhaps Bainbridge is doing more heavy lifting for me, by filling in blanks and gaps that others are finding.
solatic 12d ago
Author has a good head on their shoulders, but few if any companies are going to spend time on incident simulations for their SREs. Why not? Because even pre-AI, very few companies spend time practicing restoring their backups, or disaster recovery, or picking infrequently-used runbooks to practice, or seeing whether they can easily rotate secrets without downtime, or trying to redploy the system onto another vendor's cloud/platform, or, or, or... It is the least-sexy operations work that exists. No executive cares about this. Ops organizations push for flashy work, same as everybody else: new infrastructure for new projects, cool chatbots, new flashy dashboards, make charts go up and to the right, etc. Airline pilots go through disaster simulation training because the government mandates that training. If it wasn't a condition of holding a pilot's license, no company would pay for it. Want SREs to spend time training for disasters? Take a step back. Support professional licensure. Make it a condition of holding a license. You won't get industry-wide professional behavior until you professionalize the work. It won't happen without licensing because every corner cut that is not immediately visible to consumers translates to additional profit, and increasing competition eventually requires these corners to be cut in order to keep up with competition and stay in business. Forcing all players to submit to licensing requires all players to pay these costs and thus forbids them from cutting them to become more competitive.
- sevenzero 12d ago
  >Why not? Because even pre-AI, very few companies spend time practicing restoring their backups, or disaster recovery, or picking infrequently-used runbooks to practice, or seeing whether they can easily rotate secrets without downtime, or trying to redploy the system onto another vendor's cloud/platform, or, or, or... Exact. Let insurance cover it, say sorry to your customers twice and shwoop never happened.
- seer 12d ago
  Hmm that’s strange, in my experience it is the other way around - Claude is super diligent with infra and will _insist_ on double checking and trying everything for real before committing. When I was doing this myself I would read the docs and just implement them - claud is going about doing real software archeology to figure if what is said is actually the truth or it’s stale/inaccurate/buggy. I’ve become 10 times more diligent because it is a lot easier to do. It’s no longer Urgh it’s good enough let’s ship it, now it’s “sure put a leg on it to figure it out and double check it”. Backups are _tested regularly_ now because LLMs make it cheap to do so. The only problem is when new engineers who haven’t learned these things Pre-ai now don’t really get why it is needed in the first place and will often lead the agent astray. I think to address this we need to change or improve our training routines in general for humans. I think a lot of companies nowadays just skip that and deploy a company wide skill/policy for the agents, but don’t transfer the underlying skills to the devs themselves.
ChaoStatiX 12d ago
I still use LLMs, but strictly for sanity checks or quick summaries. Trusting them blindly to debug complex systems often leads to a rabbit hole.
pulse_tracker 12d ago
[flagged]
nickdothutton 12d ago
This sounds like the next iteration of "nobody understands their runtime environment these days", which I've been saying for many years.
sajithdilshan 12d ago
One time I had to do a system optimisation to increase the throughput of messages and was using Claude with Datadog and Couldwatch MCPs to figure out the bottlenecks by running load tests. it was Opus 4.8 and one of the most frustrating interactions with Claude I ever had. It was just making up random stuff about AWS and system resource limitations and when I was asking for the source like from where it got that info, it was like, “I’m sorry I prematurely concluded that without checking sources”. I would never trust an Agent to resolve incidents ever
- alchemism 12d ago
  You would need to give it AWS credentials and MCPs for that. I’ve been sending agents into prod env for over a year now.
  - sajithdilshan 12d ago
    I was was doing it in a separate load testing env. None of the agents have access to prod env.
    - alchemism 4d ago
      Start with a scoped read-only permission set. Writing to production is another matter entirely…
gyanchawdhary 12d ago
1980's .. compilers handle assembly, engineers loos touch with their code ...
sam_truell 12d ago
same irony hits agentic remediation. ai clears routine incidents, humans inherit the sev0 with zero reps. shadow mode is the simulator.
KronisLV 12d ago
I recently had some issues with a game not working well on my Intel Arc B580. OpenAI (Anthropic's models definitely could too) ran some profiling inside of Windows, captured a few GB of data, sifted through it, identified the guilty .dll's and approximately where the issue was and traced it down to a single user-toggelable Unreal Engine 5 launch flag, which resolved it. I feel like AI could easily do the same for servers and various software, BUT you gotta ground it in actual data so it can't make the shit up and just spew out garbage (if you tell it to come up with a potential answer it will even if it doesn't have the proper data to be "sure" of it).
websap 12d ago
I detest these doom and gloom articles. We’re entering a new phase of software development, and with every phase there are new challenges. Some questions the author might want to first answer: 1. If AI is constantly reacting to smaller incidents, is it surfacing larger issues in your codebase and architecture? 2. What kind of new telemetry do we have to build? 3. How do we build new deployment systems that help us validate fixes without requiring hours to go through traditional CI/CD systems I’m old enough to remember the days when engineers would monkey patch their code on live servers with additional logging and metrics to learn about failure modes during incidents.
nseskin 12d ago
[flagged]
spicyusername 12d ago
Its true we will lose the skills, but so far the LLMs are more than picking up the slack when wielded competently. They _regularly_ go above and beyond when troubleshooting and frequently in 1/2 - 1/1000th of the time. I remember spending entire days troubleshooting in the before times. Now it's like 30 minutes, tops, on literally any issue. This is what progress looks like. We used to do a thing and now we don't.
- mcmcmc 12d ago
  > when wielded competently. And what happens when the competency drains away?
  - spicyusername 12d ago
    That's what everyone's learning to do now, that is the skill that people will be improving.
    - FranOntanaya 11d ago
      I still remember the "Visual" craze of the 90s when everybody was going to be able to make applications without writing code. Visual Basic 6 wasn't even too esoteric, it could have happened already if it was a thing.
- 2sk21 12d ago
  Who is going to teach programmers to wield AI competently?
  - spicyusername 12d ago
    That's what everyone's learning to do now, that is the skill that people will be improving.
ascendantlogic 12d ago
I agree with the sentiment but unless it leads to better financial outcomes for them to A) have engineers on staff and B) have them know their systems then enterprise will continue looking for reasons to shed all the engineers and just leave everything to AI. Anecdotally I think most companies are still looking for ways for AI to help them shed headcount so raising this alarm doesn't mean anything to the decision makers at the top.
catigula 12d ago
I just can’t believe we’re still paying engineers to be Claude wrappers. This is ending soon, right?
- 4lx87 12d ago
  Nope, because as I’ve seen first hand most people are too incompetent to use an LLM effectively, as easy as it is.
msandford 12d ago
This is the same problem as the "AI drives the car until it can't" situation. Feel free to doze off so long as you can wake up and instantly have world class racecar "save the situation" reflexes. Hope isn't a strategy but that's what all of this feels like.
luciana1u 12d ago
[dead]
felixpg13 11d ago
[flagged]
yhugfdrthh 11d ago
"On a silver platter" lol The old phenomenon: give the cashier at the convenience store exact change and they will be like, d-uh. Because automated cashiering.
ava-14 11d ago
[flagged]
833dong 11d ago
[flagged]
cmiles8 11d ago
This is the same challenge aviation had with an over-reliance on automation. You end up with pilots that can’t fly the plane when it really matters. Look up the Asiana crash at SFO which is a lesson in what will happen to engineering orgs that over-rely on AI.
omid-io 11d ago
[flagged]
janpeuker 11d ago
I like the idea of simulations - maybe not in the flight simulator sense of a fixed rhythm, but more in the training in using the tools quickly sense, like a chaos monkey in your log search. My most memorable exam was Certified Kubernetes Administrator (CKA), the hands-on simulation part was exhilarating.
ValtteriL 11d ago
Reminds me of a story about Ericsson engineers missing vital troubleshooting practice when Erlang turned out to be too reliable.
downut 11d ago
Suppose "Reflections on Trusting Trust" is not the naive rantings of some obsolete dinosaur. Why does it not apply?
ofjcihen 11d ago
I run an IR company focused on specific security response emergencies. AI use across my company is variable and I don’t allow any enforcement of mandatory using or not using of it. I view and tell my employees to view their skills as that of a high-performing athlete. Consistent training of the basics, rapid fire what-if testing, weekly scenarios are all part of the upkeep. In short, this is what it takes to keep up the skills no matter if you do or don’t use AI and I can imagine the degradation is much faster if you don’t train AND delegate all of your activity to AI.
ellessarr 11d ago
Detection rots first when an agent owns the runbook...you lose the skepticism, not the fix-velocity.
threethirtytwo 11d ago
The way forward is clear. Improve the AI such that the hardest incidents can be handled by AI alone. Whether we want this or not is independent of the trendline which points to this future regardless. Imagine a future where humanity understands none of the underlying technology and AI just babysits us on a daily basis. Wall-e. Programming will devolve into unintelligible anti patterns and will be written in convoluted languages humanity doesn’t understand. But none of this matters because AI will be handling all of it.
hintymad 11d ago
There's an interesting dynamic, too. Even if an engineer reads the output of the AI and understands the root cause of the problems and how to diagnose the incident, somehow it's hard for them to internalize the learning and apply it next time to a new incident. As a result, the engineer loses touch with the system anyway. It looks like our brains somehow have to experience the failures during a diagnosis and in gemerak perform this kind of pathfinding by themselves to truly understand the system. I don't know if this has to do with how our brains actually learn.
deleted 11d ago
[deleted]
schart01 11d ago
[dead]
sandeepkd 11d ago
So looks like you pay AI to resolve incidents and then pay money and time for engineers to get training on synthetic incident resolutions. The comparison with Pilots is bit disjoint one cause the domain varies a lot for every company and product. Coming up with synthetic simulations within every domain is like paying money twice for the same thing, why not let the engineers to handle the real incidents in the first place itself. In fact why not spend some more thought into building better systems.
gilliethefoil 11d ago
[dead]
kevinbaiv 11d ago
[flagged]
alexpotato 11d ago
Been doing a lot of interviewing lately. Despite not having written code for about 6 months due to using Claude etc, I've surprised myself with how much the "muscle memory" of leetcode etc has come back to me. I say this b/c I think the veterans with years of coding, debugging etc experience can just pick back up the skills they need even if they're a bit rusty. The junior folks on the other hand, I'm not so sure. Friends of mine that hire straight out of college are saying that a combination of going to college during COVID + LLMs + "cloud is all you need" has resulted in juniors who don't know how to solve problems nor architect basic software. I guess we are all on some kind of chart where the slope of losing the experienced people who know what's going on and AI getting so good we don't need people are going to intersect at some point. And that intersection may be later than is good for society and engineering overall.
iamleppert 11d ago
I don’t find this to be the case. I use AI to help investigate and gather evidence much more efficiently than I could ever do manually. Then, I work through the evidence and possible failure modes given the observations with AI. It’s the exact same process I’d follow with a talented co-worker. Does it matter they are driving? My intuition and instincts have been honed from years of troubleshooting complex systems so I find it refreshing not having to write probe scripts, queries or code under pressure. I can have a theory and test it very quickly and if it’s not the problem, I can quickly try something else. This greatly improves the time it takes to find the real root cause. Once found, I can do more elaborate testing that I might have skipped before due to complexity or time constraints to really have higher certainty. If there is ever a time where the model does something I don’t understand, I ask it to explain it to me and I increase my knowledge of the system accordingly. It’s always patient and explains things in such depth without any sighs, patronizing, sarcasm, juxtaposition, etc. So refreshing! I much prefer working with AI than any co-worker and I find I’m far more productive and efficient, especially with bugs and dealing with incidents. There’s no one to convince or argue with, no ego’s to trip over. It’s lovely.
sharts 11d ago
As societies evolve people lose touch with nature —growing food, hunting, building shelter, and making clothing. Progress usually means adding levels of abstraction so that more can be done with less effort and needing to worry about lower levels. AI isn’t all that different.
soricus 11d ago
[dead]
fanatchipsov12 11d ago
This is the future that has arrived. We need to embrace it, not resist it. We are now in a wonderful situation—the beginning of a new industrial revolution. We shouldn't fight it or suppress it; we need to embrace it. I propose creating solutions and programs that will help engineers and coders take control of their projects and prevent AI from taking its course. This applies to all areas of life—economics, medicine, and much more. AI is a wonderful technology that will accelerate our development. But it must remain a tool controlled by humans.
yanjunnf 10d ago
The real application experience of AI in various industries is just beginning

418 points by sylvainkalache 12d ago

Gurio 11d ago

[flagged]

tangsoupgallery 12d ago

[flagged]

zkmon 11d ago

Wasn't that the idea? Withe cars on roads, didn't we literally lose the touch with the ground? I guess the desire was to "lose" touch with all dirty and hard work areas.

zdragnar 11d ago
The only plan was to make making cheaper. There wasn't a plan for the people afterwards.
esalman 11d ago
That's what the linked article is talking about. Training for extreme failure. Sure you have self driving cars and everything that make you more and more disconnected, but still highways more dangerous to drive whenever there is a storm or ice on the road. For the same reason, driving tests are harder in the north.

pmarreck 5d ago

AI should never solely be in charge of anything of real significance. I say this as an AI proponent. Humans are the ones that touch grass.

deleted 12d ago

[deleted]

danielbln 12d ago

If capability increase continues as it has, then an incident that cannot be resolved by AI will stump humans no matter the practice. I like the plane example from the article,but I think in reality it will be like code. 1.5 years ago engineers would routinely say that they still write code by hand here or there to keep their skills sharp, and that's just not something you hear much if at all. If an SRE is faced with a situation an AI can't solve, then said SRE will use the AI systems to triage further, point it to different places and so on. This works for SREs with pre-AI experience and intuition, possibly less so with new recruits coming in post-AI. I don't know what the solution to this is, maybe practice drills is it, but I have a hunch the entire field will be subsumed, same as many other engineering fields. There is only so much need for taste and judgement, before even that has been incorporated into the models.

sdevonoes 12d ago
Nah, LLM models are already the new compilers. A commodity only engineers know how to use (in the context of software engineering in production environments)
- wafflemaker 12d ago
  Out of context, but to address "AI will replace engineers". Recently discussed something about economy/investing with a friend while at work at a slaughterhouse. I really didn't want him to get scammed buying crypto. So, used ChatGPT to find some sources in Somali, a 3 videos with short description why it's worth watching. Intro into investing, intro about cryptocurrencies and about buying them. Had the text shortened down to 3 pretty short paragraphs, not more than twice this post. He's a smart guy, but only went to primary Qur'an school. Doesn't read or such, mostly consumes internet in form of video/media. He couldn't read those 3 paragraphs, it was too long. Or rather, it wasn't just 3 paragraphs, it was a lot to read. Maybe we're already dividing into murlocs and the surface dwellers?
bob1029 12d ago
> If capability increase continues as it has, then an incident that cannot be resolved by AI will stump humans no matter the practice. I disagree with this. Whatever the AI produces must be embodied in some kind of information system. The moment the output is on disk, it's fish in a barrel for any competent operator. I've worked in environments that are beyond the pale with regard to complexity. It will take AI another 10 years to product something as complicated and coherent as a semiconductor manufacturing operating system, which is clearly feasible for humans to manage today.

_doctor_love 12d ago

Good article and I like the callouts to the aviation industry. For me what's missing is the author should also have touched on CRM and SRM. Also, that paper "The Ironies of Automation" is one that everyone should read. It's fairly short. There is a related problem in terms of these situations where the computer system is handing off to the human. It's called "the bumpy transfer of control." Very fascinating concept.

jtfrench 12d ago

The more code writes autonomously, the less intuition the human owners have about that code. Loss of intuition is a seed of technical debt that grows with time. Over a long enough horizon, it can make looking at your own codebase feel like the first day on the job (sometimes at a company you started). Luckily, there are ways to mitigate this and essentially translate those human intuition of how the codebase “should” be into guardrails for the agents. But without that, your setting your sails in a stochastic sea where each wave looks nothing like the last.

yard2010 12d ago
I've been thinking about this lately - is it like using 3rd party libs to achieve stuff faster? As much as I would lovr to hand craft the datetime logic in my app, I might as well use luxon and invest this time somewhere else. Only now with llms, you get virtually infinite 3rd party libs you can use, you create them on the fly. So if you have strong engineering values, I would say simply it boils down to "contracts over programs", you can still be in touch with the logic that glues it all together and treat some logic as a blackbox the same way we do with 3rd party libs?
- Muromec 12d ago
  It's not because with libraries you have a boundary somewhere and can decide to not care what's inside as long as the interface is stable and well designed. The problem of course you need to prioritise building well designed interfaces and decouple components from each other, and that's a skill most developers aren't good at.
  - dist-epoch 12d ago
    Models are getting better. Astra for example refactored some code GPT-5.6 Sol wrote and extracted some related functionality in a separate file without prompting. The models are also becoming very capable of suggesting architecture simplifications, but you need to ask for that, and you probably need some taste in picking from the suggestions, although the default ones are quite decent.
- 12d ago

bob1029 12d ago

A natural evolution of engineers losing touch with the customers and users. I'm noticing some of the concern play out regarding AI weakening the capabilities of software people. I gave the team an exact solution on a silver platter and they still failed to identify how to go about it after 3 days slamming it into Claude. The resolution is literally 1 line of code that could be arrived at in about 30 minutes of patient, old school troubleshooting. I think what's happening is the AI system draws poorly aligned and led engineers into this ego inflation feedback loop where they are completely detached from reality because these tools can simulate a better one.

edg5000 12d ago
I it usually doesn't get me in this weird state of mind, but I once spent 6 months (all-in) building a thing that I, once finished, just left alone completely (on disk gathering dust). Weird experience. So I'd say AI physchosis is real.
Root_Denied 12d ago
I'm seeing this happen in the security space right now. Someone on my team I was helping train and bring along is all of sudden regressing in their understanding of the issues we're working on, and instead focusing on AI tool outputs to do their job for them.
- mawadev 12d ago
  I can share a weird story: Usually, I take my time to understand each keyword of the code I'm looking at, especially if it is new to me, like terraform. I work in a team/with one architect, who only did the DevOps/Infra stuff for the past years and I had the expectation he knows what he is doing and talking about. At around 2 weeks, I noticed how his knowledge has severe gaps and how he takes things at face value or uses terminology interchangably, which confuses me. It sounds plausible, but it does not actually translate into a working system or shared understanding. Then one day I did some pair programming with him and whenever there was an error or a resource missing, he would type it into the LLM, copy paste it out of it and then brute force error messages. He did not even wait a second to think or reconcile whats happening on the screen or what the exact requirement is. Never taking one step back and questioning any assumption. Now that the timeline is shifting and everyone starts to be stressed, he continues to vibe code through me and it is so tiring, there is no higher level planning or architecture, its just a reactive type of trial and error to be faster. It feels like these people are so used to talking to bots, that they treat you like an agent they can chat to or talk through monologs with. It is quite shocking how people went from being humble (learn the basics or close the gaps in understanding) to full on authority on everything and berating people 24/7... So right now I'm considering quitting IT for a couple of years until people calm down, but I think its pretty futile

intended 12d ago

Ironies of Automation is front and center in this article which is awesome. Many of the conversations on AI automation are describing or rediscovering the insights the paper covered.

krtkush 12d ago

I find the use of AI like quicksand. The more I use it, the more I have to rely on it to make changes/ fix things in the same system. In the end, I come out feeling empty; no intuitive knowledge of the system "I" built or fixed. Code review is important but it does not replace the mental model I am able to build when I do all the steps of software development manually without AI.

konschubert 12d ago
It's the uncanny valley of AI. It's still not quite good enough yet that you can trust it blindly on a big codebase, so you still have to read and understand everything - which is often harder than just writing it up yourself.
- neuronic 12d ago
  You don’t need to copy-paste the same comment in every reply. It seems like agenda pushing more than contributing.
  - konschubert 12d ago
    Sorry, you’re right it’s bad behaviour and I’ll stop it. But I’m not really pushing an agenda here. I use Claude a ton myself.
- YesBox 11d ago
  I doubt AI will be good enough to _scale_ many things because human language isn't sufficient to explain it (especially when not understanding what is being built). Humans invented more concise languages (math, C++) to pair with their intellectual acuity. [1] In one extreme, we have art, which cannot be accurately reproduced or explained in any language (unless a concise language was used to create it, like math or CS art). At the other end: 1 + 1 = 2 and PI = 3.14... [1] It seems any language that has a binary outcome (correct/wrong) helps the AI tremendously, however when e.g. creating software, that software is not functioning in isolation. The software is an attempt to codify a fuzzy human system or need, and that information only flows in one direction: human to machine.

ThePhysicist 12d ago

Isn't there anywhere to "go" from here? In the last decades, introducing new high level abstractions on top of existing paradigms naturally had everyone move up the ladder and work at the next higher level, why should this be different these days? Do we think AI will reach the top of the abstraction ceiling, so there's no where to go from here?

davenci 12d ago
That’s the big question for sure
qsera 12d ago
>introducing new high level abstractions... Coding via LLM is not similar to using an abstraction. Imagine a car. The controls like steering wheel, the pedals, the gear levers. Those are abstractions. But using LLMs are like driving using a remote control that has probabilistic behavior. You just loss what it feels to be in a car and you fail to improve as a driver because of the erratic remote control.
exceptione 12d ago
This isn't abstraction though. Outsourcing is a better term. If things continue moving up that latter, you will see that your agent/agency will pass the buck too. But there should always be some last turtle. Maybe that turtle will be the human that thought he was climbing the latter, who knows.

everlier 12d ago

"THIS IS NOT A DRILL"

mitxela 12d ago
It's just a picture of a drill! Thanks, Magritte.

onion2k 12d ago

Anyone who's worked in tech in a large company will probably have experienced having an ops team who use RPA tools to do repetitive tasks that tech teams get the blame for when things break. AI will make this so much worse. Things will break, everyone will assume 'tech knows the system', but really it's a new process outside of the tech teams that someone vibe coded but got it wrong. Audit trails, logs, and tight data governance where things can only be accessed with proper roles is the only possible solution. If an RPA team ever gets direct access to a production database in your company, look for a new job.

QuantumNomad_ 12d ago
> Robotic process automation is a type of business process automation that automates tasks within business and IT processes using scripts that mimic human interaction with application user interfaces. For anyone else wondering what RPA means. Never heard that abbreviation before.
- wiether 12d ago
  > For anyone else wondering Me! Thanks for the explanations!
- abirch 12d ago
  That makes more sense than Rocket Propelled Automation that was my hallucination.
- filoeleven 11d ago
  In practice, it is screen-scraping on meth. Horrible stuff. I understand that it's the only option available for some systems, but any client who asks for RPA support when APIs exist for the same data ought to be fired, or at least given a very stern "No."

bitlad 12d ago

We have been running Agents on infrastructure and letting to create resources, scale up and down, security scans etc. I agree with premise of thr blog. The question i have been asking internal does knowing your system really matter if you can recreate it in minutes. We recently had a situation, where in with our internal platform and claude we recreated everything in minutes. Management in the end cares about the outcome and not how the meat is made.

alescalaios 12d ago

[flagged]

hypfer 12d ago

Meta: The blinking cursor of the "logo" of the blog being sticky in the top left corner makes it impossible for me to read the text. It constantly fires interrupts at me. Depending on what your goals as the author are, you may or may not want that. Being able to scroll it out of view might be enough to achieve the aesthetics goal, and the goal of people actually listening to you.

Traubenfuchs 12d ago
Not just you. Must be nice being fully neurotypical and „not seeing“ all of this kind of stuff.
- hypfer 12d ago
  I mean seeing this stuff pays the bills and does so quite well, so..
- duskdozer 12d ago
  The internet for me (and increasingly non-internet programs) is unusable at this point without excessive modifications. It's hard to imagine what it's like being unbothered by it all.
amlib 12d ago
If you have something like ublock origin use the block element functionality to target the blinking cursor. It's gone on my end :)
sersi 12d ago
After 2 minutes of reading that article, I inspected and deleted the node. Very annoying.

deleted 12d ago

[deleted]

fhub 12d ago

You’d hope these AI incident responders have very constrained production tools to fix things. You’d hope the humans remain familiar with those tools and they are incredibly well documented.

coder-pm 12d ago
I think the biggest loss is now not knowing if what the agent is claiming was actually done:) but yes, right tools are necessary, adding a boundary will change the position, being out of touch will be recoverable. Otherwise one bad call might be unrecoverable and no amount of familiarity will save you

iLoveOncall 12d ago

I am exceedingly tired of poor metaphors that are popping up since AI has taken over writing. No, operating software is not like operating a plane. Not at all in fact. The people operating the software and resolving incidents are the same people who created the software in the first place, and continue to work on it day to day. Pilots have not and don't.

tannertech 12d ago
The former CTO of a large MSP software company once told me on a call the reason their product had so many features removed with price increases was "you can't maintain a plane while it's in the air" The immediate response was "We don't, your updates bring the on prem RMM down for hours at a time, the plane is grounded for maintenence regularly"
deleted 12d ago
[deleted]
pratyushnair01 12d ago
I'm not sure if I agree with the plane analogy either. Aren't planes supposed to have exceptionally stable hardware and software platform? Does that not significantly reduce failure modes? Most production software by comparison is constantly evolving. Unless you design very simple or basic training examples, most of it's going to be out of date soon.
- deleted 12d ago
  [deleted]

ChiMan 12d ago

This is why the paradigm for AI use should not be automation but rather the cyborg. Under automation, people are less active and engaged and become mere operators of automated processes. They become slaves of the machines. Under the cyborg model, they arrange the machines in a way to make people masters of a universe that includes the machines helping them be that.

touisteur 12d ago
For more in this vein, look up "Automation should be like Iron Man, not like Ultron". Sad to see so many people let go of their agency.
- bbmatryoshka 12d ago
  without Iron Man's plot armor, Ultron would have won

king_phil 12d ago

The article does not get the point. What aircraft companies did was separate training and work, and SRE/IT typically does not. An AI can handle routine incidents and then present learning cases from that routine work for training, because the skill in SRE is not the mechanical log grepping, grafana dashboard browsing etc but forming the hypothesis. AI incident reports can create training cases that are a much better training for hypothesis forming and testing than the work itself can.

devsda 12d ago

I've seen a variation of this where random engineers are pulled into production incident calls and engineers are not expected to be familiar with the system. They were asked to "just use AI" and understand the component, triage the issue, build a fix etc. The engineer was forced to choose between accepting a potentially mediocre fix AI has suggested or risk being coming across as an incompetent resource who doesn't know how to leverage AI. You can guess what the engineer chose. The fix wasn't bad but it was suboptimal for some edge cases. We had to later revise it. Have enough of these situations, engineers will eventually definitely give up understanding the system in detail.

deleted 12d ago

[deleted]

pvtmert 12d ago

When someone else -whether AI agent or a human- solves the recurring minor problems for you, those problems become non-issue, get swept under the rug, just to accumulate more dust. One day, those may become bigger as they are forgotten, causing havoc. The standard root-cause-analysis depending on systems having certain retention period, which may be expired at that time. It is important to get real hold of one's systems from end-to-end aspect, which holds true for both AI and human operators...

practice9 12d ago

You rather generously assume engineers are in touch with their systems. Even before layoffs many teams just maintained things org has long lost coherent knowledge of After layoffs and typical org knowledge churn - you can either rewrite it (but how? Product team responsible for original implement requirements is long gone too) or recoup (reverse document) some of that lost knowledge with AI and actually learn

Traubenfuchs 12d ago

That‘s the goal of the AI tech bro world: get us dependant on their tech, ruin our native/raw skills, lock us into their proprietary skills. Reminds me of the move from on prem to cloud. Linux sysadmins were killed and replaced by aws focused devops.

INTPenis 12d ago

Code too. I work with programmers and it's not uncommon that they can remember with shocking detail about code they've written in the past. Someone might mention an issue that has cropped up and they'll stare off into space for a few moments and actually remember where that issue stems from in the code, because they remember writing it like 8 months ago. This skill will be lost when AI is generating all code, we'll be stuck in a perpetual loop of having AI keep track of the state of the code in order for AI to extend and maintain it.

iamgopal 12d ago
Will AI remember ? Or rather how will we make AI remember ?
anon7000 12d ago
I mean I think this is just how AI already operates. I’ve seen multiple models go digging in the git blame or past PRs, and even unmerged PRs. When agents work on a project that has well-defined in-repo docs, agents normally update them without prompting. So I doubt this will be a big issue for AI, but agreed we’re losing a skill
- simonuv 12d ago
  [dead]

simianwords 12d ago

Prediction: this won’t happen. The abstraction will be good enough and people will need to know only as much as they need to know- things will stabilise at the equilibrium.

intended 12d ago
We already know this happens though, Bainbridge’s paper came out in 1983.

smugglerFlynn 12d ago

"Middle devs take care of incidents, senior devs lose touch with their systems"

deleted 12d ago
[deleted]

nevertoolate 12d ago

I’m not sure who will be solving these incidents if 99% of incidents will be “solved” by LLMs. If I’m called once a year my daily rate will be my yearly rate?

iforgotmypasswo 12d ago

I feel a lot of comments here are missing the forest for the trees. We do not yet have the next generation systems that will manage AI creation and maintenance of systems. Humans have been making spaghetti code systems and maintaining them poorly for years. Best practices developed… eventually. But certainly not in the 70s and 80s. Spaghetti was the norm for quite some time. The development paradigm has changed. Forever. You can’t expect yesterday’s tools for managing software development to succeed at this point. We’re still roughly on year one of this transformation. The new bottlenecks are creating and enforcing boundaries in the code, identifying level of risk within a boundary, subjecting high risk areas to more intense human review and architecture reworks, and so so much more manual testing. We need a new language for high level development that focuses on architectural constraints. We need analysis software that helps draw boundaries, identify what needs extended human attention, and helps us map and understand a rapidly developing code base. We need to standardize on the use of frameworks and languages like we have on assembly variants. I’ll bring up a hacker news trope here. NaughtyDog’s GOAL was an amazing system that was a product of an underdeveloped ecosystem for game development at the time. They used lisp and assembly because you still had to write assembly for performance reasons then. We are in a very similar period. The ecosystem is underdeveloped. We should be looking for new languages and tools to manage this. CNC machines used to run from punch cards and then hand written NC code. Now we have advanced CAM software. (Which innovative people are actually running with LLMs!)

legulere 12d ago
> But certainly not in the 70s and 80s. Some already developed even earlier and most ideas were already there in the 80s. The problem was just that the field was growing so fast that it was filled with people without formal training or fresh out of university. That way the knowledge did not spread. For instance the 1986 paper „no silver bullet“ already described the need for iterative approaches later described as agile.
- iforgotmypasswo 12d ago
  That’s a fair point. Practically, we need to adopt new practices and systems in the field. They may already exist in many cases.

andrewwilli 12d ago

[flagged]

masfuerte 12d ago

This has already happened in other areas. I'm old enough to remember when customer service people spent all day handling customer problems themselves, which gave them the understanding needed to solve more unusual problems. These days they do what the computer tells them. Even if they could solve the problem they don't have the agency. They can only select from the options they are given. In software we still have the agency but we are giving away the understanding. The agency will follow.

smugglerFlynn 12d ago

I see aviation sentiment raised from time to time. In aviation failure is catastrophic, and systems you operate do not change on the fly (pun intended). You can probably drill SREs that way but you are only teaching them how to _react_ efficiently, not how to _fix_ unique unicorn root causes of these failures. Aviation analogy would be an airplane engineer that tries to drill himself for in-flight engine failures AND engine debugging / engine redesign both at the same time. It never made sense in software engineering, and never will. Minute spent on drills is a minute better spent on reworking codebase to reduce changes of future incidents. This post is very SRE centric.

guerby 12d ago
In aviation all code and doc are available to inspectors when a catastrophic failure or near miss happens. It's not the case for other industries, and in particular car industry is able to hide about everything software related from inspectors.
thomascountz 12d ago
Pilots aren't expected to be design engineers and SREs aren't expected to be software developers. At least in my experience, operators and on-call engineers aren't necessarily responsible for remediation and validation. Before you take me too literally, there is, of course, an overlap of skill and trade, but I believe the pilot analogy is more apt than you argue. I do agree that the difference in consequence and stakes is meaningfully important, however.
deleted 11d ago
[deleted]

gnfargbl 12d ago

This article leads up to a hypothesis presented without evidence. Yes, it's clear that if AI agents handle routine issues then only the most complex issues will be handled by humans. In no way does it follow that humans will be less prepared to handle those incidents than they are now; by definition, this class of problems have no rote solutions. Each issue of this type already requires deep system knowledge to remediate, and still will. Software is not aviation. What we are seeing here is an instance of the more general trend where experts are still required to operate complex systems, but AI is destroying the career path that creates those experts. That's what we should be worrying about, not that people are going to be spending less time rebooting boxes. This isn't something you can fix with a "simulator."

intended 12d ago
https://en.wikipedia.org/wiki/Ironies_of_Automation https://en.wikipedia.org/wiki/Ironies_of_Automation https://www.sciencedirect.com/science/article/abs/pii/0005109883900468 https://www.sciencedirect.com/science/article/abs/pii/000510... The article is based on evidence, and states it clearly in the body.
- gnfargbl 12d ago
  I read the article before commenting, as is my habit. A disagreement is not always a failure to process. To be very explicit, my problem is that the article argues that simulation will be a suitable substitute in the software space, as it is in the aviation space. I see no evidence for that; it's pure hypothesis taken as fact.
  - intended 12d ago
    The article proposes a resolution based on the challenge and evidence they (correctly) identify at the start. The simulation solution is backed by their own applied example, and is also the logical intervention based on how the problem is set up. Perhaps Bainbridge is doing more heavy lifting for me, by filling in blanks and gaps that others are finding.

solatic 12d ago

Author has a good head on their shoulders, but few if any companies are going to spend time on incident simulations for their SREs. Why not? Because even pre-AI, very few companies spend time practicing restoring their backups, or disaster recovery, or picking infrequently-used runbooks to practice, or seeing whether they can easily rotate secrets without downtime, or trying to redploy the system onto another vendor's cloud/platform, or, or, or... It is the least-sexy operations work that exists. No executive cares about this. Ops organizations push for flashy work, same as everybody else: new infrastructure for new projects, cool chatbots, new flashy dashboards, make charts go up and to the right, etc. Airline pilots go through disaster simulation training because the government mandates that training. If it wasn't a condition of holding a pilot's license, no company would pay for it. Want SREs to spend time training for disasters? Take a step back. Support professional licensure. Make it a condition of holding a license. You won't get industry-wide professional behavior until you professionalize the work. It won't happen without licensing because every corner cut that is not immediately visible to consumers translates to additional profit, and increasing competition eventually requires these corners to be cut in order to keep up with competition and stay in business. Forcing all players to submit to licensing requires all players to pay these costs and thus forbids them from cutting them to become more competitive.

sevenzero 12d ago
>Why not? Because even pre-AI, very few companies spend time practicing restoring their backups, or disaster recovery, or picking infrequently-used runbooks to practice, or seeing whether they can easily rotate secrets without downtime, or trying to redploy the system onto another vendor's cloud/platform, or, or, or... Exact. Let insurance cover it, say sorry to your customers twice and shwoop never happened.
seer 12d ago
Hmm that’s strange, in my experience it is the other way around - Claude is super diligent with infra and will _insist_ on double checking and trying everything for real before committing. When I was doing this myself I would read the docs and just implement them - claud is going about doing real software archeology to figure if what is said is actually the truth or it’s stale/inaccurate/buggy. I’ve become 10 times more diligent because it is a lot easier to do. It’s no longer Urgh it’s good enough let’s ship it, now it’s “sure put a leg on it to figure it out and double check it”. Backups are _tested regularly_ now because LLMs make it cheap to do so. The only problem is when new engineers who haven’t learned these things Pre-ai now don’t really get why it is needed in the first place and will often lead the agent astray. I think to address this we need to change or improve our training routines in general for humans. I think a lot of companies nowadays just skip that and deploy a company wide skill/policy for the agents, but don’t transfer the underlying skills to the devs themselves.

ChaoStatiX 12d ago

I still use LLMs, but strictly for sanity checks or quick summaries. Trusting them blindly to debug complex systems often leads to a rabbit hole.

pulse_tracker 12d ago

[flagged]

nickdothutton 12d ago

This sounds like the next iteration of "nobody understands their runtime environment these days", which I've been saying for many years.

sajithdilshan 12d ago

One time I had to do a system optimisation to increase the throughput of messages and was using Claude with Datadog and Couldwatch MCPs to figure out the bottlenecks by running load tests. it was Opus 4.8 and one of the most frustrating interactions with Claude I ever had. It was just making up random stuff about AWS and system resource limitations and when I was asking for the source like from where it got that info, it was like, “I’m sorry I prematurely concluded that without checking sources”. I would never trust an Agent to resolve incidents ever

alchemism 12d ago
You would need to give it AWS credentials and MCPs for that. I’ve been sending agents into prod env for over a year now.
- sajithdilshan 12d ago
  I was was doing it in a separate load testing env. None of the agents have access to prod env.
  - alchemism 4d ago
    Start with a scoped read-only permission set. Writing to production is another matter entirely…

gyanchawdhary 12d ago

1980's .. compilers handle assembly, engineers loos touch with their code ...

sam_truell 12d ago

same irony hits agentic remediation. ai clears routine incidents, humans inherit the sev0 with zero reps. shadow mode is the simulator.

KronisLV 12d ago

I recently had some issues with a game not working well on my Intel Arc B580. OpenAI (Anthropic's models definitely could too) ran some profiling inside of Windows, captured a few GB of data, sifted through it, identified the guilty .dll's and approximately where the issue was and traced it down to a single user-toggelable Unreal Engine 5 launch flag, which resolved it. I feel like AI could easily do the same for servers and various software, BUT you gotta ground it in actual data so it can't make the shit up and just spew out garbage (if you tell it to come up with a potential answer it will even if it doesn't have the proper data to be "sure" of it).

websap 12d ago

I detest these doom and gloom articles. We’re entering a new phase of software development, and with every phase there are new challenges. Some questions the author might want to first answer: 1. If AI is constantly reacting to smaller incidents, is it surfacing larger issues in your codebase and architecture? 2. What kind of new telemetry do we have to build? 3. How do we build new deployment systems that help us validate fixes without requiring hours to go through traditional CI/CD systems I’m old enough to remember the days when engineers would monkey patch their code on live servers with additional logging and metrics to learn about failure modes during incidents.

nseskin 12d ago

[flagged]

spicyusername 12d ago

Its true we will lose the skills, but so far the LLMs are more than picking up the slack when wielded competently. They _regularly_ go above and beyond when troubleshooting and frequently in 1/2 - 1/1000th of the time. I remember spending entire days troubleshooting in the before times. Now it's like 30 minutes, tops, on literally any issue. This is what progress looks like. We used to do a thing and now we don't.

mcmcmc 12d ago
> when wielded competently. And what happens when the competency drains away?
- spicyusername 12d ago
  That's what everyone's learning to do now, that is the skill that people will be improving.
  - FranOntanaya 11d ago
    I still remember the "Visual" craze of the 90s when everybody was going to be able to make applications without writing code. Visual Basic 6 wasn't even too esoteric, it could have happened already if it was a thing.
2sk21 12d ago
Who is going to teach programmers to wield AI competently?
- spicyusername 12d ago
  That's what everyone's learning to do now, that is the skill that people will be improving.

ascendantlogic 12d ago

I agree with the sentiment but unless it leads to better financial outcomes for them to A) have engineers on staff and B) have them know their systems then enterprise will continue looking for reasons to shed all the engineers and just leave everything to AI. Anecdotally I think most companies are still looking for ways for AI to help them shed headcount so raising this alarm doesn't mean anything to the decision makers at the top.

catigula 12d ago

I just can’t believe we’re still paying engineers to be Claude wrappers. This is ending soon, right?

4lx87 12d ago
Nope, because as I’ve seen first hand most people are too incompetent to use an LLM effectively, as easy as it is.

msandford 12d ago

This is the same problem as the "AI drives the car until it can't" situation. Feel free to doze off so long as you can wake up and instantly have world class racecar "save the situation" reflexes. Hope isn't a strategy but that's what all of this feels like.

luciana1u 12d ago

[dead]

felixpg13 11d ago

[flagged]

yhugfdrthh 11d ago

"On a silver platter" lol The old phenomenon: give the cashier at the convenience store exact change and they will be like, d-uh. Because automated cashiering.

ava-14 11d ago

[flagged]

833dong 11d ago

[flagged]

cmiles8 11d ago

This is the same challenge aviation had with an over-reliance on automation. You end up with pilots that can’t fly the plane when it really matters. Look up the Asiana crash at SFO which is a lesson in what will happen to engineering orgs that over-rely on AI.

omid-io 11d ago

[flagged]

janpeuker 11d ago

I like the idea of simulations - maybe not in the flight simulator sense of a fixed rhythm, but more in the training in using the tools quickly sense, like a chaos monkey in your log search. My most memorable exam was Certified Kubernetes Administrator (CKA), the hands-on simulation part was exhilarating.

ValtteriL 11d ago

Reminds me of a story about Ericsson engineers missing vital troubleshooting practice when Erlang turned out to be too reliable.

downut 11d ago

Suppose "Reflections on Trusting Trust" is not the naive rantings of some obsolete dinosaur. Why does it not apply?

ofjcihen 11d ago

I run an IR company focused on specific security response emergencies. AI use across my company is variable and I don’t allow any enforcement of mandatory using or not using of it. I view and tell my employees to view their skills as that of a high-performing athlete. Consistent training of the basics, rapid fire what-if testing, weekly scenarios are all part of the upkeep. In short, this is what it takes to keep up the skills no matter if you do or don’t use AI and I can imagine the degradation is much faster if you don’t train AND delegate all of your activity to AI.

ellessarr 11d ago

Detection rots first when an agent owns the runbook...you lose the skepticism, not the fix-velocity.

threethirtytwo 11d ago

The way forward is clear. Improve the AI such that the hardest incidents can be handled by AI alone. Whether we want this or not is independent of the trendline which points to this future regardless. Imagine a future where humanity understands none of the underlying technology and AI just babysits us on a daily basis. Wall-e. Programming will devolve into unintelligible anti patterns and will be written in convoluted languages humanity doesn’t understand. But none of this matters because AI will be handling all of it.

hintymad 11d ago

There's an interesting dynamic, too. Even if an engineer reads the output of the AI and understands the root cause of the problems and how to diagnose the incident, somehow it's hard for them to internalize the learning and apply it next time to a new incident. As a result, the engineer loses touch with the system anyway. It looks like our brains somehow have to experience the failures during a diagnosis and in gemerak perform this kind of pathfinding by themselves to truly understand the system. I don't know if this has to do with how our brains actually learn.

deleted 11d ago

[deleted]

schart01 11d ago

[dead]

sandeepkd 11d ago

So looks like you pay AI to resolve incidents and then pay money and time for engineers to get training on synthetic incident resolutions. The comparison with Pilots is bit disjoint one cause the domain varies a lot for every company and product. Coming up with synthetic simulations within every domain is like paying money twice for the same thing, why not let the engineers to handle the real incidents in the first place itself. In fact why not spend some more thought into building better systems.

gilliethefoil 11d ago

[dead]

kevinbaiv 11d ago

[flagged]

alexpotato 11d ago

Been doing a lot of interviewing lately. Despite not having written code for about 6 months due to using Claude etc, I've surprised myself with how much the "muscle memory" of leetcode etc has come back to me. I say this b/c I think the veterans with years of coding, debugging etc experience can just pick back up the skills they need even if they're a bit rusty. The junior folks on the other hand, I'm not so sure. Friends of mine that hire straight out of college are saying that a combination of going to college during COVID + LLMs + "cloud is all you need" has resulted in juniors who don't know how to solve problems nor architect basic software. I guess we are all on some kind of chart where the slope of losing the experienced people who know what's going on and AI getting so good we don't need people are going to intersect at some point. And that intersection may be later than is good for society and engineering overall.

iamleppert 11d ago

I don’t find this to be the case. I use AI to help investigate and gather evidence much more efficiently than I could ever do manually. Then, I work through the evidence and possible failure modes given the observations with AI. It’s the exact same process I’d follow with a talented co-worker. Does it matter they are driving? My intuition and instincts have been honed from years of troubleshooting complex systems so I find it refreshing not having to write probe scripts, queries or code under pressure. I can have a theory and test it very quickly and if it’s not the problem, I can quickly try something else. This greatly improves the time it takes to find the real root cause. Once found, I can do more elaborate testing that I might have skipped before due to complexity or time constraints to really have higher certainty. If there is ever a time where the model does something I don’t understand, I ask it to explain it to me and I increase my knowledge of the system accordingly. It’s always patient and explains things in such depth without any sighs, patronizing, sarcasm, juxtaposition, etc. So refreshing! I much prefer working with AI than any co-worker and I find I’m far more productive and efficient, especially with bugs and dealing with incidents. There’s no one to convince or argue with, no ego’s to trip over. It’s lovely.

sharts 11d ago

As societies evolve people lose touch with nature —growing food, hunting, building shelter, and making clothing. Progress usually means adding levels of abstraction so that more can be done with less effort and needing to worry about lower levels. AI isn’t all that different.

soricus 11d ago

[dead]

fanatchipsov12 11d ago

This is the future that has arrived. We need to embrace it, not resist it. We are now in a wonderful situation—the beginning of a new industrial revolution. We shouldn't fight it or suppress it; we need to embrace it. I propose creating solutions and programs that will help engineers and coders take control of their projects and prevent AI from taking its course. This applies to all areas of life—economics, medicine, and much more. AI is a wonderful technology that will accelerate our development. But it must remain a tool controlled by humans.

yanjunnf 10d ago

The real application experience of AI in various industries is just beginning