10 ms·

AI and Infrastructure Engineering

72 points by 0megion 24d ago

ilovefanta 24d ago
this post is so relatable, especially the locals where tags are four levels deep lol
lowsong 24d ago
This is an extremely common misunderstanding about AI in software. In order for AI to "move up one layer of abstraction" it has to actually be an abstraction. But AI output will _always_ be flawed in ways that mean you still need to fully understand the layer below it.
cassianoleal 24d ago
> Four years ago I hand-wrote a nested for loop - four levels deep, tagging subnets across regions and availability zones in another AWS account - and it took me about an hour to get the syntax right I do believe this kind of engineer requires AI. I also hope I don't have to work in the same teams as them. Seasoning teaches you to make changes in other parts of the codebase rather than do this kind of data mangling on local variables, which is incredibly hard to troubleshoot and very brittle - not to mention it's a nightmare to read back and understand.
- bigstrat2003 24d ago
  Another shameful quote from the article: > Kubernetes made server management easy enough that we stopped building our own node images at all - we just use whatever the cloud provider hands us, an AWS AMI built for us, no questions asked. This dude is just bad at his job.
  - stackskipton 24d ago
    SRE here, why would you create your own node images for Kubernetes Nodes? Unless they deliver some weird kernel that would cause containers not to run, there is no reason not to take cloud provider image. If you need something extra on the nodes, 99% of the time, DaemonSets will do the job.
    - tomrod 24d ago
      In many spaces, higher security requirements.
- arjie 24d ago
  It is Terraform’s configuration language. Locals mean something specific : non-exported variables. Constructing the map or whatever prior to plan phase is pretty idiomatic. Note the key in the map is a string concatenation. Not something you would do elsewhere but a constraint of that framework.
  - cassianoleal 24d ago
    I've been using Terraform since it came out. I know what it is.
    - SOLAR_FIELDS 24d ago
      Yes, your post is more an indictment of Terraform itself, rather than the grandparent's usage of it. I've had pretty good luck with Crossplane, it has its warts, but it removes the lack of obviousness that accompanies Terraform. Unfortunately, reconciling state against infrastructure configuration seems to be a pretty tough problem to get right so no solution is amazing. But being as declarative as possible upfront and not obfuscating it behind a build time execution phase seems to help AI reason about it.
      - cassianoleal 24d ago
        I've looked into Crossplane, and I like a lot of things about it. Unfortunately when we spiked it, the providers we required were a major let down. In terms of these horrendous HCL constructs, I'm not going to pretend I haven't written them myself ever, but every single time it's self-inflicted and can be solved by redesigning the data types. As with any other tool and language, if you design your codebase for its weaknesses, you'll have a much better time.
        SOLAR_FIELDS 23d ago
        Nowadays all providers are, or can be somewhat easily, just use Upjet to convert a Terraform provider. They walked away from the custom provider maintenance model awhile back due to its lack of scaling potential and loss of network effects from Terraform being the de-jure framework for infrastructure API's I will say that yes, quite a few of the providers are half assed and have bugs. Mostly around how reconciliation works, but also gaps in the provider in how it writes fields due to some Terraform idiosyncracy. I often end up having to ignore various fields in the yaml to prevent drift. But now you can wrap your own without too much effort. HCL itself definitely footguns you into it, it's perverse, because it's intended to be a mere configuration DSL and yet it has since its inception been square pegged into a round hole to do imperative language construct things that it was never designed for. Which is why you get insane practices like "never use an array to iterate over infrastructure in a loop, because removing an element forces a recreation of every resource after that loop because the position of the element in the array shifted to the left". These sort things are completely logical things you would do in a not insane framework (who doesn't use an array to loop over things in a normal software language?), but are a well-known landmine with HCL and Terraform. You could say it was a mistake to use a language like HCL for Terraform and I would agree. It's a half baked idea that became production way too fast. Reminds me of Javascript.
- chanux 24d ago
  I would not be quick to judge the person without knowing - the kind of work environment they were in - the quality of the existing code base
anon7000 24d ago
Matches my experience working with LLMs on Kubernetes. They are amazing at writing the boilerplate yaml, and fantastic at using kubectl to monitor and debug clusters (with guardrails and readonly permissions in place of course.) if you have a staging/test cluster, you can get an extremely fast feedback loop if you let it apply changes as well. It’s a good example of how highly structured, well- documented, opinionated frameworks help LLMs do well.
anon7000 24d ago
Matches my experience working with LLMs on Kubernetes. They are amazing at writing the boilerplate yaml, and fantastic at using kubectl to monitor and debug clusters (with guardrails and readonly permissions in place of course.) if you have a staging/test cluster, you can get an extremely fast feedback loop if you let it apply changes as well. It’s a good example of how highly structured, well- documented, opinionated frameworks help LLMs do well. There’s just less room for subjective issues when there’s more structure around it.
thelastgallon 24d ago
This works because he has decades of working experience at the foundational layers and understands each layer of abstraction, what it solves and the interaction between the abstractions. Can still reason it out, if not the fastest at troubleshooting lowest layers because its no longer in muscle memory. AI can make you productive if you have the necessary experience (which takes a long time). But, AI natives have none of the knowledge of ANY layers, even at the highest layer at which they will be using AI. They only learn how to do prompts, etc and nothing else about what is being done. This is just like anyone can be a pilot because planes mostly fly themselves.
- deleted 24d ago
  [deleted]
skinfaxi 24d ago
> Did Kubernetes kill Ansible? Kind of. I haven’t written an Ansible playbook in years - if you handed me one right now I’d be squinting at the module syntax like I’d never seen it before - not because configuration management stopped mattering, but because Kubernetes made server management easy enough that we stopped building our own node images at all - we just use whatever the cloud provider hands us, an AWS AMI built for us, no questions asked. And when did I last SSH into a node to debug something? K8s did not kill Ansible because of ease. Ansible is obsolete because we moved from creating a machine and changing it to instantiating it with the changes we wanted. Ansible makes sense when you are continuously updating a server-in-place and is decidedly less useful when you spin up a new unit with the new configuration when changes are made. Even in this instance the underlying skills remain the same.
- antonvs 24d ago
  For a lot of people in practice, Kubernetes was what replaced use of Ansible (or Chef etc.), so the OP take makes sense. Also, the transition you describe isn’t the whole story. Containers meant that the node configuration essentially stopped mattering - the node contains no application-specific code or dependencies. This shift wasn’t due to Kubernetes alone, but Kubernetes became the standard way to manage such systems.
deleted 24d ago
[deleted]
Ozzie-D 24d ago
[flagged]
markbnj 24d ago
This article basically captures my current professional life to a high degree of accuracy. I still enjoy the work. I still get the satisfaction of conceiving a thing and seeing it realized. That has always been at the root of the pleasure that I get from this job. Like the author I find myself drifting away from the underlying layers that the agents has so thoroughly mastered, and like the author I wonder about its eventual mastery of the parts I still consider my domain.
stego-tech 24d ago
I’ve been in engineering long enough to know it’s cyclical: the new thing always eats layers that demanded expertise until it demands expertise, at which point something new tries eating that layer. It was shell scripting until it became Ansible, until it became HCL, until it became K8s, until Helm, until AGENTS.md, until…well, until we wound up back at building images again for local compute because cloud costs are spiraling upward and privacy/sovereignty are the focus again. It’s kind of ironic that, for all the expertise I’ve cultivated, all the skills I’ve grown, all the knowledge I’ve gained, I’ve found myself back at the start of my career cycle again in terms of manually defining and building OS images for virtualization on local hardware. AI has not been much help in this regard because of the hundreds of OS settings I have to understand and set according to security posture - I’ve been unable to get Claude/Qwen/etc to build an effective hardening profile for something as fundamental as CIS2, because it always presumes an AD environment on a CSP as opposed to a local VM. Believe me, I would love for it to do so, because this is dry, uninteresting work, but I’ve spent more time wrangling AI than I would’ve spent writing my own Powershell. AI is, at least in my realm, a neat tool for basic, repetitive, boilerplate work. Very much a sudo make me a sandwich type of deal, except for YAML. That’s not me knocking the tool, just my two cents on what function it serves at present. Put another way: we wouldn’t have made our sixth hire a Support staffer if AI could do more.
- funlang 24d ago
  [flagged]
frio 24d ago
> The honest tradeoff > That’s not a hypothetical cost - it’s > But that’s exactly the These are strong tells for an AI authored article. I know we're all tired of people saying "this is AI" as it's harmful to discussing the content of an article, but articles about AI, (seemingly!) written by AI, extolling the virtues of AI -- don't really seem to drive the discussion forward to me either.
- captainkrtek 24d ago
  It's frustrating to find AI so prevalent in writing. It takes some nugget of a human written idea, dilutes it and increases word count, only to make the actual core idea less obvious (or ultimately summarized by someone using AI).
  - ShinyLeftPad 24d ago
    Poor hyperscalers have got to make money somehow...
- danielvaughn 24d ago
  The rest of the blog post doesn't strike me as AI generated. Hard to tell, and personally I think "hard to tell" is about as good as it's going to get from here on out, so I give the author the benefit of my doubt. Agree that using AI for writing is a habit we collectively need to drop immediately.
- 0megion 24d ago
  Maybe I should have added this part. As a non-native English speaker, using AI to write RFCs, ADRs, and internal documentation, my own writing style shifted so much to the direction that my writing seems like AI.
  - frio 24d ago
    In that case, sorry for getting it wrong and throwing shade.
  - zsoltkacsandi 24d ago
purpleidea 24d ago
For most users, Kubernetes has added complexity and cost, but not value. The sooner we realize this, the sooner we start thinking about how things should be built. Whether or not you have LLMs has nothing to do with that. The LLM will happily help build your crap even faster! Maybe people will finally realize they need to start rethinking infrastructure automation. I am.
- Melatonic 24d ago
  MicroVM !
- antonvs 24d ago
  > For most users, Kubernetes has added complexity and cost, but not value. This may be true for small, simple systems, such as basic web apps, that don’t really need Kubernetes. If your system can fit on a single machine, or even two or three, why would you use a cluster orchestrator to run it? But if you’re at a real company - not a mom and pop shop or similar - and you have multiple services implemented by multiple teams, you’d be hard-pressed today to find something that makes more sense than Kubernetes. Yes, there are PaaS offerings that can hide some of the inherent complexity, but they tend to run into limits, cost issues, etc. quite quickly. Not to mention they’re often implemented on top of Kubernetes anyway. Kubernetes is worth learning if you want to understand the needs and design of non-trivial software systems. Everything it offers corresponds to requirements such systems have, and it provides an extremely well organized, unified way of managing that. It’s one of the most well-designed software products of the 21st century.
  - Melatonic 24d ago
    I think the issue is that there is a not yet emerged standard for the mid size companies that we hear less about here on HN but make up a huge amount of business in the US. Kubernetes may be the gold standard for larger companies or tech heavy companies of all sizes. But what about the mid size organisations ?
    - antonvs 23d ago
deleted 24d ago
[deleted]
Lawsonfletcher 24d ago
[flagged]

72 points by 0megion 24d ago

this post is so relatable, especially the locals where tags are four levels deep lol

This is an extremely common misunderstanding about AI in software. In order for AI to "move up one layer of abstraction" it has to actually be an abstraction. But AI output will _always_ be flawed in ways that mean you still need to fully understand the layer below it.

cassianoleal 24d ago

> Four years ago I hand-wrote a nested for loop - four levels deep, tagging subnets across regions and availability zones in another AWS account - and it took me about an hour to get the syntax right I do believe this kind of engineer requires AI. I also hope I don't have to work in the same teams as them. Seasoning teaches you to make changes in other parts of the codebase rather than do this kind of data mangling on local variables, which is incredibly hard to troubleshoot and very brittle - not to mention it's a nightmare to read back and understand.

bigstrat2003 24d ago
Another shameful quote from the article: > Kubernetes made server management easy enough that we stopped building our own node images at all - we just use whatever the cloud provider hands us, an AWS AMI built for us, no questions asked. This dude is just bad at his job.
- stackskipton 24d ago
  SRE here, why would you create your own node images for Kubernetes Nodes? Unless they deliver some weird kernel that would cause containers not to run, there is no reason not to take cloud provider image. If you need something extra on the nodes, 99% of the time, DaemonSets will do the job.
  - tomrod 24d ago
    In many spaces, higher security requirements.
arjie 24d ago
It is Terraform’s configuration language. Locals mean something specific : non-exported variables. Constructing the map or whatever prior to plan phase is pretty idiomatic. Note the key in the map is a string concatenation. Not something you would do elsewhere but a constraint of that framework.
- cassianoleal 24d ago
  I've been using Terraform since it came out. I know what it is.
  - SOLAR_FIELDS 24d ago
    Yes, your post is more an indictment of Terraform itself, rather than the grandparent's usage of it. I've had pretty good luck with Crossplane, it has its warts, but it removes the lack of obviousness that accompanies Terraform. Unfortunately, reconciling state against infrastructure configuration seems to be a pretty tough problem to get right so no solution is amazing. But being as declarative as possible upfront and not obfuscating it behind a build time execution phase seems to help AI reason about it.
    - cassianoleal 24d ago
      I've looked into Crossplane, and I like a lot of things about it. Unfortunately when we spiked it, the providers we required were a major let down. In terms of these horrendous HCL constructs, I'm not going to pretend I haven't written them myself ever, but every single time it's self-inflicted and can be solved by redesigning the data types. As with any other tool and language, if you design your codebase for its weaknesses, you'll have a much better time.
      - SOLAR_FIELDS 23d ago
        Nowadays all providers are, or can be somewhat easily, just use Upjet to convert a Terraform provider. They walked away from the custom provider maintenance model awhile back due to its lack of scaling potential and loss of network effects from Terraform being the de-jure framework for infrastructure API's I will say that yes, quite a few of the providers are half assed and have bugs. Mostly around how reconciliation works, but also gaps in the provider in how it writes fields due to some Terraform idiosyncracy. I often end up having to ignore various fields in the yaml to prevent drift. But now you can wrap your own without too much effort. HCL itself definitely footguns you into it, it's perverse, because it's intended to be a mere configuration DSL and yet it has since its inception been square pegged into a round hole to do imperative language construct things that it was never designed for. Which is why you get insane practices like "never use an array to iterate over infrastructure in a loop, because removing an element forces a recreation of every resource after that loop because the position of the element in the array shifted to the left". These sort things are completely logical things you would do in a not insane framework (who doesn't use an array to loop over things in a normal software language?), but are a well-known landmine with HCL and Terraform. You could say it was a mistake to use a language like HCL for Terraform and I would agree. It's a half baked idea that became production way too fast. Reminds me of Javascript.
chanux 24d ago
I would not be quick to judge the person without knowing - the kind of work environment they were in - the quality of the existing code base

anon7000 24d ago

Matches my experience working with LLMs on Kubernetes. They are amazing at writing the boilerplate yaml, and fantastic at using kubectl to monitor and debug clusters (with guardrails and readonly permissions in place of course.) if you have a staging/test cluster, you can get an extremely fast feedback loop if you let it apply changes as well. It’s a good example of how highly structured, well- documented, opinionated frameworks help LLMs do well.

anon7000 24d ago

thelastgallon 24d ago

This works because he has decades of working experience at the foundational layers and understands each layer of abstraction, what it solves and the interaction between the abstractions. Can still reason it out, if not the fastest at troubleshooting lowest layers because its no longer in muscle memory. AI can make you productive if you have the necessary experience (which takes a long time). But, AI natives have none of the knowledge of ANY layers, even at the highest layer at which they will be using AI. They only learn how to do prompts, etc and nothing else about what is being done. This is just like anyone can be a pilot because planes mostly fly themselves.

deleted 24d ago
[deleted]

skinfaxi 24d ago

> Did Kubernetes kill Ansible? Kind of. I haven’t written an Ansible playbook in years - if you handed me one right now I’d be squinting at the module syntax like I’d never seen it before - not because configuration management stopped mattering, but because Kubernetes made server management easy enough that we stopped building our own node images at all - we just use whatever the cloud provider hands us, an AWS AMI built for us, no questions asked. And when did I last SSH into a node to debug something? K8s did not kill Ansible because of ease. Ansible is obsolete because we moved from creating a machine and changing it to instantiating it with the changes we wanted. Ansible makes sense when you are continuously updating a server-in-place and is decidedly less useful when you spin up a new unit with the new configuration when changes are made. Even in this instance the underlying skills remain the same.

antonvs 24d ago
For a lot of people in practice, Kubernetes was what replaced use of Ansible (or Chef etc.), so the OP take makes sense. Also, the transition you describe isn’t the whole story. Containers meant that the node configuration essentially stopped mattering - the node contains no application-specific code or dependencies. This shift wasn’t due to Kubernetes alone, but Kubernetes became the standard way to manage such systems.

deleted 24d ago

[deleted]

Ozzie-D 24d ago

[flagged]

markbnj 24d ago

This article basically captures my current professional life to a high degree of accuracy. I still enjoy the work. I still get the satisfaction of conceiving a thing and seeing it realized. That has always been at the root of the pleasure that I get from this job. Like the author I find myself drifting away from the underlying layers that the agents has so thoroughly mastered, and like the author I wonder about its eventual mastery of the parts I still consider my domain.

stego-tech 24d ago

I’ve been in engineering long enough to know it’s cyclical: the new thing always eats layers that demanded expertise until it demands expertise, at which point something new tries eating that layer. It was shell scripting until it became Ansible, until it became HCL, until it became K8s, until Helm, until AGENTS.md, until…well, until we wound up back at building images again for local compute because cloud costs are spiraling upward and privacy/sovereignty are the focus again. It’s kind of ironic that, for all the expertise I’ve cultivated, all the skills I’ve grown, all the knowledge I’ve gained, I’ve found myself back at the start of my career cycle again in terms of manually defining and building OS images for virtualization on local hardware. AI has not been much help in this regard because of the hundreds of OS settings I have to understand and set according to security posture - I’ve been unable to get Claude/Qwen/etc to build an effective hardening profile for something as fundamental as CIS2, because it always presumes an AD environment on a CSP as opposed to a local VM. Believe me, I would love for it to do so, because this is dry, uninteresting work, but I’ve spent more time wrangling AI than I would’ve spent writing my own Powershell. AI is, at least in my realm, a neat tool for basic, repetitive, boilerplate work. Very much a sudo make me a sandwich type of deal, except for YAML. That’s not me knocking the tool, just my two cents on what function it serves at present. Put another way: we wouldn’t have made our sixth hire a Support staffer if AI could do more.

funlang 24d ago
[flagged]

frio 24d ago

> The honest tradeoff > That’s not a hypothetical cost - it’s > But that’s exactly the These are strong tells for an AI authored article. I know we're all tired of people saying "this is AI" as it's harmful to discussing the content of an article, but articles about AI, (seemingly!) written by AI, extolling the virtues of AI -- don't really seem to drive the discussion forward to me either.

captainkrtek 24d ago
It's frustrating to find AI so prevalent in writing. It takes some nugget of a human written idea, dilutes it and increases word count, only to make the actual core idea less obvious (or ultimately summarized by someone using AI).
- ShinyLeftPad 24d ago
  Poor hyperscalers have got to make money somehow...
danielvaughn 24d ago
The rest of the blog post doesn't strike me as AI generated. Hard to tell, and personally I think "hard to tell" is about as good as it's going to get from here on out, so I give the author the benefit of my doubt. Agree that using AI for writing is a habit we collectively need to drop immediately.
0megion 24d ago
Maybe I should have added this part. As a non-native English speaker, using AI to write RFCs, ADRs, and internal documentation, my own writing style shifted so much to the direction that my writing seems like AI.
- frio 24d ago
  In that case, sorry for getting it wrong and throwing shade.
- zsoltkacsandi 24d ago

purpleidea 24d ago

For most users, Kubernetes has added complexity and cost, but not value. The sooner we realize this, the sooner we start thinking about how things should be built. Whether or not you have LLMs has nothing to do with that. The LLM will happily help build your crap even faster! Maybe people will finally realize they need to start rethinking infrastructure automation. I am.

Melatonic 24d ago
MicroVM !
antonvs 24d ago
> For most users, Kubernetes has added complexity and cost, but not value. This may be true for small, simple systems, such as basic web apps, that don’t really need Kubernetes. If your system can fit on a single machine, or even two or three, why would you use a cluster orchestrator to run it? But if you’re at a real company - not a mom and pop shop or similar - and you have multiple services implemented by multiple teams, you’d be hard-pressed today to find something that makes more sense than Kubernetes. Yes, there are PaaS offerings that can hide some of the inherent complexity, but they tend to run into limits, cost issues, etc. quite quickly. Not to mention they’re often implemented on top of Kubernetes anyway. Kubernetes is worth learning if you want to understand the needs and design of non-trivial software systems. Everything it offers corresponds to requirements such systems have, and it provides an extremely well organized, unified way of managing that. It’s one of the most well-designed software products of the 21st century.
- Melatonic 24d ago
  I think the issue is that there is a not yet emerged standard for the mid size companies that we hear less about here on HN but make up a huge amount of business in the US. Kubernetes may be the gold standard for larger companies or tech heavy companies of all sizes. But what about the mid size organisations ?
  - antonvs 23d ago

deleted 24d ago

[deleted]

Lawsonfletcher 24d ago

[flagged]