9 ms·

Nobody has built a software factory

51 points by Mitchem 18d ago

bob1029 15d ago
Software is the factory. The utility is built right in. You can create lossless copies of the product nearly instantly and for free. It's a logistical miracle.
- ivanjermakov 15d ago
  It's an economical miracle too. Rare case where producing one sellable product is identical in costs to producing millions. And international distributuon and scaling is simple too, compared to physical goods.
- TedDallas 15d ago
  Exactly. Or at least software is a custom tooling. A factory in the traditional sense is a process and infrastructure tuned to run efficiently that produces something of value that does not require highly skilled/expensive labor.
asdev 15d ago
I think “software factory” just means automations where bots pick up tickets and spin up code changes. Maybe Stripe has that many well defined bugs but there’s no way it works if you’re building anything remotely complex. There’s just a lot of nuance that agents cannot pick up
sarreph 15d ago
I really like the premise of this article because I am intensely interested in learning about software factories. As much as "software factory" is a buzzword, it's still SOTA and it's absolutely going to happen / keep happening in the tech industry over the next few years. However, I can't read this, not because I refuse to -- I want to -- but because my the cogs in my mind just stop whirring at all of the LLM-isms, my eyes glaze over, and I stop being able to follow what the article is trying to communicate to me. I would love for the OP to call me out and correct me (that they haven't used LLMs to write this).
- CBLT 15d ago
  I was bothered by the writing and fought through it, and carefully went through everything in the article... and you didn't miss anything. It's a tweet expanded to article length.
  - deleted 15d ago
    [deleted]
- daveguy 15d ago
  Wait. It's going to happen and it's SOTA, but you can't read the article because genAI produces such garbage? Seems like a disconnect there.
  - sarreph 15d ago
    LLMs being used as a tool in general (and in the future) to automate creation and maintenance of software is not mutually exclusive to one person using LLMs to create an article.
  - steveklabnik 15d ago
    writing prose and writing code are two different things I can get good code out of LLMs. I can rarely get good writing out of them. Maybe it's a skill issue, but it's not inherently a contradiction that they're better at some things than others. It's much easier to RL coding than writing.
    - simonw 15d ago
      Being dull and predictable and repeating existing patterns are characteristics of good code, and simultaneously characteristics of bad writing.
      - SAI_Peregrinus 15d ago
        I like to delete all AI comments, and write my own. If I can't write a comment, I don't understand the code well enough to have my name on the commit. I often delete the resulting comment, because AI likes to add many more than are necessary.
measurablefunc 15d ago
This is obviously written by AI so I stopped reading at the first headline.
- jdc0589 15d ago
  > Everyone Bought the Motor. Ford Rebuilt the Floor. 110% ai
- scottmcmac 15d ago
  Yeah, I'm amazed at how quickly I spotted this one. I didn't even read any prose since the first section header was visible before I started reading and somehow my brain jumped there first and it was a dead giveaway. I think I may be developing defensive scanning strategies, that are probably useful for avoiding slop, but not good for my focus or general reading comprehension.
- submain 15d ago
  It's amazing how exhausting AI text is to read.
- QuercusMax 15d ago
  "On that footing the loop closes"
- Seattle3503 15d ago
  I ran Pangram after seeing the "Everyone Bought the Motor. Ford Rebuilt the Floor" headline. 97% AI. I have all the AI subscriptions myself, I don't need a meet proxy. ai;dr
xnx 15d ago
"Software factory" in overused and inaccurate term. Factories have an expensive tooling process and then create the same widget thousands/millions of times. Software can be copied effortlessly once made. What people are describing is more like a "software kitchen" or "software machine shop".
- samuelknight 15d ago
  Artisan software factory
- rgbrgb 15d ago
  or maybe more like a restaurant than a kitchen... with software development (like all product development) you need to bake in the user feedback loop like to torture the metaphors, i think we already have the factory (coding agents you can ask to manufacture software to a spec) but it's the whole process including QA, shipping, listening to users and iterating that has to happen over time (unless you have an accurate world simulation so you know exactly how your product will be received... and I know there are people working on that too but I'm skeptical)
nmehner 15d ago
"software factory" has always been a completely broken metaphor. Writing software is like building a factory. You try to find the optimal process to do something. This is a creative process and can never be straight forward. Using the software is running the factory. This won't change with AI, since you never now all requirements beforehand and you always will have to iterate. The only reason people are selling software factories is that customers like to imagine that this was possible. That a creative, iterative process can be perfectly planable and predictable. This is just wrong.
jdw64 15d ago
I wonder what the point is of building a software factory
john_rood 15d ago
[flagged]
ape4 15d ago
Well there's the Factory Method Pattern https://en.wikipedia.org/wiki/Factory_method_pattern https://en.wikipedia.org/wiki/Factory_method_pattern
- athenot 15d ago
  Yes, but what about the Factory Factory Pattern? https://www.danstroot.com/posts/2018-10-03-hammer-factories https://www.danstroot.com/posts/2018-10-03-hammer-factories
dash2 15d ago
Plenty of people have built prose factories, though, and God I hate them for it. Is it OK to flag this as slop?
mccoyb 15d ago
I don’t think it’s possible to build a software factory unless you truly do not care about what you’re putting out. Agent swarms, self learning, Ralph loops, execution DAGs, spending hours trying to convey my preferences into skills, yada yada — the reality is that no matter how long I spend trying to close the amortization gap between my expertise and (insert autonomous agent loop) … I always come back to see shit that I have opinions about and need to polish. If you’re keeping on top of your craft, I just don’t think general purpose pretraining / RL is going to give you something which perfectly copies you. It can get closer and closer! But I’ve never seen agent work where I didn’t immediately start thinking of refinements … I guess many things are okay if you’re okay with averages
- deleted 15d ago
  [deleted]
- Kirth 15d ago
  > I don’t think it’s possible to build a software factory unless you truly do not care about what you’re putting out In the article they talk of a swarm of agents autonomously speccing out the software and I can't help but wonder -- at that point, what are you creating, and for who??
- 27183 15d ago
  Despite the marketing, the "I" in AI doesn't (yet) stand for "intelligence". You're best off finding applications where you can easily write a rigorous, well-defined, easily evaluated acceptance criterion and then let the model, harness, etc go wild. This of course is also the case with "AI coding" technology that came before--genetic programming, NEAT, etc. It's not intelligent, and therefore limited in fundamental ways. You have to steer it to make it work. This is very fundamentally different from how factory workers operate. Therefore I can't see how the "software factories" concept is actually feasible without a technological breakthrough. We may be no closer to it now than we were 25yr ago.
- Spartan-S63 15d ago
  The fascination with "software factories" is strange. We don't have "bridge factories" or "skyscraper factories." We engineer those for the specific requirements from each individual project. I think it's rooted in the idea that AI is going to "solve" software engineering when it's really only another tool that magnifies leverage. Same thing with traditional engineering and their computer-assisted tools. It doesn't "solve" engineering, but it does automate some of the repetitive things that aren't the biggest value-add. Software isn't a manufactured product and that's where the disconnect is. You can't build software factories unless you don't care about the quality. For some companies, that might be fine, but I imagine that's a much smaller subset of the market than folks think.
PreownedPlaid 15d ago
[flagged]
ellyagg 15d ago
No true Scotsman
deadbabe 15d ago
I propose a software farm. Imagine a large multi dimensional grid. Each axis of the grid represents a spectrum of how a specific feature could be implemented. You thus have one axis per feature. Now you plot a cloud of random points all over the grid and at each point have an agent build a solution where the features are expressed according to the corresponding intensity values at the point. Now you go through and review the solutions you like, create more points around clusters where you kind of like what is going on, until the perfect product emerges.
- grim_io 15d ago
  Like The Matrix, but it's us who harvest the dreaming machines. Not a future I envisioned, but a preferable one.
- ianjbutler 15d ago
  This is the old idea of giving two teams the same project and letting a third team judge/merge a solution incorporating good parts of each. Just like the old idea, of course you would do this with infinite resources. You can already sort of do this in way that isn't insanely wasteful if your project has strong measures of correctness. But even then things like compilers have a very real axis for friction in "user/developer/agent experience".. doing it at scale involves some kind of LLM-as-Judge pointed at something which is not an objective value. So .. not only piling on additional false work, but work which is very likely to be incorrect/noisy.
- brazukadev 15d ago
  not sure how effective would that be. Multiple one-shot features won't get to the point where they are all good to deliver to compare. Unless there is an infinite amount of tokens to do the same after the one-shot for each implementation which would grow exponentially.
  - deadbabe 15d ago
    You would use LLMs baked into silicon chips.
nzoschke 15d ago
The referenced https://blog.exe.dev/replace-your-ci https://blog.exe.dev/replace-your-ci is resonating with me. I'm spending a relatively large amount of time waiting for CI to finish running, then clicking a button for an agent to fix failures, then waiting for CI to finish again. Might as well push that down into the core development loop. Do folks have advice for doing this in practice? Hooks in the harness? Pre-commit hooks? Different GitHub configurations?
- dust-jacket 15d ago
  just tell it to watch CI and fix errors?
  - nzoschke 15d ago
    Yes but this can be a slow loop. Push to GH, run CI (in GHs relatively slow / fragile runners), poll GH APIs to see status and get errors, make changes. Then repeat. The optimization is to bring running tests / fixing errors into the core agentic development loop, be much more confident that CI will pass on final pushed changes, then automatically queue / merge the changes.
- zrail 15d ago
  1. Make your tests as fast as practicable 2. Split out "syntactically correct" fast checks like linters into a standalone script and call it along with the slower checks in a full-check script 3. Set the fast-check script as a pre-commit hook and the full-check script as a pre-push hook. 4. Give the model instructions that it needs to run the fast-check script after every change and the full-check script when it thinks it's done. 5. Run full-check in CI. Then you're good as long as the model doesn't bypass the pre-push hook, and even then CI will catch it.
  - nzoschke 15d ago
    pre-commit and pre-push hooks sound smart. I like that those will work the same for any human or any harness vs harness hooks. Then push straight to main? Or do we need GitHub to help manage the queue https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/managing-a-merge-queue https://docs.github.com/en/repositories/configuring-branches...
nater5000 15d ago
I definitely agree with the sentiment of this article, give or take. I've seen it at the place a previously worked at and now the place I currently work at: everybody wants AI to make coding faster and more efficient, but nobody is willing to take the risk of changing course and bending the firm's processes around what would actually make the agents more useful. Maybe this is more general than the qualms against "software factories," but it's pretty apparent that agentic development doesn't mesh with most of the development processes everyone was following prior to LLMs. Whenever a human is in the loop, it creates THE bottleneck which ultimately sets the pace. When humans can't fathom not being in the driver seat, then they're constantly in the loop, and nothing actually improves. I DO see small instances where this is overcome, albeit, it occurs when risk is minimal, the work is non-critical, and the required environment is already available. For example, exploratory research of new features. Before, this would be quite costly and particularly cumbersome since the results may indicate that the new feature isn't viable and the work is basically "wasted." Now, you can send some agents to explore it for you and give you a "first pass" on the validity of the concept. You still end up with humans taking over immediately after (often to verify that first pass), but this has allowed a bit more efficiency in terms of understanding when investing that human time in exploratory investigation is appropriate and when it isn't. People are willing to give agents this level of control because the alternative is usually just a default "no." Regardless, it seems obvious we're in a painful transition period where everyone in software can see that there's a ton of value here, but where everyone is simultaneously constrained by existing infrastructure, processes, expectations, etc., so as to not be willing or able to tap into that value. I suspect it won't be until there's just more new companies coming in without these existing constraints will the actual new version of these processes start to emerge and MAYBE some of the existing companies will be more willing to shake things up as needed to take advantage of it.
- ardit33 15d ago
  Was this written by AI? My experience, with the latest models, they do write code that looks correct/polished, but often do not fully solve the problem, or solve the wrong thing, or introduce un-needed complexity. By often, I mean at least 10%-15%, which is still a high enough error rate, that you don't want to let loose in even a trivial service, let alone a service that is on the critical side. For now humans are absolutely needed in the loop.
daveguy 15d ago
> its self-improvement loop ends where everyone’s does: “Humans review those suggestions as PRs on the factory definition and merge improvements." That's because it's still painfully clear that genAI has no taste. It's a median content generator. And the median kinda sucks. Of course you still need people to review the output.
khelavastr 15d ago
Unironically, Microsoft is the world's software factory.
mikert89 15d ago
People have, they arent putting them up for sale. Its the same with AI sales systems, if they work, they are worth far more than what they can be sold for as a product.
- steveklabnik 15d ago
  We are also sort of at a "bespoke factory" stage where building a factory means doing it around your specific codebase, each of which has their own needs and quirks. Just taking one of these wholesale from one company and using it at another would not work. We'll see if in the future, as people begin software projects this way, if there is more standardization. I suspect that there's just too much going on too fast at the moment to do it any other way, a decent factory for Opus 4.8 looks very different than one good for Astra, I'd guess. And models are just one axis.
deleted 15d ago
[deleted]
nobodyandproud 15d ago
This entire premise assumes that we can afford the factory machine. Long term, can we? If not, what do we do once we’re addicted to the new process and it’s impossible to get out and from?
hn_submit 15d ago
I think the Japanese built a "software factory" in the 1980's, but it was staffed with humans, not LLMs (which obviously didn't exist back then). So to say that no one has built a software factory is a strong statement...and factually incorrect.
nonameiguess 15d ago
The idea of a software factory long predates LLMs or any other form of automated full-program generation. The idea was to take full CI/CD pipelines that allow for rapid deployment of new components into a running system a la popular SaaS providers and web platforms that are usually single vendor or even single application, and extend the idea to a centralized production facility used by many different application vendors and consumer endpoints. Kessel Run in the US Air Force is the largest widely-touted success story I'm aware of, but I'm sure there must have been others. Think of Netflix and Google sharing the same developer VDIs, build tooling, unit test framework, E2E test framework, promotion gates, and possibly even a common runtime platform. It's not something the commercial world tends to think much about, but in the military world you have many different vendor collaborating on a single C2 or comms system, but individually building their own production lines for each component. Centralize that and you can reduce procurement costs by de-duplicating effort, at least in theory. Plenty of these have been built. I can buy that no one has built a fully-automated set of agents running on common compute, storage, and network substrates that can create, verify, and ship arbitrary software from vague natural language desirements alone, but that isn't the same thing and it's kind of a shame they decided to use an already in-use name for this. As many have pointed out, real physical factories don't do this, either. They build and ship the same thing with the same physical design in arbitrary quantities, which is a much easier problem to solve. Any Linux distro or mobile app store already does that.
grenbys 15d ago
Mixed thoughts on this article, here's my take: There's no need to build a Software Factory. Every company already had a software factory in pre-AI – it is a socio-technical system delivering software (e.g. R&D). The input is the product ideas, the output is those ideas shipped to customers as code running in production. With AI we now have a chance to modernise the existing factory – speed it up. Code generation is solved, all the other bits of the factory are not: cheap&fast CI at scale, tech planning, code review, validation in production, incident diagnosis, etc. Naive way of "modernisation" is to give everyone a Claude Code subscription and call it done. Proper way is to figure out where the bottlenecks are and automate them with high-quality specialised AI agents. To surface the bottlenecks you need to pump in the throughput and see where it breaks – in the same way how you'd find an air leak in a tyre. The analogy with real factories is the story on how they were modernising the cotton mills when the electricity came in: retrofitting a big-ass electric engine (give everyone Claude/Codex) does not work, instead every part of the factory was modernised with smaller electric engines (specialised AI agents) – that's when the mill started to really increase its output. Humans are still there as building agents is REALLY hard, and it is not an end-state and more of a process: e.g. AI labs models are end-of-life every year and good luck just switching a model w/o re-engineering the prompts, evals, a/b tests, etc). But the emphasis really is on the SOCIO-technical system. Building those specialised AI agents is not only hard because of the non-determinism of underlying technology, but because those agents will interact with humans, create incentives, second order effects in a complex system. Every existing software factory is just a projection of a given engineering culture. Things you value as an R&D org, things you've optimised for over the years - they shape whatever the factory you have - with or w/o AI
AaronAPU 15d ago
Software factory doesn’t matter without an attention factory, and if you have that you don’t even need more software.

51 points by Mitchem 18d ago

Software is the factory. The utility is built right in. You can create lossless copies of the product nearly instantly and for free. It's a logistical miracle.

ivanjermakov 15d ago
It's an economical miracle too. Rare case where producing one sellable product is identical in costs to producing millions. And international distributuon and scaling is simple too, compared to physical goods.
TedDallas 15d ago
Exactly. Or at least software is a custom tooling. A factory in the traditional sense is a process and infrastructure tuned to run efficiently that produces something of value that does not require highly skilled/expensive labor.

asdev 15d ago

I think “software factory” just means automations where bots pick up tickets and spin up code changes. Maybe Stripe has that many well defined bugs but there’s no way it works if you’re building anything remotely complex. There’s just a lot of nuance that agents cannot pick up

sarreph 15d ago

I really like the premise of this article because I am intensely interested in learning about software factories. As much as "software factory" is a buzzword, it's still SOTA and it's absolutely going to happen / keep happening in the tech industry over the next few years. However, I can't read this, not because I refuse to -- I want to -- but because my the cogs in my mind just stop whirring at all of the LLM-isms, my eyes glaze over, and I stop being able to follow what the article is trying to communicate to me. I would love for the OP to call me out and correct me (that they haven't used LLMs to write this).

CBLT 15d ago
I was bothered by the writing and fought through it, and carefully went through everything in the article... and you didn't miss anything. It's a tweet expanded to article length.
- deleted 15d ago
  [deleted]
daveguy 15d ago
Wait. It's going to happen and it's SOTA, but you can't read the article because genAI produces such garbage? Seems like a disconnect there.
- sarreph 15d ago
  LLMs being used as a tool in general (and in the future) to automate creation and maintenance of software is not mutually exclusive to one person using LLMs to create an article.
- steveklabnik 15d ago
  writing prose and writing code are two different things I can get good code out of LLMs. I can rarely get good writing out of them. Maybe it's a skill issue, but it's not inherently a contradiction that they're better at some things than others. It's much easier to RL coding than writing.
  - simonw 15d ago
    Being dull and predictable and repeating existing patterns are characteristics of good code, and simultaneously characteristics of bad writing.
    - SAI_Peregrinus 15d ago
      I like to delete all AI comments, and write my own. If I can't write a comment, I don't understand the code well enough to have my name on the commit. I often delete the resulting comment, because AI likes to add many more than are necessary.

measurablefunc 15d ago

This is obviously written by AI so I stopped reading at the first headline.

jdc0589 15d ago
> Everyone Bought the Motor. Ford Rebuilt the Floor. 110% ai
scottmcmac 15d ago
Yeah, I'm amazed at how quickly I spotted this one. I didn't even read any prose since the first section header was visible before I started reading and somehow my brain jumped there first and it was a dead giveaway. I think I may be developing defensive scanning strategies, that are probably useful for avoiding slop, but not good for my focus or general reading comprehension.
submain 15d ago
It's amazing how exhausting AI text is to read.
QuercusMax 15d ago
"On that footing the loop closes"
Seattle3503 15d ago
I ran Pangram after seeing the "Everyone Bought the Motor. Ford Rebuilt the Floor" headline. 97% AI. I have all the AI subscriptions myself, I don't need a meet proxy. ai;dr

xnx 15d ago

"Software factory" in overused and inaccurate term. Factories have an expensive tooling process and then create the same widget thousands/millions of times. Software can be copied effortlessly once made. What people are describing is more like a "software kitchen" or "software machine shop".

samuelknight 15d ago
Artisan software factory
rgbrgb 15d ago
or maybe more like a restaurant than a kitchen... with software development (like all product development) you need to bake in the user feedback loop like to torture the metaphors, i think we already have the factory (coding agents you can ask to manufacture software to a spec) but it's the whole process including QA, shipping, listening to users and iterating that has to happen over time (unless you have an accurate world simulation so you know exactly how your product will be received... and I know there are people working on that too but I'm skeptical)

nmehner 15d ago

"software factory" has always been a completely broken metaphor. Writing software is like building a factory. You try to find the optimal process to do something. This is a creative process and can never be straight forward. Using the software is running the factory. This won't change with AI, since you never now all requirements beforehand and you always will have to iterate. The only reason people are selling software factories is that customers like to imagine that this was possible. That a creative, iterative process can be perfectly planable and predictable. This is just wrong.

jdw64 15d ago

I wonder what the point is of building a software factory

john_rood 15d ago

[flagged]

ape4 15d ago

Well there's the Factory Method Pattern https://en.wikipedia.org/wiki/Factory_method_pattern https://en.wikipedia.org/wiki/Factory_method_pattern

athenot 15d ago
Yes, but what about the Factory Factory Pattern? https://www.danstroot.com/posts/2018-10-03-hammer-factories https://www.danstroot.com/posts/2018-10-03-hammer-factories

dash2 15d ago

Plenty of people have built prose factories, though, and God I hate them for it. Is it OK to flag this as slop?

mccoyb 15d ago

I don’t think it’s possible to build a software factory unless you truly do not care about what you’re putting out. Agent swarms, self learning, Ralph loops, execution DAGs, spending hours trying to convey my preferences into skills, yada yada — the reality is that no matter how long I spend trying to close the amortization gap between my expertise and (insert autonomous agent loop) … I always come back to see shit that I have opinions about and need to polish. If you’re keeping on top of your craft, I just don’t think general purpose pretraining / RL is going to give you something which perfectly copies you. It can get closer and closer! But I’ve never seen agent work where I didn’t immediately start thinking of refinements … I guess many things are okay if you’re okay with averages

deleted 15d ago
[deleted]
Kirth 15d ago
> I don’t think it’s possible to build a software factory unless you truly do not care about what you’re putting out In the article they talk of a swarm of agents autonomously speccing out the software and I can't help but wonder -- at that point, what are you creating, and for who??
27183 15d ago
Despite the marketing, the "I" in AI doesn't (yet) stand for "intelligence". You're best off finding applications where you can easily write a rigorous, well-defined, easily evaluated acceptance criterion and then let the model, harness, etc go wild. This of course is also the case with "AI coding" technology that came before--genetic programming, NEAT, etc. It's not intelligent, and therefore limited in fundamental ways. You have to steer it to make it work. This is very fundamentally different from how factory workers operate. Therefore I can't see how the "software factories" concept is actually feasible without a technological breakthrough. We may be no closer to it now than we were 25yr ago.
Spartan-S63 15d ago
The fascination with "software factories" is strange. We don't have "bridge factories" or "skyscraper factories." We engineer those for the specific requirements from each individual project. I think it's rooted in the idea that AI is going to "solve" software engineering when it's really only another tool that magnifies leverage. Same thing with traditional engineering and their computer-assisted tools. It doesn't "solve" engineering, but it does automate some of the repetitive things that aren't the biggest value-add. Software isn't a manufactured product and that's where the disconnect is. You can't build software factories unless you don't care about the quality. For some companies, that might be fine, but I imagine that's a much smaller subset of the market than folks think.

PreownedPlaid 15d ago

[flagged]

ellyagg 15d ago

No true Scotsman

deadbabe 15d ago

I propose a software farm. Imagine a large multi dimensional grid. Each axis of the grid represents a spectrum of how a specific feature could be implemented. You thus have one axis per feature. Now you plot a cloud of random points all over the grid and at each point have an agent build a solution where the features are expressed according to the corresponding intensity values at the point. Now you go through and review the solutions you like, create more points around clusters where you kind of like what is going on, until the perfect product emerges.

grim_io 15d ago
Like The Matrix, but it's us who harvest the dreaming machines. Not a future I envisioned, but a preferable one.
ianjbutler 15d ago
This is the old idea of giving two teams the same project and letting a third team judge/merge a solution incorporating good parts of each. Just like the old idea, of course you would do this with infinite resources. You can already sort of do this in way that isn't insanely wasteful if your project has strong measures of correctness. But even then things like compilers have a very real axis for friction in "user/developer/agent experience".. doing it at scale involves some kind of LLM-as-Judge pointed at something which is not an objective value. So .. not only piling on additional false work, but work which is very likely to be incorrect/noisy.
brazukadev 15d ago
not sure how effective would that be. Multiple one-shot features won't get to the point where they are all good to deliver to compare. Unless there is an infinite amount of tokens to do the same after the one-shot for each implementation which would grow exponentially.
- deadbabe 15d ago
  You would use LLMs baked into silicon chips.

nzoschke 15d ago

The referenced https://blog.exe.dev/replace-your-ci https://blog.exe.dev/replace-your-ci is resonating with me. I'm spending a relatively large amount of time waiting for CI to finish running, then clicking a button for an agent to fix failures, then waiting for CI to finish again. Might as well push that down into the core development loop. Do folks have advice for doing this in practice? Hooks in the harness? Pre-commit hooks? Different GitHub configurations?

dust-jacket 15d ago
just tell it to watch CI and fix errors?
- nzoschke 15d ago
  Yes but this can be a slow loop. Push to GH, run CI (in GHs relatively slow / fragile runners), poll GH APIs to see status and get errors, make changes. Then repeat. The optimization is to bring running tests / fixing errors into the core agentic development loop, be much more confident that CI will pass on final pushed changes, then automatically queue / merge the changes.
zrail 15d ago
1. Make your tests as fast as practicable 2. Split out "syntactically correct" fast checks like linters into a standalone script and call it along with the slower checks in a full-check script 3. Set the fast-check script as a pre-commit hook and the full-check script as a pre-push hook. 4. Give the model instructions that it needs to run the fast-check script after every change and the full-check script when it thinks it's done. 5. Run full-check in CI. Then you're good as long as the model doesn't bypass the pre-push hook, and even then CI will catch it.
- nzoschke 15d ago
  pre-commit and pre-push hooks sound smart. I like that those will work the same for any human or any harness vs harness hooks. Then push straight to main? Or do we need GitHub to help manage the queue https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/managing-a-merge-queue https://docs.github.com/en/repositories/configuring-branches...

nater5000 15d ago

I definitely agree with the sentiment of this article, give or take. I've seen it at the place a previously worked at and now the place I currently work at: everybody wants AI to make coding faster and more efficient, but nobody is willing to take the risk of changing course and bending the firm's processes around what would actually make the agents more useful. Maybe this is more general than the qualms against "software factories," but it's pretty apparent that agentic development doesn't mesh with most of the development processes everyone was following prior to LLMs. Whenever a human is in the loop, it creates THE bottleneck which ultimately sets the pace. When humans can't fathom not being in the driver seat, then they're constantly in the loop, and nothing actually improves. I DO see small instances where this is overcome, albeit, it occurs when risk is minimal, the work is non-critical, and the required environment is already available. For example, exploratory research of new features. Before, this would be quite costly and particularly cumbersome since the results may indicate that the new feature isn't viable and the work is basically "wasted." Now, you can send some agents to explore it for you and give you a "first pass" on the validity of the concept. You still end up with humans taking over immediately after (often to verify that first pass), but this has allowed a bit more efficiency in terms of understanding when investing that human time in exploratory investigation is appropriate and when it isn't. People are willing to give agents this level of control because the alternative is usually just a default "no." Regardless, it seems obvious we're in a painful transition period where everyone in software can see that there's a ton of value here, but where everyone is simultaneously constrained by existing infrastructure, processes, expectations, etc., so as to not be willing or able to tap into that value. I suspect it won't be until there's just more new companies coming in without these existing constraints will the actual new version of these processes start to emerge and MAYBE some of the existing companies will be more willing to shake things up as needed to take advantage of it.

ardit33 15d ago
Was this written by AI? My experience, with the latest models, they do write code that looks correct/polished, but often do not fully solve the problem, or solve the wrong thing, or introduce un-needed complexity. By often, I mean at least 10%-15%, which is still a high enough error rate, that you don't want to let loose in even a trivial service, let alone a service that is on the critical side. For now humans are absolutely needed in the loop.

daveguy 15d ago

> its self-improvement loop ends where everyone’s does: “Humans review those suggestions as PRs on the factory definition and merge improvements." That's because it's still painfully clear that genAI has no taste. It's a median content generator. And the median kinda sucks. Of course you still need people to review the output.

khelavastr 15d ago

Unironically, Microsoft is the world's software factory.

mikert89 15d ago

People have, they arent putting them up for sale. Its the same with AI sales systems, if they work, they are worth far more than what they can be sold for as a product.

steveklabnik 15d ago
We are also sort of at a "bespoke factory" stage where building a factory means doing it around your specific codebase, each of which has their own needs and quirks. Just taking one of these wholesale from one company and using it at another would not work. We'll see if in the future, as people begin software projects this way, if there is more standardization. I suspect that there's just too much going on too fast at the moment to do it any other way, a decent factory for Opus 4.8 looks very different than one good for Astra, I'd guess. And models are just one axis.

deleted 15d ago

[deleted]

nobodyandproud 15d ago

This entire premise assumes that we can afford the factory machine. Long term, can we? If not, what do we do once we’re addicted to the new process and it’s impossible to get out and from?

hn_submit 15d ago

I think the Japanese built a "software factory" in the 1980's, but it was staffed with humans, not LLMs (which obviously didn't exist back then). So to say that no one has built a software factory is a strong statement...and factually incorrect.

nonameiguess 15d ago

The idea of a software factory long predates LLMs or any other form of automated full-program generation. The idea was to take full CI/CD pipelines that allow for rapid deployment of new components into a running system a la popular SaaS providers and web platforms that are usually single vendor or even single application, and extend the idea to a centralized production facility used by many different application vendors and consumer endpoints. Kessel Run in the US Air Force is the largest widely-touted success story I'm aware of, but I'm sure there must have been others. Think of Netflix and Google sharing the same developer VDIs, build tooling, unit test framework, E2E test framework, promotion gates, and possibly even a common runtime platform. It's not something the commercial world tends to think much about, but in the military world you have many different vendor collaborating on a single C2 or comms system, but individually building their own production lines for each component. Centralize that and you can reduce procurement costs by de-duplicating effort, at least in theory. Plenty of these have been built. I can buy that no one has built a fully-automated set of agents running on common compute, storage, and network substrates that can create, verify, and ship arbitrary software from vague natural language desirements alone, but that isn't the same thing and it's kind of a shame they decided to use an already in-use name for this. As many have pointed out, real physical factories don't do this, either. They build and ship the same thing with the same physical design in arbitrary quantities, which is a much easier problem to solve. Any Linux distro or mobile app store already does that.

grenbys 15d ago

Mixed thoughts on this article, here's my take: There's no need to build a Software Factory. Every company already had a software factory in pre-AI – it is a socio-technical system delivering software (e.g. R&D). The input is the product ideas, the output is those ideas shipped to customers as code running in production. With AI we now have a chance to modernise the existing factory – speed it up. Code generation is solved, all the other bits of the factory are not: cheap&fast CI at scale, tech planning, code review, validation in production, incident diagnosis, etc. Naive way of "modernisation" is to give everyone a Claude Code subscription and call it done. Proper way is to figure out where the bottlenecks are and automate them with high-quality specialised AI agents. To surface the bottlenecks you need to pump in the throughput and see where it breaks – in the same way how you'd find an air leak in a tyre. The analogy with real factories is the story on how they were modernising the cotton mills when the electricity came in: retrofitting a big-ass electric engine (give everyone Claude/Codex) does not work, instead every part of the factory was modernised with smaller electric engines (specialised AI agents) – that's when the mill started to really increase its output. Humans are still there as building agents is REALLY hard, and it is not an end-state and more of a process: e.g. AI labs models are end-of-life every year and good luck just switching a model w/o re-engineering the prompts, evals, a/b tests, etc). But the emphasis really is on the SOCIO-technical system. Building those specialised AI agents is not only hard because of the non-determinism of underlying technology, but because those agents will interact with humans, create incentives, second order effects in a complex system. Every existing software factory is just a projection of a given engineering culture. Things you value as an R&D org, things you've optimised for over the years - they shape whatever the factory you have - with or w/o AI

AaronAPU 15d ago

Software factory doesn’t matter without an attention factory, and if you have that you don’t even need more software.