8 ms·
> For example, in testing by the investment firm Millennium, Fable 5.1 found the cause of a rare crash on their internal systems that none of their engineers (o
by jumploops 15d ago
> For example, in testing by the investment firm Millennium, Fable 5.1 found the cause of a rare crash on their internal systems that none of their engineers (or any other model) had been able to explain after several years of trying.
Say what you will about LLM-generated code, but stories like this give me hope that software will never be as buggy as it once was.
- maximinus_thrax 15d ago> Say what you will about LLM-generated code, but stories like this give me hope that software will never be as buggy as it once was. Stories like these is what I now call 'Marketing slop'
- Thaxll 15d agoYou have a serious engineering problem if you're not able to find the source of a crash after years.
- enraged_camel 15d agoYou are either seriously naive, or have never worked on any large and complex legacy codebase.
- DaiPlusPlus 15d agoI’ve been living in a bubble with my .NET day-job, where debugging/tracing/postmortems are a breeze. Compare with, say, a CORBA or DCOM system, deployed to prod with uber-optimized binaries without any debugging-symbols. So it’s not that I haven’t worked on large-scale, complex legacy systems - but that I haven’t worked on any large-scale, complex legacy systems written in languages bereft of runtime reflection and verbose error reporting. ————— It’s also possible that the bug was never found because its impact was so minimal: e.g. 1 crash per year, each causing 3 minutes’ downtime in a noncritical system: that’s something that will never get investigated fully.
- Thaxll 13d agoSorry but I don't buy it, any crash can be found and fix by one or more humans, if it was not it's either they're incompetent ( I doubt that ) or they did not beleive it was important enough to fix. A crash is actually the easiest kind of problem to fix since you have a crash. It means stacktrace, core dump, kernel error etc..
- brokencode 15d agoIf it’s rare and the impact is low, then it’s not getting prioritized. It doesn’t matter how much time passes if you decide not to spend time investigating.
- slashdave 15d agoIf only!
- DarmokTanagra 15d ago[flagged]
- ruszki 15d agoMy experience with such problems, is that they stay for this long because nobody cares, not because it's impossible, or even technically too difficult. So hopefully, LLM will improve things, but that quote is a lie.
- mark_l_watson 14d agoI use Kimi K3 and GLM 5.3 to find old bugs and improve old code, and would try Fable 5.1 if I could afford $50 for 1M output tokens (I am retired and all my research is self funded now) I like to start by prompting with “examine this code base for problems and improvements and write to IMPROVEMENTS.MD” and then carefully look over the suggestions, and either fix myself or let the model+coding harness try.
- vablings 15d agoSoftware will be buggier than ever but also way less buggy.
- luckystarr 15d agoTranslation: As the number of bugs per line of code is a constant depending on language and project, we will experience a world full of bugs now that LLMs generate so much more code. On the other hand, LLMs are so quick at fixing them, the number of bugs should go down. I wonder which trend will be winning though. I personally won't bet on quality.
- unglaublich 15d agoIt's going to be 50% less buggy, but we're going to write 10x as much code too.
- alasano 15d agoGood software will be good-er. Bad software will be nightmare fuel.
- farkerhaiku 15d agobad software will be replacable.
- pphysch 15d agoBad software, as in stateless programs, doesn't actually matter and never did. They can be replaced trivially. The problem is the real world isn't made of stateless programs, but lots of important data in bespoke formats/schemas, and if you change the shitty software that interacts with the important data, in the wrong way, you can lose everything.
- visarga 15d ago> Good software will be good-er. Bad software will be nightmare fuel. We are talking about a moving target here ... they get better every few months, so I expect the super-LLMs from 2035 will write amazing code even with sloppy prompting.
- PedroBatista 15d agoI sometimes have that feeling too, then ask another LLM to do a code and vulnerability review and OMG: rookie mistakes, over complications and security gaps even a 1st year student would not make regularly. So.. one more year of untreated bipolar AI psychosis I guess..
- brokencode 15d agoAt least we are at a point where we can have AI review code and reliably find real problems. That alone is incredibly valuable.
- chpatrick 15d agoI think these kinds of comments really need to say which LLM that is. There's an enormous difference in skill between the frontier ones and say the Google search AI.
- PedroBatista 15d agoCodex Luna, Terra and Sol. Claude Opus, Sonnet and sometime Fable. They all work, they all are "good", they all are both "smart" and commit incredible basic mistakes a fair amount of times. Then there's the cost situation..
- chpatrick 15d agoAlso true for human developers.
- aennassiri 15d agoWe will have more bugs. Even the best models with the best software engineers will produce bugs. There are two reasons : first the pressure to produce more and second LLMs will always produce slop
- anthonyrstevens 15d ago>> LLMs will always produce slop Such a low-quality comment
- aennassiri 15d agoWhat are your arguments then ? What are your thoughts ? I use fable everyday and it is always coming up with changes on thousands of files for simple things, overall the code does the job but there is always marginal slop or unnecessary code to be addressed.
- nchie 14d agoThat honestly sounds like it's more about either the quality of the codebase, or the driver.
- aennassiri 14d agoTo be more concrete, the last two examples were that Fable added every time a function called utcnow in newly added Python modules, and it always ended up adding constants. I'm saying it takes a lot of effort to make sure Fable or other frontier models don't produce such marginal slop and to keep things concise.
- kilroy123 15d agoI think we'll have lots of bugs. They'll just be found and closed way sooner. You'll have an agent that watchs for issues, then opens a PR fixing it.
- exabrial 15d agoThe marketing here trick is, if they spent the same money on humans they'd have found it years ago. Instead, the lurking variable here is new budget was added. With the new budget, they added a new tool, and the bug was located. The difference here was budget.
- aenis 15d agoNope. The difference here is stamina. Those models never tire while working on an issue, people do. And I doubt finding that bug cost more than $1k or so. Even if $10k. Thats nothing for a large department in a multinational company. Thats maybe 2 weeks of fully loaded costs of an engineer. Thats a single business trip across the Atlantic. Thats about two company issued macbooks, or one, if the company is nice. Nope. Not budget.
- efficax 15d agothe budget for allowing a single engineer to deep dive on a bug that is annoying but also not bad enough that you can live with it for years is pretty big. $10k a month or more. My budget for Claude is $200/mo.
- coder-pm 15d agoThat kind of one shot capability is impressive but how does it work for my typical work style? The way I work is to build a huge roadmap with goals and hand it to my agent to execute (often over night). I don't care that much about the benchmarks, what I care about is how often Fable 5.1 is making a baffling decision and destroys my plan, not respecting stop conditions or goals. I would seek for behavioral reliability over long autonomous runs, not eval scores. Anyone have that kind of feedback and observations?
- visarga 15d agoYou can engineer loops that have it, but it depends on a case by case basis. Does your loop have strong validation? if it's all vibes nothing can stop it from diverging.
- coder-pm 15d agoAgree on the validation, my loops are already gated. My concerns are about the cases when model is passing validation and quietly abandoning the goal. The second scenario is rewriting the plan to fit what was already done.