7 ms·

> All of the AI projects we have observed as a team are failing. Every single one – we have seen 0% success in a year and a half, not only amongst projects we h

by A1kmm 2mo ago

> All of the AI projects we have observed as a team are failing. Every single one – we have seen 0% success in a year and a half, not only amongst projects we have been asked to participate in, but even within projects that we have observed in passing while doing totally unrelated work. That's got to be hyperbole, which blows out their credibility. They chose to say 'AI' rather than, for example, LLM, or Transformer model, or Diffusion model. This means they are including a huge swathe of things dating back to Expert Systems in their claim. And who hasn't seen productivity gains from more established AI technology - at least things like semantic search? Who hasn't seen diffusion models generating content in roles that might have done the work by hand before? Who hasn't seen some kind of regression algorithm (even using linear regression in a supervised context counts as AI - so you can absolutely do AI even in tools like Excel) improve operation productivity? Even if they narrowed it to the Transformer model LLMs which re-ignited recent public interest in AI, less ambitious projects to give them to engineering staff to automate easy but boring tasks in the background generally have been a success. More ambitious ones that are beyond what you'd reasonably expect the models to be able to do - for sure, those tend to fail. For most of these, the failure is predictable in advance, while some are at the boundary of what's possible, and so it is harder to predict (these are rationally genuine R&D projects).

calvinmorrison 2mo ago
zero percent of statistics online are made up!
- fragmede 2mo ago
  [citation needed]
  - ElProlactin 2mo ago
    https://hermit-tech.com/business-owners https://hermit-tech.com/business-owners
Aurornis 2mo ago
If you read the footnote, they follow up to say they've rejected 100% of the AI projects brought to them. Go to their home page and one of their consulting selling points is recovering struggling projects. One of their front-page selling points is that they use "ancient techniques" from books written prior to the year 2000, because presumably everything newer than that is bad? > For non-executive management who might be struggling to deliver things that feel beyond their control, we have ancient techniques (see: books written between 1986 and 1999) to turn your team into the envy of the organisation, and we can drop in directly to get your team the resources it needs to save a struggling project. This is entirely a selection bias issue that they've created for themselves: Advertise a consulting service for saving failing projects to companies that don't have internal expertise to handle it, then write blog posts that 100% of the projects you see are failing. Also refuse to help them, to guarantee they can't be converted to successful projects to keep the success number at 0%.
- teravor 2mo ago
  it's not surprising as LLM's will generally fail you when you yourself don't know what you want. it's surprising how many people and organizations just blunder around scribbling code without clear goals in mind - for such people LLM's are a net negative as they will just end up with even more scribbles and no hope to make sense of it all.
  - BobbyTables2 2mo ago
    Cheaper than consultants!
  - szundi 2mo ago
    [dead]
- ElProlactin 2mo ago
  The prominent "Business Owners" link on this company's website is broken. "Not Found". Perhaps this will get downvoted but I personally take with a grain of salt anything written/stated by a company that can't even get the most basic functions (like running a simple website) correct. And the even bigger irony here is that the author has a ranty blog post in which he claims he saved his employer $500,000 by clicking a button. "[It] is fucking wild that an inefficiency that took me five minutes to solve in a GUI configuration panel was allowed to persist," he wrote. https://ludic.mataroa.blog/blog/i-accidentally-saved-half-a-million-dollars/ https://ludic.mataroa.blog/blog/i-accidentally-saved-half-a-...
j45 2mo ago
I would love to learn about and from the details of their projects.
simonw 2mo ago
I think there's a big difference between individual employees using AI tools to boost their productivity - with things like Claude Code and Codex - and "AI projects" where companies build custom software on top of LLMs. The former is easy to get right. Any software engineer (at least provided they aren't actively resisting the technology )can get useful results out of Claude Code these days. The latter is really hard. LLMs are a strange beast to build software on, and most of the obvious projects - like the internal chatbots described in this article - are easy to have over-promise and under-deliver.
jauntywundrkind 2mo ago
Their previous article on AI shows a pretty strident ham-handedness. https://news.ycombinator.com/item?id=48002795 https://news.ycombinator.com/item?id=48002795 In general I find their submissions tend towards extreme grandiosity. I find I really appreciate people who have some nuance about the world, can see some duality, and the many many many submissions here are (I admit) often quite fun and enjoyable, but spoken much more from a bully pulpit perspective, with a zeal and self certainty that I find rarely coincides with truth-seeking.
- muvlon 2mo ago
  That's a completely different person, they (ludic and smileplease) just both happen to be publishing on mataroa.blog. This is a blogging website anybody can sign up for, basically Medium but for nerds.
darth_avocado 2mo ago
> And who hasn't seen productivity gains from more established AI technology You have to be very careful about claiming productivity gains. There may have been some instances of gains in a specific part of a workflow but does it slow down others or result in overall gains is yet to be empirically measured and validated. We’re seeing metrics like more lines of code, better unit tests, documentation, faster PRs etc. but the actual gains of businesses are still a question mark. Do more PRs lead to faster features being shipped or does it lead to slower reviews or bug ridden code that breaks user experience? I’ve see a lot of companies tout their metrics around more code being shipped but the same companies aren’t talking about how that translates to an actual dollar amount.
- MagicMoonlight 2mo ago
  [dead]
- dd8601fn 2mo ago
  These things have been "good" for a while now. And yet, companies like Amazon and Microsoft aren't showing significant improvement in their most visible products. If it's not obviously showing for the all-in, AI-selling companies, I simply don't expect serious improvement for everyone else. They're undeniably neat tools, but so far there's no observable evidence that they're transformative.
  - Lord-Jobo 2mo ago
    Amazon and Microsoft have shown considerable decay in their most visible products. Maybe it’s AI, maybe it’s just momentum, maybe it’s something else. But they AI clearly isn’t helping them and they have near infinite access(compared to me anyway), so why should I expect it to help me?
andai 2mo ago
>semantic search I'm doing fundraising for my tf-idf startup. It's named after a very big number!
- oblio 2mo ago
  Outside of Google, has semantic search succeeded anywhere else?
deleted 2mo ago
[deleted]
dannersy 2mo ago
I get what you are saying, and while I'll say it is definitely not 0%, I have seen very little in the way of useful software that is primarily generated. The vast majority just does not go the distance for whatever reason. I could explain many reasons, but I am getting really tired of explaining myself. If the tools were as great as everyone says, we'd be going through a software Renaissance, but we're not. I would argue a software dark ages since it feels like things are getting worse and I find bugs in what was historically very long running and stable applications. But, whatever. I think the author is clearly talking about modern AI, I don't think they need to be explicit about models.
noodletheworld 2mo ago
Look, if they have data and it says 0%, and you have vibes that say that can’t be true, who should we believe? Do you work with lots of companies and see large AI success stories? Or do you just vibe that you personally find AI useful so it must also be a business success? Look, I honestly don’t care, but I think “it must be false” is also unsubstantiated hyperbole. If an agency says they see no AI success, I see no particular reason to believe they’re lying. They’re not saying AI can’t be a success. They’re saying they haven’t seen it. That matches my experience too. Proven AI success stories seem… vague, when you dig into the details, in my personal experience. It doesn’t seem surprising to me.
- moepstar 2mo ago
  ...especially considering someone has to justify their token spend...
mcv 2mo ago
He does employ a lot of hyperbole. Some of his previous blog posts have titles like "I Will Fucking Dropkick You If You Use That Spreadsheet" (https://ludic.mataroa.blog/blog/i-will-fucking-dropkick-you-if-you-use-that-spreadsheet/ https://ludic.mataroa.blog/blog/i-will-fucking-dropkick-you-...) and "I Will Fucking Piledrive You If You Mention AI Again" (https://ludic.mataroa.blog/blog/i-will-fucking-piledrive-you-if-you-mention-ai-again/ https://ludic.mataroa.blog/blog/i-will-fucking-piledrive-you...). He's not a big fan of polite, corporate speech, I suspect. But I like his style and he often has some insightful points that go against some popular industry practices.
- projektfu 2mo ago
  Yes, from the first link: "This is unreasonable. "Do I seem like a reasonable person?"

by A1kmm 2mo ago

calvinmorrison 2mo ago

zero percent of statistics online are made up!

fragmede 2mo ago
[citation needed]
- ElProlactin 2mo ago
  https://hermit-tech.com/business-owners https://hermit-tech.com/business-owners

Aurornis 2mo ago

If you read the footnote, they follow up to say they've rejected 100% of the AI projects brought to them. Go to their home page and one of their consulting selling points is recovering struggling projects. One of their front-page selling points is that they use "ancient techniques" from books written prior to the year 2000, because presumably everything newer than that is bad? > For non-executive management who might be struggling to deliver things that feel beyond their control, we have ancient techniques (see: books written between 1986 and 1999) to turn your team into the envy of the organisation, and we can drop in directly to get your team the resources it needs to save a struggling project. This is entirely a selection bias issue that they've created for themselves: Advertise a consulting service for saving failing projects to companies that don't have internal expertise to handle it, then write blog posts that 100% of the projects you see are failing. Also refuse to help them, to guarantee they can't be converted to successful projects to keep the success number at 0%.

teravor 2mo ago
it's not surprising as LLM's will generally fail you when you yourself don't know what you want. it's surprising how many people and organizations just blunder around scribbling code without clear goals in mind - for such people LLM's are a net negative as they will just end up with even more scribbles and no hope to make sense of it all.
- BobbyTables2 2mo ago
  Cheaper than consultants!
- szundi 2mo ago
  [dead]
ElProlactin 2mo ago
The prominent "Business Owners" link on this company's website is broken. "Not Found". Perhaps this will get downvoted but I personally take with a grain of salt anything written/stated by a company that can't even get the most basic functions (like running a simple website) correct. And the even bigger irony here is that the author has a ranty blog post in which he claims he saved his employer $500,000 by clicking a button. "[It] is fucking wild that an inefficiency that took me five minutes to solve in a GUI configuration panel was allowed to persist," he wrote. https://ludic.mataroa.blog/blog/i-accidentally-saved-half-a-million-dollars/ https://ludic.mataroa.blog/blog/i-accidentally-saved-half-a-...

j45 2mo ago

I would love to learn about and from the details of their projects.

simonw 2mo ago

I think there's a big difference between individual employees using AI tools to boost their productivity - with things like Claude Code and Codex - and "AI projects" where companies build custom software on top of LLMs. The former is easy to get right. Any software engineer (at least provided they aren't actively resisting the technology )can get useful results out of Claude Code these days. The latter is really hard. LLMs are a strange beast to build software on, and most of the obvious projects - like the internal chatbots described in this article - are easy to have over-promise and under-deliver.

jauntywundrkind 2mo ago

Their previous article on AI shows a pretty strident ham-handedness. https://news.ycombinator.com/item?id=48002795 https://news.ycombinator.com/item?id=48002795 In general I find their submissions tend towards extreme grandiosity. I find I really appreciate people who have some nuance about the world, can see some duality, and the many many many submissions here are (I admit) often quite fun and enjoyable, but spoken much more from a bully pulpit perspective, with a zeal and self certainty that I find rarely coincides with truth-seeking.

muvlon 2mo ago
That's a completely different person, they (ludic and smileplease) just both happen to be publishing on mataroa.blog. This is a blogging website anybody can sign up for, basically Medium but for nerds.

darth_avocado 2mo ago

> And who hasn't seen productivity gains from more established AI technology You have to be very careful about claiming productivity gains. There may have been some instances of gains in a specific part of a workflow but does it slow down others or result in overall gains is yet to be empirically measured and validated. We’re seeing metrics like more lines of code, better unit tests, documentation, faster PRs etc. but the actual gains of businesses are still a question mark. Do more PRs lead to faster features being shipped or does it lead to slower reviews or bug ridden code that breaks user experience? I’ve see a lot of companies tout their metrics around more code being shipped but the same companies aren’t talking about how that translates to an actual dollar amount.

MagicMoonlight 2mo ago
[dead]
dd8601fn 2mo ago
These things have been "good" for a while now. And yet, companies like Amazon and Microsoft aren't showing significant improvement in their most visible products. If it's not obviously showing for the all-in, AI-selling companies, I simply don't expect serious improvement for everyone else. They're undeniably neat tools, but so far there's no observable evidence that they're transformative.
- Lord-Jobo 2mo ago
  Amazon and Microsoft have shown considerable decay in their most visible products. Maybe it’s AI, maybe it’s just momentum, maybe it’s something else. But they AI clearly isn’t helping them and they have near infinite access(compared to me anyway), so why should I expect it to help me?

andai 2mo ago

>semantic search I'm doing fundraising for my tf-idf startup. It's named after a very big number!

oblio 2mo ago
Outside of Google, has semantic search succeeded anywhere else?

deleted 2mo ago

[deleted]

dannersy 2mo ago

I get what you are saying, and while I'll say it is definitely not 0%, I have seen very little in the way of useful software that is primarily generated. The vast majority just does not go the distance for whatever reason. I could explain many reasons, but I am getting really tired of explaining myself. If the tools were as great as everyone says, we'd be going through a software Renaissance, but we're not. I would argue a software dark ages since it feels like things are getting worse and I find bugs in what was historically very long running and stable applications. But, whatever. I think the author is clearly talking about modern AI, I don't think they need to be explicit about models.

noodletheworld 2mo ago

Look, if they have data and it says 0%, and you have vibes that say that can’t be true, who should we believe? Do you work with lots of companies and see large AI success stories? Or do you just vibe that you personally find AI useful so it must also be a business success? Look, I honestly don’t care, but I think “it must be false” is also unsubstantiated hyperbole. If an agency says they see no AI success, I see no particular reason to believe they’re lying. They’re not saying AI can’t be a success. They’re saying they haven’t seen it. That matches my experience too. Proven AI success stories seem… vague, when you dig into the details, in my personal experience. It doesn’t seem surprising to me.

moepstar 2mo ago
...especially considering someone has to justify their token spend...

mcv 2mo ago

He does employ a lot of hyperbole. Some of his previous blog posts have titles like "I Will Fucking Dropkick You If You Use That Spreadsheet" (https://ludic.mataroa.blog/blog/i-will-fucking-dropkick-you-if-you-use-that-spreadsheet/ https://ludic.mataroa.blog/blog/i-will-fucking-dropkick-you-...) and "I Will Fucking Piledrive You If You Mention AI Again" (https://ludic.mataroa.blog/blog/i-will-fucking-piledrive-you-if-you-mention-ai-again/ https://ludic.mataroa.blog/blog/i-will-fucking-piledrive-you...). He's not a big fan of polite, corporate speech, I suspect. But I like his style and he often has some insightful points that go against some popular industry practices.

projektfu 2mo ago
Yes, from the first link: "This is unreasonable. "Do I seem like a reasonable person?"