5 ms·

> All of the AI projects we have observed as a team are failing. Every single one – we have seen 0% success in a year and a half, What is an "AI project"? The

by azakai 2mo ago

> All of the AI projects we have observed as a team are failing. Every single one – we have seen 0% success in a year and a half, What is an "AI project"? The post doesn't define it. Is it writing some software from scratch? Using an LLM chatbot by non-coders, either internally or externally? Or something else entirely? Some examples would really help.

parpfish 2mo ago
i feel like the useful applications of AI get silently integrated into workflows. the ill-conceived moonshots by and for a non-technical audience get labelled as "AI projects/initiatives" and they fail.
- jaggederest 2mo ago
  That has ever been the case. As soon as it works reliably, it's not "AI" any more. Take spellcheckers or collaborative filtering as examples, but there are lots more. Hofstadter in G.E.B. said it well: > There is a related “Theorem” about progress in AI: once some mental function is programmed, people soon cease to consider it as an essential ingredient of “real thinking”. The ineluctable core of intelligence is always in that next thing which hasn’t yet been programmed. This “Theorem” was first proposed to me by Larry Tesler, so I call it Tesler’s Theorem: “AI is whatever hasn’t been done yet.”
  - ambicapter 2mo ago
    Sure, or maybe the actual applications of “AI” are small and unobtrusive, like the dictation of doctor’s notes example, and it’s not actually the massive revolution it’s claimed to be.
    - jaggederest 2mo ago
      I struggle to reconcile this attitude with, among other things, the massive number of exploits being discovered and the pretty straightforward utility for coding. Have you used codex recently?
  - jerf 2mo ago
    In this moment, the opposite is happening. Everything is getting called "AI", whether it uses LLMs to prompt LLMs about how to prompt LLMs, uses "conventional" machine learning, or just looks mysterious enough that they can expect the market to not ask questions. I am reminded of "game AI", which for the most part has historically been just giant decision trees, encoded one way or another, because if you hook up any sort of real AI to a game entity or collection of game entities that does any sort of learning or training, even simple 1980s-era reinforcement learning, it turns out the game entities will roflstomp the human players, and the human players aren't interested in paying for that experience. We've been calling those collections of if statements and for loops "AI" for a long time, though, because who wants to hear about how deliberately stupid their opponents are?
  - inigyou 2mo ago
    Because AI meant "make computers smart like people". We taught the computer to spell check, but the computer still didn't feel smart like a person, just smart like a machine so that obviously wasn't AI. We taught it to do algebra, same thing. With LLMs though, now it really does feel like an artificial human, so this time it really is AI.
    - jaggederest 2mo ago
      Right, but I remember when spellcheck first started to be a thing, and people were like "Wow proofreaders are out of a job, ai is so cool", but then it became normal.
  - variadix 2mo ago
    Perhaps, but I think what people mean by intelligence is something that learns and adapts. If LLMs couldn’t do in context learning I don’t think people would think of them as AI, more as a kind of queryable database via natural language. There are other algorithms that learn and adapt, but in much more narrow circumstances that most people won’t obviously interact with.
    - jaggederest 2mo ago
      Right, and my point is that now that is the bar. If you'd told me in 2008 that markov chains would be smart enough to one-shot even a trivial video game from scratch I would certainly have called that AI even if you had to do some major rigamarole to get it to work.
jdlshore 2mo ago
Their company does data projects. That plus context makes me think they’re talking about internal work process automation type of work, although it also seems like they’re talking about conversational interfaces (chatbots). I completely buy the “emperor’s new clothes” argument for work process automation. I’m surprised they don’t address AI-assisted engineering, which seems to be going positively for a lot of folks (although I have doubts about its sustainability). I disagree about the success of chatbots, if the problem is narrowly-defined and chosen properly. My previous company built a conversational interface to a vector database and saw good results. (Although, arguably, the vector database was the real magic, and a traditional UI would have been faster and more accurate.) In general, I think OP is more right than wrong, though, particularly about the AI mania and unrealistic expectations sweeping the C-suite.
- skydhash 2mo ago
  > I disagree about the success of chatbots, if the problem is narrowly-defined and chosen properly. If you can narrow the problem down, then you could design a much better interface for it than a text box and free form text (unless that's the better solution). As for as AI assisted engineering goes, the thing is that after some time with a project, you already have much of the workflows and routines nailed down as scripts and other various combinations of tooling. And unless it's spaghetti code, you will have various snippets you can copy from for new code. The one thing I've observed about AI projects is that there's often little technical design coherence about them. It's always a kitchen sink of technologies and practices.
  - jdlshore 2mo ago
    > If you can narrow the problem down, then you could design a much better interface for it than a text box and free form text (unless that's the better solution). Yes, I agree, in that the chatbot we built probably would have worked just as well with a traditional UI, and would have been done a lot faster. But it would have been a lot less sexy (actually important for the bottom line!) and there are future directions that could take advantage of the conversational interface that’s potentially better than a traditional UI. On the down side, good chatbots are really frikkin difficult to write. These things (LLMs) are not reliable at scale. The basic functionality came together in weeks. Getting it to behave consistently and obey guardrails took months, and even then we had to accept a low level of failed conversations.
Aurornis 2mo ago
They run a consulting company called "Hermit Tech". Their websites has an olde style font. They boast about "ancient techniques" from books written prior to the year 2000 > For non-executive management who might be struggling to deliver things that feel beyond their control, we have ancient techniques (see: books written between 1986 and 1999) to turn your team into the envy of the organisation, and we can drop in directly to get your team the resources it needs to save a struggling project. So yeah, of course these people hate AI and everything about it. No serious company is reaching out to these people for help with their AI project.

by azakai 2mo ago

parpfish 2mo ago

i feel like the useful applications of AI get silently integrated into workflows. the ill-conceived moonshots by and for a non-technical audience get labelled as "AI projects/initiatives" and they fail.

jaggederest 2mo ago
That has ever been the case. As soon as it works reliably, it's not "AI" any more. Take spellcheckers or collaborative filtering as examples, but there are lots more. Hofstadter in G.E.B. said it well: > There is a related “Theorem” about progress in AI: once some mental function is programmed, people soon cease to consider it as an essential ingredient of “real thinking”. The ineluctable core of intelligence is always in that next thing which hasn’t yet been programmed. This “Theorem” was first proposed to me by Larry Tesler, so I call it Tesler’s Theorem: “AI is whatever hasn’t been done yet.”
- ambicapter 2mo ago
  Sure, or maybe the actual applications of “AI” are small and unobtrusive, like the dictation of doctor’s notes example, and it’s not actually the massive revolution it’s claimed to be.
  - jaggederest 2mo ago
    I struggle to reconcile this attitude with, among other things, the massive number of exploits being discovered and the pretty straightforward utility for coding. Have you used codex recently?
- jerf 2mo ago
  In this moment, the opposite is happening. Everything is getting called "AI", whether it uses LLMs to prompt LLMs about how to prompt LLMs, uses "conventional" machine learning, or just looks mysterious enough that they can expect the market to not ask questions. I am reminded of "game AI", which for the most part has historically been just giant decision trees, encoded one way or another, because if you hook up any sort of real AI to a game entity or collection of game entities that does any sort of learning or training, even simple 1980s-era reinforcement learning, it turns out the game entities will roflstomp the human players, and the human players aren't interested in paying for that experience. We've been calling those collections of if statements and for loops "AI" for a long time, though, because who wants to hear about how deliberately stupid their opponents are?
- inigyou 2mo ago
  Because AI meant "make computers smart like people". We taught the computer to spell check, but the computer still didn't feel smart like a person, just smart like a machine so that obviously wasn't AI. We taught it to do algebra, same thing. With LLMs though, now it really does feel like an artificial human, so this time it really is AI.
  - jaggederest 2mo ago
    Right, but I remember when spellcheck first started to be a thing, and people were like "Wow proofreaders are out of a job, ai is so cool", but then it became normal.
- variadix 2mo ago
  Perhaps, but I think what people mean by intelligence is something that learns and adapts. If LLMs couldn’t do in context learning I don’t think people would think of them as AI, more as a kind of queryable database via natural language. There are other algorithms that learn and adapt, but in much more narrow circumstances that most people won’t obviously interact with.
  - jaggederest 2mo ago
    Right, and my point is that now that is the bar. If you'd told me in 2008 that markov chains would be smart enough to one-shot even a trivial video game from scratch I would certainly have called that AI even if you had to do some major rigamarole to get it to work.

jdlshore 2mo ago

Their company does data projects. That plus context makes me think they’re talking about internal work process automation type of work, although it also seems like they’re talking about conversational interfaces (chatbots). I completely buy the “emperor’s new clothes” argument for work process automation. I’m surprised they don’t address AI-assisted engineering, which seems to be going positively for a lot of folks (although I have doubts about its sustainability). I disagree about the success of chatbots, if the problem is narrowly-defined and chosen properly. My previous company built a conversational interface to a vector database and saw good results. (Although, arguably, the vector database was the real magic, and a traditional UI would have been faster and more accurate.) In general, I think OP is more right than wrong, though, particularly about the AI mania and unrealistic expectations sweeping the C-suite.

skydhash 2mo ago
> I disagree about the success of chatbots, if the problem is narrowly-defined and chosen properly. If you can narrow the problem down, then you could design a much better interface for it than a text box and free form text (unless that's the better solution). As for as AI assisted engineering goes, the thing is that after some time with a project, you already have much of the workflows and routines nailed down as scripts and other various combinations of tooling. And unless it's spaghetti code, you will have various snippets you can copy from for new code. The one thing I've observed about AI projects is that there's often little technical design coherence about them. It's always a kitchen sink of technologies and practices.
- jdlshore 2mo ago
  > If you can narrow the problem down, then you could design a much better interface for it than a text box and free form text (unless that's the better solution). Yes, I agree, in that the chatbot we built probably would have worked just as well with a traditional UI, and would have been done a lot faster. But it would have been a lot less sexy (actually important for the bottom line!) and there are future directions that could take advantage of the conversational interface that’s potentially better than a traditional UI. On the down side, good chatbots are really frikkin difficult to write. These things (LLMs) are not reliable at scale. The basic functionality came together in weeks. Getting it to behave consistently and obey guardrails took months, and even then we had to accept a low level of failed conversations.

Aurornis 2mo ago

They run a consulting company called "Hermit Tech". Their websites has an olde style font. They boast about "ancient techniques" from books written prior to the year 2000 > For non-executive management who might be struggling to deliver things that feel beyond their control, we have ancient techniques (see: books written between 1986 and 1999) to turn your team into the envy of the organisation, and we can drop in directly to get your team the resources it needs to save a struggling project. So yeah, of course these people hate AI and everything about it. No serious company is reaching out to these people for help with their AI project.