6 ms·

LLMs and their capabilities are very impressive and definitely useful. The productivity gains often seem to be smaller than intuitively expected though. For exa

LLMs and their capabilities are very impressive and definitely useful. The productivity gains often seem to be smaller than intuitively expected though. For example, using ChatGPT to get a response to a random question like "How do I do XYZ" is much more convenient than googling it, but the time savings are often not that relevant for your overall productivity. Before LLMs you were usually already able to find the information quickly and even a 10x speed up does not really have too much of an impact on your overall productivity, because the time it took was already negligible.

direwolf20 8mo ago
This is partly because Google is past the enshittification hump and ChatGPT is just starting to climb up it - they just announced ads.
- robofanatic 8mo ago
  > they just announced ads wondering how is it going to work when they "search the web" to get the information, are they essentially going to take ad revenue away from the source website?
- hodgesrm 8mo ago
  This. And the wonderful thing about LLMs is that they can be trained to bend responses in specific directions, say toward using Oracle Cloud solutions. There's fertile ground for commercial value extraction that goes far beyond ads. Think of it as product placement on steroid.
  - direwolf20 8mo ago
    You don't even need training — you can add steering vectors in the middle of the otherwise-unmodified computation. Remember Golden Gate Claude?
- foobarchu 8mo ago
  Not to be a dick, but enshittification is not a hump you get past, it's a constant climb until the product is abandoned. Did you just mean growing pains?
linuxftw 8mo ago
You're overestimating the mean person's ability to search the web effectively.
- jgalt212 8mo ago
  And perhaps both are overestimating the mean person's ability to detect a hallucinated solution vs a genuine one.
  - linuxftw 8mo ago
    I think hallucination is grossly overstated as a problem at this point, most models will actively search the web and reason about the results. You're much more likely to get the incorrect solution browsing stack overflow than you are asking AI.
    - danudey 8mo ago
      Gemini hallucinated a method name in a rust crate then spent several minutes googling the method name + 'rust example' trying to find documentation about the method it made up. Unsurprisingly it didn't find any, and then it just gave up and commented out the entire function and called it done.
      - linuxftw 8mo ago
        Comparing the free tier of Gemini to the latest premium coding models will give you drastically different results.
avaer 8mo ago
The difference is LLMs let you "run Google" on your own data with copy paste. Which you could not do before. If you're using ChatGPT like you use Google then I agree with you. But IMO comparing ChatGPT to Google means you haven't had the "aha" moment yet. As a concrete example, a lot of my work these days involves asking ChatGPT to produce me an obscure micro-app to process my custom data. Which it usually does and renders in one shot. This app could not exist before I asked for it. The productivity gains over coding this myself are immense. And the experience is nothing like using Google.
- bryanrasmussen 8mo ago
  there have been various solutions that allow you to "run Google" on your own data for quite a while, what is the "aha" moment related to that?
  - avaer 8mo ago
    By "run Google" I don't mean "index your data into a search engine". I mean the experience of being able to semantically extract and process data at "internet scale", in seconds. It might seem quaint today but one example might be fact checking a piece of text. Google effectively has a pretty good internal representation of whether any particular document concords with other documents on the internet, on account of massive crawling and indexing over decades. But LLMs let you run the same process nearly instantly on your own data, and that's the difference.
  - dumbmrblah 8mo ago
    But before I needed to be a programmer or have a team of data analysts analyze the data for me, now I can just process that data on my own and gather my own insights. That was my aha moment.
- MadDemon 8mo ago
  It's great for you that you were able to create this app that wouldn't otherwise exist, but does that app dramatically increase your overall productivity? And can you imagine that a significant chunk of the population would experience a similar productivity boost? I'm not saying that there is no productivity gain, but big tech has promised MASSIVE productivity gains. I just feel like the productivity gains are more modest for now, similar to other technologies. Maybe one day AGI comes along and changes everything, but I feel like we'll need a few more break throughs before that.
drzaiusx11 8mo ago
If only search engine AI output didn't constantly haluciate nonexistent APIs, it might be a net productivity gain for me...but it's not. I've been bit enough times by their false "example" output for it to be a significant net time loss vs using traditional search results.
- skybrian 8mo ago
  Using ChatGPT and phrasing it like a search seems like a better way? “Can you find documentation about an API that does X?”
  - lazide 8mo ago
    It will often literally just make up the documentation. If you ask for a link, it may hallucinate the link. And unlike a search engine where someone had to previously think of, and then make some page with the fake content on it, it will happily make it up on the fly so you'll end up with a new/unique bit of fake documentation/url! At that point, you would have been way better off just... using a search engine?
    - taude 8mo ago
      how is it hallucinating links? The links are direct links to the webpage that they vectorized or whatever as input to the LLM query. In fact, on almost all LLM responses DuckDuckGo and Google, the links are right there as sited sources that you click on (i know because I'm almost always clicking on the source link to read the original details, and not the made up one
      - madcaptenor 8mo ago
        I would imagine links can be hallucinated because the original URLs in the training data get broken up into tokens - so it's not hard to come up with a URL that has the right format (say https://arxiv.org/abs/2512.01234 https://arxiv.org/abs/2512.01234 - which is a real paper but I just made up that URL) and a plausible-sounding title.
sylware 8mo ago
That makes me think about the development of much software out there: the development time is often several orders of magnitude smaller than its life cycle.
skybrian 8mo ago
I use ChatGPT “thinking” mode as a way to run multiple searches and summarize the results. It takes some time, but I can do other stuff in another tab and come back. It’s for queries that are unlikely to be satisfied in a single search. I don’t think it would be a negligible amount of time if you did it yourself.
- Gud 8mo ago
  This is the way. I do it the same way for development. The main point is I can run multiple tasks in parallel(myself + LLM(s)). I let Claude and ChatGPT type out code for me, while I focus on my research
- entropicdrifter 8mo ago
  I find Gemini to be the most consistent at actually using the search results for this, in "Deep Research" mode
- Incipient 8mo ago
  But for large searches, I then have to spend a lot of time validating the output - which I'd normally do while reading the content etc as I searched (discarding dodgy websites etc). On the other hand, where I think llms are going to excel, is you roll the dice, trust the output, and don't validate it. If it works out yayy you're ahead of everyone else that did bother to validate it. I think this is how vibe coded apps are going to go. If the app blows up, shut down the company and start a new one.
binary132 8mo ago
The difference is that in the past that information had to come from what people wrote and are writing about, and now it can come from a derivative of an archive of what people once wrote, upon a time. So if they just stop doing that — whether because they must, or because they no longer have any reason to, or because they are now drowned out in a massive ocean of slop, or simply because they themselves have turned into slopslaves — no new information will be generated, only derivative slop, milled from derivative slop. I think we all understand that at this point, so I question deeply why anyone acts like they don’t.
HarHarVeryFunny 8mo ago
> For example, using ChatGPT to get a response to a random question like "How do I do XYZ" is much more convenient than googling it More convenient than traditional search? Maybe. Quicker than traditional search? Maybe not. Asking random questions is exactly where you run into time-wasting hallucinations since the models don't seem to be very good at deciding when to use a search tool and when just to rely on their training data. For example, just now I was asking Gemini how to fix a bunch of Ubuntu/Xfce annoyances after a major upgrade, and it was a very mixed bag. One example: the default date and time display is in an unreadably small "date stacked over time" format (using a few pixel high font so this fits into the menu bar), and Gemini's advice was to enable the "Display date and time on single line" option ... but there is no such option (it just hallucinated it), and it also hallucinated a bunch of other suggestions until I finally figured out what you need to do is to configure it to display "Time only" rather than "Data and Time", then change the "Time" format to display both data and time! Just to experiment, I then told Gemini about this fix and amusingly the response was basically "Good to know - this'll be useful for anyone reading this later"! More examples, from yesterday (these are not rare exceptions): 1) I asked Gemini (generally considered one of the smartest models - better than ChatGPT, and rapidly taking away market share from it - 20% shift in last month or so) to look at the GitHub codebase for an Anthropic optimization challenge, to summarize and discuss etc, and it appeared to have looked at the codebase until I got more into the weeds and was questioning it where it got certain details from (what file), and it became apparent it had some (search based?) knowledge of the problem, but seemingly hadn't actually looked at it (wasn't able to?). 2) I was asking Gemini about chemically fingerprinting (via impurities, isotopes) roman silver coins to the mines that produced the silver, and it confidently (as always) comes up with a bunch of academic references that it claimed made the connection, but none or references (which did at least exist) actually contained what it claimed (just partial information), and when I pointed this out it just kept throwing out different references. So, it's convenient to be able to chat with your "search engine" to drill down and clarify, etc, but a big time waste if a lot of it is hallucination. Search vs Chat has anyways really become a difference without a difference since Google now gives you the "AI Overview" (a diving off point into "AI Mode"), or you can just click on "AI Mode" in the first place - which is Gemini.
- NoGravitas 8mo ago
palmotea 8mo ago
> For example, using ChatGPT to get a response to a random question like "How do I do XYZ" is much more convenient than googling it, but the time savings are often not that relevant for your overall productivity. Before LLMs you were usually already able to find the information quickly and even a 10x speed up does not really have too much of an impact on your overall productivity, because the time it took was already negligible. I'd even question that. The pre-LLM solutions were in most cases better. Searching a maintained database of curated and checked information is far better than LLM output (which is possibly bullshit). Ditto to software engineering. In software, we have things call libraries: you write the code once, test it, then you trust it and can use it as many times as you want forever for free. Why use LLM generated code when you have a library? And if you're asking for anything complex, you're probably just getting a plagiarized and bastardized version of some library anyway. The only thing where LLMs shine is a kind of simple, lazy "mash this up so I don't have to think about it" cases. And sometimes it might be better to just do it yourself and develop your own skills instead of use an LLM.
- mountainriver 8mo ago
  Why not have the LLM generate the library?
  - javcasas 8mo ago
    Because every time you run the LLM it will generate a new library, with new and surprising bugs. It's better to take an existing, already curated and tested library. Which, yes, may have been generated by an LLM, but has been curated beyond the skill of the LLM.
  - techblueberry 8mo ago
    The value of a library is not in the code it’s in the operations. It’s been curated and tested by multiple people in multiple environments. One of the ways to code at a high level is to delegate the cognitive load. If you really can one shot it and it’s simple(left-pad). Great. But most things aren’t my that simple, the third time you have to think about it, it’s probably a net loss.
- 8mo ago
CyberDildonics 8mo ago
The real benefit to a search engine is to rework and launder other people's information and make it your information. Now instead of the wikipedia article you are reading the exact same thing from google's home page and you don't click on anything.
robofanatic 8mo ago
It really depends on what 'XYZ' is and how many hoops you need to jump through to get to the answer. ChatGPT gets information from various places and gives you the answer as well as the explanation at each step. Without tools like ChatGPT its definitely not negligible in a lot of cases.
SkiFire13 8mo ago
I think you're underestimating how many people don't know how to properly search on google (i.e. finding the proper keywords, selecting the reputable results, etc etc). Those are probably also the same people that will blindly believe anything a LLM says unfortunately.
- 1718627440 8mo ago
  True, I do not know how two properly search something on google.com in 2025. I only know how to do it on startpage.com in 2025, kagi.com in 2025 or google.com in 2015.
zelos 8mo ago
LLM output is quickly rendering google search unusable, so it's kind of creating its own speedup multiplier.

direwolf20 8mo ago

This is partly because Google is past the enshittification hump and ChatGPT is just starting to climb up it - they just announced ads.

robofanatic 8mo ago
> they just announced ads wondering how is it going to work when they "search the web" to get the information, are they essentially going to take ad revenue away from the source website?
hodgesrm 8mo ago
This. And the wonderful thing about LLMs is that they can be trained to bend responses in specific directions, say toward using Oracle Cloud solutions. There's fertile ground for commercial value extraction that goes far beyond ads. Think of it as product placement on steroid.
- direwolf20 8mo ago
  You don't even need training — you can add steering vectors in the middle of the otherwise-unmodified computation. Remember Golden Gate Claude?
foobarchu 8mo ago
Not to be a dick, but enshittification is not a hump you get past, it's a constant climb until the product is abandoned. Did you just mean growing pains?

linuxftw 8mo ago

You're overestimating the mean person's ability to search the web effectively.

jgalt212 8mo ago
And perhaps both are overestimating the mean person's ability to detect a hallucinated solution vs a genuine one.
- linuxftw 8mo ago
  I think hallucination is grossly overstated as a problem at this point, most models will actively search the web and reason about the results. You're much more likely to get the incorrect solution browsing stack overflow than you are asking AI.
  - danudey 8mo ago
    Gemini hallucinated a method name in a rust crate then spent several minutes googling the method name + 'rust example' trying to find documentation about the method it made up. Unsurprisingly it didn't find any, and then it just gave up and commented out the entire function and called it done.
    - linuxftw 8mo ago
      Comparing the free tier of Gemini to the latest premium coding models will give you drastically different results.

avaer 8mo ago

The difference is LLMs let you "run Google" on your own data with copy paste. Which you could not do before. If you're using ChatGPT like you use Google then I agree with you. But IMO comparing ChatGPT to Google means you haven't had the "aha" moment yet. As a concrete example, a lot of my work these days involves asking ChatGPT to produce me an obscure micro-app to process my custom data. Which it usually does and renders in one shot. This app could not exist before I asked for it. The productivity gains over coding this myself are immense. And the experience is nothing like using Google.

bryanrasmussen 8mo ago
there have been various solutions that allow you to "run Google" on your own data for quite a while, what is the "aha" moment related to that?
- avaer 8mo ago
  By "run Google" I don't mean "index your data into a search engine". I mean the experience of being able to semantically extract and process data at "internet scale", in seconds. It might seem quaint today but one example might be fact checking a piece of text. Google effectively has a pretty good internal representation of whether any particular document concords with other documents on the internet, on account of massive crawling and indexing over decades. But LLMs let you run the same process nearly instantly on your own data, and that's the difference.
- dumbmrblah 8mo ago
  But before I needed to be a programmer or have a team of data analysts analyze the data for me, now I can just process that data on my own and gather my own insights. That was my aha moment.
MadDemon 8mo ago
It's great for you that you were able to create this app that wouldn't otherwise exist, but does that app dramatically increase your overall productivity? And can you imagine that a significant chunk of the population would experience a similar productivity boost? I'm not saying that there is no productivity gain, but big tech has promised MASSIVE productivity gains. I just feel like the productivity gains are more modest for now, similar to other technologies. Maybe one day AGI comes along and changes everything, but I feel like we'll need a few more break throughs before that.

drzaiusx11 8mo ago

If only search engine AI output didn't constantly haluciate nonexistent APIs, it might be a net productivity gain for me...but it's not. I've been bit enough times by their false "example" output for it to be a significant net time loss vs using traditional search results.

skybrian 8mo ago
Using ChatGPT and phrasing it like a search seems like a better way? “Can you find documentation about an API that does X?”
- lazide 8mo ago
  It will often literally just make up the documentation. If you ask for a link, it may hallucinate the link. And unlike a search engine where someone had to previously think of, and then make some page with the fake content on it, it will happily make it up on the fly so you'll end up with a new/unique bit of fake documentation/url! At that point, you would have been way better off just... using a search engine?
  - taude 8mo ago
    how is it hallucinating links? The links are direct links to the webpage that they vectorized or whatever as input to the LLM query. In fact, on almost all LLM responses DuckDuckGo and Google, the links are right there as sited sources that you click on (i know because I'm almost always clicking on the source link to read the original details, and not the made up one
    - madcaptenor 8mo ago
      I would imagine links can be hallucinated because the original URLs in the training data get broken up into tokens - so it's not hard to come up with a URL that has the right format (say https://arxiv.org/abs/2512.01234 https://arxiv.org/abs/2512.01234 - which is a real paper but I just made up that URL) and a plausible-sounding title.

sylware 8mo ago

That makes me think about the development of much software out there: the development time is often several orders of magnitude smaller than its life cycle.

skybrian 8mo ago

I use ChatGPT “thinking” mode as a way to run multiple searches and summarize the results. It takes some time, but I can do other stuff in another tab and come back. It’s for queries that are unlikely to be satisfied in a single search. I don’t think it would be a negligible amount of time if you did it yourself.

Gud 8mo ago
This is the way. I do it the same way for development. The main point is I can run multiple tasks in parallel(myself + LLM(s)). I let Claude and ChatGPT type out code for me, while I focus on my research
entropicdrifter 8mo ago
I find Gemini to be the most consistent at actually using the search results for this, in "Deep Research" mode
Incipient 8mo ago
But for large searches, I then have to spend a lot of time validating the output - which I'd normally do while reading the content etc as I searched (discarding dodgy websites etc). On the other hand, where I think llms are going to excel, is you roll the dice, trust the output, and don't validate it. If it works out yayy you're ahead of everyone else that did bother to validate it. I think this is how vibe coded apps are going to go. If the app blows up, shut down the company and start a new one.

binary132 8mo ago

The difference is that in the past that information had to come from what people wrote and are writing about, and now it can come from a derivative of an archive of what people once wrote, upon a time. So if they just stop doing that — whether because they must, or because they no longer have any reason to, or because they are now drowned out in a massive ocean of slop, or simply because they themselves have turned into slopslaves — no new information will be generated, only derivative slop, milled from derivative slop. I think we all understand that at this point, so I question deeply why anyone acts like they don’t.

HarHarVeryFunny 8mo ago

> For example, using ChatGPT to get a response to a random question like "How do I do XYZ" is much more convenient than googling it More convenient than traditional search? Maybe. Quicker than traditional search? Maybe not. Asking random questions is exactly where you run into time-wasting hallucinations since the models don't seem to be very good at deciding when to use a search tool and when just to rely on their training data. For example, just now I was asking Gemini how to fix a bunch of Ubuntu/Xfce annoyances after a major upgrade, and it was a very mixed bag. One example: the default date and time display is in an unreadably small "date stacked over time" format (using a few pixel high font so this fits into the menu bar), and Gemini's advice was to enable the "Display date and time on single line" option ... but there is no such option (it just hallucinated it), and it also hallucinated a bunch of other suggestions until I finally figured out what you need to do is to configure it to display "Time only" rather than "Data and Time", then change the "Time" format to display both data and time! Just to experiment, I then told Gemini about this fix and amusingly the response was basically "Good to know - this'll be useful for anyone reading this later"! More examples, from yesterday (these are not rare exceptions): 1) I asked Gemini (generally considered one of the smartest models - better than ChatGPT, and rapidly taking away market share from it - 20% shift in last month or so) to look at the GitHub codebase for an Anthropic optimization challenge, to summarize and discuss etc, and it appeared to have looked at the codebase until I got more into the weeds and was questioning it where it got certain details from (what file), and it became apparent it had some (search based?) knowledge of the problem, but seemingly hadn't actually looked at it (wasn't able to?). 2) I was asking Gemini about chemically fingerprinting (via impurities, isotopes) roman silver coins to the mines that produced the silver, and it confidently (as always) comes up with a bunch of academic references that it claimed made the connection, but none or references (which did at least exist) actually contained what it claimed (just partial information), and when I pointed this out it just kept throwing out different references. So, it's convenient to be able to chat with your "search engine" to drill down and clarify, etc, but a big time waste if a lot of it is hallucination. Search vs Chat has anyways really become a difference without a difference since Google now gives you the "AI Overview" (a diving off point into "AI Mode"), or you can just click on "AI Mode" in the first place - which is Gemini.

NoGravitas 8mo ago

palmotea 8mo ago

> For example, using ChatGPT to get a response to a random question like "How do I do XYZ" is much more convenient than googling it, but the time savings are often not that relevant for your overall productivity. Before LLMs you were usually already able to find the information quickly and even a 10x speed up does not really have too much of an impact on your overall productivity, because the time it took was already negligible. I'd even question that. The pre-LLM solutions were in most cases better. Searching a maintained database of curated and checked information is far better than LLM output (which is possibly bullshit). Ditto to software engineering. In software, we have things call libraries: you write the code once, test it, then you trust it and can use it as many times as you want forever for free. Why use LLM generated code when you have a library? And if you're asking for anything complex, you're probably just getting a plagiarized and bastardized version of some library anyway. The only thing where LLMs shine is a kind of simple, lazy "mash this up so I don't have to think about it" cases. And sometimes it might be better to just do it yourself and develop your own skills instead of use an LLM.

mountainriver 8mo ago
Why not have the LLM generate the library?
- javcasas 8mo ago
  Because every time you run the LLM it will generate a new library, with new and surprising bugs. It's better to take an existing, already curated and tested library. Which, yes, may have been generated by an LLM, but has been curated beyond the skill of the LLM.
- techblueberry 8mo ago
  The value of a library is not in the code it’s in the operations. It’s been curated and tested by multiple people in multiple environments. One of the ways to code at a high level is to delegate the cognitive load. If you really can one shot it and it’s simple(left-pad). Great. But most things aren’t my that simple, the third time you have to think about it, it’s probably a net loss.
8mo ago

CyberDildonics 8mo ago

The real benefit to a search engine is to rework and launder other people's information and make it your information. Now instead of the wikipedia article you are reading the exact same thing from google's home page and you don't click on anything.

robofanatic 8mo ago

It really depends on what 'XYZ' is and how many hoops you need to jump through to get to the answer. ChatGPT gets information from various places and gives you the answer as well as the explanation at each step. Without tools like ChatGPT its definitely not negligible in a lot of cases.

SkiFire13 8mo ago

I think you're underestimating how many people don't know how to properly search on google (i.e. finding the proper keywords, selecting the reputable results, etc etc). Those are probably also the same people that will blindly believe anything a LLM says unfortunately.

1718627440 8mo ago
True, I do not know how two properly search something on google.com in 2025. I only know how to do it on startpage.com in 2025, kagi.com in 2025 or google.com in 2015.

zelos 8mo ago

LLM output is quickly rendering google search unusable, so it's kind of creating its own speedup multiplier.