7 ms·

I totally agree with the author. Sadly, I feel like that's not what the majority of LLM users tend to view LLMs. And it's definitely not what AI companies marke

I totally agree with the author. Sadly, I feel like that's not what the majority of LLM users tend to view LLMs. And it's definitely not what AI companies marketing. > The key thing is to develop an intuition for questions it can usefully answer vs questions that are at a level of detail where the lossiness matters the problem is that in order to develop an intuition for questions that LLMs can answer, the user will at least need to know something about the topic beforehand. I believe that this lack of initial understanding of the user input is what can lead to taking LLM output as factual. If one side of the exchange knows nothing about the subject, the other side can use jargon and even present random facts or lossy facts which can almost guarantee to impress the other side. > The way to solve this particular problem is to make a correct example available to it. My question is how much effort would it take to make a correct example available for the LLM before it can output quality and useful data? If the effort I put in is more than what I would get in return, then I feel like it's best to write and reason it myself.

theshrike79 1y ago
> the problem is that in order to develop an intuition for questions that LLMs can answer, the user will at least need to know something about the topic beforehand This is why simonw (The author) has his "pelican on a bike" -test, it's not 100% accurate but it is a good indicator. I have a set of my own standard queries and problems (no counting characters or algebra crap) I feed to new LLMs I'm testing None of the questions exist outside of my own Obsidian note so they can't be gamed by LLM authors. And I've tested multiple different LLMs using them so I have a "feeling" on what the answer should look like. And I personally know the correct answer so I can immediately validate them.
- barapa 1y ago
  They are training on your queries. So they may have some exposure to them going forward.
  - keysdev 1y ago
    Not if one ollama pull to ur machine.
  - franktankbank 1y ago
    Even if your queries are hidden via a local running model you must have some humility that your queries are not actually unique. For this reason I have a very difficult time believing that a basic LLM will be able to properly reason about complex topics, it can regurgitate to whatever level its been trained. That doesn't make it less useful though. But on the edge case how do we know the query its ingesting gets trained with a suitable answer? Wouldn't this constitute an over-fitting in these cases and be terribly self-reinforcing?
giancarlostoro 1y ago
> the user will at least need to know something about the topic beforehand. This is why I've said a few times here on HN and elsewhere, if you're using an LLM you need to think of yourself as an architect guiding a Junior to Mid Level developer. Juniors can do amazing things, they can also goof up hard. What's really funny is you can make them audit their own code in a new context window, and give you a detailed answer as to why that code is awful. I use it mostly on personal projects especially since I can prototype quickly as needed.
- skydhash 1y ago
  > if you're using an LLM you need to think of yourself as an architect guiding a Junior to Mid Level developer. The thing is coding can (and should) be part of the design process. Many times, I though I have a good idea of what the solution should look like, then while coding, I got exposed more to the libraries and other parts of the code, which led me to a more refined approach. This exposure is what you will miss and it will quickly result in unfamiliar code.
  - giancarlostoro 1y ago
    I agree. I mostly use it for scaffolding, I don't like letting it do all the work for me.
    - codr7 1y ago
      No friction, no improvements; that only guarantees you'll never find a better way to solve the problem.
cj 1y ago
> the user will at least need to know something about the topic beforehand. I used ChatGPT 5 over the weekend to double check dosing guidelines for a specific medication. "Provide dosage guidelines for medication [insert here]" It spit back dosing guidelines that were an order of magnitude wrong (suggested 100mcg instead of 1mg). When I saw 100mcg, I was suspicious and said "I don't think that's right" and it quickly corrected itself and provided the correct dosing guidelines. These are the kind of innocent errors that can be dangerous if users trust it blindly. The main challenge is LLMs aren't able to gauge confidence in its answers, so it can't adjust how confidently it communicates information back to you. It's like compressing a photo, and a photographer wrongly saying "here's the best quality image I have!" - do you trust the photographer at their word, or do you challenge him to find a better quality image?
- dncornholio 1y ago
  Using a LLM for medical research is just as dangerous as Googling it. Always ask your doctors!
  - yojo 1y ago
    This is the terrifying part: doctors do this too! I have an MD friend that told me she uses ChatGPT to retrieve dosing info. I asked her to please, please not do that.
    - ozgrakkurt 1y ago
      Find good doctors. A solution doesn’t have to be perfect. A doctor doing better than regular joe with a computer is much higher as you can see in research around this topic
      - SequoiaHope 1y ago
        I have noticed that my doctor is getting busier and busier lately. I worry that cost cutting will have doctors so frantic that they are forced to rely on things like ChatGPT, and “find good doctors” will be an option only for an elite few.
netcan 1y ago
>the problem is that in order to develop an intuition for questions that LLMs can answer, the user will at least need to know something about the topic beforehand. I believe that this lack of initial understanding of the user input I think there's a parallel here for the internet as an i formation source. It delivered on "unlimited knowledge at the tip of everyone's fingertips" but lowering the bar also lowered the bar. That access "works" only when the user is capable of doing their part too. Evaluating sources, integrating knowledge. Validating. Cross examining. Now we are just more used to recognizing that accessibility comes with its own problem. Some of this is down to general education. Some to domain expertize. Personality plays a big part. The biggest factor is, i think, intelligence. There's a lot of 2nd and 3rd order thinking required to simultaneously entertain a curiosity, consider of how the LLM works, and exercise different levels of skepticism depending on the types of errors LLMs are likely to make. Using LLMs correctly and incorrectly is.. subtle.
HarHarVeryFunny 1y ago
> The key thing is to develop an intuition for questions it can usefully answer vs questions that are at a level of detail where the lossiness matters It's also useful to have an intuition for what things an LLM is liable to get wrong/hallucinate, one of which is questions where the question itself suggests one or more obvious answers (which may or may not be correct), which the LLM may well then hallucinate, and sound reasonable, if it doesn't "know".
- felipeerias 1y ago
  LLMs are very sensitive to leading questions. A small hint of that the expected answer looks like will tend to produce exactly that answer.
  - SAI_Peregrinus 1y ago
    As a consequence LLMs are extremely unlikely to recognize an X-Y problem.
  - giantrobot 1y ago
    You don't even need a leading direct question. You can easily lead an LLM just by having some statements (even at times single words) in the context window.
bobbylarrybobby 1y ago
https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect
- puchatek 1y ago
  I was thinking the exact same thing.
estimator7292 1y ago
It's really strange to me that the only way to effectively use LLMs is if you already have all the knowledge and skill to do the task yourself. I can't think of any other tools like this. An LLM can multiply your efforts, but only if you were capable of doing it yourself. Wild.

theshrike79 1y ago

> the problem is that in order to develop an intuition for questions that LLMs can answer, the user will at least need to know something about the topic beforehand This is why simonw (The author) has his "pelican on a bike" -test, it's not 100% accurate but it is a good indicator. I have a set of my own standard queries and problems (no counting characters or algebra crap) I feed to new LLMs I'm testing None of the questions exist outside of my own Obsidian note so they can't be gamed by LLM authors. And I've tested multiple different LLMs using them so I have a "feeling" on what the answer should look like. And I personally know the correct answer so I can immediately validate them.

barapa 1y ago
They are training on your queries. So they may have some exposure to them going forward.
- keysdev 1y ago
  Not if one ollama pull to ur machine.
- franktankbank 1y ago
  Even if your queries are hidden via a local running model you must have some humility that your queries are not actually unique. For this reason I have a very difficult time believing that a basic LLM will be able to properly reason about complex topics, it can regurgitate to whatever level its been trained. That doesn't make it less useful though. But on the edge case how do we know the query its ingesting gets trained with a suitable answer? Wouldn't this constitute an over-fitting in these cases and be terribly self-reinforcing?

giancarlostoro 1y ago

> the user will at least need to know something about the topic beforehand. This is why I've said a few times here on HN and elsewhere, if you're using an LLM you need to think of yourself as an architect guiding a Junior to Mid Level developer. Juniors can do amazing things, they can also goof up hard. What's really funny is you can make them audit their own code in a new context window, and give you a detailed answer as to why that code is awful. I use it mostly on personal projects especially since I can prototype quickly as needed.

skydhash 1y ago
> if you're using an LLM you need to think of yourself as an architect guiding a Junior to Mid Level developer. The thing is coding can (and should) be part of the design process. Many times, I though I have a good idea of what the solution should look like, then while coding, I got exposed more to the libraries and other parts of the code, which led me to a more refined approach. This exposure is what you will miss and it will quickly result in unfamiliar code.
- giancarlostoro 1y ago
  I agree. I mostly use it for scaffolding, I don't like letting it do all the work for me.
  - codr7 1y ago
    No friction, no improvements; that only guarantees you'll never find a better way to solve the problem.

cj 1y ago

> the user will at least need to know something about the topic beforehand. I used ChatGPT 5 over the weekend to double check dosing guidelines for a specific medication. "Provide dosage guidelines for medication [insert here]" It spit back dosing guidelines that were an order of magnitude wrong (suggested 100mcg instead of 1mg). When I saw 100mcg, I was suspicious and said "I don't think that's right" and it quickly corrected itself and provided the correct dosing guidelines. These are the kind of innocent errors that can be dangerous if users trust it blindly. The main challenge is LLMs aren't able to gauge confidence in its answers, so it can't adjust how confidently it communicates information back to you. It's like compressing a photo, and a photographer wrongly saying "here's the best quality image I have!" - do you trust the photographer at their word, or do you challenge him to find a better quality image?

dncornholio 1y ago
Using a LLM for medical research is just as dangerous as Googling it. Always ask your doctors!
- yojo 1y ago
  This is the terrifying part: doctors do this too! I have an MD friend that told me she uses ChatGPT to retrieve dosing info. I asked her to please, please not do that.
  - ozgrakkurt 1y ago
    Find good doctors. A solution doesn’t have to be perfect. A doctor doing better than regular joe with a computer is much higher as you can see in research around this topic
    - SequoiaHope 1y ago
      I have noticed that my doctor is getting busier and busier lately. I worry that cost cutting will have doctors so frantic that they are forced to rely on things like ChatGPT, and “find good doctors” will be an option only for an elite few.

netcan 1y ago

>the problem is that in order to develop an intuition for questions that LLMs can answer, the user will at least need to know something about the topic beforehand. I believe that this lack of initial understanding of the user input I think there's a parallel here for the internet as an i formation source. It delivered on "unlimited knowledge at the tip of everyone's fingertips" but lowering the bar also lowered the bar. That access "works" only when the user is capable of doing their part too. Evaluating sources, integrating knowledge. Validating. Cross examining. Now we are just more used to recognizing that accessibility comes with its own problem. Some of this is down to general education. Some to domain expertize. Personality plays a big part. The biggest factor is, i think, intelligence. There's a lot of 2nd and 3rd order thinking required to simultaneously entertain a curiosity, consider of how the LLM works, and exercise different levels of skepticism depending on the types of errors LLMs are likely to make. Using LLMs correctly and incorrectly is.. subtle.

HarHarVeryFunny 1y ago

> The key thing is to develop an intuition for questions it can usefully answer vs questions that are at a level of detail where the lossiness matters It's also useful to have an intuition for what things an LLM is liable to get wrong/hallucinate, one of which is questions where the question itself suggests one or more obvious answers (which may or may not be correct), which the LLM may well then hallucinate, and sound reasonable, if it doesn't "know".

felipeerias 1y ago
LLMs are very sensitive to leading questions. A small hint of that the expected answer looks like will tend to produce exactly that answer.
- SAI_Peregrinus 1y ago
  As a consequence LLMs are extremely unlikely to recognize an X-Y problem.
- giantrobot 1y ago
  You don't even need a leading direct question. You can easily lead an LLM just by having some statements (even at times single words) in the context window.

bobbylarrybobby 1y ago

https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect

puchatek 1y ago
I was thinking the exact same thing.

estimator7292 1y ago

It's really strange to me that the only way to effectively use LLMs is if you already have all the knowledge and skill to do the task yourself. I can't think of any other tools like this. An LLM can multiply your efforts, but only if you were capable of doing it yourself. Wild.