22 ms·

Just yesterday I had a moment Claude's code in a conversation said - “Yes. I just looked at tag names and sorted them by gut feeling into buckets. No systema

Just yesterday I had a moment Claude's code in a conversation said - “Yes. I just looked at tag names and sorted them by gut feeling into buckets. No systematic reasoning behind it.” It has gut feelings now? I confronted for a minute - but pulled out. I walked away from my desk for an hour to not get pulled into the AInsanity.

boxedemp 6mo ago
It has a lot. I find by challenging it often, getting it to explain it's assumptions, it's usually guessing. This can be overcome by continuously asking it to justify everything, but even then...
- aisengard 6mo ago
  It's almost like an emergent feature of a tool that's literally built on best guesses is...guesswork. Not what you want out of a tool that's supposed to be replacing professionals!
  - boxedemp 6mo ago
    Interesting perspective. I guess I'm more interested in understanding what it can and can't do.
- reg_dunlop 6mo ago
  Trust shouldn't be inherent in our adoption of these models. However, constant skepticism is an interesting habit to develop. I agree, continually asking it to justify may seem tiresome, especially if there's a deadline. Though with less pressure, "slow is smooth...". Just this evening, a model gave an example of 2 different things with a supposed syntax difference, with no discernible syntax difference to my eyes. While prompting for a 'sanity check', the model relented: "oops, my bad; i copied the same line twice". smh
  - boxedemp 6mo ago
    I don't find it tiresome at all. What I was getting at was, even with constant justifications you need to remain vigilant.
unselect5917 6mo ago
>It has gut feelings now? I would say hard no. It doesn't. But it's been trained on humans saying that in explaining their behavior, so that is "reasonable" text to generate and spit out at you. It has no concept of the idea that a human-serving language model should not be saying it to a human because it's not a useful answer. It doesn't know that it's not a useful answer. It knows that based on the language its been trained on that's a "reasonable" (in terms of matrix math, not actual reasoning) response. Way too many people think that it's really thinking and I don't think that most of them are. My abstract understanding is that they're basically still upjumped Markov chains.
Phlogistique 6mo ago
Even when used by humans, "gut feelings" is still a metaphor.

boxedemp 6mo ago

It has a lot. I find by challenging it often, getting it to explain it's assumptions, it's usually guessing. This can be overcome by continuously asking it to justify everything, but even then...

aisengard 6mo ago
It's almost like an emergent feature of a tool that's literally built on best guesses is...guesswork. Not what you want out of a tool that's supposed to be replacing professionals!
- boxedemp 6mo ago
  Interesting perspective. I guess I'm more interested in understanding what it can and can't do.
reg_dunlop 6mo ago
Trust shouldn't be inherent in our adoption of these models. However, constant skepticism is an interesting habit to develop. I agree, continually asking it to justify may seem tiresome, especially if there's a deadline. Though with less pressure, "slow is smooth...". Just this evening, a model gave an example of 2 different things with a supposed syntax difference, with no discernible syntax difference to my eyes. While prompting for a 'sanity check', the model relented: "oops, my bad; i copied the same line twice". smh
- boxedemp 6mo ago
  I don't find it tiresome at all. What I was getting at was, even with constant justifications you need to remain vigilant.

unselect5917 6mo ago

>It has gut feelings now? I would say hard no. It doesn't. But it's been trained on humans saying that in explaining their behavior, so that is "reasonable" text to generate and spit out at you. It has no concept of the idea that a human-serving language model should not be saying it to a human because it's not a useful answer. It doesn't know that it's not a useful answer. It knows that based on the language its been trained on that's a "reasonable" (in terms of matrix math, not actual reasoning) response. Way too many people think that it's really thinking and I don't think that most of them are. My abstract understanding is that they're basically still upjumped Markov chains.

Phlogistique 6mo ago

Even when used by humans, "gut feelings" is still a metaphor.