5 ms·

I have to wonder how many CEOs and other executives are low-key bouncing their bad ideas off of ChatGPT, not realizing it’s only going to tell them what they wa

I have to wonder how many CEOs and other executives are low-key bouncing their bad ideas off of ChatGPT, not realizing it’s only going to tell them what they want to hear and not give genuine critical feedback.

throwanem 1y ago
"Low-key?" About that. https://futurism.com/openai-investor-chatgpt-mental-health https://futurism.com/openai-investor-chatgpt-mental-health
- Ajedi32 1y ago
  That's bizarre. I wonder if the use of AI was actually a contributing factor to his psychotic break as the article implies, or if the guy was already developing schizophrenia and the chat bot just controlled what direction he went after that. I'm vaguely reminded of people getting sucked down conspiracy theory rabbit holes, though this seems way more extreme in how unhinged it is.
  - throwanem 1y ago
    In form, the conversation he had (which appears to have ended five days ago along with all other public footprint) appears to me very much like a heavily refined and customizable version of "Qanon," [1] complete with intermittent reinforcement. That conspiracy theory was structurally novel in its "growth hacking" style of rollout, where ARG and influencer techniques were leveraged to build interest and develop a narrative in conjunction with the audience. That stuff was incredibly compelling when the Lost producers did it in 2010, and it worked just as well a decade later. Of course, in 2020, it required people behind the scenes doing the work to produce the "drops." Now any LLM can be convinced with a bit of effort to participate in a "role-playing game" of this type with its user, and since Qanon itself was heavily covered and its subject matter broadly archived, even the actual structure is available as a reference. I think it would probably be pretty easy to get an arbitrary model to start spitting out stuff like this, especially if you conditioned the initial context carefully to work around whatever after-the-fact safety measures may be in place, or just use one of the models that's been modified or finetuned to "decensor" it. There are collections of "jailbreak" prompts that go around, and I would expect Mr. Jawline Fillers here to be in social circles where that stuff would be pretty easy to come by. For it to become self-reinforcing doesn't seem too difficult to mentally model from there, and I don't think pre-existing organic disorder is really required. How would anyone handle a machine that specializes in telling them exactly what they want to hear, and never ever gets tired of doing so? Elsewhere in this thread, I proposed a somewhat sanguine mental model for LLMs. Here's another, much less gory, and with which I think people probably are a lot more intuitively familiar: https://harrypotter.fandom.com/wiki/Mirror_of_Erised https://harrypotter.fandom.com/wiki/Mirror_of_Erised [1] https://en.wikipedia.org/wiki/QAnon#Origin_and_spread https://en.wikipedia.org/wiki/QAnon#Origin_and_spread
    - Ajedi32 1y ago
      I love the analogy of the Mirror of Erised. Obviously not quite the same thing, but similar tendencies, and with similar dangers. Very fitting!
      - throwanem 1y ago
        You know, it's odd? I missed the whole initial fad, and only around 2018 or so got around to reading the books to see what all the hype had been about. (Even for a millennial I'm old, and I grew up in a pretty backward corner of the country; I've never played a Pokemon game, either...) So why's it me, and not an actual fan, who should be the one to come up with Rowling's serial-numbers-filed-off Echo and Narcissus as the example?
- dclowd9901 1y ago
  > As such, if he really is suffering a mental health crisis related to his use of OpenAI's product, his situation could serve as an immense optical problem for the company, which has so far downplayed concerns about the mental health of its users. Yikes. Not just an optics* problem, but one has to consider if they're pouring so much money into the company because he feels he "needs" to (whatever basis of coercion exists to support his need to get to the "truth").
- butlike 1y ago
  Is "futurism.com" a trustworthy publication? I've never heard of it. I read the article and it didn't seem like the writing had the hallmarks of top-tier journalism.
  - throwanem 1y ago
    I'm not familiar with the publication either, but the claims I've examined, most notably those relevant to the subject's presently public X.com The Everything App account, appear to check out, as does that the account appears to have been inactive since the day before the linked article was published last week. It isn't clear to me where the reputation of the source becomes relevant.
pjerem 1y ago
I think I have a precise figure : it's a lot.
- ninetyninenine 1y ago
  [flagged]
- sitkack 1y ago
  And we thought having access to someones internet searches was good intel. Now we have a direct feed to their brain stem along with a way to manipulate it. Good thing that narcissistic sociopaths have such a low expression in the overall population.
- isoprophlex 1y ago
  Yeah. If HN is your primary source of in depth AI discussion, you get a pretty balanced take IMO compared to other channels out there. We (the HN crowd) should take into account that if you take "people commenting on HN" as a group, you are implicitly selecting for people that are able to read, parse and contextualise written comment threads. This is NOT your average mid-to-high level corpo management exec, who can for more than 80% (from experience) be placed in the "rise of the business idiot" cohort, fed on prime linkedin brainrot. self-reinforcing hopium addicts with an mba. Nor is it the great masses of random earth dwellers who are not always able to resist excess sugar, nicotine, mcdonalds, youtube, fentanyl, my-car-is-bigger-than-yours credit card capitalism, free pornography, you name it. And now RLHF: Validation as a service. Not sure if humanity is ready for this. (Disclosure: my mum has a chatgpt instance that she named and I'm deeply concerned about the spiritual convos she has with it; random people keep calling me on the level of "can you build me an app that uses llms to predict Funko Pop futures".)
kovezd 1y ago
[flagged]
bpodgursky 1y ago
It depends what model you use. o3 pushes back reasonably well. 4o doesn't.
sanitycheck 1y ago
Then it'll be no different than when they bounce their bad ideas off their human subordinates.
- game_the0ry 1y ago
  This. Their immediate subordinates will be just a as sycophantic, if not more.
  - NoGravitas 1y ago
    cf Celene's Second Law.
petesergeant 1y ago
I feel like I’ve had good results for getting feedback on technical writing by claiming the author is a third party and I need to understand the strengths and weaknesses of their work. I should probably formally test this.
game_the0ry 1y ago
Fun fact, my manager wrote my annual review with our internal LLM tool, which itself is just a wrapper around GTP-4o. (My manager told me when I asked him)
- yojo 1y ago
  This already seems somewhat widespread. I have a friend working at a mid-tier tech co that has a handful of direct reports. He showed me that the interface to his eval app had a “generate review” button, which he clicked, then moved on to the next one. Honestly, I’m fine with this as long as I also get a “generate self review” button. I just wish I could get back all the time I’ve spent massaging a small number of data points into pages of prose.
  - game_the0ry 1y ago
    That makes you wonder why we go through the ritual of an annual review if no one takes it seriously.
Cthulhu_ 1y ago
TBH if they have sycophants and a lot of money it's probably the same. How many bullshit startups have there been, how many dumb ideas came from higher up before LLMs?

throwanem 1y ago

"Low-key?" About that. https://futurism.com/openai-investor-chatgpt-mental-health https://futurism.com/openai-investor-chatgpt-mental-health

Ajedi32 1y ago
That's bizarre. I wonder if the use of AI was actually a contributing factor to his psychotic break as the article implies, or if the guy was already developing schizophrenia and the chat bot just controlled what direction he went after that. I'm vaguely reminded of people getting sucked down conspiracy theory rabbit holes, though this seems way more extreme in how unhinged it is.
- throwanem 1y ago
  In form, the conversation he had (which appears to have ended five days ago along with all other public footprint) appears to me very much like a heavily refined and customizable version of "Qanon," [1] complete with intermittent reinforcement. That conspiracy theory was structurally novel in its "growth hacking" style of rollout, where ARG and influencer techniques were leveraged to build interest and develop a narrative in conjunction with the audience. That stuff was incredibly compelling when the Lost producers did it in 2010, and it worked just as well a decade later. Of course, in 2020, it required people behind the scenes doing the work to produce the "drops." Now any LLM can be convinced with a bit of effort to participate in a "role-playing game" of this type with its user, and since Qanon itself was heavily covered and its subject matter broadly archived, even the actual structure is available as a reference. I think it would probably be pretty easy to get an arbitrary model to start spitting out stuff like this, especially if you conditioned the initial context carefully to work around whatever after-the-fact safety measures may be in place, or just use one of the models that's been modified or finetuned to "decensor" it. There are collections of "jailbreak" prompts that go around, and I would expect Mr. Jawline Fillers here to be in social circles where that stuff would be pretty easy to come by. For it to become self-reinforcing doesn't seem too difficult to mentally model from there, and I don't think pre-existing organic disorder is really required. How would anyone handle a machine that specializes in telling them exactly what they want to hear, and never ever gets tired of doing so? Elsewhere in this thread, I proposed a somewhat sanguine mental model for LLMs. Here's another, much less gory, and with which I think people probably are a lot more intuitively familiar: https://harrypotter.fandom.com/wiki/Mirror_of_Erised https://harrypotter.fandom.com/wiki/Mirror_of_Erised [1] https://en.wikipedia.org/wiki/QAnon#Origin_and_spread https://en.wikipedia.org/wiki/QAnon#Origin_and_spread
  - Ajedi32 1y ago
    I love the analogy of the Mirror of Erised. Obviously not quite the same thing, but similar tendencies, and with similar dangers. Very fitting!
    - throwanem 1y ago
      You know, it's odd? I missed the whole initial fad, and only around 2018 or so got around to reading the books to see what all the hype had been about. (Even for a millennial I'm old, and I grew up in a pretty backward corner of the country; I've never played a Pokemon game, either...) So why's it me, and not an actual fan, who should be the one to come up with Rowling's serial-numbers-filed-off Echo and Narcissus as the example?
dclowd9901 1y ago
> As such, if he really is suffering a mental health crisis related to his use of OpenAI's product, his situation could serve as an immense optical problem for the company, which has so far downplayed concerns about the mental health of its users. Yikes. Not just an optics* problem, but one has to consider if they're pouring so much money into the company because he feels he "needs" to (whatever basis of coercion exists to support his need to get to the "truth").
butlike 1y ago
Is "futurism.com" a trustworthy publication? I've never heard of it. I read the article and it didn't seem like the writing had the hallmarks of top-tier journalism.
- throwanem 1y ago
  I'm not familiar with the publication either, but the claims I've examined, most notably those relevant to the subject's presently public X.com The Everything App account, appear to check out, as does that the account appears to have been inactive since the day before the linked article was published last week. It isn't clear to me where the reputation of the source becomes relevant.

pjerem 1y ago

I think I have a precise figure : it's a lot.

ninetyninenine 1y ago
[flagged]
sitkack 1y ago
And we thought having access to someones internet searches was good intel. Now we have a direct feed to their brain stem along with a way to manipulate it. Good thing that narcissistic sociopaths have such a low expression in the overall population.
isoprophlex 1y ago
Yeah. If HN is your primary source of in depth AI discussion, you get a pretty balanced take IMO compared to other channels out there. We (the HN crowd) should take into account that if you take "people commenting on HN" as a group, you are implicitly selecting for people that are able to read, parse and contextualise written comment threads. This is NOT your average mid-to-high level corpo management exec, who can for more than 80% (from experience) be placed in the "rise of the business idiot" cohort, fed on prime linkedin brainrot. self-reinforcing hopium addicts with an mba. Nor is it the great masses of random earth dwellers who are not always able to resist excess sugar, nicotine, mcdonalds, youtube, fentanyl, my-car-is-bigger-than-yours credit card capitalism, free pornography, you name it. And now RLHF: Validation as a service. Not sure if humanity is ready for this. (Disclosure: my mum has a chatgpt instance that she named and I'm deeply concerned about the spiritual convos she has with it; random people keep calling me on the level of "can you build me an app that uses llms to predict Funko Pop futures".)

kovezd 1y ago

[flagged]

bpodgursky 1y ago

It depends what model you use. o3 pushes back reasonably well. 4o doesn't.

sanitycheck 1y ago

Then it'll be no different than when they bounce their bad ideas off their human subordinates.

game_the0ry 1y ago
This. Their immediate subordinates will be just a as sycophantic, if not more.
- NoGravitas 1y ago
  cf Celene's Second Law.

petesergeant 1y ago

I feel like I’ve had good results for getting feedback on technical writing by claiming the author is a third party and I need to understand the strengths and weaknesses of their work. I should probably formally test this.

game_the0ry 1y ago

Fun fact, my manager wrote my annual review with our internal LLM tool, which itself is just a wrapper around GTP-4o. (My manager told me when I asked him)

yojo 1y ago
This already seems somewhat widespread. I have a friend working at a mid-tier tech co that has a handful of direct reports. He showed me that the interface to his eval app had a “generate review” button, which he clicked, then moved on to the next one. Honestly, I’m fine with this as long as I also get a “generate self review” button. I just wish I could get back all the time I’ve spent massaging a small number of data points into pages of prose.
- game_the0ry 1y ago
  That makes you wonder why we go through the ritual of an annual review if no one takes it seriously.

Cthulhu_ 1y ago

TBH if they have sycophants and a lot of money it's probably the same. How many bullshit startups have there been, how many dumb ideas came from higher up before LLMs?