6 ms·

Seems like a fairly direct parable against AI, suggesting that even if it's right it's better to think for yourself.

by fourthark 2mo ago

beepbooptheory 2mo ago
You mean like LLM AI? Seems a little too early for that no? Either way, what is it right about? In your mind?
- margalabargala 2mo ago
  It's not subtle, it's clearly saying "if you outsource your thinking to something else, your mind will wither". People complaining about this have a long and storied history, this is the same as Plato complaining philosophers were ruining their minds by writing things down to remember them.
  - beepbooptheory 2mo ago
    This is not what Plato thought, this is not even what Plato wrote that Socrates thought. You are referring to the Phaedrus. You should read it! Its much better than this story, trust me. But either way, people's minds do not wither in this story, it explicitly works out fine for them?
    - Dylan16807 2mo ago
      > But either way, people's minds do not wither in this story, it explicitly works out fine for them? Huh? "when the corpses of the wearers were being prepared for burial, it was noted that their brains were curiously deformed: the neocortexes had wasted away, and the bulk of their mass was an abnormally hypertrophied mid- and lower-brain, especially the parts associated with reflexive action" You could infer their minds fading away even without that paragraph, but it's very explicit. Also you have another comment where you insist nobody has explained what the story is arguing when multiple people told you what the story is arguing, what the heck.
      - beepbooptheory 2mo ago
        OK, so there is a magical device that provides you with deep contentment no matter what. But then, when you die and they cut your head open, the physical, biological brain has shrunk. And that's bad. Is that right? Is this what it's about? Is the story actually arguing something? I have just been literally asking what it even presumes itself to be about. Is it the gloss above? If it's about agency, I don't get it. It seems silly/facile to assert the "right" decision is the thing that matters without making the thing so magic it doesn't even make sense. Does agency stop you from getting hit by a runaway bus? Help you from being persecuted for arbitrary reasons? The story tells us they all, unfailingly, live "abnormally successful" lives. That's all supposed to be because of their (non) agency because of the earring device?
        Dylan16807 2mo ago
        Is it actually contentment or are you fading away so much you cease to exist? The story is not about physical brain changes, no. It's about losing yourself. It's arguing that to exist is to make decisions. Following the device too much leads to a life that looks successful from the outside but inside no longer has a person enjoying the success, just a puppet.
        beepbooptheory 2mo ago
        But it explicitly says it provides the wearer with happiness, it is "supernaturally effective" at it. Where as a reader are we supposed to gather this is not an internal happiness? Come to think of it, its not even clear there is a question of agency or like some inverted solipsism as you suggest. Wouldn't the wearer need to be compelled for that to be at stake? Kadmi-nomai can wear it and take it off, right? Aren't they, from the inside and outside, just listening to magically good advice and getting along great because of it?
        Dylan16807 2mo ago
        > But it explicitly says it provides the wearer with happiness It gives you one of the best options for happiness for any decision you face, which is not quite the same. And don't forget it tells you to take it off very first thing. > Where as a reader are we supposed to gather this is not an internal happiness? The rest of the story. > Come to think of it, its not even clear there is a question of agency or like some inverted solipsism as you suggest. Wouldn't the wearer need to be compelled for that to be at stake? Kadmi-nomai can wear it and take it off, right? It's a problem if you wear and follow it for a very long time. He talked to it for a couple hours. > Aren't they, from the inside and outside, just listening to magically good advice and getting along great because of it? Your neocortex is pretty important. You can argue that you think loss of self is unrealistic, but the story is very clear in its judgement. Don't turn a disagreement with the story into "not understanding" the story.
- Sharlin 2mo ago
  This is from 2012, so no LLM AI in particular, just AGI/strong AI in general.
dudisubekti 2mo ago
Eh I think the writer should visit poor countries and live among locals. Mistakes and poor judgements compound, and it leads to horrific living conditions (malnutrition, nonexistent hygiene, quack medicines, short-sighted violence, etc) In fact developed countries are built on top of these "earrings": it's called math (and by extension science in general). If the math is right, it's always right, and it has spared their lucky citizens a LOT of needless misery and wheel-reinvention. "Thinking for yourself" is overrated if you don't have good framework. See antivaxxers and their "research"
- thinking_cactus 2mo ago
  No, this story takes it to the extreme, that you end up basically having no brain. Take it this way. People seem to judge that LLMs are not conscious and thus have no real inner experience. And that humans do. So if you replaced a human brain with a very powerful LLM (with some simply residual neural matter), you've basically killed the human. He isn't feeling anything at all anymore, he's just an automaton, basically a flesh robot. I think this story is a weirdly good discussion point because many people still don't seem to "get it" that success is completely fundamentally irrelevant, and what matters is having a good inner life. You are what happens in your mind -- overall good thoughts and inner life = good life; bad thoughts and inner life = bad life. You can theoretically be the richest man in the world and about the most miserable, and very poor and living the best life. If we don't kill our definition of good life and success (i.e. reaffirming value as pertaining exclusively to the mind), it will sooner or later kill us, or kill what's good in us. The more we can orient our entire society toward good inner lives and not anything else, the better our lives will be.
- ameliaquining 2mo ago
  Scott Alexander has actually practiced medicine in Haiti as part of a relief effort.
  - dudisubekti 2mo ago
    He was in Haiti for one week. https://web.archive.org/web/20150407223525/http://squid314.livejournal.com/297579.html https://web.archive.org/web/20150407223525/http://squid314.l... Then went back to a society safeguarded by the "earrings" (OSHA, FDA, CDC, etc). Poor countries usually don't have the resources and enough HDI to enforce public safety. Not saying his experience doesnt count, but 1 week is not enough to experience that crushing learned helplessness when everyone around you keep doing stupid sh*t and sabotaging themselves and others near them. Which is why Trump defunding public safety agencies is baffling to me, a person from a poor country. Such a decadent behavior.
Ari_Rahikkala 2mo ago
I'm a big fan of Scott's, but I haven't been happy to see how this story's failure to say what it means has in some ways only made it less relevant over the years as we've come closer to building something like the item that, on the surface, it talks about. Not to say I think it's a bad thing at all that people have a name for this concern for giving up agency to an external system, even when that system really has your best intent in mind. You can call Claude a whispering earring and people will understand what you mean pretty well. The actual consequences you're looking out for are maybe a bit subtler than your brain atrophying, I really really never liked that detail in the story, but that's fine, people will get it. By the Death of the Author, the story's a great contribution as it is. But it just feels like a loss that there really was a different thing that the author meant, and it got lost in translation. http://web.archive.org/web/20121007235422/http://squid314.livejournal.com/333168.html http://web.archive.org/web/20121007235422/http://squid314.li... clarifies it. It's quite similar to a Yudkowskyan fun theory concern (and IMO really not surprising that the story was released a couple of years after the fun theory sequence): If making the right choices and doing the right thing eventually comes down to a mechanical computation, and you've got all of your resource limitations and psychological hang-ups and such things out of the way, then there's no room left to do the right thing, and you've turned yourself into a mechanism. I don't think that's really a possible state that any system with any trace of humanity left in it will ever enter, there's too many too large search processes in the way to fit in a finite universe. So it's a pretty abstract concern even among its peers. But it's one that's fun to talk about sometimes, and one lesson we've learned from the response to this story is that mysterious, easily misinterpreted parallels in fact do not point people in the right direction to see it.

Seems like a fairly direct parable against AI, suggesting that even if it's right it's better to think for yourself.

beepbooptheory 2mo ago

You mean like LLM AI? Seems a little too early for that no? Either way, what is it right about? In your mind?

margalabargala 2mo ago
It's not subtle, it's clearly saying "if you outsource your thinking to something else, your mind will wither". People complaining about this have a long and storied history, this is the same as Plato complaining philosophers were ruining their minds by writing things down to remember them.
- beepbooptheory 2mo ago
  This is not what Plato thought, this is not even what Plato wrote that Socrates thought. You are referring to the Phaedrus. You should read it! Its much better than this story, trust me. But either way, people's minds do not wither in this story, it explicitly works out fine for them?
  - Dylan16807 2mo ago
    > But either way, people's minds do not wither in this story, it explicitly works out fine for them? Huh? "when the corpses of the wearers were being prepared for burial, it was noted that their brains were curiously deformed: the neocortexes had wasted away, and the bulk of their mass was an abnormally hypertrophied mid- and lower-brain, especially the parts associated with reflexive action" You could infer their minds fading away even without that paragraph, but it's very explicit. Also you have another comment where you insist nobody has explained what the story is arguing when multiple people told you what the story is arguing, what the heck.
    - beepbooptheory 2mo ago
      OK, so there is a magical device that provides you with deep contentment no matter what. But then, when you die and they cut your head open, the physical, biological brain has shrunk. And that's bad. Is that right? Is this what it's about? Is the story actually arguing something? I have just been literally asking what it even presumes itself to be about. Is it the gloss above? If it's about agency, I don't get it. It seems silly/facile to assert the "right" decision is the thing that matters without making the thing so magic it doesn't even make sense. Does agency stop you from getting hit by a runaway bus? Help you from being persecuted for arbitrary reasons? The story tells us they all, unfailingly, live "abnormally successful" lives. That's all supposed to be because of their (non) agency because of the earring device?
      - Dylan16807 2mo ago
        Is it actually contentment or are you fading away so much you cease to exist? The story is not about physical brain changes, no. It's about losing yourself. It's arguing that to exist is to make decisions. Following the device too much leads to a life that looks successful from the outside but inside no longer has a person enjoying the success, just a puppet.
        beepbooptheory 2mo ago
        But it explicitly says it provides the wearer with happiness, it is "supernaturally effective" at it. Where as a reader are we supposed to gather this is not an internal happiness? Come to think of it, its not even clear there is a question of agency or like some inverted solipsism as you suggest. Wouldn't the wearer need to be compelled for that to be at stake? Kadmi-nomai can wear it and take it off, right? Aren't they, from the inside and outside, just listening to magically good advice and getting along great because of it?
        Dylan16807 2mo ago
        > But it explicitly says it provides the wearer with happiness It gives you one of the best options for happiness for any decision you face, which is not quite the same. And don't forget it tells you to take it off very first thing. > Where as a reader are we supposed to gather this is not an internal happiness? The rest of the story. > Come to think of it, its not even clear there is a question of agency or like some inverted solipsism as you suggest. Wouldn't the wearer need to be compelled for that to be at stake? Kadmi-nomai can wear it and take it off, right? It's a problem if you wear and follow it for a very long time. He talked to it for a couple hours. > Aren't they, from the inside and outside, just listening to magically good advice and getting along great because of it? Your neocortex is pretty important. You can argue that you think loss of self is unrealistic, but the story is very clear in its judgement. Don't turn a disagreement with the story into "not understanding" the story.
Sharlin 2mo ago
This is from 2012, so no LLM AI in particular, just AGI/strong AI in general.

dudisubekti 2mo ago

Eh I think the writer should visit poor countries and live among locals. Mistakes and poor judgements compound, and it leads to horrific living conditions (malnutrition, nonexistent hygiene, quack medicines, short-sighted violence, etc) In fact developed countries are built on top of these "earrings": it's called math (and by extension science in general). If the math is right, it's always right, and it has spared their lucky citizens a LOT of needless misery and wheel-reinvention. "Thinking for yourself" is overrated if you don't have good framework. See antivaxxers and their "research"

thinking_cactus 2mo ago
No, this story takes it to the extreme, that you end up basically having no brain. Take it this way. People seem to judge that LLMs are not conscious and thus have no real inner experience. And that humans do. So if you replaced a human brain with a very powerful LLM (with some simply residual neural matter), you've basically killed the human. He isn't feeling anything at all anymore, he's just an automaton, basically a flesh robot. I think this story is a weirdly good discussion point because many people still don't seem to "get it" that success is completely fundamentally irrelevant, and what matters is having a good inner life. You are what happens in your mind -- overall good thoughts and inner life = good life; bad thoughts and inner life = bad life. You can theoretically be the richest man in the world and about the most miserable, and very poor and living the best life. If we don't kill our definition of good life and success (i.e. reaffirming value as pertaining exclusively to the mind), it will sooner or later kill us, or kill what's good in us. The more we can orient our entire society toward good inner lives and not anything else, the better our lives will be.
ameliaquining 2mo ago
Scott Alexander has actually practiced medicine in Haiti as part of a relief effort.
- dudisubekti 2mo ago
  He was in Haiti for one week. https://web.archive.org/web/20150407223525/http://squid314.livejournal.com/297579.html https://web.archive.org/web/20150407223525/http://squid314.l... Then went back to a society safeguarded by the "earrings" (OSHA, FDA, CDC, etc). Poor countries usually don't have the resources and enough HDI to enforce public safety. Not saying his experience doesnt count, but 1 week is not enough to experience that crushing learned helplessness when everyone around you keep doing stupid sh*t and sabotaging themselves and others near them. Which is why Trump defunding public safety agencies is baffling to me, a person from a poor country. Such a decadent behavior.

Ari_Rahikkala 2mo ago

I'm a big fan of Scott's, but I haven't been happy to see how this story's failure to say what it means has in some ways only made it less relevant over the years as we've come closer to building something like the item that, on the surface, it talks about. Not to say I think it's a bad thing at all that people have a name for this concern for giving up agency to an external system, even when that system really has your best intent in mind. You can call Claude a whispering earring and people will understand what you mean pretty well. The actual consequences you're looking out for are maybe a bit subtler than your brain atrophying, I really really never liked that detail in the story, but that's fine, people will get it. By the Death of the Author, the story's a great contribution as it is. But it just feels like a loss that there really was a different thing that the author meant, and it got lost in translation. http://web.archive.org/web/20121007235422/http://squid314.livejournal.com/333168.html http://web.archive.org/web/20121007235422/http://squid314.li... clarifies it. It's quite similar to a Yudkowskyan fun theory concern (and IMO really not surprising that the story was released a couple of years after the fun theory sequence): If making the right choices and doing the right thing eventually comes down to a mechanical computation, and you've got all of your resource limitations and psychological hang-ups and such things out of the way, then there's no room left to do the right thing, and you've turned yourself into a mechanism. I don't think that's really a possible state that any system with any trace of humanity left in it will ever enter, there's too many too large search processes in the way to fit in a finite universe. So it's a pretty abstract concern even among its peers. But it's one that's fun to talk about sometimes, and one lesson we've learned from the response to this story is that mysterious, easily misinterpreted parallels in fact do not point people in the right direction to see it.