8 ms·

Playing in the Creek

347 points by c1ccccc1 1y ago

profsummergig 1y ago
Requesting someone to please explain the "coquina" metaphor.
- ern 1y ago
  Maybe I’m not smart enough, or too tired to decode these metaphors, so I plugged the essay into ChatGPT and got a clear explanation from 4o.
  - profsummergig 1y ago
    Ah. Should have thought of that. Going to do that now. Thanks.
  - criddell 1y ago
    Are you at all concerned that plugging stuff like this into ChatGPT is leaving you with weaker cognitive muscles? Or is it more similar to what people do when they see a new word and reach for their dictionary?
    - adwn 1y ago
      > Are you at all concerned that plugging stuff like this into ChatGPT is leaving you with weaker cognitive muscles? Couldn't this very same argument have been used against any form of mental augmentation, like written language and computers? Or, in an extended interpretation, against any form of physical augmentation, like tool use?
      - criddell 1y ago
        You can argue whatever you want to argue. I make my living with my brain so I do worry about the downsides of removing boredom and mental struggle from my days.
        Workaccount2 1y ago
        It's almost certainly going to be bad, and almost certainly going to be unavoidable. I can't spell for shit anymore. Ever since auto correct became omnipresent in pretty much all writing fields, my brain just kinda ditched remembering how to spell words. buuuttt Manual labor has been obsolete for at least 100 years now for certain classes of people, and fitness is still an enormous recreational activity people partake in. So even in an AI heavy society, I still strongly suspect there will be "brain games" that people still enjoy and regularly play.
        criddell 1y ago
        We aren't talking about something like spelling or digging a hole. We're talking about a fundamental cognitive skill: reading eight short paragraphs of text and extracting meaning from it.
        profsummergig 1y ago
        > eight short paragraphs of text Fair point. But they are heavily metaphor-laden paragraphs. Textual interpretation is a highly subjective activity. Entire careers consist of interpreting, reinterpreting, and discussing texts that others have already interpreted. Film critics, book reviewers, political pundits, TV anchors, podcasters, etc. 'In 1972, Chinese premier Zhou Enlai was asked about the impact of the French Revolution. "Too early to say," he replied' I had my own sense of what the "coquina" metaphor stood for. I wanted to see other peoples' interpretations. Turns out my interpretation was wrong.
        profsummergig 1y ago
        > I can't spell for shit anymore. This is increasingly happening to me every day. Hope the alien overlords don't have spelling tests (as their version of IQ tests) to separate the serfs from the field-masters.
        steve_adams_86 1y ago
        Me too. There is another side to this, which is maybe we don’t need to know a lot of things. It was true with search engines already, but maybe truer with LLMs. That thing you’re querying probably doesn’t actually matter. It’s neurotic digging and searching for an object you will never use or benefit from. The urge to seek is strong but you won’t find the thing you’re searching for this way. You might learn more by just going for a walk.
      - TimorousBestie 1y ago
        In fact it has been, dating all the way back to Phaedrus. > If men learn [writing], it will implant forgetfulness in their souls. They will cease to exercise memory because they rely on that which is written, calling things to remembrance no longer from within themselves, but by means of external marks.
    - ern 1y ago
      I see AI like the reading glasses I’ll soon need — not because I can’t think clearly, but because it helps cut through things faster when my brain’s juggling too much. A few years ago, I’d have quietly filed this kind of article under “too hard” or passed a log analysis request from the CIO down the line. Now? I get AI to draft the query, check it, run it, and move on. It’s not about thinking less — it’s about clearing the clutter so I can focus where it counts.
- hecanjog 1y ago
  I think that they're saying a little bit of playing around with replacing thinking and composing with automated tools is recoverable, but at an industrial or societal scale the damage is significant. Like the difference between shoveling away some sand with your hands to bury the small creatures temporarily and actually destroying their habitat by "lobbying city council members to put in a groin or seawall, and seriously move that beach sand."
  - profsummergig 1y ago
    I skimmed the Anthropic report and didn't catch the negative effects. Did they mention any? Good on them if they did.
    - hecanjog 1y ago
      Yes, they mention a few times the concern that students are offloading critical thinking rather than using the tool for learning.
      - Cthulhu_ 1y ago
        I just hope the educational institutions catch on, stick with their principles and don't give them the paperwork. The paper / title should be evidence of students' learning and thinking abilities, not of just their output.
- xmprt 1y ago
  My understanding is that the author is this superior being trying to accomplish a massive task (damming a beach) while knowing that it could cause problems for these clams. In the real world, Anthropic is trying to accomplish a massive task (building AGI) and they're finally starting to notice the potential impacts this has on people.
- jjcob 1y ago
  Coquinas are clams that bury themselves in the sand very close to the surface [1]. The author worries that while they are playing with the sand, they might accidentally bury coquina clams too deep and kill them because they can no longer reach the surface. Anthropic apparently is starting to notice the possible danger to others of their work. I'm not sure what they are referring to. [1]: https://www.youtube.com/watch?v=KZUlf7quu3o https://www.youtube.com/watch?v=KZUlf7quu3o
  - profsummergig 1y ago
    > Anthropic apparently is starting to notice the possible danger to others of their work. I'm not sure what they are referring to. Are they being vague about the danger? If possible, please link to a communique from them. I've missed it somehow. Thanks.
    - vermilingua 1y ago
      https://www.anthropic.com/news/anthropic-education-report-how-university-students-use-claude https://www.anthropic.com/news/anthropic-education-report-ho... Discussed here yesterday: https://news.ycombinator.com/item?id=43633383 https://news.ycombinator.com/item?id=43633383
      - profsummergig 1y ago
        Thank you.
  - deathanatos 1y ago
    As a child at the beach, I would think noticing the clams would result in attempting to unearth them. Childhood curiosity about why there are bubbles. Your explanation makes more sense, however.
- cubefox 1y ago
  Anthropic (Claude.ai) is mentioning in their report on LLMs and education that students use Claude to cheat and do their work for them: https://www.anthropic.com/news/anthropic-education-report-how-university-students-use-claude https://www.anthropic.com/news/anthropic-education-report-ho...
doctoboggan 1y ago
This is an excellent essay, and I feel similar to the author but couldn't express it as nicely. However if we are counting on AI researchers to take the advice and slow down then I wouldn't hold my breath waiting. The author indicated they stepped away from a high paying finance job for moral reasons, which is admirable. But wallstreet continues on and does not lack for people willing to play the "make as much money as you can" game.
- yapyap 1y ago
  > However if we are counting on AI researchers to take the advice and slow down then I wouldn't hold my breath waiting. The author indicated they stepped away from a high paying finance job for moral reasons, which is admirable. But wallstreet continues on and does not lack for people willing to play the "make as much money as you can" game. I doubt OP is counting on it, it is moreso expressing what an optimal world would look like so people can work towards it if they would feel like it or just to put the idea out there.
- dachris 1y ago
  The paperclip maximizers are already here, but they are maximizing money. One recent HN comment [0] comparing corporations and institutions to AI really stuck with me - those are already superhuman intelligences. [0] https://news.ycombinator.com/item?id=43580681 https://news.ycombinator.com/item?id=43580681
  - actionfromafar 1y ago
    I could imagine a Star Trek episode where someone says "I always assumed the paperclip optimizer was a parable for unchecked capitalism?"
  - bitethecutebait 1y ago
    > those are already superhuman intelligence(s) ... only because "unsafe" and "leaky" are a Ponzi's best-and-loves-to-be-roofied-and-abused friend ... you see, intelligence is only good when it doesn't irreversibly break everything to the point where most of the variety of the physical structure that evolved it and maintains it is lost. you could argue, of course, and this is an abbreviated version, that a new physical structure then evolves a new intelligence that is adapted (emerged from and adjusts to) to the challenges of the new environment but that's not the point of already capable self-healing systems; except if the destructive part of the superhuman intelligence is more successful with it's methods of sabotage and disruption of (a) 'truthy' information flow and b) individual and collective super-rational agency -- for the good of as many systems-internal entities as possible, as a precaution due to always living in uncertainty and being surrounded by an endless amount of variables currently tagged "noise" -- than it's counterpart is in enabling and propagating a) and b) ... in simpler words, if the red team FUBARS the blue team or vice versa, the superhuman intelligence can be assumed to have cancer or that at least some vital part of the force is corrupted otherwise.
unwind 1y ago
Ah, this [1] meaning of tillering (bending wood to form a bow), not this [2] (production of side shoots in grasses). The joys of new words. [1]: https://www.howtomakealongbow.co.uk/part-5-tillering https://www.howtomakealongbow.co.uk/part-5-tillering [2]: https://en.wikipedia.org/wiki/Tiller_(botany) https://en.wikipedia.org/wiki/Tiller_(botany)
- defrost 1y ago
  As I recall tillering is more about the shaping of the bow to achieve an optimal bend and force delivery on release. It's an iterative process of bending and shaping, bending again, and wood removal in stages.
DanAtC 1y ago
[flagged]
axpvms 1y ago
My backyard creek also had crocodiles in it.
- seafoamteal 1y ago
  Florida?
  - tilne 1y ago
    No they’ve got a little place on the Nile
MrBuddyCasino 1y ago
That was a well written essay with a non-sequitur AI Safety thing tacked to the end. His real world examples were concrete, and the reason to stop escalating easy to understand ("don't flood the neighbourhood by building a real dam"). The AI angle is not only even hypothetical: there is no attempt to describe or reason about a concrete "x leading to y", just "see, the same principle probably extrapolates". There is no argument there that is sounder than "the high velocities of steam locomotives might kill you" that people made 200 years ago.
- luc4sdreyer 1y ago
  > the high velocities of steam locomotives might kill you This obviously seems silly in hindsight. Warnings about radium watches or asbestos sound less silly, or even wise. But neither had any solid scientific studies showing clear hazard and risk. Just people being good Bayesian agents, trying to ride the middle of the exploration vs. exploitation curve. Maybe it makes sense to spend some percentage of AI development resources on trying to understand how they work, and how they can fail.
  - ripe 1y ago
    > This [steam locomotives might kill you] obviously seems silly in hindsight. To be fair, many people did die on level crossings and by wandering on to the tracks. We learned over time to put in place safety fences and tunnels.
    - Gracana 1y ago
      People thought that the speed itself was dangerous, that the wind and vibration and landscape screaming by at 25mph would cause physical and mental harm.
  - MrBuddyCasino 1y ago
    > Warnings about radium watches or asbestos sound less silly, or even wise. But neither had any solid scientific studies showing clear hazard and risk. In the case of asbestos, this is incorrect. Many people knew it was deadly, but the corporations selling it hid it for decades, killing thousands of people. There are quite a few other examples besides asbestos, like leaded fuel or cigarettes.
khazhoux 1y ago
Parents: you know how every day you look at your child and you’re struck with wonder at the amazing and quirky and unique person your little one is? I swear that’s what lesswrong posters see every day in the mirror.
DrSiemer 1y ago
So many articles and comments claim Ai will destroy critical thinking in our youths. Is there any evidence that this conviction that many people share is even remotely true? To me it just seems like the same old knee-jerk luddite response people have to any powerful new technology that challenges that status quo since the dawn of time. The calculator did not erase math wizards, the television did not replace books and so on. It just made us better, faster, more productive. Sometimes there is an adjustment period (we still haven't figured out how to deal with short dopamine hits from certain types of entertainment and social media), but things will balance themselves out eventually. Some people may go full-on Wall-E, but I for one will never stop tinkering, and many of my friends won't either. The things I could have done if I had had an LLM as a kid... I think I've learned more in the past two years than ever before.
- iNic 1y ago
  I don't think you got the point of the article? It is saying that we as wise humans know (sometimes) when to stop optimizing for a goal, due to the negative side effects. AIs (and as some other people have pointed out corporations) do not naturally have this line in their head, and we must draw such lines carefully and with purpose for these superhuman beings.
- dsign 1y ago
  > Ai will destroy critical thinking in our youths I don't think that's the argument the article was making. It was, to my understanding, a more nuanced question about if we want to destroy or severely disturb systems at equilibrium by letting AI systems infiltrate our society. > Sometimes there is an adjustment period (we still haven't figured out how to deal with short dopamine hits from certain types of entertainment and social media), but things will balance themselves out eventually. One can zoom out a little bit. The issue didn't start with social media, nor AI. "Star Wars, A New Hope", is, to my understanding, an incredibly good film. It came out in 1977 and it's a great story made to be appreciated by the masses. And in trying to achieve that goal, it really wasn't intellectually challenging. We have continued in that downhill for a bit, and now we are in 16 second stingers in TikTok and Youtube. So, the way I see it, things are not balancing out. Worse, people in USA elected D.J. Trump because somehow they couldn't understand how this real-world Emperor Palpatine was the bad guy.
BrenBarn 1y ago
It's a nice article. In a way though it kind of bypasses what I see as the main takeaways. It's not about AI development, it's about something mentioned earlier in the article: "make as much money as I can". The problems that we see with AI have little to do with AI "development", they have to do with AI marketing and promulgation. If the author had gone ahead and dammed the creek with a shovel, or blown off his hand, that would have been bad, but not that bad. Those kinds of mistakes are self-limiting because if you're doing something for the enjoyment or challenge of it, you won't do it at a scale that creates more enjoyment than you personally can experience. In the parable of the CEO and the fisherman, the fisherman stops at what he can tangibly appreciate. If everyone working on and using AI were approaching it like damming a creek for fun, we would have no problems. The AI models we had might be powerful, but they would be funky and disjointed because people would be more interested in tinkering with them than making money from them. We see tons of posts on HN every day about remarkable things people do for the gusto. We'd see a bunch of posts about new AI models and people would talk about how cool they are and go on not using them in any load-bearing way. As soon as people start trying to use anything, AI or not, to make as much money as possible, we have a problem. The second missed takeaway is at the end. He says Anthropic is noticing the coquinas as if that means they're going to somehow self-regulate. But in most of the examples he gives, he wasn't stopped by his own realization, but by an external authority (like parents) telling him to stop. Most people are not as self-reflective as this author and won't care about "winning zero sum games against people who don't necessarily deserve to lose", let alone about coquinas. They need a parent to step in and take the shovel away. As long as we keep treating "making as much money as you can" as some kind of exception to the principle of "you can't keep doing stuff until you break something", we'll have these problems, AI or not.
- noduerme 1y ago
  This is such a well-written response. There's something intentionally soothing about this post that slowly turns into a jarring form of self-congratulation as it goes along. Congratulations for knowing there's a limit to wrecking your parents' property. Congratulations for being able to appreciate the sand on the beach, in some no doubt instagrammable moment of existential simplicity. Congratulations for being so smart that you could have blown up your hand. And for "Leetcoding", whatever the fuck that means. And for claiming you quit a shady job because you got bored (but possibly also grew a conscience). And then topped off by the final turn: "This is, of course, about artificial intelligence development". I'd only add one thing to your analysis: We've got a demo right here of a psyche that would prefer love to money (but mostly both), and it's still determined to foist bad things onto the world in a load-bearing way, as a bid for either, or whatever it can get. My parents used to call that "a kid that doesn't care if he gets good or bad attention, as long as he gets attention." I think that's the root driver for almost all the tech billionaires of the past 20 years, and the one thing that unites Bezos, Zuck, Jobs, Dorsey, Musk... it's: "Look dad, I didn't just take your money. I'm so smart I could'a blown off my hand with all those fireworks you bought me, but see? Two hands! Look how much money I made from your money! Why aren't you proud of me?! Where can I find love? Maybe if I tell people what a leetcoder I am and how I could be making BAD AI but I'm just making GOOD AI, then everyone will love me." Don't get me wrong, I'm not immune to these feelings either. I want to do good work and I want people to love what I do. But there's something so... so fucking nakedly exhibitionist and narcissistic about these kinds of posts. Like, so, GO FUCKING LAY WITH CLAMS, write a novel, the world is waiting for it if you're really a genius. Have the courage to say you have a conscience if you actually do. Leave the rest of us alone and stop polluting a world you don't understand with your childish greed and self-obsession.
praptak 1y ago
If there's money to be made, there will always be someone with a shovel or a truckload of sparklers who is willing to take the risk (especially if the risk can be externalized to the public) and reap the reward.
A_D_E_P_T 1y ago
The author seems concerned about AI risk -- as in, "they're going to kill us all" -- and that's a common LW trope. Yet, as a regular user of SOTA AI models, it's far from clear to me that the risk exists on any foreseeable time horizon. Even today's best models are credulous and lack a certain insight and originality. As Dwarkesh once asked: > One question I had for you while we were talking about the intelligence stuff was, as a scientist yourself, what do you make of the fact that these things have basically the entire corpus of human knowledge memorized and they haven’t been able to make a single new connection that has led to a discovery? Whereas if even a moderately intelligent person had this much stuff memorized, they would notice — Oh, this thing causes this symptom. This other thing also causes this symptom. There’s a medical cure right here. > Shouldn’t we be expecting that kind of stuff? I noticed this myself just the other day. I asked GPT-4.5 "Deep Research" what material would make the best [mechanical part]. The top response I got was directly copied from a laughably stupid LinkedIn essay. The second response was derived from some marketingslop press release. There was no original insight at all. What I took away from my prompt was that I'd have to do the research and experimentation myself. Point is, I don't think that LLMs are capable of coming up with terrifyingly novel ways to kill all humans. And this hasn't changed at all over the past five years. Now they're able to trawl LinkedIn posts and browse the web for press releases, is all. More than that, these models lack independent volition and they have no temporal/spatial sense. It's not clear, from first principles, that they can operate as truly independent agents.
- tvc015 1y ago
  Aren’t semiautonomous drones already killing soldiers in Ukraine? Can you not imagine a future with more conflict and automated killing? Maybe that’s not seen as AI risk per se?
  - A_D_E_P_T 1y ago
    That's not "AI risk" because they're still tools that lack independent volition. Somebody's building them and setting them loose. They're not building themselves and setting themselves loose, and it's far from clear how to get there from here. Dumb bombs kill people just as easily. One 80-year old nuke is, at least potentially, more effective than the entirety of the world's drones.
FollowingTheDao 1y ago
"It was only once I got it that I realized I no longer could play the game "make as much money as I can."" Funny, that is what my father taught me when I was 12 because we had compassion. What is it with glorifying all these logic loving Spock like people? Don't you know Captain Kirk was the real hero of Star Trek? Because he had compassion? It is no wonder the Zizians were birthed from LW.
appleorchard46 1y ago
Could someone explain the metaphor? I'm struggling to see the connection between AI and the rest of the post.
- ido 1y ago
  That AI is dangerous and the closer we get to the danger zone the better it would be if the companies developing these technologies understand it might be better to slow down and make sure it's safe vs pushing ahead at maximum speed.
  - appleorchard46 1y ago
    Thank you.
  - nektro 1y ago
    if they were capable of that they wouldn't have started. but the second best time to stop is now so here's hoping theyre open to our calls
curtisszmania 1y ago
[dead]
migueldeicaza 1y ago
Vonnegut said it best: https://richardswsmith.wordpress.com/2017/11/18/we-are-here-on-earth-to-fart-around-and-dont-let-anybody-tell-you-any-different/ https://richardswsmith.wordpress.com/2017/11/18/we-are-here-...
- broabprobe 1y ago
  huh, I wonder if he has relayed this story multiple times, I’m only familiar with this version, https://www.goodreads.com/quotes/12020560-talking-about-when-he-tells-his-wife-he-s-going-out https://www.goodreads.com/quotes/12020560-talking-about-when... “(talking about when he tells his wife he’s going out to buy an envelope) Oh, she says well, you’re not a poor man. You know, why don’t you go online and buy a hundred envelopes and put them in the closet? And so I pretend not to hear her. And go out to get an envelope because I’m going to have a hell of a good time in the process of buying one envelope. I meet a lot of people. And, see some great looking babes. And a fire engine goes by. And I give them the thumbs up. And, and ask a woman what kind of dog that is. And, and I don’t know. The moral of the story is, is we’re here on Earth to fart around. And, of course, the computers will do us out of that. And, what the computer people don’t realize, or they don’t care, is we’re dancing animals.” ― Kurt Vonnegut
  - Thorrez 1y ago
    He ignores his wife's suggestion because, among other things, he wants to see some great looking babes. Maybe this isn't a guy whose philosophy I want to follow.
    - ryandrake 1y ago
      Looks like you're completely missing the point of the quote and instead rat-holing on one word that you don't like. HN in a nutshell.
      - Thorrez 1y ago
        I understand the point of the quote. My point was that if someone advocates for something I consider immoral, I prefer to disregard the overall philosophy that lead the person to advocate for that. If the quote threw in some casual racism, or advocated for stealing the envelope instead of paying for it, I would similarly disregard the overall philosophy.
Isamu 1y ago
>After I cracked the trick of tillering Guide to Bow Tillering: https://straightgrainedboard.com/beginners-guide-on-bow-tillering/ https://straightgrainedboard.com/beginners-guide-on-bow-till...
red_admiral 1y ago
The story of playing at damming the creek or on the sand at the seaside is wholesome and brought a smile to my face. Cracking the "puzzle" is almost the bad ending of the game, if you don't get any fun at playing it anymore. People should spend more of their time doing things because they're fun, not because they want to get better at it. Maybe the apocalypse will happen in our lifetime, maybe not. I intend to have fun as much as I can in my life either way.
bogdanoff_2 1y ago
The solution to this problem is to choose a "game" that you 100% believe will positively impact the world.
ziofill 1y ago
This is a tangent, but I would love so much to be able to give my kids memories of playing in a creek in the backyard...

347 points by c1ccccc1 1y ago

profsummergig 1y ago

Requesting someone to please explain the "coquina" metaphor.

ern 1y ago
Maybe I’m not smart enough, or too tired to decode these metaphors, so I plugged the essay into ChatGPT and got a clear explanation from 4o.
- profsummergig 1y ago
  Ah. Should have thought of that. Going to do that now. Thanks.
- criddell 1y ago
  Are you at all concerned that plugging stuff like this into ChatGPT is leaving you with weaker cognitive muscles? Or is it more similar to what people do when they see a new word and reach for their dictionary?
  - adwn 1y ago
    > Are you at all concerned that plugging stuff like this into ChatGPT is leaving you with weaker cognitive muscles? Couldn't this very same argument have been used against any form of mental augmentation, like written language and computers? Or, in an extended interpretation, against any form of physical augmentation, like tool use?
    - criddell 1y ago
      You can argue whatever you want to argue. I make my living with my brain so I do worry about the downsides of removing boredom and mental struggle from my days.
      - Workaccount2 1y ago
        It's almost certainly going to be bad, and almost certainly going to be unavoidable. I can't spell for shit anymore. Ever since auto correct became omnipresent in pretty much all writing fields, my brain just kinda ditched remembering how to spell words. buuuttt Manual labor has been obsolete for at least 100 years now for certain classes of people, and fitness is still an enormous recreational activity people partake in. So even in an AI heavy society, I still strongly suspect there will be "brain games" that people still enjoy and regularly play.
        criddell 1y ago
        We aren't talking about something like spelling or digging a hole. We're talking about a fundamental cognitive skill: reading eight short paragraphs of text and extracting meaning from it.
        profsummergig 1y ago
        > eight short paragraphs of text Fair point. But they are heavily metaphor-laden paragraphs. Textual interpretation is a highly subjective activity. Entire careers consist of interpreting, reinterpreting, and discussing texts that others have already interpreted. Film critics, book reviewers, political pundits, TV anchors, podcasters, etc. 'In 1972, Chinese premier Zhou Enlai was asked about the impact of the French Revolution. "Too early to say," he replied' I had my own sense of what the "coquina" metaphor stood for. I wanted to see other peoples' interpretations. Turns out my interpretation was wrong.
        profsummergig 1y ago
        > I can't spell for shit anymore. This is increasingly happening to me every day. Hope the alien overlords don't have spelling tests (as their version of IQ tests) to separate the serfs from the field-masters.
      - steve_adams_86 1y ago
        Me too. There is another side to this, which is maybe we don’t need to know a lot of things. It was true with search engines already, but maybe truer with LLMs. That thing you’re querying probably doesn’t actually matter. It’s neurotic digging and searching for an object you will never use or benefit from. The urge to seek is strong but you won’t find the thing you’re searching for this way. You might learn more by just going for a walk.
    - TimorousBestie 1y ago
      In fact it has been, dating all the way back to Phaedrus. > If men learn [writing], it will implant forgetfulness in their souls. They will cease to exercise memory because they rely on that which is written, calling things to remembrance no longer from within themselves, but by means of external marks.
  - ern 1y ago
    I see AI like the reading glasses I’ll soon need — not because I can’t think clearly, but because it helps cut through things faster when my brain’s juggling too much. A few years ago, I’d have quietly filed this kind of article under “too hard” or passed a log analysis request from the CIO down the line. Now? I get AI to draft the query, check it, run it, and move on. It’s not about thinking less — it’s about clearing the clutter so I can focus where it counts.
hecanjog 1y ago
I think that they're saying a little bit of playing around with replacing thinking and composing with automated tools is recoverable, but at an industrial or societal scale the damage is significant. Like the difference between shoveling away some sand with your hands to bury the small creatures temporarily and actually destroying their habitat by "lobbying city council members to put in a groin or seawall, and seriously move that beach sand."
- profsummergig 1y ago
  I skimmed the Anthropic report and didn't catch the negative effects. Did they mention any? Good on them if they did.
  - hecanjog 1y ago
    Yes, they mention a few times the concern that students are offloading critical thinking rather than using the tool for learning.
    - Cthulhu_ 1y ago
      I just hope the educational institutions catch on, stick with their principles and don't give them the paperwork. The paper / title should be evidence of students' learning and thinking abilities, not of just their output.
xmprt 1y ago
My understanding is that the author is this superior being trying to accomplish a massive task (damming a beach) while knowing that it could cause problems for these clams. In the real world, Anthropic is trying to accomplish a massive task (building AGI) and they're finally starting to notice the potential impacts this has on people.
jjcob 1y ago
Coquinas are clams that bury themselves in the sand very close to the surface [1]. The author worries that while they are playing with the sand, they might accidentally bury coquina clams too deep and kill them because they can no longer reach the surface. Anthropic apparently is starting to notice the possible danger to others of their work. I'm not sure what they are referring to. [1]: https://www.youtube.com/watch?v=KZUlf7quu3o https://www.youtube.com/watch?v=KZUlf7quu3o
- profsummergig 1y ago
  > Anthropic apparently is starting to notice the possible danger to others of their work. I'm not sure what they are referring to. Are they being vague about the danger? If possible, please link to a communique from them. I've missed it somehow. Thanks.
  - vermilingua 1y ago
    https://www.anthropic.com/news/anthropic-education-report-how-university-students-use-claude https://www.anthropic.com/news/anthropic-education-report-ho... Discussed here yesterday: https://news.ycombinator.com/item?id=43633383 https://news.ycombinator.com/item?id=43633383
    - profsummergig 1y ago
      Thank you.
- deathanatos 1y ago
  As a child at the beach, I would think noticing the clams would result in attempting to unearth them. Childhood curiosity about why there are bubbles. Your explanation makes more sense, however.
cubefox 1y ago
Anthropic (Claude.ai) is mentioning in their report on LLMs and education that students use Claude to cheat and do their work for them: https://www.anthropic.com/news/anthropic-education-report-how-university-students-use-claude https://www.anthropic.com/news/anthropic-education-report-ho...

doctoboggan 1y ago

This is an excellent essay, and I feel similar to the author but couldn't express it as nicely. However if we are counting on AI researchers to take the advice and slow down then I wouldn't hold my breath waiting. The author indicated they stepped away from a high paying finance job for moral reasons, which is admirable. But wallstreet continues on and does not lack for people willing to play the "make as much money as you can" game.

yapyap 1y ago
> However if we are counting on AI researchers to take the advice and slow down then I wouldn't hold my breath waiting. The author indicated they stepped away from a high paying finance job for moral reasons, which is admirable. But wallstreet continues on and does not lack for people willing to play the "make as much money as you can" game. I doubt OP is counting on it, it is moreso expressing what an optimal world would look like so people can work towards it if they would feel like it or just to put the idea out there.
dachris 1y ago
The paperclip maximizers are already here, but they are maximizing money. One recent HN comment [0] comparing corporations and institutions to AI really stuck with me - those are already superhuman intelligences. [0] https://news.ycombinator.com/item?id=43580681 https://news.ycombinator.com/item?id=43580681
- actionfromafar 1y ago
  I could imagine a Star Trek episode where someone says "I always assumed the paperclip optimizer was a parable for unchecked capitalism?"
- bitethecutebait 1y ago
  > those are already superhuman intelligence(s) ... only because "unsafe" and "leaky" are a Ponzi's best-and-loves-to-be-roofied-and-abused friend ... you see, intelligence is only good when it doesn't irreversibly break everything to the point where most of the variety of the physical structure that evolved it and maintains it is lost. you could argue, of course, and this is an abbreviated version, that a new physical structure then evolves a new intelligence that is adapted (emerged from and adjusts to) to the challenges of the new environment but that's not the point of already capable self-healing systems; except if the destructive part of the superhuman intelligence is more successful with it's methods of sabotage and disruption of (a) 'truthy' information flow and b) individual and collective super-rational agency -- for the good of as many systems-internal entities as possible, as a precaution due to always living in uncertainty and being surrounded by an endless amount of variables currently tagged "noise" -- than it's counterpart is in enabling and propagating a) and b) ... in simpler words, if the red team FUBARS the blue team or vice versa, the superhuman intelligence can be assumed to have cancer or that at least some vital part of the force is corrupted otherwise.

unwind 1y ago

Ah, this [1] meaning of tillering (bending wood to form a bow), not this [2] (production of side shoots in grasses). The joys of new words. [1]: https://www.howtomakealongbow.co.uk/part-5-tillering https://www.howtomakealongbow.co.uk/part-5-tillering [2]: https://en.wikipedia.org/wiki/Tiller_(botany) https://en.wikipedia.org/wiki/Tiller_(botany)

defrost 1y ago
As I recall tillering is more about the shaping of the bow to achieve an optimal bend and force delivery on release. It's an iterative process of bending and shaping, bending again, and wood removal in stages.

DanAtC 1y ago

[flagged]

axpvms 1y ago

My backyard creek also had crocodiles in it.

seafoamteal 1y ago
Florida?
- tilne 1y ago
  No they’ve got a little place on the Nile

MrBuddyCasino 1y ago

That was a well written essay with a non-sequitur AI Safety thing tacked to the end. His real world examples were concrete, and the reason to stop escalating easy to understand ("don't flood the neighbourhood by building a real dam"). The AI angle is not only even hypothetical: there is no attempt to describe or reason about a concrete "x leading to y", just "see, the same principle probably extrapolates". There is no argument there that is sounder than "the high velocities of steam locomotives might kill you" that people made 200 years ago.

luc4sdreyer 1y ago
> the high velocities of steam locomotives might kill you This obviously seems silly in hindsight. Warnings about radium watches or asbestos sound less silly, or even wise. But neither had any solid scientific studies showing clear hazard and risk. Just people being good Bayesian agents, trying to ride the middle of the exploration vs. exploitation curve. Maybe it makes sense to spend some percentage of AI development resources on trying to understand how they work, and how they can fail.
- ripe 1y ago
  > This [steam locomotives might kill you] obviously seems silly in hindsight. To be fair, many people did die on level crossings and by wandering on to the tracks. We learned over time to put in place safety fences and tunnels.
  - Gracana 1y ago
    People thought that the speed itself was dangerous, that the wind and vibration and landscape screaming by at 25mph would cause physical and mental harm.
- MrBuddyCasino 1y ago
  > Warnings about radium watches or asbestos sound less silly, or even wise. But neither had any solid scientific studies showing clear hazard and risk. In the case of asbestos, this is incorrect. Many people knew it was deadly, but the corporations selling it hid it for decades, killing thousands of people. There are quite a few other examples besides asbestos, like leaded fuel or cigarettes.

khazhoux 1y ago

Parents: you know how every day you look at your child and you’re struck with wonder at the amazing and quirky and unique person your little one is? I swear that’s what lesswrong posters see every day in the mirror.

DrSiemer 1y ago

So many articles and comments claim Ai will destroy critical thinking in our youths. Is there any evidence that this conviction that many people share is even remotely true? To me it just seems like the same old knee-jerk luddite response people have to any powerful new technology that challenges that status quo since the dawn of time. The calculator did not erase math wizards, the television did not replace books and so on. It just made us better, faster, more productive. Sometimes there is an adjustment period (we still haven't figured out how to deal with short dopamine hits from certain types of entertainment and social media), but things will balance themselves out eventually. Some people may go full-on Wall-E, but I for one will never stop tinkering, and many of my friends won't either. The things I could have done if I had had an LLM as a kid... I think I've learned more in the past two years than ever before.

iNic 1y ago
I don't think you got the point of the article? It is saying that we as wise humans know (sometimes) when to stop optimizing for a goal, due to the negative side effects. AIs (and as some other people have pointed out corporations) do not naturally have this line in their head, and we must draw such lines carefully and with purpose for these superhuman beings.
dsign 1y ago
> Ai will destroy critical thinking in our youths I don't think that's the argument the article was making. It was, to my understanding, a more nuanced question about if we want to destroy or severely disturb systems at equilibrium by letting AI systems infiltrate our society. > Sometimes there is an adjustment period (we still haven't figured out how to deal with short dopamine hits from certain types of entertainment and social media), but things will balance themselves out eventually. One can zoom out a little bit. The issue didn't start with social media, nor AI. "Star Wars, A New Hope", is, to my understanding, an incredibly good film. It came out in 1977 and it's a great story made to be appreciated by the masses. And in trying to achieve that goal, it really wasn't intellectually challenging. We have continued in that downhill for a bit, and now we are in 16 second stingers in TikTok and Youtube. So, the way I see it, things are not balancing out. Worse, people in USA elected D.J. Trump because somehow they couldn't understand how this real-world Emperor Palpatine was the bad guy.

BrenBarn 1y ago

It's a nice article. In a way though it kind of bypasses what I see as the main takeaways. It's not about AI development, it's about something mentioned earlier in the article: "make as much money as I can". The problems that we see with AI have little to do with AI "development", they have to do with AI marketing and promulgation. If the author had gone ahead and dammed the creek with a shovel, or blown off his hand, that would have been bad, but not that bad. Those kinds of mistakes are self-limiting because if you're doing something for the enjoyment or challenge of it, you won't do it at a scale that creates more enjoyment than you personally can experience. In the parable of the CEO and the fisherman, the fisherman stops at what he can tangibly appreciate. If everyone working on and using AI were approaching it like damming a creek for fun, we would have no problems. The AI models we had might be powerful, but they would be funky and disjointed because people would be more interested in tinkering with them than making money from them. We see tons of posts on HN every day about remarkable things people do for the gusto. We'd see a bunch of posts about new AI models and people would talk about how cool they are and go on not using them in any load-bearing way. As soon as people start trying to use anything, AI or not, to make as much money as possible, we have a problem. The second missed takeaway is at the end. He says Anthropic is noticing the coquinas as if that means they're going to somehow self-regulate. But in most of the examples he gives, he wasn't stopped by his own realization, but by an external authority (like parents) telling him to stop. Most people are not as self-reflective as this author and won't care about "winning zero sum games against people who don't necessarily deserve to lose", let alone about coquinas. They need a parent to step in and take the shovel away. As long as we keep treating "making as much money as you can" as some kind of exception to the principle of "you can't keep doing stuff until you break something", we'll have these problems, AI or not.

noduerme 1y ago
This is such a well-written response. There's something intentionally soothing about this post that slowly turns into a jarring form of self-congratulation as it goes along. Congratulations for knowing there's a limit to wrecking your parents' property. Congratulations for being able to appreciate the sand on the beach, in some no doubt instagrammable moment of existential simplicity. Congratulations for being so smart that you could have blown up your hand. And for "Leetcoding", whatever the fuck that means. And for claiming you quit a shady job because you got bored (but possibly also grew a conscience). And then topped off by the final turn: "This is, of course, about artificial intelligence development". I'd only add one thing to your analysis: We've got a demo right here of a psyche that would prefer love to money (but mostly both), and it's still determined to foist bad things onto the world in a load-bearing way, as a bid for either, or whatever it can get. My parents used to call that "a kid that doesn't care if he gets good or bad attention, as long as he gets attention." I think that's the root driver for almost all the tech billionaires of the past 20 years, and the one thing that unites Bezos, Zuck, Jobs, Dorsey, Musk... it's: "Look dad, I didn't just take your money. I'm so smart I could'a blown off my hand with all those fireworks you bought me, but see? Two hands! Look how much money I made from your money! Why aren't you proud of me?! Where can I find love? Maybe if I tell people what a leetcoder I am and how I could be making BAD AI but I'm just making GOOD AI, then everyone will love me." Don't get me wrong, I'm not immune to these feelings either. I want to do good work and I want people to love what I do. But there's something so... so fucking nakedly exhibitionist and narcissistic about these kinds of posts. Like, so, GO FUCKING LAY WITH CLAMS, write a novel, the world is waiting for it if you're really a genius. Have the courage to say you have a conscience if you actually do. Leave the rest of us alone and stop polluting a world you don't understand with your childish greed and self-obsession.

praptak 1y ago

If there's money to be made, there will always be someone with a shovel or a truckload of sparklers who is willing to take the risk (especially if the risk can be externalized to the public) and reap the reward.

A_D_E_P_T 1y ago

The author seems concerned about AI risk -- as in, "they're going to kill us all" -- and that's a common LW trope. Yet, as a regular user of SOTA AI models, it's far from clear to me that the risk exists on any foreseeable time horizon. Even today's best models are credulous and lack a certain insight and originality. As Dwarkesh once asked: > One question I had for you while we were talking about the intelligence stuff was, as a scientist yourself, what do you make of the fact that these things have basically the entire corpus of human knowledge memorized and they haven’t been able to make a single new connection that has led to a discovery? Whereas if even a moderately intelligent person had this much stuff memorized, they would notice — Oh, this thing causes this symptom. This other thing also causes this symptom. There’s a medical cure right here. > Shouldn’t we be expecting that kind of stuff? I noticed this myself just the other day. I asked GPT-4.5 "Deep Research" what material would make the best [mechanical part]. The top response I got was directly copied from a laughably stupid LinkedIn essay. The second response was derived from some marketingslop press release. There was no original insight at all. What I took away from my prompt was that I'd have to do the research and experimentation myself. Point is, I don't think that LLMs are capable of coming up with terrifyingly novel ways to kill all humans. And this hasn't changed at all over the past five years. Now they're able to trawl LinkedIn posts and browse the web for press releases, is all. More than that, these models lack independent volition and they have no temporal/spatial sense. It's not clear, from first principles, that they can operate as truly independent agents.

tvc015 1y ago
Aren’t semiautonomous drones already killing soldiers in Ukraine? Can you not imagine a future with more conflict and automated killing? Maybe that’s not seen as AI risk per se?
- A_D_E_P_T 1y ago
  That's not "AI risk" because they're still tools that lack independent volition. Somebody's building them and setting them loose. They're not building themselves and setting themselves loose, and it's far from clear how to get there from here. Dumb bombs kill people just as easily. One 80-year old nuke is, at least potentially, more effective than the entirety of the world's drones.

FollowingTheDao 1y ago

"It was only once I got it that I realized I no longer could play the game "make as much money as I can."" Funny, that is what my father taught me when I was 12 because we had compassion. What is it with glorifying all these logic loving Spock like people? Don't you know Captain Kirk was the real hero of Star Trek? Because he had compassion? It is no wonder the Zizians were birthed from LW.

appleorchard46 1y ago

Could someone explain the metaphor? I'm struggling to see the connection between AI and the rest of the post.

ido 1y ago
That AI is dangerous and the closer we get to the danger zone the better it would be if the companies developing these technologies understand it might be better to slow down and make sure it's safe vs pushing ahead at maximum speed.
- appleorchard46 1y ago
  Thank you.
- nektro 1y ago
  if they were capable of that they wouldn't have started. but the second best time to stop is now so here's hoping theyre open to our calls

curtisszmania 1y ago

[dead]

migueldeicaza 1y ago

Vonnegut said it best: https://richardswsmith.wordpress.com/2017/11/18/we-are-here-on-earth-to-fart-around-and-dont-let-anybody-tell-you-any-different/ https://richardswsmith.wordpress.com/2017/11/18/we-are-here-...

broabprobe 1y ago
huh, I wonder if he has relayed this story multiple times, I’m only familiar with this version, https://www.goodreads.com/quotes/12020560-talking-about-when-he-tells-his-wife-he-s-going-out https://www.goodreads.com/quotes/12020560-talking-about-when... “(talking about when he tells his wife he’s going out to buy an envelope) Oh, she says well, you’re not a poor man. You know, why don’t you go online and buy a hundred envelopes and put them in the closet? And so I pretend not to hear her. And go out to get an envelope because I’m going to have a hell of a good time in the process of buying one envelope. I meet a lot of people. And, see some great looking babes. And a fire engine goes by. And I give them the thumbs up. And, and ask a woman what kind of dog that is. And, and I don’t know. The moral of the story is, is we’re here on Earth to fart around. And, of course, the computers will do us out of that. And, what the computer people don’t realize, or they don’t care, is we’re dancing animals.” ― Kurt Vonnegut
- Thorrez 1y ago
  He ignores his wife's suggestion because, among other things, he wants to see some great looking babes. Maybe this isn't a guy whose philosophy I want to follow.
  - ryandrake 1y ago
    Looks like you're completely missing the point of the quote and instead rat-holing on one word that you don't like. HN in a nutshell.
    - Thorrez 1y ago
      I understand the point of the quote. My point was that if someone advocates for something I consider immoral, I prefer to disregard the overall philosophy that lead the person to advocate for that. If the quote threw in some casual racism, or advocated for stealing the envelope instead of paying for it, I would similarly disregard the overall philosophy.

Isamu 1y ago

>After I cracked the trick of tillering Guide to Bow Tillering: https://straightgrainedboard.com/beginners-guide-on-bow-tillering/ https://straightgrainedboard.com/beginners-guide-on-bow-till...

red_admiral 1y ago

The story of playing at damming the creek or on the sand at the seaside is wholesome and brought a smile to my face. Cracking the "puzzle" is almost the bad ending of the game, if you don't get any fun at playing it anymore. People should spend more of their time doing things because they're fun, not because they want to get better at it. Maybe the apocalypse will happen in our lifetime, maybe not. I intend to have fun as much as I can in my life either way.

bogdanoff_2 1y ago

The solution to this problem is to choose a "game" that you 100% believe will positively impact the world.

ziofill 1y ago

This is a tangent, but I would love so much to be able to give my kids memories of playing in a creek in the backyard...