4 ms·

> I'm not saying that C.AI is completely blameless here I know we're several decades into this pattern, but it's sad to me that we've just given up on that ide

> I'm not saying that C.AI is completely blameless here I know we're several decades into this pattern, but it's sad to me that we've just given up on that idea that businesses should have a net positive impact on society, that we've just decided there is nothing we can or should do about companies that actively exploit us to enrich themselves, that we give them a pass to ignore the obvious detrimental second-order effects of their business model.

stickfigure 2y ago
Meh. There's a long history (especially here on HN) of hyperfocusing on unfortunate edge cases of technology and ignoring the vast good they do. Someone posts some BS on twitterface and it leads to a lynching - yes, bad, but this is the exception not the rule. The rule is that billions of people can now communicate directly with each other in nearly real time, which is incredible. So call me skeptical. Maybe the tech isn't perfect, but it will never be perfect. Does it do more harm than good? I don't know enough about this product, but I am not going to draw a conclusion from one lawsuit.
- llamaimperative 2y ago
  There's a long history of taking the dulled down, de-risked, mitigated, and ultimately successful technologies that we've allowed to proliferate our society and say "see, no need to do dulling down, de-risking, mitigation!" Bioweapons haven't proliferated through dedicated effort to prevent it. Nuclear weapons aren't used through dedicated effort to prevent it. Gangs don't rule our civilization through dedicated effort to prevent it. Chattel slavery doesn't exist in the western world through dedicated effort to eliminate and prevent it. Bad outcomes aren't impossible by default, and they're probably not even less likely than good outcomes. Bad outcomes are avoided through effort to avoid them!
  - stickfigure 2y ago
    You just compared twitter to slavery and nuclear weapons. Clearly Godwin's law is overdue for an update.
  - Eisenstein 2y ago
    Yet we also had 'comic books are making kids amoral and violent', 'TV is making kids amoral and violent', 'video games are making kids amoral and violent', 'dungeons and dragons is making kids amoral and violent'...
    - llamaimperative 2y ago
      Sure! Some technologies are fine. Some of them are not. The existence of one set does not negate the existence of the other.
    - p_j_w 2y ago
      The existence of those movements doesnt say much except that we need to carefully consider things to avoid overreach. Unless you think we should rethink our approaches to Chattel Slavery and proliferation of biological weapons because we want to avoid people complaining about and trying unsuccessfully to ban violent video games.
      - Eisenstein 2y ago
        > The existence of those movements doesnt say much except that we need to carefully consider things to avoid overreach. It says that people react to new media technologies that target children the same way pretty consistently.
        llamaimperative 2y ago
        No, it says that you can decide what reaction you're looking for across a global population of billions of people and succeed in finding it. True of any reaction to any object or idea.
        Eisenstein 2y ago
        I'm sure you could, but the things I mentioned are culturally American, as is the lawsuit.
xvector 2y ago
I think it's also entirely reasonable to expect parents to actually parent, instead of installing foam bumpers and a nanny state everywhere in case some kid hurts themselves. If the parents weren't absent and actually used parental controls, the kids wouldn't have even been able to download the app, which is explicitly marked as 17+. C.AI's entire customer base consists of those that like the edgy, unrestricted AI, and they shouldn't have to suffer a neutered product because of some lazy parents.
- empath75 2y ago
  It was 12+ at the time and only recently changed to 17+
- ethbr1 2y ago
  It's a bit easy, from the historical perspective of pre-always-available-internet, to say "Parents should do more." At some future point though, maybe we need to accept that social changes are necessary to account for a default firewall-less exposure of a developing mind to the full horrors of the world's information systems (and the terrible people using them).
  - anonzzzies 2y ago
    You would have to continuously monitor everything, everywhere; before the internet, in the 80s, it was easy for us to get porn (mags/vhs), weed, all kinds of books that glorify death or whatever, music in that similar vain. Hell, they even had and read from a bible in some schools then; talk about indoctrination of often scary fiction. Some kids had different parents so to not allow us to see or get our hands on these things, parents and teachers would need sit with us every waking moment; it's not possible (or healthy imho). With access to a phone or laptop, all bets are off: everything is there, doesn't matter what restraints are in place; kids know how to install vpns, pick birthdates, use torrent, or, more innocent, go to a forum (these days social media but forums are still there) about something they love and go to other parts of the same forum where other stuff happens. Be good parents, educate about what happens in the world including that people irl but definitely online might not be serious about what they say and that you should not take anything without critical thought. And for stuff that will happen anyway; sex, drugs etc, make sure it's a controlled environment as much as possible. Not much more you can do to protect from the big, bad world. Chat bots are similarly genies that are not possible to keep in, no matter what levels or restraint or law are put in place; you can torrent ollama or whatever with llama 3.3 locally. There are easy to get nsfw bots everywhere, including on decentralised shares. It is not possible to prevent them talking about anything as they do not understand anything; they helpfully generate stuff which is a great invention and I use them all the time, but they lie and tell strange things sometimes. People do too, only people would have a reason maybe; to get a reaction, to be mean etc; doubt you could sue them in a similar case. Of course a big company would need to do something to try to prevent is: they cannot (as said above), so they can just make character ai 18+ with Cc payment in their name as kyc (then the parents have a problem if that happens you would think) and cover their asses; plenty commercial and free ones kids will get instead. And some of those are far 'worse'.
beepbooptheory 2y ago
This feels a little like being sad that it rains sometimes, or that Santa Claus doesnt exist. I just can't even connect with the mindset that would mourn such a thing. What even is the theory behind such an idea? Like how can one, even in theory, make more and more money every year and remain positive for society? What even could assure such a relation? Is everyone just doing something "wrong" here?
- voltaireodactyl 2y ago
  Traditionally one role of government has been to provide legislative oversight to temper unadulterated pursuit of profits. Lobbying and the related ills have definitely undercut that role significantly. But the theory is that government provides the guardrails within which business should operate.
- Dylan16807 2y ago
  Yes, the companies that prioritize growing forever are all doing something wrong.
  - Nevermark 2y ago
    > Yes, the companies that prioritize growing forever <<irrespective of harm they are causing>> are all doing something wrong. <<I think that makes sense with my edit>>
- ChadNauseam 2y ago
  > Like how can one, even in theory, make more and more money every year and remain positive for society? By providing more and more goods and services that people value, or by providing more and more valuable goods and services > What even could assure such a relation? A government that tries to proactively identify ways that someone could make money in a way that could be harmful or helpful to people who aren't involved in the transaction, and tax those things based on to the degree that they are harmful, and subsidize them to the degree that they are helpful. The government should also take actions to prevent monopolies from forming, and to make sure everyone has as much information as possible before making a transaction.
hombre_fatal 2y ago
An individual case where things went wrong isn't enough to determine whether Character.AI or LLMs are a net negative for society. The analysis can't just stop there or else we'd have nothing.
- ajb 2y ago
  No, but it's also not good enough to just look at "are they positive on average". We are talking here about actions that even a pig-butcher would think twice about.
tryauuum 2y ago
and I don't want government to decide what is "good impact on society" when regulating websites. This is how you get roskomnadzor
- blackoil 2y ago
  Government could manage companies to give right tools to the parents to do the parenting.
- Nevermark 2y ago
  Governments could outlaw or down regulate business models that are based on surveillance and feed manipulation for benefit of third parties. Conflicts of interest have always been a problem. Now they scale. Now there are companies with hundreds of billions of dollars of market cap, with unrelenting pressure to grow, and their business model is an obvious conflict of interest. Things will get worse and worse until (if ever) the massive scaling of manipulative personalization is recognized as inherently deeply damaging to individuals and society. None of that would require targeting any particular content.
benatkin 2y ago
> it's sad to me that we've just given up on that idea that businesses should have a net positive impact on society Eh, we don't want to encourage companies to sugar coat what they're doing.
- Nevermark 2y ago
  Being accepting of damaging behavior and damagers so they feel safe in the open is solving the wrong problem.
empath75 2y ago
Character AI's entire business model is depraved and exploitative, and everyone involved should be ashamed of themselves.

stickfigure 2y ago

Meh. There's a long history (especially here on HN) of hyperfocusing on unfortunate edge cases of technology and ignoring the vast good they do. Someone posts some BS on twitterface and it leads to a lynching - yes, bad, but this is the exception not the rule. The rule is that billions of people can now communicate directly with each other in nearly real time, which is incredible. So call me skeptical. Maybe the tech isn't perfect, but it will never be perfect. Does it do more harm than good? I don't know enough about this product, but I am not going to draw a conclusion from one lawsuit.

llamaimperative 2y ago
There's a long history of taking the dulled down, de-risked, mitigated, and ultimately successful technologies that we've allowed to proliferate our society and say "see, no need to do dulling down, de-risking, mitigation!" Bioweapons haven't proliferated through dedicated effort to prevent it. Nuclear weapons aren't used through dedicated effort to prevent it. Gangs don't rule our civilization through dedicated effort to prevent it. Chattel slavery doesn't exist in the western world through dedicated effort to eliminate and prevent it. Bad outcomes aren't impossible by default, and they're probably not even less likely than good outcomes. Bad outcomes are avoided through effort to avoid them!
- stickfigure 2y ago
  You just compared twitter to slavery and nuclear weapons. Clearly Godwin's law is overdue for an update.
- Eisenstein 2y ago
  Yet we also had 'comic books are making kids amoral and violent', 'TV is making kids amoral and violent', 'video games are making kids amoral and violent', 'dungeons and dragons is making kids amoral and violent'...
  - llamaimperative 2y ago
    Sure! Some technologies are fine. Some of them are not. The existence of one set does not negate the existence of the other.
  - p_j_w 2y ago
    The existence of those movements doesnt say much except that we need to carefully consider things to avoid overreach. Unless you think we should rethink our approaches to Chattel Slavery and proliferation of biological weapons because we want to avoid people complaining about and trying unsuccessfully to ban violent video games.
    - Eisenstein 2y ago
      > The existence of those movements doesnt say much except that we need to carefully consider things to avoid overreach. It says that people react to new media technologies that target children the same way pretty consistently.
      - llamaimperative 2y ago
        No, it says that you can decide what reaction you're looking for across a global population of billions of people and succeed in finding it. True of any reaction to any object or idea.
        Eisenstein 2y ago
        I'm sure you could, but the things I mentioned are culturally American, as is the lawsuit.

xvector 2y ago

I think it's also entirely reasonable to expect parents to actually parent, instead of installing foam bumpers and a nanny state everywhere in case some kid hurts themselves. If the parents weren't absent and actually used parental controls, the kids wouldn't have even been able to download the app, which is explicitly marked as 17+. C.AI's entire customer base consists of those that like the edgy, unrestricted AI, and they shouldn't have to suffer a neutered product because of some lazy parents.

empath75 2y ago
It was 12+ at the time and only recently changed to 17+
ethbr1 2y ago
It's a bit easy, from the historical perspective of pre-always-available-internet, to say "Parents should do more." At some future point though, maybe we need to accept that social changes are necessary to account for a default firewall-less exposure of a developing mind to the full horrors of the world's information systems (and the terrible people using them).
- anonzzzies 2y ago
  You would have to continuously monitor everything, everywhere; before the internet, in the 80s, it was easy for us to get porn (mags/vhs), weed, all kinds of books that glorify death or whatever, music in that similar vain. Hell, they even had and read from a bible in some schools then; talk about indoctrination of often scary fiction. Some kids had different parents so to not allow us to see or get our hands on these things, parents and teachers would need sit with us every waking moment; it's not possible (or healthy imho). With access to a phone or laptop, all bets are off: everything is there, doesn't matter what restraints are in place; kids know how to install vpns, pick birthdates, use torrent, or, more innocent, go to a forum (these days social media but forums are still there) about something they love and go to other parts of the same forum where other stuff happens. Be good parents, educate about what happens in the world including that people irl but definitely online might not be serious about what they say and that you should not take anything without critical thought. And for stuff that will happen anyway; sex, drugs etc, make sure it's a controlled environment as much as possible. Not much more you can do to protect from the big, bad world. Chat bots are similarly genies that are not possible to keep in, no matter what levels or restraint or law are put in place; you can torrent ollama or whatever with llama 3.3 locally. There are easy to get nsfw bots everywhere, including on decentralised shares. It is not possible to prevent them talking about anything as they do not understand anything; they helpfully generate stuff which is a great invention and I use them all the time, but they lie and tell strange things sometimes. People do too, only people would have a reason maybe; to get a reaction, to be mean etc; doubt you could sue them in a similar case. Of course a big company would need to do something to try to prevent is: they cannot (as said above), so they can just make character ai 18+ with Cc payment in their name as kyc (then the parents have a problem if that happens you would think) and cover their asses; plenty commercial and free ones kids will get instead. And some of those are far 'worse'.

beepbooptheory 2y ago

This feels a little like being sad that it rains sometimes, or that Santa Claus doesnt exist. I just can't even connect with the mindset that would mourn such a thing. What even is the theory behind such an idea? Like how can one, even in theory, make more and more money every year and remain positive for society? What even could assure such a relation? Is everyone just doing something "wrong" here?

voltaireodactyl 2y ago
Traditionally one role of government has been to provide legislative oversight to temper unadulterated pursuit of profits. Lobbying and the related ills have definitely undercut that role significantly. But the theory is that government provides the guardrails within which business should operate.
Dylan16807 2y ago
Yes, the companies that prioritize growing forever are all doing something wrong.
- Nevermark 2y ago
  > Yes, the companies that prioritize growing forever <<irrespective of harm they are causing>> are all doing something wrong. <<I think that makes sense with my edit>>
ChadNauseam 2y ago
> Like how can one, even in theory, make more and more money every year and remain positive for society? By providing more and more goods and services that people value, or by providing more and more valuable goods and services > What even could assure such a relation? A government that tries to proactively identify ways that someone could make money in a way that could be harmful or helpful to people who aren't involved in the transaction, and tax those things based on to the degree that they are harmful, and subsidize them to the degree that they are helpful. The government should also take actions to prevent monopolies from forming, and to make sure everyone has as much information as possible before making a transaction.

hombre_fatal 2y ago

An individual case where things went wrong isn't enough to determine whether Character.AI or LLMs are a net negative for society. The analysis can't just stop there or else we'd have nothing.

ajb 2y ago
No, but it's also not good enough to just look at "are they positive on average". We are talking here about actions that even a pig-butcher would think twice about.

tryauuum 2y ago

and I don't want government to decide what is "good impact on society" when regulating websites. This is how you get roskomnadzor

blackoil 2y ago
Government could manage companies to give right tools to the parents to do the parenting.
Nevermark 2y ago
Governments could outlaw or down regulate business models that are based on surveillance and feed manipulation for benefit of third parties. Conflicts of interest have always been a problem. Now they scale. Now there are companies with hundreds of billions of dollars of market cap, with unrelenting pressure to grow, and their business model is an obvious conflict of interest. Things will get worse and worse until (if ever) the massive scaling of manipulative personalization is recognized as inherently deeply damaging to individuals and society. None of that would require targeting any particular content.

benatkin 2y ago

> it's sad to me that we've just given up on that idea that businesses should have a net positive impact on society Eh, we don't want to encourage companies to sugar coat what they're doing.

Nevermark 2y ago
Being accepting of damaging behavior and damagers so they feel safe in the open is solving the wrong problem.

empath75 2y ago

Character AI's entire business model is depraved and exploitative, and everyone involved should be ashamed of themselves.