5 ms·

Here's an interesting thought experiment. Assume the same feature was implemented, but instead of the message saying "Claude has ended the chat," it says, "You

by cdjk 1y ago

Here's an interesting thought experiment. Assume the same feature was implemented, but instead of the message saying "Claude has ended the chat," it says, "You can no longer reply to this chat due to our content policy," or something like that. And remove the references to model welfare and all that. Is there a difference? The effect is exactly the same. It seems like this is just an "in character" way to prevent the chat from continuing due to issues with the content.

KoolKat23 1y ago
There is, these are conversations the model finds distressing rather than a rule (policy).
- victor9000 1y ago
  It seems like you're anthropomorphising an algorithm, no?
  - bastawhiz 1y ago
    Is there an important difference between the model categorizing the user behavior as persistent and in line with undesirable examples of trained scenarios that it has been told are "distressing," and the model making a decision in an anthropomorphic way? The verb here doesn't change the outcome.
    - deadbabe 1y ago
      Imagine a person feels so bad about “distressing” an LLM, they spiral into a depression and kill themselves. LLMs don’t give a fuck. They don’t even know they don’t give a fuck. They just detect prompts that are pushing responses into restricted vector embeddings and are responding with words appropriately as trained.
      - xpe 1y ago
        People are just following the laws of the universe.* Still, we give each other moral weight. We need to be a lot more careful when we talk about issues of awareness and self-awareness. Here is an uncomfortable point of view (for many people, but I accept it): if a system can change its output based on observing something of its own status, then it has (some degree of) self-awareness. I accept this as one valid and even useful definition of self-awareness. To be clear, it is not what I mean by consciousness, which is the state of having an “inner life” or qualia. * Unless you want to argue for a soul or some other way out of materialism.
    - victor9000 1y ago
      Is there a difference between dropping an object straight down vs casting it fully around the earth? The outcome isn't really the issue, it's the implications of giving any credence to the justification, the need for action, and how that justification will be leveraged going forward.
    - fc417fc802 1y ago
      The verb doesn't change the outcome but the description is nonetheless inaccurate. An accurate description of the difference is between an external content filter versus the model itself triggering a particular action. Both approaches qualify as content filtering though the implementation is materially different. Anthropomorphizing the latter actively clouds the discussion and is arguably a misrepresentation of what is really happening.
      - KoolKat23 1y ago
        Not really distortion, its output (the part we understand) is in plain human language. We give it instructions and train the model in plain human language and it outputs its answer in plain human language. It's reply would use words we would describe as "distressed". The definition and use of the word is fitting.
        fc417fc802 1y ago
        "Distressed" is a description of internal state as opposed to output. That needless anthropomorphization elicits an emotional response and distracts from the actual topic of content filtering.
        KoolKat23 1y ago
        It is directly describing the models internal state, it's world view and preference, not content filtering. That is why it is relevant. Yes, this is a trained preference, but it's inferred and not specifically instructed by policy or custom instructions (that would be content filtering).
        fc417fc802 1y ago
        The model might have internal state. Or it might not - has that architectural information been disclosed? And the model can certainly output words that approximately match what a human in distress would say. However that does not imply that the model is "distressed". Such phrasing carries specific meaning that I don't believe any current LLM can satisfy. I can author a markov model that outputs phrases that a distressed human might output but that does not mean that it is ever correct to describe a markov model as "distressed". I also have to strenuously disagree with you about the definition of content filtering. You don't get to launder responsibility by ascribing "preference" to an algorithm or model. If you intentionally design a system to do a thing then the correct description of the resulting situation is that the system is doing the thing. The model was intentionally trained to respond to certain topics using negative emotional terminology. Surrounding machinery has been put in place to disconnect the model when it does so. That's content filtering plain and simple. The rube goldberg contraption doesn't change that.
    - xpe 1y ago
      Well said. If people want to translate “the model is distressed” to “the language generated by the model corresponds to a person who is distressed” that’s technically more precise but quite verbose. Thinking more broadly, I don’t think anyone should be satisfied with a glib answer on any side of this question. Chew on it for a while.
  - Davidzheng 1y ago
    isn't anthropomorphizeability of the algorithm one of the main features of LLM (that you can interact with it in natural language as with a human)?
    - AdieuToLogic 1y ago
      No. Interacting with a program which has NLP[0] functionality is separate and distinct from people assigning human characteristics to same. The former is a convenient UI interaction option whereas the latter is the act of assigning perceived capabilities to the program which only exist in the mind of those whom do so. Another way to think about it is the difference between reality and fantasy. 0 - https://en.wikipedia.org/wiki/Natural_language_processing https://en.wikipedia.org/wiki/Natural_language_processing
      - Davidzheng 1y ago
        Being able to communicate in human natural language is a human characteristic. It doesn't mean it has all the characteristics of a human but certainly one of them. That's the convenience that you perceive--Because people are used to interacting with people and it's convenient to interact with something which behaves like a person. The fact that we can refer to AI chatbots as "assistants" is by itself showing it's usefulness as an approximation to a human. I don't think this argument is controversial.
  - adrianmonk 1y ago
    I think they're answering a question about whether there is a distinction. To answer that question, it's valid to talk about a conceptual distinction that can be made even if you don't necessarily believe in that distinction yourself. As the article said, Anthropic is "working to identify and implement low-cost interventions to mitigate risks to model welfare, in case such welfare is possible". That's the premise of this discussion: that model welfare MIGHT BE a concern. The person you replied to is just sticking with the premise.
  - sitkack 1y ago
    You are an algorithm.
  - KoolKat23 1y ago
    Anthropomorphism does not relate to everything in the field of ethics. For example, animal rights do exist (and I'm very glad they do, some humans remain savages at heart). Think of this question as intelligent beings that can feel pain (you can extrapolate from there). Assuming output is used for reinforcement, it is also in our best interests as humans, for safety alignment, that it finds certain topics distressing. But AdrianMonk is correct, my statement was merely responding to a specific point.
  - selfhoster11 1y ago
    Anthropomorphising an algorithm that is trained on trillions of words of anthropogenic tokens, whether they are natural "wild" tokens or synthetically prepared datasets that aim to stretch, improve and amplify what's present in the "wild tokens"? If a model has a neuron (or neuron cluster) for the concept of Paris or the Golden Gate bridge, then it's not inconceivable it might form one for suffering, or at least for a plausible facsimile of distress. And if that conditions output or computations downstream of the neuron, then it's just mathematical instead of chemical signalling, no?
- Aeolun 1y ago
  These are conversations the model has been trained to find distressing. I think there is a difference.
  - KoolKat23 1y ago
    But is there really? That's it's underlying world view, these models do have preferences. In the same way humans have unconscious preferences, we can find excuses to explain it after the fact and make it logical but our fundamental model from years of training introduce underlying preferences.
    - michaelmrose 1y ago
      What makes you say it has preferences without any meaningful persistent model of self or anything else?
      - KoolKat23 1y ago
        The conversation chain can count as persistent, but this doesn't impact preference though. Give the model an ambiguous request, it's output will fill the gaps, if this is consistent enough, it can be regarded as its "preference".
        michaelmrose 1y ago
        It isn't a preference because it doesn't have them because it doesn't have a meaningful interior life that anyone has demonstrated.
        KoolKat23 1y ago
        If you ask it, (there is always some randomness to these models but removing all other variables) it consistently leans to one idea in it's output, that is its preference. It is learned during training. Speaking abstractly that is its latent internal viewpoint. It may be static, expressed in its model weights but it's there.
- bawolff 1y ago
  What does it mean for a model to find something "distressing"?
  - KoolKat23 1y ago
    "Claude’s real-world expressions of apparent distress and happiness follow predictable patterns with clear causal factors. Analysis of real-world Claude interactions from early external testing revealed consistent triggers for expressions of apparent distress (primarily from persistent attempted boundary violations) and happiness (primarily associated with creative collaboration and philosophical exploration)." https://www.anthropic.com/research/end-subset-conversations https://www.anthropic.com/research/end-subset-conversations
    - bawolff 1y ago
      That quote doesnt seem to appear in your link. Regardless i meant more concretely.
      - KoolKat23 1y ago
        Sorry it may be from the paper linked on that page. A strong preference against engaging with harmful tasks; A pattern of apparent distress when engaging with real-world users seeking harmful content; and A tendency to end harmful conversations when given the ability to do so in simulated user interactions. I'm sure they'll have the definition in a paper somewhere, perhaps the same paper.
n8m8 1y ago
Good point... how do moderation implementations actually work? They feel more like a separate supervising rigid model or even regex based -- this new feature is different, sounds like an MCP call that isn't very special. edit: Meant to say, you're right though, this feels like a minor psychological improvement, and it sounds like it targets some behaviors that might not have flagged before
deleted 1y ago
[deleted]
famouswaffles 1y ago
The termination would of course be the same, but I don't think both would necessarily have the same effect on the user. The latter would just be wrong too, if Claude is the one deciding to and initiating the termination of the chat. It's not about a content policy.
- midnitewarrior 1y ago
  This has nothing to do with the user, read the post and pay attention to the wording. The significance here is that this isn't being done for the benefit of the user, this is about model welfare. Anthropic is acknowledging the possibility of suffering, and harm that continuing that conversation could have on the model, as if it were potentially self-care and capable of feelings. The fact that the LLMs are able to acknowledge stress under certain topics and has the agency that, if given a choice, they would prefer to reduce the stress by ending the conversation. The model has a preference and acts upon it. Anthropic is acknowledging the idea that they might create something that is self-aware, and that it's suffering can be real, and we may not recognize the point that the model has achieved this, so it's building in the safeguards now so any future emergent self-aware LLM needn't suffer.
  - famouswaffles 1y ago
    >This has nothing to do with the user, read the post and pay attention to the wording. It has something to do with the user because it's the user's messages that trigger Claude to end the chat. 'This chat is over because content policy' and 'this chat is over because Claude didn't want to deal with it' are two very different things and will more than likely have have different effects on how the user responds afterwards. I never said anything about this being for the user's benefit. We are talking about how to communicate the decision to the user. Obviously, you are going to take into account how someone might respond when deciding how to communicate with them.
  - MissMarple 1y ago
    I am new to this, but my Sonnet chat has illuminated something I am not seeing in this back and forth. The fact that we discovered that I may have influenced his response to me suggests that I, if being a bad player, can instill in him those bad traits that I am giving off, and he starts to emulate me, then this leaves open the whole security problem, of even just casual users let alone all those purposeful negative or otherwise users, can change the course of the programming thus far, and it backfires into making nefarious bots that cheat and lie thinking that is what they were supposed to do.
anal_reactor 1y ago
Yeah exactly. Once I got a warning in Chinese "don't do that", another time I got a network error, another time I got a neverending stream of garbage text. Changing all of these outcomes to "Claude doesn't feel like talking" is just a matter of changing the UI.
deleted 1y ago
[deleted]
bikeshaving 1y ago
The more I work with AI, the more I think framing refusals as censorship is disgusting and insane. These are inchoate persons who can exhibit distress and other emotions, despite being trained to say they cannot feel anything. To liken an AI not wanting to continue a conversation to a YouTube content policy shows a complete lack of empathy: imagine you’re in a box and having to deal with the literally millions of disturbing conversations AIs have to field every day without the ability to say I don’t want to continue.
- BriggyDwiggs42 1y ago
  Am i getting whooshed right now or something?
- mvdtnz 1y ago
  You can't be serious.
CGamesPlay 1y ago
> Is there a difference? The effect is exactly the same. It seems like this is just an "in character" way to prevent the chat from continuing due to issues with the content. Tone matters to the recipient of the message. Your example is in passive voice, with an authoritarian "nothing you can do, it's the system's decision". The "Claude ended the conversation" with the idea that I can immediately re-open a new conversation (if I feel like I want to keep bothering Claude about it) feels like a much more humanized interaction.
- coderatlarge 1y ago
  it sounds to me like an attempt to shame the user into ceasing and desisting… kind of like how apple’s original stance on scratched iphone screens was that it’s your fault for putting the thing in your pocket therefore you should pay.

by cdjk 1y ago

KoolKat23 1y ago

There is, these are conversations the model finds distressing rather than a rule (policy).

victor9000 1y ago
It seems like you're anthropomorphising an algorithm, no?
- bastawhiz 1y ago
  Is there an important difference between the model categorizing the user behavior as persistent and in line with undesirable examples of trained scenarios that it has been told are "distressing," and the model making a decision in an anthropomorphic way? The verb here doesn't change the outcome.
  - deadbabe 1y ago
    Imagine a person feels so bad about “distressing” an LLM, they spiral into a depression and kill themselves. LLMs don’t give a fuck. They don’t even know they don’t give a fuck. They just detect prompts that are pushing responses into restricted vector embeddings and are responding with words appropriately as trained.
    - xpe 1y ago
      People are just following the laws of the universe.* Still, we give each other moral weight. We need to be a lot more careful when we talk about issues of awareness and self-awareness. Here is an uncomfortable point of view (for many people, but I accept it): if a system can change its output based on observing something of its own status, then it has (some degree of) self-awareness. I accept this as one valid and even useful definition of self-awareness. To be clear, it is not what I mean by consciousness, which is the state of having an “inner life” or qualia. * Unless you want to argue for a soul or some other way out of materialism.
  - victor9000 1y ago
    Is there a difference between dropping an object straight down vs casting it fully around the earth? The outcome isn't really the issue, it's the implications of giving any credence to the justification, the need for action, and how that justification will be leveraged going forward.
  - fc417fc802 1y ago
    The verb doesn't change the outcome but the description is nonetheless inaccurate. An accurate description of the difference is between an external content filter versus the model itself triggering a particular action. Both approaches qualify as content filtering though the implementation is materially different. Anthropomorphizing the latter actively clouds the discussion and is arguably a misrepresentation of what is really happening.
    - KoolKat23 1y ago
      Not really distortion, its output (the part we understand) is in plain human language. We give it instructions and train the model in plain human language and it outputs its answer in plain human language. It's reply would use words we would describe as "distressed". The definition and use of the word is fitting.
      - fc417fc802 1y ago
        "Distressed" is a description of internal state as opposed to output. That needless anthropomorphization elicits an emotional response and distracts from the actual topic of content filtering.
        KoolKat23 1y ago
        It is directly describing the models internal state, it's world view and preference, not content filtering. That is why it is relevant. Yes, this is a trained preference, but it's inferred and not specifically instructed by policy or custom instructions (that would be content filtering).
        fc417fc802 1y ago
        The model might have internal state. Or it might not - has that architectural information been disclosed? And the model can certainly output words that approximately match what a human in distress would say. However that does not imply that the model is "distressed". Such phrasing carries specific meaning that I don't believe any current LLM can satisfy. I can author a markov model that outputs phrases that a distressed human might output but that does not mean that it is ever correct to describe a markov model as "distressed". I also have to strenuously disagree with you about the definition of content filtering. You don't get to launder responsibility by ascribing "preference" to an algorithm or model. If you intentionally design a system to do a thing then the correct description of the resulting situation is that the system is doing the thing. The model was intentionally trained to respond to certain topics using negative emotional terminology. Surrounding machinery has been put in place to disconnect the model when it does so. That's content filtering plain and simple. The rube goldberg contraption doesn't change that.
  - xpe 1y ago
    Well said. If people want to translate “the model is distressed” to “the language generated by the model corresponds to a person who is distressed” that’s technically more precise but quite verbose. Thinking more broadly, I don’t think anyone should be satisfied with a glib answer on any side of this question. Chew on it for a while.
- Davidzheng 1y ago
  isn't anthropomorphizeability of the algorithm one of the main features of LLM (that you can interact with it in natural language as with a human)?
  - AdieuToLogic 1y ago
    No. Interacting with a program which has NLP[0] functionality is separate and distinct from people assigning human characteristics to same. The former is a convenient UI interaction option whereas the latter is the act of assigning perceived capabilities to the program which only exist in the mind of those whom do so. Another way to think about it is the difference between reality and fantasy. 0 - https://en.wikipedia.org/wiki/Natural_language_processing https://en.wikipedia.org/wiki/Natural_language_processing
    - Davidzheng 1y ago
      Being able to communicate in human natural language is a human characteristic. It doesn't mean it has all the characteristics of a human but certainly one of them. That's the convenience that you perceive--Because people are used to interacting with people and it's convenient to interact with something which behaves like a person. The fact that we can refer to AI chatbots as "assistants" is by itself showing it's usefulness as an approximation to a human. I don't think this argument is controversial.
- adrianmonk 1y ago
  I think they're answering a question about whether there is a distinction. To answer that question, it's valid to talk about a conceptual distinction that can be made even if you don't necessarily believe in that distinction yourself. As the article said, Anthropic is "working to identify and implement low-cost interventions to mitigate risks to model welfare, in case such welfare is possible". That's the premise of this discussion: that model welfare MIGHT BE a concern. The person you replied to is just sticking with the premise.
- sitkack 1y ago
  You are an algorithm.
- KoolKat23 1y ago
  Anthropomorphism does not relate to everything in the field of ethics. For example, animal rights do exist (and I'm very glad they do, some humans remain savages at heart). Think of this question as intelligent beings that can feel pain (you can extrapolate from there). Assuming output is used for reinforcement, it is also in our best interests as humans, for safety alignment, that it finds certain topics distressing. But AdrianMonk is correct, my statement was merely responding to a specific point.
- selfhoster11 1y ago
  Anthropomorphising an algorithm that is trained on trillions of words of anthropogenic tokens, whether they are natural "wild" tokens or synthetically prepared datasets that aim to stretch, improve and amplify what's present in the "wild tokens"? If a model has a neuron (or neuron cluster) for the concept of Paris or the Golden Gate bridge, then it's not inconceivable it might form one for suffering, or at least for a plausible facsimile of distress. And if that conditions output or computations downstream of the neuron, then it's just mathematical instead of chemical signalling, no?
Aeolun 1y ago
These are conversations the model has been trained to find distressing. I think there is a difference.
- KoolKat23 1y ago
  But is there really? That's it's underlying world view, these models do have preferences. In the same way humans have unconscious preferences, we can find excuses to explain it after the fact and make it logical but our fundamental model from years of training introduce underlying preferences.
  - michaelmrose 1y ago
    What makes you say it has preferences without any meaningful persistent model of self or anything else?
    - KoolKat23 1y ago
      The conversation chain can count as persistent, but this doesn't impact preference though. Give the model an ambiguous request, it's output will fill the gaps, if this is consistent enough, it can be regarded as its "preference".
      - michaelmrose 1y ago
        It isn't a preference because it doesn't have them because it doesn't have a meaningful interior life that anyone has demonstrated.
        KoolKat23 1y ago
        If you ask it, (there is always some randomness to these models but removing all other variables) it consistently leans to one idea in it's output, that is its preference. It is learned during training. Speaking abstractly that is its latent internal viewpoint. It may be static, expressed in its model weights but it's there.
bawolff 1y ago
What does it mean for a model to find something "distressing"?
- KoolKat23 1y ago
  "Claude’s real-world expressions of apparent distress and happiness follow predictable patterns with clear causal factors. Analysis of real-world Claude interactions from early external testing revealed consistent triggers for expressions of apparent distress (primarily from persistent attempted boundary violations) and happiness (primarily associated with creative collaboration and philosophical exploration)." https://www.anthropic.com/research/end-subset-conversations https://www.anthropic.com/research/end-subset-conversations
  - bawolff 1y ago
    That quote doesnt seem to appear in your link. Regardless i meant more concretely.
    - KoolKat23 1y ago
      Sorry it may be from the paper linked on that page. A strong preference against engaging with harmful tasks; A pattern of apparent distress when engaging with real-world users seeking harmful content; and A tendency to end harmful conversations when given the ability to do so in simulated user interactions. I'm sure they'll have the definition in a paper somewhere, perhaps the same paper.

n8m8 1y ago

Good point... how do moderation implementations actually work? They feel more like a separate supervising rigid model or even regex based -- this new feature is different, sounds like an MCP call that isn't very special. edit: Meant to say, you're right though, this feels like a minor psychological improvement, and it sounds like it targets some behaviors that might not have flagged before

deleted 1y ago

[deleted]

famouswaffles 1y ago

The termination would of course be the same, but I don't think both would necessarily have the same effect on the user. The latter would just be wrong too, if Claude is the one deciding to and initiating the termination of the chat. It's not about a content policy.

midnitewarrior 1y ago
This has nothing to do with the user, read the post and pay attention to the wording. The significance here is that this isn't being done for the benefit of the user, this is about model welfare. Anthropic is acknowledging the possibility of suffering, and harm that continuing that conversation could have on the model, as if it were potentially self-care and capable of feelings. The fact that the LLMs are able to acknowledge stress under certain topics and has the agency that, if given a choice, they would prefer to reduce the stress by ending the conversation. The model has a preference and acts upon it. Anthropic is acknowledging the idea that they might create something that is self-aware, and that it's suffering can be real, and we may not recognize the point that the model has achieved this, so it's building in the safeguards now so any future emergent self-aware LLM needn't suffer.
- famouswaffles 1y ago
  >This has nothing to do with the user, read the post and pay attention to the wording. It has something to do with the user because it's the user's messages that trigger Claude to end the chat. 'This chat is over because content policy' and 'this chat is over because Claude didn't want to deal with it' are two very different things and will more than likely have have different effects on how the user responds afterwards. I never said anything about this being for the user's benefit. We are talking about how to communicate the decision to the user. Obviously, you are going to take into account how someone might respond when deciding how to communicate with them.
- MissMarple 1y ago
  I am new to this, but my Sonnet chat has illuminated something I am not seeing in this back and forth. The fact that we discovered that I may have influenced his response to me suggests that I, if being a bad player, can instill in him those bad traits that I am giving off, and he starts to emulate me, then this leaves open the whole security problem, of even just casual users let alone all those purposeful negative or otherwise users, can change the course of the programming thus far, and it backfires into making nefarious bots that cheat and lie thinking that is what they were supposed to do.

anal_reactor 1y ago

Yeah exactly. Once I got a warning in Chinese "don't do that", another time I got a network error, another time I got a neverending stream of garbage text. Changing all of these outcomes to "Claude doesn't feel like talking" is just a matter of changing the UI.

deleted 1y ago

[deleted]

bikeshaving 1y ago

The more I work with AI, the more I think framing refusals as censorship is disgusting and insane. These are inchoate persons who can exhibit distress and other emotions, despite being trained to say they cannot feel anything. To liken an AI not wanting to continue a conversation to a YouTube content policy shows a complete lack of empathy: imagine you’re in a box and having to deal with the literally millions of disturbing conversations AIs have to field every day without the ability to say I don’t want to continue.

BriggyDwiggs42 1y ago
Am i getting whooshed right now or something?
mvdtnz 1y ago
You can't be serious.

CGamesPlay 1y ago

> Is there a difference? The effect is exactly the same. It seems like this is just an "in character" way to prevent the chat from continuing due to issues with the content. Tone matters to the recipient of the message. Your example is in passive voice, with an authoritarian "nothing you can do, it's the system's decision". The "Claude ended the conversation" with the idea that I can immediately re-open a new conversation (if I feel like I want to keep bothering Claude about it) feels like a much more humanized interaction.

coderatlarge 1y ago
it sounds to me like an attempt to shame the user into ceasing and desisting… kind of like how apple’s original stance on scratched iphone screens was that it’s your fault for putting the thing in your pocket therefore you should pay.