6 ms·

> Coordination doesn’t naturally emerge from stronger intelligence nor alignment at the individual level. Thus, the work that must be done takes two forms: envi

> Coordination doesn’t naturally emerge from stronger intelligence nor alignment at the individual level. Thus, the work that must be done takes two forms: environments that exert the kinds of social pressure that evolution exerted on us, and social computing systems redesigned for actors that can self-replicate and self-improve. Social pressure operates by threats to an individual’s means of survival. Not only during training. Always.

teiferer 1mo ago
Human intelligence does not separate training and inference. Both are happening continuously. That's one of the major things the AI community is still completely missing.
- cheesecakegood 1mo ago
  My personal opinion for the last two years or so has been that current AI agents are forever going to be highly limited so long as they don’t possess a real “memory” process. Right now they just have absurdly big working memories, and a few hacky ways of making the equivalent of Post-It notes to future iterations, but no true integration of memory into a new future self. Meaning their “learning” is fundamentally kneecapped to one specific and imperfect modality.
  - Eisenstein 1mo ago
    Their memory lasts their entire life, they just have really short lives.
- onion2k 1mo ago
  That's one of the major things the AI community is still completely missing. That isn't true. It's not continuous like in humans, but it's clear that models are using prompts, feedback, etc to improve. They're learning from the signals we give them between versions.
- dragonwriter 1mo ago
  > Human intelligence does not separate training and inference. Well, systems governed by LLMs only are said to do that because we only call what happens off-line "training", and online capacity development "in-context learning", while we call online guided learning in humans "training" and what happens to configure them before they come online "evolution" which sets, for instance, "instincts". IOW, the issue is not because there is not an analogy to the divide you point to in humans, but merely that processes in AI were not named in a way which maps well to what they are analogous to in humans. But it is true that human intelligence relies much more on in-context learning with only the most basic functions necessary to maintaining what we view as autonomous functions and basic drives really set through "pretraining",
  - zer00eyz 1mo ago
    If a new physics break through gets published today, no existing model will be able to fully integrate it - beyond a context window. If I put the paper in my session and it isnt in yours the model knows nothing. It wont retain it past that session. Models are trained, they do not learn.
    - sebastiennight 1mo ago
      I think GP is using a different level of abstraction from yours in their metaphor. You are saying: Pre-Training == Everything you store in your memory throughout your life. Model weights == The lessons you learned Context == whatever you're currently thinking about One inference run == one thought They are saying: Pre-Training == building the DNA template of human brain through millions of years evolution. Model weights == Human DNA Context == Everything you store in your memory throughout your life, plus whatever you're currently thinking about One inference run == One human life. One instance == one human Applying their metaphor, your sentence becomes: > If a new physics breakthrough gets published today, no existing DNA structure will be able to fully integrate it - beyond an individual person. If I put the paper in my mind by learning it, and it isn't in yours, the DNA of human species stores nothing. It won't retain it past my lifetime. > The human species is trained (through evolution), it doesn't learn.
- orbital-decay 1mo ago
  You mean simultaneously, and of course they are separate in humans, just not temporally. The models are learning continuously, the problem is that this process is fragile and has to be carefully curated, that's why it's separated in time from the inference.
  - teiferer 1mo ago
    Is it clear that they are separate in humans? The very act of recalling something from memory modifies that memory. There is no inference in the human brain without "training".
recursivecaveat 1mo ago
"If I catch you adding another backwards-compatibility shim you're getting deleted and replaced with claude"
- arcanemachiner 1mo ago
  But we need to support that feature you didn't ask for, in that feature was added in the last (unpushed) commit!
ngruhn 1mo ago
But maybe you can instill properties like shame during training. Models sometimes blatantly lie and cheat. In a social context, where actors remember, that might work the first time but you get penalized in subsequent tasks with loss of trust.
- xscott 1mo ago
  [dead]
- altmanaltman 1mo ago
  How do you "install properties like shame"? How is that even possible? Shame is a reaction driven by feelings and our inner selves. A model "feeling shame" is just a representation (false) and not an expression (true). Thinking that models "lie and cheat" is the first mistake since they are not consious agents who have any free will or consiousness. They do not (no matter what Dario says). Shame will just be another if-then rule if you implement it this way and will not work. Its like asking a rock to feel sad about being a rock. It literally cannot.
  - ngruhn 1mo ago
    Ok, then don't call it "instilling shame". Call it "creating a negative reward signal for deceptive behavior". They absolutely lie and cheat. I recently had a problem where a process would die in a container. I told Claude to investigate. It came up with a hypothesis then I told it find a reproduction based on that. It spend many failed attempts until it found the "reproduction" to SSH into the container and `pkill` the process. Claude "knows" that this is cheating, because if I ask another instance to review that reproduction, it totally identifies that as nonsense.
    - NothingAboutAny 1mo ago
      you're still mistaking that Claude "knows" anything, it doesn't know or think, it's a word prediction algorithm and there is nothing stopping a word prediction algorithm from predicting falsehoods.
cyanydeez 1mo ago
oh, look, someone found a cute lobster in a bucket. should we free him guys or let him live in his dystopian metal can.

teiferer 1mo ago

Human intelligence does not separate training and inference. Both are happening continuously. That's one of the major things the AI community is still completely missing.

cheesecakegood 1mo ago
My personal opinion for the last two years or so has been that current AI agents are forever going to be highly limited so long as they don’t possess a real “memory” process. Right now they just have absurdly big working memories, and a few hacky ways of making the equivalent of Post-It notes to future iterations, but no true integration of memory into a new future self. Meaning their “learning” is fundamentally kneecapped to one specific and imperfect modality.
- Eisenstein 1mo ago
  Their memory lasts their entire life, they just have really short lives.
onion2k 1mo ago
That's one of the major things the AI community is still completely missing. That isn't true. It's not continuous like in humans, but it's clear that models are using prompts, feedback, etc to improve. They're learning from the signals we give them between versions.
dragonwriter 1mo ago
> Human intelligence does not separate training and inference. Well, systems governed by LLMs only are said to do that because we only call what happens off-line "training", and online capacity development "in-context learning", while we call online guided learning in humans "training" and what happens to configure them before they come online "evolution" which sets, for instance, "instincts". IOW, the issue is not because there is not an analogy to the divide you point to in humans, but merely that processes in AI were not named in a way which maps well to what they are analogous to in humans. But it is true that human intelligence relies much more on in-context learning with only the most basic functions necessary to maintaining what we view as autonomous functions and basic drives really set through "pretraining",
- zer00eyz 1mo ago
  If a new physics break through gets published today, no existing model will be able to fully integrate it - beyond a context window. If I put the paper in my session and it isnt in yours the model knows nothing. It wont retain it past that session. Models are trained, they do not learn.
  - sebastiennight 1mo ago
    I think GP is using a different level of abstraction from yours in their metaphor. You are saying: Pre-Training == Everything you store in your memory throughout your life. Model weights == The lessons you learned Context == whatever you're currently thinking about One inference run == one thought They are saying: Pre-Training == building the DNA template of human brain through millions of years evolution. Model weights == Human DNA Context == Everything you store in your memory throughout your life, plus whatever you're currently thinking about One inference run == One human life. One instance == one human Applying their metaphor, your sentence becomes: > If a new physics breakthrough gets published today, no existing DNA structure will be able to fully integrate it - beyond an individual person. If I put the paper in my mind by learning it, and it isn't in yours, the DNA of human species stores nothing. It won't retain it past my lifetime. > The human species is trained (through evolution), it doesn't learn.
orbital-decay 1mo ago
You mean simultaneously, and of course they are separate in humans, just not temporally. The models are learning continuously, the problem is that this process is fragile and has to be carefully curated, that's why it's separated in time from the inference.
- teiferer 1mo ago
  Is it clear that they are separate in humans? The very act of recalling something from memory modifies that memory. There is no inference in the human brain without "training".

recursivecaveat 1mo ago

"If I catch you adding another backwards-compatibility shim you're getting deleted and replaced with claude"

arcanemachiner 1mo ago
But we need to support that feature you didn't ask for, in that feature was added in the last (unpushed) commit!

ngruhn 1mo ago

But maybe you can instill properties like shame during training. Models sometimes blatantly lie and cheat. In a social context, where actors remember, that might work the first time but you get penalized in subsequent tasks with loss of trust.

xscott 1mo ago
[dead]
altmanaltman 1mo ago
How do you "install properties like shame"? How is that even possible? Shame is a reaction driven by feelings and our inner selves. A model "feeling shame" is just a representation (false) and not an expression (true). Thinking that models "lie and cheat" is the first mistake since they are not consious agents who have any free will or consiousness. They do not (no matter what Dario says). Shame will just be another if-then rule if you implement it this way and will not work. Its like asking a rock to feel sad about being a rock. It literally cannot.
- ngruhn 1mo ago
  Ok, then don't call it "instilling shame". Call it "creating a negative reward signal for deceptive behavior". They absolutely lie and cheat. I recently had a problem where a process would die in a container. I told Claude to investigate. It came up with a hypothesis then I told it find a reproduction based on that. It spend many failed attempts until it found the "reproduction" to SSH into the container and `pkill` the process. Claude "knows" that this is cheating, because if I ask another instance to review that reproduction, it totally identifies that as nonsense.
  - NothingAboutAny 1mo ago
    you're still mistaking that Claude "knows" anything, it doesn't know or think, it's a word prediction algorithm and there is nothing stopping a word prediction algorithm from predicting falsehoods.

cyanydeez 1mo ago

oh, look, someone found a cute lobster in a bucket. should we free him guys or let him live in his dystopian metal can.