6 ms·

LLMs and other learning models have likely already been trained on AI output. These self reinforcing feedback loops are going to have to be accounted for at som

LLMs and other learning models have likely already been trained on AI output. These self reinforcing feedback loops are going to have to be accounted for at some point in AI development.

sbierwagen 4y ago
Here's a LLM fine-tuned on its own output that improved on several tasks: https://arxiv.org/abs/2210.11610 https://arxiv.org/abs/2210.11610
- Cardinal7167 4y ago
  Oh this is interesting! Thanks for sharing. To be clear: it’s not that I think it would necessarily degrade the quality of the training, but that the biases of training on other AI output need to be taken into account.
  - feoren 4y ago
    > it’s not that I think it would necessarily degrade the quality of the training It absolutely would necessarily degrade the quality of the training in the long-term. It's lossy knowledge compression. There is no lossy compression that gets better when you feed its output back into its own input over and over. It's basic information theory. I admit I don't understand why the linked article had those results -- if the results replicate, it must be somehow squeezing out a bit more usefulness from the training set, similar to the slight perturbations they sometimes give images. Or the model was previously too "unsure" of itself, and it's just amping up its own confidence in itself. If the training set was poisoned, all it would do is squeeze out more poison, or become more confidently wrong. But those are just hunches as to what's going on.
    - notpushkin 4y ago
      You're forgetting that not all AI outputs are getting posted. Therefore, training next generation AI on these outputs will reinforce the “share-ability” aspect a bit more.
    - zarzavat 4y ago
      How was the data created in the first place? A human took their prior knowledge, thought, and wrote something. There is no conservation law for knowledge. We would expect that when AIs become advanced enough, feeding their own output to them decreases loss, just as it does for humanity. In fact, this is a good definition of intelligence.
    - SideQuark 4y ago
      > There is no lossy compression that gets better when you feed its output back into its own input over and over. It's basic information theory Sure there is. The goal of AI is not to memorize all information, but to make the ability to generalize, so "lossy" doesn't make sense. For example, noisy data can be improved by successive filtering. In fact, from an information theory argument, noisy channel coding shows exactly that information can be improved via multiple lossy passes: many modern error correcting codes have iterative decoders, each stage lossy at the level of input to out stages, yet each stage gets closer to the correct original message. So the "lossy" and "information theory" argument doesn't work.
    - sbierwagen 4y ago
      AlphaZero plays games of chess against itself over and over, feeding the output of the neural network back into the input, and now it's vastly more powerful than any chess engine that's ever existed. What's the Kolmogorov complexity of the standard model? If you start with thousands of terabytes of training data, why wouldn't the accurate representation be dramatically smaller than that? A schoolchild is expected to memorize every word of a text and faithfully repeat it on command. Is that the same thing as understanding a book?
      - _dain_ 4y ago
        > AlphaZero plays games of chess against itself over and over, feeding the output of the neural network back into the input, and now it's vastly more powerful than any chess engine that's ever existed. not a good comparison. alphazero's loss function never changed as it was playing itself. it was always just "win this game given these rules". but LLM loss function rewards it for predicting the next token. and now "the next token" might be something that an AI wrote previously.
        SideQuark 4y ago
        >alphazero's loss function never changed Yes, it did. Alpha zero's loss function changes with every update. The loss function works on the actual outcome versus the predicted outcome, and the predicted outcome is a result of previous learning. So with every step it takes, which are random (and the initial weights are also random), it modifies it's own loss function for future walks in the space of weights. Rerun the entire training with different initial random weights, and look at the sequence of loss functions, and you will get a different sequence. The paper: https://arxiv.org/pdf/1712.01815.pdf https://arxiv.org/pdf/1712.01815.pdf
        _dain_ 4y ago
        i mean, there's always an objective ground truth because of the rules of chess never change. did it win or lose? but the "ground truth" of the english corpus changes all the time. and is changing right now as LLMs emit words into the noosphere. so I don't see how this counters my point.
    - wongarsu 4y ago
      "Distillation and amplification" is a somewhat popluar AI technique. For example if you have a chess engine with a heuristic to choose which moves to investigate, you can explore the 20 best paths according to your heuristic, see which moves ended in the best result, and use that to train the heuristic for the first move. Doing the same thing with LLMs isn't out of the question, but for it to work well you need some kind of reward function that doesn't depend on the model you train. Training on LLM texts that humans conciously chose to publish might already provide that, you just have to somehow filter out the content farms that lack any human review.
      - feoren 4y ago
        Chess is "data mineable" -- you can get more training data just by having computers play chess against themselves. There's clear winners and losers. If you programmed in the rules of chess, a sufficiently powerful AI could learn everything there is to know about the game just by playing itself -- mining its own training data. There's no analogue with language. The system can't determine whether what was said makes sense or is true on its own. Maybe you could program in "the rules of grammar" and have AIs invent their own languages, but they'd have nothing to say to each other, so don't expect a translation for "a broken clock is right twice a day". Besides, that's not what anyone is doing. This is why I'm saying any technique like this that works, must work by "squeezing out" more information from the training data (very likely overfitting in the process). You simply cannot data-mine new useful language training data like you can data-mine bitcoin or 1v1 game data. > for it to work well you need some kind of reward function that doesn't depend on the model you train. Training on LLM texts that humans conciously chose to publish might already provide that Of course adding more human-curated data can improve the model. But the whole idea of the arxiv article is whether these AIs can improve themselves. It seems patently clear to me that the answer is "only if they're underfit to the data, and only to a limit, after which they will start to overfit on their own excrement". I really just don't see how there's any other possibility that doesn't rely on ChatGPT magically having actually reached the singularity. Look, even humans don't get perpetually more intelligent just by talking to themselves. After a certain point, all they get is more entrenched in bad ideas, more cult-like, more superstitious. Humans get more intelligent by interacting with the environment and seeing what happens.
stingraycharles 4y ago
Could someone explain to me what the advantages are of that? How could an AI model trained on input generated by an AI ever be any better than the original AI? Is it about costs of getting clean datasets?
- barbazoo 4y ago
  > Could someone explain to me what the advantages are of that? How could an AI model trained on input generated by an AI ever be any better than the original AI? Is it about costs of getting clean datasets? I think parent was suggesting that the fact that LLM will inadvertently be trained on AI generated output needs to be accounted for.
- gameman144 4y ago
  It's not about advantages, it's about the fact that if you pull in any wild dataset for a language model, there's an overwhelming chance that some of that wild data was produced by an AI. E.g. if you're training an LLM on Reddit, there's a good shot that some of the Reddit posts you're pulling in were generated by an AI.
  - amelius 4y ago
    Then train on posts only if they were upvoted.
    - gkbrk 4y ago
      You are assuming AI-generated content doesn't get upvoted. It's entirely possible that they even get more upvotes than real humans producing organic content.
      - amelius 4y ago
        Well, if it got upvoted (by humans) then isn't it reasonable to train on it? Of course, one challenge is to make sure AI doesn't interfere with moderation. Perhaps this whole situation will spur more work on moderation systems.

LLMs and other learning models have likely already been trained on AI output. These self reinforcing feedback loops are going to have to be accounted for at some point in AI development.

sbierwagen 4y ago

Here's a LLM fine-tuned on its own output that improved on several tasks: https://arxiv.org/abs/2210.11610 https://arxiv.org/abs/2210.11610

Cardinal7167 4y ago
Oh this is interesting! Thanks for sharing. To be clear: it’s not that I think it would necessarily degrade the quality of the training, but that the biases of training on other AI output need to be taken into account.
- feoren 4y ago
  > it’s not that I think it would necessarily degrade the quality of the training It absolutely would necessarily degrade the quality of the training in the long-term. It's lossy knowledge compression. There is no lossy compression that gets better when you feed its output back into its own input over and over. It's basic information theory. I admit I don't understand why the linked article had those results -- if the results replicate, it must be somehow squeezing out a bit more usefulness from the training set, similar to the slight perturbations they sometimes give images. Or the model was previously too "unsure" of itself, and it's just amping up its own confidence in itself. If the training set was poisoned, all it would do is squeeze out more poison, or become more confidently wrong. But those are just hunches as to what's going on.
  - notpushkin 4y ago
    You're forgetting that not all AI outputs are getting posted. Therefore, training next generation AI on these outputs will reinforce the “share-ability” aspect a bit more.
  - zarzavat 4y ago
    How was the data created in the first place? A human took their prior knowledge, thought, and wrote something. There is no conservation law for knowledge. We would expect that when AIs become advanced enough, feeding their own output to them decreases loss, just as it does for humanity. In fact, this is a good definition of intelligence.
  - SideQuark 4y ago
    > There is no lossy compression that gets better when you feed its output back into its own input over and over. It's basic information theory Sure there is. The goal of AI is not to memorize all information, but to make the ability to generalize, so "lossy" doesn't make sense. For example, noisy data can be improved by successive filtering. In fact, from an information theory argument, noisy channel coding shows exactly that information can be improved via multiple lossy passes: many modern error correcting codes have iterative decoders, each stage lossy at the level of input to out stages, yet each stage gets closer to the correct original message. So the "lossy" and "information theory" argument doesn't work.
  - sbierwagen 4y ago
    AlphaZero plays games of chess against itself over and over, feeding the output of the neural network back into the input, and now it's vastly more powerful than any chess engine that's ever existed. What's the Kolmogorov complexity of the standard model? If you start with thousands of terabytes of training data, why wouldn't the accurate representation be dramatically smaller than that? A schoolchild is expected to memorize every word of a text and faithfully repeat it on command. Is that the same thing as understanding a book?
    - _dain_ 4y ago
      > AlphaZero plays games of chess against itself over and over, feeding the output of the neural network back into the input, and now it's vastly more powerful than any chess engine that's ever existed. not a good comparison. alphazero's loss function never changed as it was playing itself. it was always just "win this game given these rules". but LLM loss function rewards it for predicting the next token. and now "the next token" might be something that an AI wrote previously.
      - SideQuark 4y ago
        >alphazero's loss function never changed Yes, it did. Alpha zero's loss function changes with every update. The loss function works on the actual outcome versus the predicted outcome, and the predicted outcome is a result of previous learning. So with every step it takes, which are random (and the initial weights are also random), it modifies it's own loss function for future walks in the space of weights. Rerun the entire training with different initial random weights, and look at the sequence of loss functions, and you will get a different sequence. The paper: https://arxiv.org/pdf/1712.01815.pdf https://arxiv.org/pdf/1712.01815.pdf
        _dain_ 4y ago
        i mean, there's always an objective ground truth because of the rules of chess never change. did it win or lose? but the "ground truth" of the english corpus changes all the time. and is changing right now as LLMs emit words into the noosphere. so I don't see how this counters my point.
  - wongarsu 4y ago
    "Distillation and amplification" is a somewhat popluar AI technique. For example if you have a chess engine with a heuristic to choose which moves to investigate, you can explore the 20 best paths according to your heuristic, see which moves ended in the best result, and use that to train the heuristic for the first move. Doing the same thing with LLMs isn't out of the question, but for it to work well you need some kind of reward function that doesn't depend on the model you train. Training on LLM texts that humans conciously chose to publish might already provide that, you just have to somehow filter out the content farms that lack any human review.
    - feoren 4y ago
      Chess is "data mineable" -- you can get more training data just by having computers play chess against themselves. There's clear winners and losers. If you programmed in the rules of chess, a sufficiently powerful AI could learn everything there is to know about the game just by playing itself -- mining its own training data. There's no analogue with language. The system can't determine whether what was said makes sense or is true on its own. Maybe you could program in "the rules of grammar" and have AIs invent their own languages, but they'd have nothing to say to each other, so don't expect a translation for "a broken clock is right twice a day". Besides, that's not what anyone is doing. This is why I'm saying any technique like this that works, must work by "squeezing out" more information from the training data (very likely overfitting in the process). You simply cannot data-mine new useful language training data like you can data-mine bitcoin or 1v1 game data. > for it to work well you need some kind of reward function that doesn't depend on the model you train. Training on LLM texts that humans conciously chose to publish might already provide that Of course adding more human-curated data can improve the model. But the whole idea of the arxiv article is whether these AIs can improve themselves. It seems patently clear to me that the answer is "only if they're underfit to the data, and only to a limit, after which they will start to overfit on their own excrement". I really just don't see how there's any other possibility that doesn't rely on ChatGPT magically having actually reached the singularity. Look, even humans don't get perpetually more intelligent just by talking to themselves. After a certain point, all they get is more entrenched in bad ideas, more cult-like, more superstitious. Humans get more intelligent by interacting with the environment and seeing what happens.

stingraycharles 4y ago

Could someone explain to me what the advantages are of that? How could an AI model trained on input generated by an AI ever be any better than the original AI? Is it about costs of getting clean datasets?

barbazoo 4y ago
> Could someone explain to me what the advantages are of that? How could an AI model trained on input generated by an AI ever be any better than the original AI? Is it about costs of getting clean datasets? I think parent was suggesting that the fact that LLM will inadvertently be trained on AI generated output needs to be accounted for.
gameman144 4y ago
It's not about advantages, it's about the fact that if you pull in any wild dataset for a language model, there's an overwhelming chance that some of that wild data was produced by an AI. E.g. if you're training an LLM on Reddit, there's a good shot that some of the Reddit posts you're pulling in were generated by an AI.
- amelius 4y ago
  Then train on posts only if they were upvoted.
  - gkbrk 4y ago
    You are assuming AI-generated content doesn't get upvoted. It's entirely possible that they even get more upvotes than real humans producing organic content.
    - amelius 4y ago
      Well, if it got upvoted (by humans) then isn't it reasonable to train on it? Of course, one challenge is to make sure AI doesn't interfere with moderation. Perhaps this whole situation will spur more work on moderation systems.