8 ms·

AI-Generated Data Can Poison Future AI Models

147 points by meany 3y ago

hermitcrab 3y ago
See also: https://news.ycombinator.com/item?id=39422528 https://news.ycombinator.com/item?id=39422528
Der_Einzige 3y ago
Certain words, like "groundbreaking", have been totally ruined for me by LLMs which are too often trained to sound like each other.
buo 3y ago
I think it's interesting that human minds generally (though not always!) improve when exposed to the output of other human minds. It seems to be the opposite for current LLMs.
- diggan 3y ago
  Maybe it's less about "Human VS Robot" and more about exposure to "Original thoughts VS mass-produced average thoughts". I don't think a human mind would be improving if they're in a echo-chamber with no new information. I think the reason the human mind is improving is because we're exposed to new, original and/or different thoughts, that we hadn't considered or come across before. Meanwhile, a LLM will just regurgitate the most likely token based on the previous one, so there isn't any originality there, hence any output from a LLM cannot improve another LLM. There is nothing new to be learned, basically.
  - bluefirebrand 3y ago
    > I don't think a human mind would be improving if they're in a echo-chamber with no new information If this were true of humans, we would have never made it this far Humans are very capable of looking around themselves and thinking "I can do better than this", and then trying to come up with ways how LLMs are not
    - diggan 3y ago
      > Humans are very capable of looking around themselves and thinking "I can do better than this" Doesn't this require at least some perspective of what "better than this" means, which you could only know with at least a bit of outside influence in one way or another?
      - esafak 3y ago
        Parsimony, explanatory power, and aesthetics. These are things that could be taught to a computer, and I think we will. We had to evolve them too.
      - Jensson 3y ago
        Every human has feelings and instincts, they answer what "better than this" means. Yes, even in math and science, those were built on top of our feeling of "better than this" iterated over thousands of years.
- throwaway74432 3y ago
  Different loss function
- KolmogorovComp 3y ago
  A more appropriate analogy would be isolating someone from the rest of the world and only being able to read their own writings from now on. While some persons can strive in these kind of environment (think Kant for example), many would become crazy.
- ausbah 3y ago
  humans haven’t been had the same set of all encompassing “training experiences” like LLMs have. we each a subset of knowledge that may overlap with some other’s knowledge, but is largely unique. so when we interact with each other we can learn new things, but with LLMs I imagine it is a group of experienced but antiquated professors developing their own set of out of touch ideas
- NortySpock 3y ago
  I do get to choose what I read, though.
- BobaFloutist 3y ago
  I mean it makes sense that (even impressively functional) statistical approximations would degrade when recursed. If anything I think this just demonstrates yet again that these aren't actually analogous to what humans think of as "minds", even if they're able to replicate more of the output than makes us comfortable.
- mewpmewp2 3y ago
  Have you ever heard of the telephone game? This is what is going on here. Or imagine an original story of something that really happened. If it goes by 100 people in a chain, how much do you think the story will resemble the original one?
- orbital-decay 3y ago
  Humans exhibit very similar behavior. Prolonged sensory deprivation can drive a single individual insane. Fully isolated/monolithic/connected communities easily become detached from reality and are susceptible to mass psychosis. Etc etc etc. Humans need some minimum amount of external data to keep them in check as well.
- ben_w 3y ago
  Reproductive analogy: A sequence of AI models trained on each other's output gets mutations, which might help or hurt, but if there's one dominant model at any given time then it's like asexual reproduction with only living descendant in each generation (and all the competing models being failures to reproduce). A photocopy of a photocopy of a photocopy — this seems to me to also be the incorrect model which Intelligent Design proponents seem to mistakenly think is how evolution is supposed to work. A huge number of competing models that never rise to dominance would be more like plants spreading pollen in the wind. A huge number of AI there are each smart enough to decide what to include in its training set would be more like animal reproduction. The fittest memes survive. Memetic mode collapses still happen in individual AI (they still happen in humans, we're not magic), but that manifests as certain AI ceasing to be useful and others replacing them economically. A few mega-minds is a memetic monoculture, fragile in all the same ways as a biological monoculture.
  - nonrandomstring 3y ago
    A different biological analogy occurred to me which I've mentioned before in a security context. It isn't model degeneration but the amplification of invisible nasties that don't become a problem until way down the line. Natural examples are prions such as Bovine spongiform encephalopathy [0] or sheep scrapie. This seems to really become a problem in systems with a strong and fast positive feedback loop with some selector. In the case of cattle it was feeding rendered bonemeal from dead cattle back to livestock. Prions are immune to high temperature removal so are selected for and concentrated by the feedback process. To really feel the horror of this, read Ken Thompson's "Reflections on Trusting Trust" [1] and ponder the ways that a trojan can be replicated iteratively (like a worm) but undetectably. It isn't loss functions we should worry about. It's gain functions. [0] https://en.wikipedia.org/wiki/Bovine_spongiform_encephalopathy https://en.wikipedia.org/wiki/Bovine_spongiform_encephalopat... [1] https://tebibyte.media/blog/reflections-on-trusting-trust/ https://tebibyte.media/blog/reflections-on-trusting-trust/
- analog31 3y ago
  This might be my biases speaking, but I have a hunch that there's still more potential for human generated content to poison our minds, than AI.
- JohnFen 3y ago
  It's almost as if LLMs and human minds operate entirely differently from each other.
GaggiX 3y ago
Unless the internet is no longer useful because there is no way to find anything reliable, there would be enough signal to train and align models.
- Iulioh 3y ago
  Dead internet theory is closer and closer I don't remember wich YouTuber made a interesting video about it but basically communities are moving away from the free web in private communities (think discord or even sites that you are forced to register to to read the content) It's an interesting thing but I think queries on searche engines are becoming worse for this reason too.
- hackerlight 3y ago
  I question whether it'll matter. There is so much language data already, unlocking a little more isn't going to be the difference maker for AGI.
sophrocyne 3y ago
Some perspectives from someone working in the image space. These tests don't feel practical - That is, they seem intended to collapse the model, not demonstrate "in the wild" performance. The assumption is that all content is black or white - AI or not AI - and that you treat all content as equally worth retraining on. It offers no room for assumptions around data augmentation, human-guided quality discrimination, or anything else that might alter the set of outputs to mitigate the "poison"
- data-ottawa 3y ago
  > Use the model to generate some AI output. Then use that output to train a new instance of the model and use the resulting output to train a third version, and so forth. With each iteration, errors build atop one another. The 10th model, prompted to write about historical English architecture, spews out gibberish about jackrabbits. That this happens doesn't surprise me, but I'd love to see a curve of how each organic vs machine content mixe ratio results in model collapse over N generations.
- MacsHeadroom 3y ago
  This is exactly right. Model collapse does not exist in practice. In fact, LLMs trained on newer web scrapes have increased capabilities thanks to the generated output in their training data. For example, "base" pretrained models trained on scrapes which include generated outputs can 0-shot instruction follow and score higher on reasoning benchmarks. Intentionally produced synthetic training data takes this a step further. For SoTA LLMs the majority of, or all of, their training data is generated. Phi-2 and Claude 3 for example.
  - pavel_lishin 3y ago
    What happens if you train a model on nothing but AI-generated output, recursively? Does it eventually get inbred?
    - Kuinox 3y ago
      Without human input, yes.
ipython 3y ago
This reminds me of how fascinated I was as a kid of the artifacts you get from recursively photocopying a piece of paper.
- sshine 3y ago
  I watched someone in the printer room at the computer science department gradually photocopy from white to black, and back again, over the span of 300 pieces of paper, by altering the thresholds of the photocopyer. They didn’t graduate to become computer scientists, but did indeed get admitted to the royal school of art the year after. I found it strangely therapeutic.
doubloon 3y ago
reminds me of sheep and cows being fed their bretherens own brain matter developing spongiform encepalopathy (brain disease) or of course cannibals developing kuru. except a purely 'software' form.
chmike 3y ago
And human generated data may not ?
jxdxbx 3y ago
How does this relate to synthetic data?
add-sub-mul-div 3y ago
You'd think we'd be concerned about it poisoning the culture, well before any concerns that it would start to interfere with the rich continuing to be able to profit from it doing so.
cortesoft 3y ago
Human created content is also filled with gibberish and false information and random noise… how is AI generated content worse?
- Libcat99 3y ago
  Imagine you have a calculator that outputs a result that is off by one percent. That's ai right now. If you use the results of each calculation in additional calculations, the result will skew further and further from reality with each error. That's ai training on itself.
  - richk449 3y ago
    In many areas of communication and information, this exact problem is dealt with through error correction codes. Do AI models have built in ECC?
    - Libcat99 3y ago
      The trouble is "truth" and math are different. You can verify a mathematical result. You can run the calculations a second time on a separate calculator (in fact some computers do this) to verify the result, or use a built in check like ecc. There's no such mathematical test for truth for an ai to run.
      - ben_w 3y ago
        There's no fully general test for truth for an AI to run. In some specific domains such tests exist — and the result is, generally, computers wildly outperforming humans. But I get the impression from using them that current LLMs didn't take full advantage of this during training.
      - richk449 3y ago
        Error correction doesn’t insure truth. At least in communication, it insures that the final version matches the original version. For AI, you wouldn’t be doing EC to make sure the AI was saying truth, you would be doing EC to ensure that the AI hasn’t drifted due to the 1% error rate. Of course I have no idea how to actually do it - if it isn’t being done now, it is probably hard or impossible.
richk449 3y ago
Kessler syndrome for the internet?
ur-whale 3y ago
> AI-Generated Data Can Poison Future AI Models Looks like we didn't learn anything from the mad cow disease!
ein0p 3y ago
I also wonder what search engines are going to do about all this. Sounds to me, actually, traditional, non-intelligent search might be on its way out, although of course it'll take time. Future search engines will have to be quite adept at trying to figure out whether the text they index is bullshit or not.
beeboobaa 3y ago
It shouldn't be a problem if you only train on legally acquired data. You will know the authors name and can contact them if you so wish.
- theferalrobot 3y ago
  I don't think any of the major players could do that for all their data and they are acquiring it legally.
- astrange 3y ago
  There aren't any laws that require "acquiring" something in a way that "knows the author's name".
- pests 3y ago
  What? How do you know the data your buying isn't AI generated by the sellers? If they are scamming and you contact them, of course they will lie. So how does this work?
coldcode 3y ago
I think AI-generated images are worse for training AI generative models than LLMs, since there are so many now on the internet (see Instagram art related hashtags if you want to see nothing but AI art) compared to the quantity of images downloaded prior to 2021 (for those AI that did that). Text will always be more varied than seeing 10m versions of the same ideas that people make for fun. AI text can also be partial (like AI-assisted writing) but the images will all be essentially 100% generated.
- ToucanLoucan 3y ago
  That's far from unique to instagram. I loathe Stable Diffiusion and co solely because they've utterly FLOODED every cool art-adjacent website with endless mediocre derivative shit. Like there was always low-effort content of course, but holy fuck, there is SO MUCH MORE now. And some of these people are trying to CHARGE for this uninspired junk!!!
  - 7moritz7 3y ago
    I agree with this despite using SD a lot myself. It's fun to use until you realize the majority of people posting stuff generated with it have almost no creativity, all generating the same things over and over again, mostly without any manual work involved. that uncanny realism style with the generic Stable Diffusion face and one of 5 different poses. The number of people putting any sort of effort into it is way, way lower than the number of users thinking they are making art. It's more of a slot machine in the majority of cases
    - vunderba 3y ago
      Unfortunately, yeah 99.9% of images you're going to see generated from stable diffusion models are going to be either selfies, portraits, or porn. What's you're not going to see is things like "a divine gigantic textile loom sewing together a white horse and a black horse in an interlaced pattern to create a zebra." for example.
    - ToucanLoucan 3y ago
nestorD 3y ago
I believe that this is a non-problem pushed forward by small-scale experiments that are not representative of what people actually do with AI generation. A lot of new content, while AI generated, has been hand picked and polished by a human (for example, while you might commit AI generated code to your codebase, you ensure that it is correct and follows your preferred style). Content farms will push gibberish out, but they did so, and worse, before and the first generation of models was able to train on the internet anyway.
- x86x87 3y ago
  i think it's pretty much a problem and it's going to ruin any chance of high quality, original content. look at the original internet content and what seo has done to it. google and search in general results are trash nowadays. this is what genAI is going to do over long term. garbage in garbage out.
randcraw 3y ago
It's fascinating that error can accumulate through repeated trainings that 1) is undetected by humans and 2) can degrade LLM or diffusion models (or any transformer model?) so completely. This implies that not only do we not understand how latent knowledge is actually representated in deep nets, we don't know it forms or how it changes during training. If we did, we could have predicted the destructive impact of recycling of output as input. IMO, this suggests we should demand rigorous validation of deep nets (especially generative ones) before relying on them to behave responsibly.
- x86x87 3y ago
  The effect is not new. We have known about it ever since we've had basic machine learning. The way to look at it is somewhat novel but not surprising at all.
esafak 3y ago
Computers need to be able to learn from the world at large, not just their own output. World models are needed to make progress.
- astrange 3y ago
  There's no such thing as a world model. People do not have world models. This is a confused term made up by 70s AI researchers, who had the continual problem that they didn't know any philosophy and kept making up their own metaphors for how intelligence might work, and then deciding that because they'd made it up it must be true, and also that if they wrote a computer program that had the same metaphors it must work. "World model" just vaguely points at something people might do and assumes that if you make up a new thing it vaguely points at it'd help.
  - Jensson 3y ago
    On what basis are you saying that? I have a model in my head mapping out the world around me, so I know where things are etc and what I can do with all those things. How is that not a world model? Are you using a very strange definition of "world model"?
    - astrange 3y ago
      > On what basis are you saying that? This is a longstanding critique of GOFAI; see Hubert Dreyfuss and Phil Agre. https://pages.gseis.ucla.edu/faculty/agre/critical.html https://pages.gseis.ucla.edu/faculty/agre/critical.html > I have a model in my head mapping out the world around me, so I know where things are etc and what I can do with all those things. No you don't; a map is not the territory, and is necessarily wrong, which means that if you had such a model and were actually relying on it you wouldn't be able to do things you obviously can do in real life. You have an inaccurate memory of the world and you update it, only as much as you need to[0], as you go, in order to do a specific task. [0] probably a little less than you need to, because you want to save thinking energy
      - 3y ago
Bjorkbat 3y ago
I'm not sure how much of a risk this is to LLMs in particular, but I feel like we're already seeing the impact on image AI models. Even though they're getting better at generating hands that make sense and other fine details, you can generally tell that an image is AI generated because it has a certain "style". Can't help but wonder if this is partly due to generated images contaminating the training data and causing subsequent AI image generators to stylistically converge over time.
- astrange 3y ago
  It's because the models don't have an optimal aesthetic policy. Which would be difficult, but if they did have one, it wouldn't matter how much bad input data you added during pretraining.
p5v 3y ago
Is there a standard objective metric that can help determine that the quality of a model has degraded over time. In that case, much like source code, you just revert to the old version.
RecycledEle 3y ago
Synthetic data is a disaster. If you want foom (fast self-improvement in AI) use AIs to filter the training data for the next generation of AIs.
deleted 3y ago
[deleted]

147 points by meany 3y ago

See also: https://news.ycombinator.com/item?id=39422528 https://news.ycombinator.com/item?id=39422528

Certain words, like "groundbreaking", have been totally ruined for me by LLMs which are too often trained to sound like each other.

buo 3y ago

I think it's interesting that human minds generally (though not always!) improve when exposed to the output of other human minds. It seems to be the opposite for current LLMs.

diggan 3y ago
Maybe it's less about "Human VS Robot" and more about exposure to "Original thoughts VS mass-produced average thoughts". I don't think a human mind would be improving if they're in a echo-chamber with no new information. I think the reason the human mind is improving is because we're exposed to new, original and/or different thoughts, that we hadn't considered or come across before. Meanwhile, a LLM will just regurgitate the most likely token based on the previous one, so there isn't any originality there, hence any output from a LLM cannot improve another LLM. There is nothing new to be learned, basically.
- bluefirebrand 3y ago
  > I don't think a human mind would be improving if they're in a echo-chamber with no new information If this were true of humans, we would have never made it this far Humans are very capable of looking around themselves and thinking "I can do better than this", and then trying to come up with ways how LLMs are not
  - diggan 3y ago
    > Humans are very capable of looking around themselves and thinking "I can do better than this" Doesn't this require at least some perspective of what "better than this" means, which you could only know with at least a bit of outside influence in one way or another?
    - esafak 3y ago
      Parsimony, explanatory power, and aesthetics. These are things that could be taught to a computer, and I think we will. We had to evolve them too.
    - Jensson 3y ago
      Every human has feelings and instincts, they answer what "better than this" means. Yes, even in math and science, those were built on top of our feeling of "better than this" iterated over thousands of years.
throwaway74432 3y ago
Different loss function
KolmogorovComp 3y ago
A more appropriate analogy would be isolating someone from the rest of the world and only being able to read their own writings from now on. While some persons can strive in these kind of environment (think Kant for example), many would become crazy.
ausbah 3y ago
humans haven’t been had the same set of all encompassing “training experiences” like LLMs have. we each a subset of knowledge that may overlap with some other’s knowledge, but is largely unique. so when we interact with each other we can learn new things, but with LLMs I imagine it is a group of experienced but antiquated professors developing their own set of out of touch ideas
NortySpock 3y ago
I do get to choose what I read, though.
BobaFloutist 3y ago
I mean it makes sense that (even impressively functional) statistical approximations would degrade when recursed. If anything I think this just demonstrates yet again that these aren't actually analogous to what humans think of as "minds", even if they're able to replicate more of the output than makes us comfortable.
mewpmewp2 3y ago
Have you ever heard of the telephone game? This is what is going on here. Or imagine an original story of something that really happened. If it goes by 100 people in a chain, how much do you think the story will resemble the original one?
orbital-decay 3y ago
Humans exhibit very similar behavior. Prolonged sensory deprivation can drive a single individual insane. Fully isolated/monolithic/connected communities easily become detached from reality and are susceptible to mass psychosis. Etc etc etc. Humans need some minimum amount of external data to keep them in check as well.
ben_w 3y ago
Reproductive analogy: A sequence of AI models trained on each other's output gets mutations, which might help or hurt, but if there's one dominant model at any given time then it's like asexual reproduction with only living descendant in each generation (and all the competing models being failures to reproduce). A photocopy of a photocopy of a photocopy — this seems to me to also be the incorrect model which Intelligent Design proponents seem to mistakenly think is how evolution is supposed to work. A huge number of competing models that never rise to dominance would be more like plants spreading pollen in the wind. A huge number of AI there are each smart enough to decide what to include in its training set would be more like animal reproduction. The fittest memes survive. Memetic mode collapses still happen in individual AI (they still happen in humans, we're not magic), but that manifests as certain AI ceasing to be useful and others replacing them economically. A few mega-minds is a memetic monoculture, fragile in all the same ways as a biological monoculture.
- nonrandomstring 3y ago
  A different biological analogy occurred to me which I've mentioned before in a security context. It isn't model degeneration but the amplification of invisible nasties that don't become a problem until way down the line. Natural examples are prions such as Bovine spongiform encephalopathy [0] or sheep scrapie. This seems to really become a problem in systems with a strong and fast positive feedback loop with some selector. In the case of cattle it was feeding rendered bonemeal from dead cattle back to livestock. Prions are immune to high temperature removal so are selected for and concentrated by the feedback process. To really feel the horror of this, read Ken Thompson's "Reflections on Trusting Trust" [1] and ponder the ways that a trojan can be replicated iteratively (like a worm) but undetectably. It isn't loss functions we should worry about. It's gain functions. [0] https://en.wikipedia.org/wiki/Bovine_spongiform_encephalopathy https://en.wikipedia.org/wiki/Bovine_spongiform_encephalopat... [1] https://tebibyte.media/blog/reflections-on-trusting-trust/ https://tebibyte.media/blog/reflections-on-trusting-trust/
analog31 3y ago
This might be my biases speaking, but I have a hunch that there's still more potential for human generated content to poison our minds, than AI.
JohnFen 3y ago
It's almost as if LLMs and human minds operate entirely differently from each other.

GaggiX 3y ago

Unless the internet is no longer useful because there is no way to find anything reliable, there would be enough signal to train and align models.

Iulioh 3y ago
Dead internet theory is closer and closer I don't remember wich YouTuber made a interesting video about it but basically communities are moving away from the free web in private communities (think discord or even sites that you are forced to register to to read the content) It's an interesting thing but I think queries on searche engines are becoming worse for this reason too.
hackerlight 3y ago
I question whether it'll matter. There is so much language data already, unlocking a little more isn't going to be the difference maker for AGI.

sophrocyne 3y ago

Some perspectives from someone working in the image space. These tests don't feel practical - That is, they seem intended to collapse the model, not demonstrate "in the wild" performance. The assumption is that all content is black or white - AI or not AI - and that you treat all content as equally worth retraining on. It offers no room for assumptions around data augmentation, human-guided quality discrimination, or anything else that might alter the set of outputs to mitigate the "poison"

data-ottawa 3y ago
> Use the model to generate some AI output. Then use that output to train a new instance of the model and use the resulting output to train a third version, and so forth. With each iteration, errors build atop one another. The 10th model, prompted to write about historical English architecture, spews out gibberish about jackrabbits. That this happens doesn't surprise me, but I'd love to see a curve of how each organic vs machine content mixe ratio results in model collapse over N generations.
MacsHeadroom 3y ago
This is exactly right. Model collapse does not exist in practice. In fact, LLMs trained on newer web scrapes have increased capabilities thanks to the generated output in their training data. For example, "base" pretrained models trained on scrapes which include generated outputs can 0-shot instruction follow and score higher on reasoning benchmarks. Intentionally produced synthetic training data takes this a step further. For SoTA LLMs the majority of, or all of, their training data is generated. Phi-2 and Claude 3 for example.
- pavel_lishin 3y ago
  What happens if you train a model on nothing but AI-generated output, recursively? Does it eventually get inbred?
  - Kuinox 3y ago
    Without human input, yes.

ipython 3y ago

This reminds me of how fascinated I was as a kid of the artifacts you get from recursively photocopying a piece of paper.

sshine 3y ago
I watched someone in the printer room at the computer science department gradually photocopy from white to black, and back again, over the span of 300 pieces of paper, by altering the thresholds of the photocopyer. They didn’t graduate to become computer scientists, but did indeed get admitted to the royal school of art the year after. I found it strangely therapeutic.

doubloon 3y ago

reminds me of sheep and cows being fed their bretherens own brain matter developing spongiform encepalopathy (brain disease) or of course cannibals developing kuru. except a purely 'software' form.

chmike 3y ago

And human generated data may not ?

jxdxbx 3y ago

How does this relate to synthetic data?

add-sub-mul-div 3y ago

You'd think we'd be concerned about it poisoning the culture, well before any concerns that it would start to interfere with the rich continuing to be able to profit from it doing so.

cortesoft 3y ago

Human created content is also filled with gibberish and false information and random noise… how is AI generated content worse?

Libcat99 3y ago
Imagine you have a calculator that outputs a result that is off by one percent. That's ai right now. If you use the results of each calculation in additional calculations, the result will skew further and further from reality with each error. That's ai training on itself.
- richk449 3y ago
  In many areas of communication and information, this exact problem is dealt with through error correction codes. Do AI models have built in ECC?
  - Libcat99 3y ago
    The trouble is "truth" and math are different. You can verify a mathematical result. You can run the calculations a second time on a separate calculator (in fact some computers do this) to verify the result, or use a built in check like ecc. There's no such mathematical test for truth for an ai to run.
    - ben_w 3y ago
      There's no fully general test for truth for an AI to run. In some specific domains such tests exist — and the result is, generally, computers wildly outperforming humans. But I get the impression from using them that current LLMs didn't take full advantage of this during training.
    - richk449 3y ago
      Error correction doesn’t insure truth. At least in communication, it insures that the final version matches the original version. For AI, you wouldn’t be doing EC to make sure the AI was saying truth, you would be doing EC to ensure that the AI hasn’t drifted due to the 1% error rate. Of course I have no idea how to actually do it - if it isn’t being done now, it is probably hard or impossible.

richk449 3y ago

Kessler syndrome for the internet?

ur-whale 3y ago

> AI-Generated Data Can Poison Future AI Models Looks like we didn't learn anything from the mad cow disease!

ein0p 3y ago

I also wonder what search engines are going to do about all this. Sounds to me, actually, traditional, non-intelligent search might be on its way out, although of course it'll take time. Future search engines will have to be quite adept at trying to figure out whether the text they index is bullshit or not.

beeboobaa 3y ago

It shouldn't be a problem if you only train on legally acquired data. You will know the authors name and can contact them if you so wish.

theferalrobot 3y ago
I don't think any of the major players could do that for all their data and they are acquiring it legally.
astrange 3y ago
There aren't any laws that require "acquiring" something in a way that "knows the author's name".
pests 3y ago
What? How do you know the data your buying isn't AI generated by the sellers? If they are scamming and you contact them, of course they will lie. So how does this work?

coldcode 3y ago

I think AI-generated images are worse for training AI generative models than LLMs, since there are so many now on the internet (see Instagram art related hashtags if you want to see nothing but AI art) compared to the quantity of images downloaded prior to 2021 (for those AI that did that). Text will always be more varied than seeing 10m versions of the same ideas that people make for fun. AI text can also be partial (like AI-assisted writing) but the images will all be essentially 100% generated.

ToucanLoucan 3y ago
That's far from unique to instagram. I loathe Stable Diffiusion and co solely because they've utterly FLOODED every cool art-adjacent website with endless mediocre derivative shit. Like there was always low-effort content of course, but holy fuck, there is SO MUCH MORE now. And some of these people are trying to CHARGE for this uninspired junk!!!
- 7moritz7 3y ago
  I agree with this despite using SD a lot myself. It's fun to use until you realize the majority of people posting stuff generated with it have almost no creativity, all generating the same things over and over again, mostly without any manual work involved. that uncanny realism style with the generic Stable Diffusion face and one of 5 different poses. The number of people putting any sort of effort into it is way, way lower than the number of users thinking they are making art. It's more of a slot machine in the majority of cases
  - vunderba 3y ago
    Unfortunately, yeah 99.9% of images you're going to see generated from stable diffusion models are going to be either selfies, portraits, or porn. What's you're not going to see is things like "a divine gigantic textile loom sewing together a white horse and a black horse in an interlaced pattern to create a zebra." for example.
  - ToucanLoucan 3y ago

nestorD 3y ago

I believe that this is a non-problem pushed forward by small-scale experiments that are not representative of what people actually do with AI generation. A lot of new content, while AI generated, has been hand picked and polished by a human (for example, while you might commit AI generated code to your codebase, you ensure that it is correct and follows your preferred style). Content farms will push gibberish out, but they did so, and worse, before and the first generation of models was able to train on the internet anyway.

x86x87 3y ago
i think it's pretty much a problem and it's going to ruin any chance of high quality, original content. look at the original internet content and what seo has done to it. google and search in general results are trash nowadays. this is what genAI is going to do over long term. garbage in garbage out.

randcraw 3y ago

It's fascinating that error can accumulate through repeated trainings that 1) is undetected by humans and 2) can degrade LLM or diffusion models (or any transformer model?) so completely. This implies that not only do we not understand how latent knowledge is actually representated in deep nets, we don't know it forms or how it changes during training. If we did, we could have predicted the destructive impact of recycling of output as input. IMO, this suggests we should demand rigorous validation of deep nets (especially generative ones) before relying on them to behave responsibly.

x86x87 3y ago
The effect is not new. We have known about it ever since we've had basic machine learning. The way to look at it is somewhat novel but not surprising at all.

esafak 3y ago

Computers need to be able to learn from the world at large, not just their own output. World models are needed to make progress.

astrange 3y ago
There's no such thing as a world model. People do not have world models. This is a confused term made up by 70s AI researchers, who had the continual problem that they didn't know any philosophy and kept making up their own metaphors for how intelligence might work, and then deciding that because they'd made it up it must be true, and also that if they wrote a computer program that had the same metaphors it must work. "World model" just vaguely points at something people might do and assumes that if you make up a new thing it vaguely points at it'd help.
- Jensson 3y ago
  On what basis are you saying that? I have a model in my head mapping out the world around me, so I know where things are etc and what I can do with all those things. How is that not a world model? Are you using a very strange definition of "world model"?
  - astrange 3y ago
    > On what basis are you saying that? This is a longstanding critique of GOFAI; see Hubert Dreyfuss and Phil Agre. https://pages.gseis.ucla.edu/faculty/agre/critical.html https://pages.gseis.ucla.edu/faculty/agre/critical.html > I have a model in my head mapping out the world around me, so I know where things are etc and what I can do with all those things. No you don't; a map is not the territory, and is necessarily wrong, which means that if you had such a model and were actually relying on it you wouldn't be able to do things you obviously can do in real life. You have an inaccurate memory of the world and you update it, only as much as you need to[0], as you go, in order to do a specific task. [0] probably a little less than you need to, because you want to save thinking energy
    - 3y ago

Bjorkbat 3y ago

I'm not sure how much of a risk this is to LLMs in particular, but I feel like we're already seeing the impact on image AI models. Even though they're getting better at generating hands that make sense and other fine details, you can generally tell that an image is AI generated because it has a certain "style". Can't help but wonder if this is partly due to generated images contaminating the training data and causing subsequent AI image generators to stylistically converge over time.

astrange 3y ago
It's because the models don't have an optimal aesthetic policy. Which would be difficult, but if they did have one, it wouldn't matter how much bad input data you added during pretraining.

p5v 3y ago

Is there a standard objective metric that can help determine that the quality of a model has degraded over time. In that case, much like source code, you just revert to the old version.

RecycledEle 3y ago

Synthetic data is a disaster. If you want foom (fast self-improvement in AI) use AIs to filter the training data for the next generation of AIs.

deleted 3y ago

[deleted]