6 ms·

People are resistant because: 1. There's this huge misconception that LLMs are literally just memorizing stuff and repeating patterns from their training data

People are resistant because: 1. There's this huge misconception that LLMs are literally just memorizing stuff and repeating patterns from their training data 2. People glamorize math and feel like advancements in it would "be AGI" They don't realize that having it generate "new math" is not much harder than having it generate "new programs." Instead of writing something in Python, it's writing something in Lean.

martin-t 1y ago
1) They absolutely do sometimes repeat training data verbatim.[0] 2) That's not even the point. The point is being trained on stolen data without permission, pretending that the resulting model of the training data is not a derived work of the training data and that the output of the model plus a prompt is not derived work of the training data. Point 1 is just an extreme edge case which is a symptom of point 2 and yet people still have trouble accepting it. GPL was about user freedom and now if derived work no longer applies as long as you run code through a sufficiently complex plagiarism automator, plagiarism is unprovable and GPL is broken. Great, we lost another freedom. [0]: I recall a study or court document with 100 examples of plagiarising multiple whole paragraphs from the New York Times, don't have time to look for it now
- danielbarla 1y ago
  You're completely correct in your two points, however people _do_ regularly assert that LLMs cannot possibly generate anything novel: "they are just regurgitating and recombining the original". I mean, sure. But so am I (in what is likely a far more advanced manner, but still). I also find it somewhat funny that I am also partially trained on stolen data without permission. I also jaywalk occasionally (perhaps I am trivializing the topic too much, but show me a researcher who hasn't _once_ downloaded a paper they really needed, in less than perfectly legal ways).
  - martin-t 1y ago
    Human time is valuable, LLM time is not. If you spend hundreds of hours creating something, nobody should have the right to copy (verbatim or with automatic modifications) it unless you allow them. Human rights are valuable. LLMs allow laundering GPL code (removing both attribution and users' rights to inspect and modify the code). Free software cannot compete against proprietary in a world where making a copy is trivial but proving it's a copy is nearly impossible.
- JanisErdmanis 1y ago
  > trained on stolen data without permission My sympathies to academic publishers ;)
- fkyoureadthedoc 1y ago
  > I recall a study or court document with 100 examples of plagiarising multiple whole paragraphs from the New York Times, don't have time to look for it now Convenient. Well then, I recall two studies that said the opposite. Unfortunately pressed for time as well.
  - martin-t 1y ago
    https://en.lmgtfy2.com/query/?q=ONE+HUNDRED+EXAMPLES+OF+GPT-4+MEMORIZING+CONTENT+FROM+THE+NEW+YORK+TIMES https://en.lmgtfy2.com/query/?q=ONE+HUNDRED+EXAMPLES+OF+GPT-... You didn't have to be rudely dismissive and lie, you chose to. I would happily respond politely to a polite request. Please be mindful of your behavior next time. --- Link for everyone else: https://nytco-assets.nytimes.com/2023/12/Lawsuit-Document-dkt-1-68-Ex-J.pdf https://nytco-assets.nytimes.com/2023/12/Lawsuit-Document-dk...
    - fkyoureadthedoc 1y ago
      Not very convincing. If you prompt GPT-4 (nobody uses it) with a huge chunk of an article (nobody does this), sometimes it'll output another chunk of said article. Conveniently omitted, how many attempts did not result in this behavior, how much of the the articles were not repeated (you can see they cut off mid answer)
- ComplexSystems 1y ago
  This all seems totally orthogonal to the statement: "I don't get why people are so resistant to the idea that AI can prove new mathematical theorems." I don't necessarily disagree about the copyleft stuff. Transformers do sometimes overfit to exact token sequences from training data, but that isn't really what they the architecture does in general.
  - martin-t 1y ago
    When you say new mathematical theorems, they absolutely can. So can infinite monkeys on typewriters, though LLMs have a much better heuristic to arrive at valid trheorems. The same applies to valid new programs. The issue I have with this is pretending that the word "new" is sufficient justification for giving all the credit/attribution and subsequent reward (reputational, financial, etc.) to the person who wrote the prompt instead of distributing it to the people in the whole chain of work according to how much work and what quality of work they did. How many man-hours did it take to create the training data? How many to create the LLM training algorithm and the electricity to run it? How many to write the prompts? The most work by many, many orders of magnitude was put in by the first group. They often did it with altruistic goals in mind and released their work under permissive or copyleft licenses. And now somebody found a way to monetize this effort without giving them anything in return. In fact, they will have to pay to access the LLMs which are based on their own work. Copyright or plagiarism are perhaps the wrong terms to use when talking about it. I think copyright should absolutely apply but it was designed to protect creative works, not code in the first place. Either way it's a form of industrialized exploitation and we should use all available tools to defend against it.
marliechiller 1y ago
>1. There's this huge misconception that LLMs are literally just memorizing stuff and repeating patterns from their training data For a lay person, what are they actually doing instead?
- dcre 1y ago
  I don't think anyone really knows, but I also don't think it's quite an either/or. To me a more interesting way to put the question is to ask what it would mean to say that GPT-5 is just applying patterns from its training data when it finds bugs in 1000 lines of new Rust code that were missed by multiple human reviewers. "Applying a memorized pattern" seems well-defined because it is an everyday concept but I don't think it really is well-defined. If the bug "fits a pattern" but is expressed in a different programming language, with different variable names, different context, etc., recognizing that and applying the pattern doesn't seem to me like a merely mechanical process. Kant has an argument in the Critique of Pure Reason that reason cannot be reducible to the application of rules, because in order to apply rule A to a situation, you would need a rule B to follow for applying rule A, and a rule C for applying rule B, and this is an infinite regress. I think the same is true here: any reasonable characterization of "applying a pattern" that would succeed at reducing what LLMs do to something mechanical is vulnerable to the regress argument. In short: even if you want to say it's pattern matching, retrieving a pattern and applying it requires something a lot closer to intelligence than the phrase makes it sound.
- whalee 1y ago
  This question becomes difficult whenever a system becomes sufficiently complex. Take any chaotic system, like a double pendulum, and press play at step 100,000. You ask 'what is it doing'? Well, it's just applying it's rule. Step to step. Zoom out and look at it's trajectory over those 100,00 steps and ask again. The answer is something alien. Probabilistically it is certain the description of its behavior is not going to exist in a space we as humans can understand. Maybe if we were god beings we could say 'No no, you see the behavior of the double pendulum isn't seemingly random, you just have to look at it like this'. Encryption is a decent analogy here. We're fooled into thinking we can understand these systems because we forced them to speak English. Under the hood is a different story.
0points 1y ago
> 1. There's this huge misconception that LLMs are literally just memorizing stuff and repeating patterns from their training data So then, what are they doing? I'm seeing people creating full apps with GPT-5-pro, but nothing is novel. Just discussed the "impressiveness" of it creating a gameboy emulator from scratch. (There's over 3500 gameboy emulators on github. I would be suprised if it failed to produce a solution with that much training data). Where's the novel break-throughs? As it stands today, I'm sure it can produce a new ssl implementation or whatever it has been trained on, but to what benefit???

martin-t 1y ago

1) They absolutely do sometimes repeat training data verbatim.[0] 2) That's not even the point. The point is being trained on stolen data without permission, pretending that the resulting model of the training data is not a derived work of the training data and that the output of the model plus a prompt is not derived work of the training data. Point 1 is just an extreme edge case which is a symptom of point 2 and yet people still have trouble accepting it. GPL was about user freedom and now if derived work no longer applies as long as you run code through a sufficiently complex plagiarism automator, plagiarism is unprovable and GPL is broken. Great, we lost another freedom. [0]: I recall a study or court document with 100 examples of plagiarising multiple whole paragraphs from the New York Times, don't have time to look for it now

danielbarla 1y ago
You're completely correct in your two points, however people _do_ regularly assert that LLMs cannot possibly generate anything novel: "they are just regurgitating and recombining the original". I mean, sure. But so am I (in what is likely a far more advanced manner, but still). I also find it somewhat funny that I am also partially trained on stolen data without permission. I also jaywalk occasionally (perhaps I am trivializing the topic too much, but show me a researcher who hasn't _once_ downloaded a paper they really needed, in less than perfectly legal ways).
- martin-t 1y ago
  Human time is valuable, LLM time is not. If you spend hundreds of hours creating something, nobody should have the right to copy (verbatim or with automatic modifications) it unless you allow them. Human rights are valuable. LLMs allow laundering GPL code (removing both attribution and users' rights to inspect and modify the code). Free software cannot compete against proprietary in a world where making a copy is trivial but proving it's a copy is nearly impossible.
JanisErdmanis 1y ago
> trained on stolen data without permission My sympathies to academic publishers ;)
fkyoureadthedoc 1y ago
> I recall a study or court document with 100 examples of plagiarising multiple whole paragraphs from the New York Times, don't have time to look for it now Convenient. Well then, I recall two studies that said the opposite. Unfortunately pressed for time as well.
- martin-t 1y ago
  https://en.lmgtfy2.com/query/?q=ONE+HUNDRED+EXAMPLES+OF+GPT-4+MEMORIZING+CONTENT+FROM+THE+NEW+YORK+TIMES https://en.lmgtfy2.com/query/?q=ONE+HUNDRED+EXAMPLES+OF+GPT-... You didn't have to be rudely dismissive and lie, you chose to. I would happily respond politely to a polite request. Please be mindful of your behavior next time. --- Link for everyone else: https://nytco-assets.nytimes.com/2023/12/Lawsuit-Document-dkt-1-68-Ex-J.pdf https://nytco-assets.nytimes.com/2023/12/Lawsuit-Document-dk...
  - fkyoureadthedoc 1y ago
    Not very convincing. If you prompt GPT-4 (nobody uses it) with a huge chunk of an article (nobody does this), sometimes it'll output another chunk of said article. Conveniently omitted, how many attempts did not result in this behavior, how much of the the articles were not repeated (you can see they cut off mid answer)
ComplexSystems 1y ago
This all seems totally orthogonal to the statement: "I don't get why people are so resistant to the idea that AI can prove new mathematical theorems." I don't necessarily disagree about the copyleft stuff. Transformers do sometimes overfit to exact token sequences from training data, but that isn't really what they the architecture does in general.
- martin-t 1y ago
  When you say new mathematical theorems, they absolutely can. So can infinite monkeys on typewriters, though LLMs have a much better heuristic to arrive at valid trheorems. The same applies to valid new programs. The issue I have with this is pretending that the word "new" is sufficient justification for giving all the credit/attribution and subsequent reward (reputational, financial, etc.) to the person who wrote the prompt instead of distributing it to the people in the whole chain of work according to how much work and what quality of work they did. How many man-hours did it take to create the training data? How many to create the LLM training algorithm and the electricity to run it? How many to write the prompts? The most work by many, many orders of magnitude was put in by the first group. They often did it with altruistic goals in mind and released their work under permissive or copyleft licenses. And now somebody found a way to monetize this effort without giving them anything in return. In fact, they will have to pay to access the LLMs which are based on their own work. Copyright or plagiarism are perhaps the wrong terms to use when talking about it. I think copyright should absolutely apply but it was designed to protect creative works, not code in the first place. Either way it's a form of industrialized exploitation and we should use all available tools to defend against it.

marliechiller 1y ago

>1. There's this huge misconception that LLMs are literally just memorizing stuff and repeating patterns from their training data For a lay person, what are they actually doing instead?

dcre 1y ago
I don't think anyone really knows, but I also don't think it's quite an either/or. To me a more interesting way to put the question is to ask what it would mean to say that GPT-5 is just applying patterns from its training data when it finds bugs in 1000 lines of new Rust code that were missed by multiple human reviewers. "Applying a memorized pattern" seems well-defined because it is an everyday concept but I don't think it really is well-defined. If the bug "fits a pattern" but is expressed in a different programming language, with different variable names, different context, etc., recognizing that and applying the pattern doesn't seem to me like a merely mechanical process. Kant has an argument in the Critique of Pure Reason that reason cannot be reducible to the application of rules, because in order to apply rule A to a situation, you would need a rule B to follow for applying rule A, and a rule C for applying rule B, and this is an infinite regress. I think the same is true here: any reasonable characterization of "applying a pattern" that would succeed at reducing what LLMs do to something mechanical is vulnerable to the regress argument. In short: even if you want to say it's pattern matching, retrieving a pattern and applying it requires something a lot closer to intelligence than the phrase makes it sound.
whalee 1y ago
This question becomes difficult whenever a system becomes sufficiently complex. Take any chaotic system, like a double pendulum, and press play at step 100,000. You ask 'what is it doing'? Well, it's just applying it's rule. Step to step. Zoom out and look at it's trajectory over those 100,00 steps and ask again. The answer is something alien. Probabilistically it is certain the description of its behavior is not going to exist in a space we as humans can understand. Maybe if we were god beings we could say 'No no, you see the behavior of the double pendulum isn't seemingly random, you just have to look at it like this'. Encryption is a decent analogy here. We're fooled into thinking we can understand these systems because we forced them to speak English. Under the hood is a different story.

0points 1y ago

> 1. There's this huge misconception that LLMs are literally just memorizing stuff and repeating patterns from their training data So then, what are they doing? I'm seeing people creating full apps with GPT-5-pro, but nothing is novel. Just discussed the "impressiveness" of it creating a gameboy emulator from scratch. (There's over 3500 gameboy emulators on github. I would be suprised if it failed to produce a solution with that much training data). Where's the novel break-throughs? As it stands today, I'm sure it can produce a new ssl implementation or whatever it has been trained on, but to what benefit???