7 ms·

What is the "compilers argument"?

by idiliv 8d ago

troupo 8d ago
People keep saying that "models are just compilers, and I don't see you complsining about compilers". Which is such a bullshit argument
- lelanthran 8d ago
  > People keep saying that "models are just compilers, and I don't see you complsining about compilers". Which is such a bullshit argument At what point do we normalise the message "This is a stupid line of reasoning and you should feel stupid for suggesting it, stupid!" I mean, all the reasoned and logical arguments in the world doesn't change a religious follower's faith, but emotive ones regularly work! At what point can we start using shaming language on people who apparently don't know how neither an LLM works nor how a compiler works, but still trot out this argument as a cognitive kill switch?
- vincston 8d ago
  Can you elaborate? Is it because the models are inherently not deterministic? But then, will it not get better with time? I mean arent we just at the beginning of the research here?
  - troupo 8d ago
    > Is it because the models are inherently not deterministic This is the main reason (and yes, many modern compilers and CPUs carry some non-determinism which actually quite well explained and specified). > But then, will it not get better with time? No, it won't. Bacuse that is that is the actual literal limitation/feature of LLMs. > I mean arent we just at the beginning of the research here? In general? Yes. With LLMs? We can reasonably say that they will never be deterministic. There is a way to get a non-determenistic output: first question on a temp 0 local model on a completely new session will give you the same answer. The second answer in that same session will already be different on every session (even if it's the same question).
  - eudamoniac 7d ago
    It is because English is an ambiguous input unlike a high level programming language. The output cannot help but be wrong sometimes when the input is ambiguous. See my comment here https://news.ycombinator.com/item?id=49491797 https://news.ycombinator.com/item?id=49491797 Determinism isn't the issue; it's that the prompt does not contain enough information to know the correct way to do the thing, even if it did the thing the same way with the same inputs every time.
big-chungus4 8d ago
I think it's that when some code compiles to say assembler, the compiler doesn't prioritize readability and maintainability of the assembler code, since people are not expected to read and maintain it directly
prerok 8d ago
Back in the day, the argument was that compilers produce unreadable assembly, so people used to writing assembly were arguing against the use of compilers. Compilers also had bugs, so we still had to debug the assembly to understand how to fix the problem. Nowadays, almost nobody has to resort to those steps, except of course compiler developers. But that is just a testament to the quality of compilers. Comparing LLMs to compilers is a take I often see, but I am not sure the comparison quite holds. The problem is that LLMs are inherently non-deterministic, so we always get a different output on the same prompt. Maybe if LLMs are powerful enough it won't matter. I doubt it but we will see.
- tripledry 8d ago
  What you say is true, the comparison indeed doesn't hold. But is it relevant? does it matter from a product perspective if LLMs are non-deterministic. You don't need to one shot the correct result, english is ambiguous and LLMs non-deterministic, but you can iterate. If it's possible to iterate fast and cheap enough, even ambiguous language can produce the results you want, given enough iterations. There are a lot of ifs and buts here, just a thought on the compiler argument.
  - prerok 8d ago
    I think it matters, because nowadays we don't look at assembly any more. I mean, I don't recall the last time I was tracking down a compiler bug but it's definitely been more than 15 years ago. We do have to look at the LLMs' output, though, and, as you already pointed out, iterate to get the correct results. What this means is that the output must still be readable, must be analyzed by someone and I don't see it going away any time soon. The problem is that the analysis is not cheap. Sometimes, with boilerplate, it is easy, but many times it is not and that's where we get only slight gains by using LLMs.
    - tripledry 8d ago
      I agree from a programmers perspective. But from a broad market and product perspective, for most things you don't need to look at the code. If the product kinda does what it's supposed to. For example, in my game projects I don't look at the CMakeLists anymore, or python scripts that move assets here and there, I can run my game and just see that it did what I expect it to do (renders assets etc). Similar with frontend, I don't care that much what the code looks like anymore, mostly that the site looks and feels as I expect it, and the correct network calls are happening. TLDR; I'm thinking there are levels to this, in some projects it matters, in others it doesn't, it's kinda two different things. Programming wasn't replaced, LLMs just brought a new paradigm of doing things on the side. I'm just rambling at this point, my thoughts on this are not super clear, sorry for that :D
IceDane 8d ago
Lots of really silly people love to compare LLMs to compilers. "You don't look at the compiled code either" and "Back in the day, people also had negative reactions to compilers and wanted to keep writing assembly by hand" and other such nonsense.
- recursivecaveat 8d ago
  Indeed, the compiler does not have to ingest its own output, figure it out, and insert modifications in the middle. Source code is the medium that LLMs work in.
- bluetomcat 8d ago
  They are ignorant about the elephant in the room. The input language of a compiler is a formally-specified grammar with well-defined semantics for each operation. It generally abstracts the computation process over a von-Neumann machine, adding convenience features and bells and whistles. What "a=1" means in an HLL is, store that value in a particular storage location, so that it can be accessed later by the same name. The slop machine reads lousy natural human language that can mean different things in different contexts. That lousy language is then statistically probed for the most likely output correspondence, producing shit that needs to be externally verified.
sampullman 8d ago
Hand writing assembly produces more efficient and concise code, at the cost of developer time and required expertise. It was true for a long time, now not so much.
OtomotO 8d ago
I assume it's the idea that initially you wrote machine code, later assembly and then the "high level languages" started to gain traction. And for each iteration there were scepticals... But I am curious myself, what OP meant by this.

What is the "compilers argument"?

by idiliv 8d ago

What is the "compilers argument"?

People keep saying that "models are just compilers, and I don't see you complsining about compilers". Which is such a bullshit argument

lelanthran 8d ago
> People keep saying that "models are just compilers, and I don't see you complsining about compilers". Which is such a bullshit argument At what point do we normalise the message "This is a stupid line of reasoning and you should feel stupid for suggesting it, stupid!" I mean, all the reasoned and logical arguments in the world doesn't change a religious follower's faith, but emotive ones regularly work! At what point can we start using shaming language on people who apparently don't know how neither an LLM works nor how a compiler works, but still trot out this argument as a cognitive kill switch?
vincston 8d ago
Can you elaborate? Is it because the models are inherently not deterministic? But then, will it not get better with time? I mean arent we just at the beginning of the research here?
- troupo 8d ago
  > Is it because the models are inherently not deterministic This is the main reason (and yes, many modern compilers and CPUs carry some non-determinism which actually quite well explained and specified). > But then, will it not get better with time? No, it won't. Bacuse that is that is the actual literal limitation/feature of LLMs. > I mean arent we just at the beginning of the research here? In general? Yes. With LLMs? We can reasonably say that they will never be deterministic. There is a way to get a non-determenistic output: first question on a temp 0 local model on a completely new session will give you the same answer. The second answer in that same session will already be different on every session (even if it's the same question).
- eudamoniac 7d ago
  It is because English is an ambiguous input unlike a high level programming language. The output cannot help but be wrong sometimes when the input is ambiguous. See my comment here https://news.ycombinator.com/item?id=49491797 https://news.ycombinator.com/item?id=49491797 Determinism isn't the issue; it's that the prompt does not contain enough information to know the correct way to do the thing, even if it did the thing the same way with the same inputs every time.

big-chungus4 8d ago

I think it's that when some code compiles to say assembler, the compiler doesn't prioritize readability and maintainability of the assembler code, since people are not expected to read and maintain it directly

prerok 8d ago

Back in the day, the argument was that compilers produce unreadable assembly, so people used to writing assembly were arguing against the use of compilers. Compilers also had bugs, so we still had to debug the assembly to understand how to fix the problem. Nowadays, almost nobody has to resort to those steps, except of course compiler developers. But that is just a testament to the quality of compilers. Comparing LLMs to compilers is a take I often see, but I am not sure the comparison quite holds. The problem is that LLMs are inherently non-deterministic, so we always get a different output on the same prompt. Maybe if LLMs are powerful enough it won't matter. I doubt it but we will see.

tripledry 8d ago
What you say is true, the comparison indeed doesn't hold. But is it relevant? does it matter from a product perspective if LLMs are non-deterministic. You don't need to one shot the correct result, english is ambiguous and LLMs non-deterministic, but you can iterate. If it's possible to iterate fast and cheap enough, even ambiguous language can produce the results you want, given enough iterations. There are a lot of ifs and buts here, just a thought on the compiler argument.
- prerok 8d ago
  I think it matters, because nowadays we don't look at assembly any more. I mean, I don't recall the last time I was tracking down a compiler bug but it's definitely been more than 15 years ago. We do have to look at the LLMs' output, though, and, as you already pointed out, iterate to get the correct results. What this means is that the output must still be readable, must be analyzed by someone and I don't see it going away any time soon. The problem is that the analysis is not cheap. Sometimes, with boilerplate, it is easy, but many times it is not and that's where we get only slight gains by using LLMs.
  - tripledry 8d ago
    I agree from a programmers perspective. But from a broad market and product perspective, for most things you don't need to look at the code. If the product kinda does what it's supposed to. For example, in my game projects I don't look at the CMakeLists anymore, or python scripts that move assets here and there, I can run my game and just see that it did what I expect it to do (renders assets etc). Similar with frontend, I don't care that much what the code looks like anymore, mostly that the site looks and feels as I expect it, and the correct network calls are happening. TLDR; I'm thinking there are levels to this, in some projects it matters, in others it doesn't, it's kinda two different things. Programming wasn't replaced, LLMs just brought a new paradigm of doing things on the side. I'm just rambling at this point, my thoughts on this are not super clear, sorry for that :D

IceDane 8d ago

Lots of really silly people love to compare LLMs to compilers. "You don't look at the compiled code either" and "Back in the day, people also had negative reactions to compilers and wanted to keep writing assembly by hand" and other such nonsense.

recursivecaveat 8d ago
Indeed, the compiler does not have to ingest its own output, figure it out, and insert modifications in the middle. Source code is the medium that LLMs work in.
bluetomcat 8d ago
They are ignorant about the elephant in the room. The input language of a compiler is a formally-specified grammar with well-defined semantics for each operation. It generally abstracts the computation process over a von-Neumann machine, adding convenience features and bells and whistles. What "a=1" means in an HLL is, store that value in a particular storage location, so that it can be accessed later by the same name. The slop machine reads lousy natural human language that can mean different things in different contexts. That lousy language is then statistically probed for the most likely output correspondence, producing shit that needs to be externally verified.

sampullman 8d ago

Hand writing assembly produces more efficient and concise code, at the cost of developer time and required expertise. It was true for a long time, now not so much.

OtomotO 8d ago

I assume it's the idea that initially you wrote machine code, later assembly and then the "high level languages" started to gain traction. And for each iteration there were scepticals... But I am curious myself, what OP meant by this.