5 ms·

Eliminating duplication and cleaning code is a different type of refactoring than supporting concurrency, which is much much harder. Cleaning up code also foll

Eliminating duplication and cleaning code is a different type of refactoring than supporting concurrency, which is much much harder. Cleaning up code also follows some well established patterns, performance work is much less pattern-y. Codebases like FFMPEG are one of the kind. I bet you need 10 or 100 times more understanding than the react thing you mentioned above. One day maybe AI can do it, but it probably won't be LLM. It would be something which can understand symbols and math.

ctoth 3y ago
Ah, we're having some classic goalpost moving! > Because refactoring requires understanding, which LLMs completely lack. <demonstration that an LLM can refactor code> > Cleaning up code also follows some well established patterns, performance work is much less pattern-y. Just as writing shitty react apps follow patterns, low-level performance and concurrency work also follow patterns. See [0] for a sample. > I bet you need 10 or 100 times more understanding Okay, so a 10 or 100 times larger model? Sounds like something we'll have next year, and certainly within a decade. > One day maybe AI can do it, but it probably won't be LLM. It would be something which can understand symbols and math. You do understand that the reason some of the earlier GPTs had trouble with symbols and math was the tokenization scheme, completely separate from how they work in general, right? [0]: C++ Concurrency in Action: Practical Multithreading 1st Edition https://www.amazon.com/C-Concurrency-Action-Practical-Multithreading/dp/1933988770 https://www.amazon.com/C-Concurrency-Action-Practical-Multit...
- kcbanner 3y ago
  > Because refactoring requires understanding, which LLMs completely lack. It's obvious from context here that the refactoring that was mentioned was specifically around concurrency, not simply cleaning up code.
  - ctoth 3y ago
    So if I show you an LLM implementing concurrency, will you concede the point? Is this your true objection? https://chat.openai.com/share/7c41f59a-c21c-4abd-876c-c95647d68026 https://chat.openai.com/share/7c41f59a-c21c-4abd-876c-c95647...
    - bigbillheck 3y ago
      Did you learn anything from that exercise? Are you a better programmer now for having seen that solution? Because if not, this seems like a great way for getting the fabled "one year of experience, twenty times"
    - arp242 3y ago
      That ... didn't even refactor the code. It just returned some generic Python concurrency methods which vaguely fit the posted code.
    - malcolmgreaves 3y ago
      Hope you're looking for good-faith discussion here. I'll assume that you're looking for a response where someone has taken the time to read through your previous messages and also the linked ChatGPT interaction logs. What you've shown is actually a great example of the what folks mean that LLMs lack any sort of understanding. They're fundamentally predict-the-next-token machines; they regurgitate and mix parts of their training data in order to satisfy the token prediction loss function they were trained with. In the linked example you provided, *you* are the one that needs to provide the understanding. It's a rather lengthly back-and-forth to get that code into a somewhat useable state. Importantly, if you didn't tell it to fix things (sqlite connections over threads, etc.), it would have failed. And while it's concurrent, it's using threads, so it's not going to be doing any work in parallel. The example you have mixes some IO and compute-bound looking operations. So, if your need was to refactor your original code to _actually be fast_, ChatGPT demonstrated it doesn't understand nearly enough to actually make this happen. This thread conversation got started around correcting the misnomer that an LLM would actually ever be able to possess enough knowledge to do actually valuable, complex refactoring and programming. While I believe that LLMs can be good tools for a variety of usecases, they have to be used in short bursts. Since their output is fundamentally unreliable, someone always has to read -- then comprehend -- its output. Giving it too much context and then prompting it in such a way to align its next token prediction with a complex outcome is a highly variable and unstable process. If it outputs millions of tokens, how is someone going to actually review all of this? In my experience using ChatGPT, GPT4, and a few other LLMs, I've found that it's pretty good at coming up with little bits to jog one's own thinking and problem solving. But doing an actual complex task with lots of nuance and semantics-to-be-understood outright? The technology is not quite there yet.
- never_inline 3y ago
  Ok bro I am not the parent commenter who set the goalpost. Let's see how your smooth talking LLM is going to do with things that are not web development or leetcode medium, for which so much stuff has been written. All the best.

ctoth 3y ago

Ah, we're having some classic goalpost moving! > Because refactoring requires understanding, which LLMs completely lack. <demonstration that an LLM can refactor code> > Cleaning up code also follows some well established patterns, performance work is much less pattern-y. Just as writing shitty react apps follow patterns, low-level performance and concurrency work also follow patterns. See [0] for a sample. > I bet you need 10 or 100 times more understanding Okay, so a 10 or 100 times larger model? Sounds like something we'll have next year, and certainly within a decade. > One day maybe AI can do it, but it probably won't be LLM. It would be something which can understand symbols and math. You do understand that the reason some of the earlier GPTs had trouble with symbols and math was the tokenization scheme, completely separate from how they work in general, right? [0]: C++ Concurrency in Action: Practical Multithreading 1st Edition https://www.amazon.com/C-Concurrency-Action-Practical-Multithreading/dp/1933988770 https://www.amazon.com/C-Concurrency-Action-Practical-Multit...

kcbanner 3y ago
> Because refactoring requires understanding, which LLMs completely lack. It's obvious from context here that the refactoring that was mentioned was specifically around concurrency, not simply cleaning up code.
- ctoth 3y ago
  So if I show you an LLM implementing concurrency, will you concede the point? Is this your true objection? https://chat.openai.com/share/7c41f59a-c21c-4abd-876c-c95647d68026 https://chat.openai.com/share/7c41f59a-c21c-4abd-876c-c95647...
  - bigbillheck 3y ago
    Did you learn anything from that exercise? Are you a better programmer now for having seen that solution? Because if not, this seems like a great way for getting the fabled "one year of experience, twenty times"
  - arp242 3y ago
    That ... didn't even refactor the code. It just returned some generic Python concurrency methods which vaguely fit the posted code.
  - malcolmgreaves 3y ago
    Hope you're looking for good-faith discussion here. I'll assume that you're looking for a response where someone has taken the time to read through your previous messages and also the linked ChatGPT interaction logs. What you've shown is actually a great example of the what folks mean that LLMs lack any sort of understanding. They're fundamentally predict-the-next-token machines; they regurgitate and mix parts of their training data in order to satisfy the token prediction loss function they were trained with. In the linked example you provided, *you* are the one that needs to provide the understanding. It's a rather lengthly back-and-forth to get that code into a somewhat useable state. Importantly, if you didn't tell it to fix things (sqlite connections over threads, etc.), it would have failed. And while it's concurrent, it's using threads, so it's not going to be doing any work in parallel. The example you have mixes some IO and compute-bound looking operations. So, if your need was to refactor your original code to _actually be fast_, ChatGPT demonstrated it doesn't understand nearly enough to actually make this happen. This thread conversation got started around correcting the misnomer that an LLM would actually ever be able to possess enough knowledge to do actually valuable, complex refactoring and programming. While I believe that LLMs can be good tools for a variety of usecases, they have to be used in short bursts. Since their output is fundamentally unreliable, someone always has to read -- then comprehend -- its output. Giving it too much context and then prompting it in such a way to align its next token prediction with a complex outcome is a highly variable and unstable process. If it outputs millions of tokens, how is someone going to actually review all of this? In my experience using ChatGPT, GPT4, and a few other LLMs, I've found that it's pretty good at coming up with little bits to jog one's own thinking and problem solving. But doing an actual complex task with lots of nuance and semantics-to-be-understood outright? The technology is not quite there yet.
never_inline 3y ago
Ok bro I am not the parent commenter who set the goalpost. Let's see how your smooth talking LLM is going to do with things that are not web development or leetcode medium, for which so much stuff has been written. All the best.