7 ms·

It’s cool but there’s a good chance it’s just copying someone else’s homework albeit in an elaborate round about way.

luke5441 7mo ago
It looks like a much more progressed/complete version of https://github.com/kidoz/smdc-toolchain/tree/master/crates/smdc/src https://github.com/kidoz/smdc-toolchain/tree/master/crates/s... . But that one is only a month old. So a bit confused there. Maybe that was also created via LLM?
- madmax911 7mo ago
  [dead]
nomel 7mo ago
I would claim that LLMs desperately need proprietary code in their training, before we see any big gains in quality. There's some incredible source available code out there. Statistically, I think there's a LOT more not so great source available code out there, because the majority of output of seasoned/high skill developers is proprietary. To me, a surprising portion of Claude 4.5 output definitely looks like student homework answers, because I think that's closer to the mean of the code population.
- andai 7mo ago
  Let's start with the source code for the Flash IDE :)
- bhadass 7mo ago
  yeah, but isn't the whole point of claude code to get people to provide preference data/telemetry data to anthropic (unless you opt out?). same w/ other providers. i'm guessing most of the gains we've seen recently are post training rather than pretraining.
  - nomel 7mo ago
    Yes, but you have the problem that a good portion of that is going to be AI generated. But, I naively assume most orgs would opt out. I know some orgs have a proxy in place that will prevent certain proprietary code from passing through! This makes me curious if, in the allow case, Anthropic is recording generated output, to maybe down-weight it if it's seen in the training data (or something similar)?
- typ 7mo ago
  I'd bet, on average, the quality of proprietary code is worse than open-source code. There have been decades of accumulated slop generated by human agents with wildly varied skill levels, all vibe-coded by ruthless, incompetent corporate bosses.
  - Manouchehri 7mo ago
    There's only very niche fields where closed-source code quality is often better than open-source code. Exploits and HFT are the two examples I can think of. Both are usually closed source because of the financial incentives.
    - ozim 7mo ago
      Here we can start debating what means better code. I haven’t seen HFT code but I have seen examples of exploit codes and most of it is amateur hour when it comes to building big size systems. They are of course efficient in getting to the goal. But exploits are one off code that is not there to be maintained.
  - Take8435 7mo ago
    Not to mention, a team member is (surprise!) fired or let go, and no knowledge transfer exists. Womp, womp. Codebase just gets worse as the organization or team flails. Seen this way too often.
    - icedchai 7mo ago
      Developers are often treated as cogs. Anyone should be able to step in a pick things up instantly. It’s just typing, right? /s
  - hirvi74 7mo ago
    In my time, I have potentially written code that some legal jurisdictions might classify as a "crime against humanity" due to the quality.
  - kortilla 7mo ago
- bearjaws 7mo ago
  I will say many closed source repos are probably equally as poor as open source ones. Even worse in many cases because they are so over engineered nobody understands how they work.
  - hirvi74 7mo ago
    I firmly agree with your first sentence. I can just think about the various modders that have created patches and performance enhancing mods for games with budgets of tens to hundreds of millions of dollars. But to give other devs and myself some grace, I do believe plenty of bad code can likely be explained by bad deadlines. After all, what's the Russian idiom? "There is nothing more permanent than the temporary."
  - deleted 7mo ago
    [deleted]
- dcre 7mo ago
  This is dead wrong: essentially the entirety of the huge gains in coding performance in the past year have come from RL, not from new sources of training data. I echo the other commenters that proprietary code isn’t any better, plus it doesn’t matter because when you use LLMs to work on proprietary code, it has the code right there.
  - elevation 7mo ago
    > it doesn’t matter because when you use LLMs to work on proprietary code, it has the code right there The quality of the existing code base makes a huge difference. On a recent greenfield effort, Claude emitted an MVP that matched the design semantics, but the code was not up to standards. For example, it repeatedly loaded a large file into memory in different areas where it was needed (rather than loading once and passing a reference.) However, after an early refactor, the subsequently generated code vastly improved. It honors the testing and performance paradigms, and it's so clean there's nothing for the linter to do.
  - nextos 7mo ago
    Progress with RL is very interesting, but it's still too inefficient. Current models do OK on simple boring linear code. But they output complete nonsense when presented with some compact but mildly complex code, e.g. a NumPyro model with some nesting and einsums. For this reason, to be truly useful, model outputs need to be verifiable. Formal verification with languages like Dafny , F*, or Isabelle might offer some solutions [1]. Otherwise, a gigantic software artifact such as a compiler is going to have a critical correctness bugs with far-fetched consequences if deployed in production. Right now, I think treating a LLM like something different than a very useful information retrieval system with excellent semantic capabilities is not something I am comfortable with. [1] https://risemsr.github.io/blog/2026-02-04-nik-agentic-pop https://risemsr.github.io/blog/2026-02-04-nik-agentic-pop
    - dcre 7mo ago
      Human-written compilers have bugs too! It takes decades of use to iron them out, and we’re introducing new ones all the time.
wvenable 7mo ago
This is cool and actually demonstrates real utility. Using AI to take something that already exists and create it for a different library / framework / platform is cool. I'm sure there's a lot of training data in there for just this case. But I wonder how it would fare given a language specification for a non-existent non-trivial language and build a compiler for that instead?
- nmstoker 7mo ago
  If you come up with a realistic language spec and wait maybe six months, by then it'll probably be approach being cheap enough that you could test the scenario yourself!
kreelman 7mo ago
..A small thing, but it won't compile the RISCV version of hello.c if the source isn't installed on the machine it's running on. It is standing on the shoulders of giants (all of the compilers of the past, built into it's training data... and the recent learnings about getting these agents to break up tasks) to get itself going. Still fairly impressive. On a side-quest, I wonder where Anthropic is getting there power from. The whole energy debacle in the US at the moment probably means it made some CO2 in the process. Would be hard to avoid?
nlawalker 7mo ago
I see that as the point that all this is proving - most people, most of the time, are essentially reinventing the wheel at some scope and scale or another, so we’d all benefit from being able to find and copy each others’ homework more efficiently.
computerex 7mo ago
And the goal post shifts.

It looks like a much more progressed/complete version of https://github.com/kidoz/smdc-toolchain/tree/master/crates/smdc/src https://github.com/kidoz/smdc-toolchain/tree/master/crates/s... . But that one is only a month old. So a bit confused there. Maybe that was also created via LLM?

madmax911 7mo ago
[dead]

nomel 7mo ago

I would claim that LLMs desperately need proprietary code in their training, before we see any big gains in quality. There's some incredible source available code out there. Statistically, I think there's a LOT more not so great source available code out there, because the majority of output of seasoned/high skill developers is proprietary. To me, a surprising portion of Claude 4.5 output definitely looks like student homework answers, because I think that's closer to the mean of the code population.

andai 7mo ago
Let's start with the source code for the Flash IDE :)
bhadass 7mo ago
yeah, but isn't the whole point of claude code to get people to provide preference data/telemetry data to anthropic (unless you opt out?). same w/ other providers. i'm guessing most of the gains we've seen recently are post training rather than pretraining.
- nomel 7mo ago
  Yes, but you have the problem that a good portion of that is going to be AI generated. But, I naively assume most orgs would opt out. I know some orgs have a proxy in place that will prevent certain proprietary code from passing through! This makes me curious if, in the allow case, Anthropic is recording generated output, to maybe down-weight it if it's seen in the training data (or something similar)?
typ 7mo ago
I'd bet, on average, the quality of proprietary code is worse than open-source code. There have been decades of accumulated slop generated by human agents with wildly varied skill levels, all vibe-coded by ruthless, incompetent corporate bosses.
- Manouchehri 7mo ago
  There's only very niche fields where closed-source code quality is often better than open-source code. Exploits and HFT are the two examples I can think of. Both are usually closed source because of the financial incentives.
  - ozim 7mo ago
    Here we can start debating what means better code. I haven’t seen HFT code but I have seen examples of exploit codes and most of it is amateur hour when it comes to building big size systems. They are of course efficient in getting to the goal. But exploits are one off code that is not there to be maintained.
- Take8435 7mo ago
  Not to mention, a team member is (surprise!) fired or let go, and no knowledge transfer exists. Womp, womp. Codebase just gets worse as the organization or team flails. Seen this way too often.
  - icedchai 7mo ago
    Developers are often treated as cogs. Anyone should be able to step in a pick things up instantly. It’s just typing, right? /s
- hirvi74 7mo ago
  In my time, I have potentially written code that some legal jurisdictions might classify as a "crime against humanity" due to the quality.
- kortilla 7mo ago
bearjaws 7mo ago
I will say many closed source repos are probably equally as poor as open source ones. Even worse in many cases because they are so over engineered nobody understands how they work.
- hirvi74 7mo ago
  I firmly agree with your first sentence. I can just think about the various modders that have created patches and performance enhancing mods for games with budgets of tens to hundreds of millions of dollars. But to give other devs and myself some grace, I do believe plenty of bad code can likely be explained by bad deadlines. After all, what's the Russian idiom? "There is nothing more permanent than the temporary."
- deleted 7mo ago
  [deleted]
dcre 7mo ago
This is dead wrong: essentially the entirety of the huge gains in coding performance in the past year have come from RL, not from new sources of training data. I echo the other commenters that proprietary code isn’t any better, plus it doesn’t matter because when you use LLMs to work on proprietary code, it has the code right there.
- elevation 7mo ago
  > it doesn’t matter because when you use LLMs to work on proprietary code, it has the code right there The quality of the existing code base makes a huge difference. On a recent greenfield effort, Claude emitted an MVP that matched the design semantics, but the code was not up to standards. For example, it repeatedly loaded a large file into memory in different areas where it was needed (rather than loading once and passing a reference.) However, after an early refactor, the subsequently generated code vastly improved. It honors the testing and performance paradigms, and it's so clean there's nothing for the linter to do.
- nextos 7mo ago
  Progress with RL is very interesting, but it's still too inefficient. Current models do OK on simple boring linear code. But they output complete nonsense when presented with some compact but mildly complex code, e.g. a NumPyro model with some nesting and einsums. For this reason, to be truly useful, model outputs need to be verifiable. Formal verification with languages like Dafny , F*, or Isabelle might offer some solutions [1]. Otherwise, a gigantic software artifact such as a compiler is going to have a critical correctness bugs with far-fetched consequences if deployed in production. Right now, I think treating a LLM like something different than a very useful information retrieval system with excellent semantic capabilities is not something I am comfortable with. [1] https://risemsr.github.io/blog/2026-02-04-nik-agentic-pop https://risemsr.github.io/blog/2026-02-04-nik-agentic-pop
  - dcre 7mo ago
    Human-written compilers have bugs too! It takes decades of use to iron them out, and we’re introducing new ones all the time.

wvenable 7mo ago

This is cool and actually demonstrates real utility. Using AI to take something that already exists and create it for a different library / framework / platform is cool. I'm sure there's a lot of training data in there for just this case. But I wonder how it would fare given a language specification for a non-existent non-trivial language and build a compiler for that instead?

nmstoker 7mo ago
If you come up with a realistic language spec and wait maybe six months, by then it'll probably be approach being cheap enough that you could test the scenario yourself!

kreelman 7mo ago

..A small thing, but it won't compile the RISCV version of hello.c if the source isn't installed on the machine it's running on. It is standing on the shoulders of giants (all of the compilers of the past, built into it's training data... and the recent learnings about getting these agents to break up tasks) to get itself going. Still fairly impressive. On a side-quest, I wonder where Anthropic is getting there power from. The whole energy debacle in the US at the moment probably means it made some CO2 in the process. Would be hard to avoid?

nlawalker 7mo ago

I see that as the point that all this is proving - most people, most of the time, are essentially reinventing the wheel at some scope and scale or another, so we’d all benefit from being able to find and copy each others’ homework more efficiently.

computerex 7mo ago

And the goal post shifts.