4 ms·

There is still publicly available code and documentation to draw from. As models get smarter and bootstrapped on top of older models, they should need less and

There is still publicly available code and documentation to draw from. As models get smarter and bootstrapped on top of older models, they should need less and less training data. In theory, just providing the grammar for a new programming language should be enough for a sufficiently smart LLM to answer problems in that language. Unlike freeform writing tasks, coding also has a strong feedback loop (i.e. does the code compile, run successfully, and output a result?), which means it is probably easier to generate synthetic training data for models.

layer8 2y ago
> In theory, just providing the grammar for a new programming language should be enough for a sufficiently smart LLM to answer problems in that language. I doubt it. Take a language like Rust or Haskell or even modern Java or Python. Without prolonged experience with the language, you have no idea how the various features interact in practice, what the best practices and typical pitfalls are, what common patterns and habits have been established by its practitioners, and so on. At best, the system would have to simulate building a number of nontrivial systems using the language in order to discover that knowledge, and in the end it would still be like someone locked in a room without knowledge of how the language is actually applied in the real world.
oblio 2y ago
> sufficiently smart LLM Cousin of the sufficiently smart compiler? :-p