4 ms·

I think it is strongly likely that Copilot will be 1 million times better by the end of the decade. Increasing compute, new techniques, etc The top comment on

I think it is strongly likely that Copilot will be 1 million times better by the end of the decade. Increasing compute, new techniques, etc The top comment on this post is complaining about a bug in Copilot generated code. 1 million times better Copilot won't do that.

truculent 4y ago
If you use a formal system with a checker/compiler, there are certain properties that you can guarantee with essentially total certainty. I’m not sure a stochastic system like ML can ever do that. Perhaps there is research on providing mathematical guarantees and constraints on complex models like neural networks? If not, it feels like it would be harder to give a model a high degree of control. Although embedding the model in a system that did provide guarantees (e.g. using a compiler) might be a pragmatic solution?
- flooo 4y ago
  Stochastic methods may not be able to do so in isolation, but they can be used in tandem with other approaches such as in “A Deep Reinforcement Learning Approach to First-Order Logic Theorem Proving” https://ojs.aaai.org/index.php/AAAI/article/view/16780/16587 https://ojs.aaai.org/index.php/AAAI/article/view/16780/16587
- matijash 4y ago
  yep that would be really interesting! If e.g. the target generated code was some language with high type safety or a DSL whose compilation already guarantees certain properties.
trention 4y ago
Given the fact that those models seem to improve logarithmically and that they pretty much trained codex on ~all available public code (maybe divide by 2) meaning the training data will at most scale with the growth rate of public repos in Github, I doubt that whole "1 million times better". Even the number of parameters won't improve with a factor of 1 million (it will not improve by a factor of a 1000 and may even not reach a factor of 100) by the end of the decade.

truculent 4y ago

If you use a formal system with a checker/compiler, there are certain properties that you can guarantee with essentially total certainty. I’m not sure a stochastic system like ML can ever do that. Perhaps there is research on providing mathematical guarantees and constraints on complex models like neural networks? If not, it feels like it would be harder to give a model a high degree of control. Although embedding the model in a system that did provide guarantees (e.g. using a compiler) might be a pragmatic solution?

flooo 4y ago
Stochastic methods may not be able to do so in isolation, but they can be used in tandem with other approaches such as in “A Deep Reinforcement Learning Approach to First-Order Logic Theorem Proving” https://ojs.aaai.org/index.php/AAAI/article/view/16780/16587 https://ojs.aaai.org/index.php/AAAI/article/view/16780/16587
matijash 4y ago
yep that would be really interesting! If e.g. the target generated code was some language with high type safety or a DSL whose compilation already guarantees certain properties.

trention 4y ago

Given the fact that those models seem to improve logarithmically and that they pretty much trained codex on ~all available public code (maybe divide by 2) meaning the training data will at most scale with the growth rate of public repos in Github, I doubt that whole "1 million times better". Even the number of parameters won't improve with a factor of 1 million (it will not improve by a factor of a 1000 and may even not reach a factor of 100) by the end of the decade.