5 ms·

[I'm one of the Microsoft Research people who worked on this] All code is from public repositories. We only used publicly available APIs to obtain the data, so

[I'm one of the Microsoft Research people who worked on this] All code is from public repositories. We only used publicly available APIs to obtain the data, so that others can reproduce the results / build on top of the data extraction pipeline we built: https://github.com/github/CodeSearchNet/tree/master/function_parser https://github.com/github/CodeSearchNet/tree/master/function...

throwaway744678 7y ago
What about code from public repository that don't have some kind of open source licence? Won't you be running into a copyright issue somewhere?
- mabrocks 7y ago
  We only used code from repos that github has marked as using a non-copyleft open source license (i.e., we had an explicit license whitelist and used only repos matching that).
  - gmueckl 7y ago
    So how do you handle repositories that apply different licenses to different subsections of code? E.g. what haopens when an optional helper library is in a subdirectory and licensed differently?
jacquesm 7y ago
Thank you for responding. So, then the next question is whether or not the code found in public repositories is also representative for the code found in private ones. I can see arguments for it going either way (better, worse), which might have substantial impact on the applicability of the models for code from the class that it never saw.
- mabrocks 7y ago
  That's an interesting question which we haven't studied so far. There is a good argument to be made that ML models trained on public data do not work well on private data on the interesting queries, where I consider those queries interesting that only make sense in the specific project. The core issue there is around the specialised vocabulary that you would only see in the (private) project, which the model would be unfamiliar with. This could be mitigated by using absurd amounts of data, models that can generalise more easily (character-level/BPE/subtokens/...), and finetuning on project-specific data, but it's an open question nonetheless. It's also easy to test without access to private repos, by just splitting some repos off as a separate test set, so should hopefully be something that other people can also make progress on!
  - jacquesm 7y ago
    You could test on GH and MS's own private repos to get some idea of the performance. These might still not be representative but it would be an interesting data point.

throwaway744678 7y ago

What about code from public repository that don't have some kind of open source licence? Won't you be running into a copyright issue somewhere?

mabrocks 7y ago
We only used code from repos that github has marked as using a non-copyleft open source license (i.e., we had an explicit license whitelist and used only repos matching that).
- gmueckl 7y ago
  So how do you handle repositories that apply different licenses to different subsections of code? E.g. what haopens when an optional helper library is in a subdirectory and licensed differently?

jacquesm 7y ago

Thank you for responding. So, then the next question is whether or not the code found in public repositories is also representative for the code found in private ones. I can see arguments for it going either way (better, worse), which might have substantial impact on the applicability of the models for code from the class that it never saw.

mabrocks 7y ago
That's an interesting question which we haven't studied so far. There is a good argument to be made that ML models trained on public data do not work well on private data on the interesting queries, where I consider those queries interesting that only make sense in the specific project. The core issue there is around the specialised vocabulary that you would only see in the (private) project, which the model would be unfamiliar with. This could be mitigated by using absurd amounts of data, models that can generalise more easily (character-level/BPE/subtokens/...), and finetuning on project-specific data, but it's an open question nonetheless. It's also easy to test without access to private repos, by just splitting some repos off as a separate test set, so should hopefully be something that other people can also make progress on!
- jacquesm 7y ago
  You could test on GH and MS's own private repos to get some idea of the performance. These might still not be representative but it would be an interesting data point.