13 ms·

Hello folks, this is Hamel from GitHub -- I’m one of the Machine Learning Engineers who worked on this project. The reason we are excited to host this data is t

Hello folks, this is Hamel from GitHub -- I’m one of the Machine Learning Engineers who worked on this project. The reason we are excited to host this data is that we believe the community will be able to innovate and advance the state of the art much faster if it is provided in a tractable format for machine learning researchers. This data is already public, however there is specialized knowledge required to acquire, parse, dedupe, and clean code from many programming languages at a massive scale on GitHub. We strived to reduce these barriers to encourage greater involvement. While we present the task of information retrieval as one possible use case of this dataset, we know there could be other practical applications of this data (i.e. code summarization). While we went through great lengths to pre-process the data for the community, the data is still messy and often you will find that there might not be high-quality comments that are aligned with a code snippet we parsed. However, we believe this is part of the excitement of the dataset it poses challenges that machine learning practitioners will have to address. Code is very different than natural language with regard to its structure and syntactic rules and may benefit from different approaches relative to natural language processing. Our baseline models and benchmarks mostly treat code as natural language, however we are aware that there could be an opportunity to innovate on this front. If anyone creates any interesting projects from this dataset, please do get in touch. Happy to answer any questions!

airocker 7y ago
Could you have code reviews in the data?
- mloncode 7y ago
  The unit of observation for this data are functions or methods and their associated docstrings or top-level comments. So there are no code reviews in the data. However, we do include metadata including the SHA and owner/repo which would allow you to retrieve this information! What were you thinking of doing with code reviews?
  - airocker 7y ago
    We can try to write an automatic code review system?
dchichkov 7y ago
How would you make sure that any derived works (resulting from the use of this dataset) are properly licensed? It is very likely that this dataset (as per 1909.09436.pdf: "2 million functions, obtained from mechanically scraping and preprocessing associated function documentation") is contaminated by code with dubious licenses, right? And any derived work as a result would also be contaminated, right? What's the plan to deal with this issue?
- mabrocks 7y ago
  [I'm one of the Microsoft Research people who worked on this] We used the repo-level license information from github to filter to free non-copyleft licenses, and the license files are actually stored next to the extracted data corpus (see https://github.com/github/CodeSearchNet#Licenses https://github.com/github/CodeSearchNet#Licenses).
  - panny 7y ago
    Are you doing anything to help copyleft licensed code repos find infringers? We all know it's common practice for unscrupulous developers and companies to take copyleft code, copy it, and then try to relicensed under permissive licenses on a regular basis. Like Google did with Oracle's GPLed Java source code. It seems irresponsible for Github to continue distributing infringing code when it clearly has the tools to prevent that happening. Youtube has fingerprinting for infringing music and video. Github should be doing the same with source code.
    - desiderantes 7y ago
      >Like Google did with Oracle's GPLed Java source code. What are you talking about?
      - panny 7y ago
        https://news.ycombinator.com/item?id=20066269 https://news.ycombinator.com/item?id=20066269
sytelus 7y ago
<rant> GitHub has one of the worse search experience in the modern history of the internet. It doesn't need fancy ML or DL. Just get the stupid dedupping done, for God's sake. A simple TF-IDF similarity metric would make your search 10,000% better. You even know forks, names of files, folders, number of stars. You have no excuse to so badly that a teenager can beat you by simple handcrafted rules. First learn and do the basics and then come here to talk about your need for ML. </rant> PS: Sorry, I had to get this out. I've spent better part of my productivity in past two years shifting through page number 50 and 100 of undedupped results that GitHub mercylessly throws at you. No one needs to suffer like this.
- zamfi 7y ago
  Seriously, how dare someone at GitHub release a large dataset that might be interesting while their website’s basic search functionality is still terrible!? Why isn’t everyone at GitHub working to solve just this problem?? C’mon, cut these folks some slack — GitHub is huge, and one annoying adjacent feature doesn’t give you the right to attack these folks like this.
  - mattigames 7y ago
    This is not an "attack", is just straight forward feedback, and really one I agree with, a company that has received more than 100 millions on investment shouldn't have such a bad search feature
    - zamfi 7y ago
      Really? Did you read the same post I did? > Just get the stupid dedupping done, for God's sake. > You have no excuse to so badly that a teenager can beat you by simple handcrafted rules. > First learn and do the basics and then come here to talk about your need for ML. > No one needs to suffer like this. Let’s see, the OP belittles the original authors for something they’re not responsible for, implies that they’re incompetent, tells them to go away until their problem is fixed, and, oh, for god measure: invokes a deity. This is not “straight forward feedback” to me. How about we just cut all the insults and shift the tone a bit: > GitHub has a really poor search experience compared with most modern web services. Search doesn't need fancy ML or DL, and dedupping alone would help a ton. A simple TF-IDF similarity metric would make GitHub’s search 10,000% better. You even know forks, names of files, folders, number of stars. > PS: Sorry, I had to get this out. I've spent better part of my productivity in past two years shifting through page number 50 and 100 of undedupped results that GitHub returns and it’s been extremely frustrating. See how this doesn’t blame the authors for something that isn’t their fault? Doesn’t imply they’re stupid or incompetent for some other feature they have nothing to do with? Doesn’t tell them to go away? And a bonus: It presents some approaches to solve a problem the author is frustrated by. My experience ranting on the internet is that personal attacks don’t lead to feature improvements.
jacquesm 7y ago
Have you used code from private repositories in the training of this model?
- mabrocks 7y ago
  [I'm one of the Microsoft Research people who worked on this] All code is from public repositories. We only used publicly available APIs to obtain the data, so that others can reproduce the results / build on top of the data extraction pipeline we built: https://github.com/github/CodeSearchNet/tree/master/function_parser https://github.com/github/CodeSearchNet/tree/master/function...
  - throwaway744678 7y ago
    What about code from public repository that don't have some kind of open source licence? Won't you be running into a copyright issue somewhere?
    - mabrocks 7y ago
      We only used code from repos that github has marked as using a non-copyleft open source license (i.e., we had an explicit license whitelist and used only repos matching that).
      - gmueckl 7y ago
        So how do you handle repositories that apply different licenses to different subsections of code? E.g. what haopens when an optional helper library is in a subdirectory and licensed differently?
  - jacquesm 7y ago
    Thank you for responding. So, then the next question is whether or not the code found in public repositories is also representative for the code found in private ones. I can see arguments for it going either way (better, worse), which might have substantial impact on the applicability of the models for code from the class that it never saw.
marmaduke 7y ago
thanks for the dataset release! So many questions. Have you thought about any approaches to assess quality/accuracy of comments by using the data itself? One could assume a distribution of accuracy and look for correlated metrics on the comment? Other information such as author or project could be helpful if present, by simplying regularizing comment quality over author/project? If GitHub would be a primary beneficiary of any techniques developed with this dataset, would they volunteer any compute resources toward running models?
- mallamanis 7y ago
  [I'm one of the Microsoft Research people who worked on this] Thanks for your questions! We have thought of many heuristics but we didn't want to constrain the dataset release on some heuristic that we picked, possibly ruining the dataset. Participants in the challenge should feel free to apply additional filters as they see fit. For example, this [1] work could be useful as a filtering method. Unfortunately, we do not have the budget to provide any compute resources to help with running the models at this time. Note that any techniques developed with this dataset will be owned by those who develop them and it's up to them how/if they will make them available/open-source. [1] https://arxiv.org/abs/1806.04616 https://arxiv.org/abs/1806.04616
flohofwoe 7y ago
I know this sounds silly because code on github is visible to everybody anyway and that's a good thing, but I would appreciate a way to opt out my own code from such automated data gathering programs for machine learning purposes. Something like a robots.txt for github projects. Not that anybody would really care, only to make my intent clear that I don't support this sort of mass data gathering nonsense.
- kogir 7y ago
  You opt out by using a restrictive license or making your repo private.
  - flohofwoe 7y ago
    Using a restrictive license seems a bit extreme when the whole point of a public github repo is to make the code available for other programmers. I consider this mass data gathering use for machine learning an edge case which should not harm the 'legitimate' users of the repository.
    - bpt3 7y ago
      Why do you see this effort as not "legitimate"?
- zelly 7y ago
  Self-host your own Git browser. Gitlab or Cgit is still good which is still used by the biggest project in the world (linux kernel).
tallanvor 7y ago
Why are you hosting the data in AWS instead of on Azure?
deleted 7y ago
[deleted]
ArtWomb 7y ago
Absolute game changer. We are witnessing the birth of "Big Code" From the paper, it's not really clear how the resulting binary or intermediate representation fits into the picture? Wouldn't a robust model have to exploit features of the underlying computer architecture as well? Besides the possible applications mentioned such as recommendation systems and code generation. I am wondering about automatic code annotation and documentation. With the eventual goal of creating a "code tutor" that would assist students in real-time as they type. I think what's interesting about this is that the most talented CS grads tend to have a high correlation with deep understanding not only of the history of digital logic design, but also math, physics etc. Fantastic work ;)

airocker 7y ago

Could you have code reviews in the data?

mloncode 7y ago
The unit of observation for this data are functions or methods and their associated docstrings or top-level comments. So there are no code reviews in the data. However, we do include metadata including the SHA and owner/repo which would allow you to retrieve this information! What were you thinking of doing with code reviews?
- airocker 7y ago
  We can try to write an automatic code review system?

dchichkov 7y ago

How would you make sure that any derived works (resulting from the use of this dataset) are properly licensed? It is very likely that this dataset (as per 1909.09436.pdf: "2 million functions, obtained from mechanically scraping and preprocessing associated function documentation") is contaminated by code with dubious licenses, right? And any derived work as a result would also be contaminated, right? What's the plan to deal with this issue?

mabrocks 7y ago
[I'm one of the Microsoft Research people who worked on this] We used the repo-level license information from github to filter to free non-copyleft licenses, and the license files are actually stored next to the extracted data corpus (see https://github.com/github/CodeSearchNet#Licenses https://github.com/github/CodeSearchNet#Licenses).
- panny 7y ago
  Are you doing anything to help copyleft licensed code repos find infringers? We all know it's common practice for unscrupulous developers and companies to take copyleft code, copy it, and then try to relicensed under permissive licenses on a regular basis. Like Google did with Oracle's GPLed Java source code. It seems irresponsible for Github to continue distributing infringing code when it clearly has the tools to prevent that happening. Youtube has fingerprinting for infringing music and video. Github should be doing the same with source code.
  - desiderantes 7y ago
    >Like Google did with Oracle's GPLed Java source code. What are you talking about?
    - panny 7y ago
      https://news.ycombinator.com/item?id=20066269 https://news.ycombinator.com/item?id=20066269

sytelus 7y ago

<rant> GitHub has one of the worse search experience in the modern history of the internet. It doesn't need fancy ML or DL. Just get the stupid dedupping done, for God's sake. A simple TF-IDF similarity metric would make your search 10,000% better. You even know forks, names of files, folders, number of stars. You have no excuse to so badly that a teenager can beat you by simple handcrafted rules. First learn and do the basics and then come here to talk about your need for ML. </rant> PS: Sorry, I had to get this out. I've spent better part of my productivity in past two years shifting through page number 50 and 100 of undedupped results that GitHub mercylessly throws at you. No one needs to suffer like this.

zamfi 7y ago
Seriously, how dare someone at GitHub release a large dataset that might be interesting while their website’s basic search functionality is still terrible!? Why isn’t everyone at GitHub working to solve just this problem?? C’mon, cut these folks some slack — GitHub is huge, and one annoying adjacent feature doesn’t give you the right to attack these folks like this.
- mattigames 7y ago
  This is not an "attack", is just straight forward feedback, and really one I agree with, a company that has received more than 100 millions on investment shouldn't have such a bad search feature
  - zamfi 7y ago
    Really? Did you read the same post I did? > Just get the stupid dedupping done, for God's sake. > You have no excuse to so badly that a teenager can beat you by simple handcrafted rules. > First learn and do the basics and then come here to talk about your need for ML. > No one needs to suffer like this. Let’s see, the OP belittles the original authors for something they’re not responsible for, implies that they’re incompetent, tells them to go away until their problem is fixed, and, oh, for god measure: invokes a deity. This is not “straight forward feedback” to me. How about we just cut all the insults and shift the tone a bit: > GitHub has a really poor search experience compared with most modern web services. Search doesn't need fancy ML or DL, and dedupping alone would help a ton. A simple TF-IDF similarity metric would make GitHub’s search 10,000% better. You even know forks, names of files, folders, number of stars. > PS: Sorry, I had to get this out. I've spent better part of my productivity in past two years shifting through page number 50 and 100 of undedupped results that GitHub returns and it’s been extremely frustrating. See how this doesn’t blame the authors for something that isn’t their fault? Doesn’t imply they’re stupid or incompetent for some other feature they have nothing to do with? Doesn’t tell them to go away? And a bonus: It presents some approaches to solve a problem the author is frustrated by. My experience ranting on the internet is that personal attacks don’t lead to feature improvements.

jacquesm 7y ago

Have you used code from private repositories in the training of this model?

mabrocks 7y ago
[I'm one of the Microsoft Research people who worked on this] All code is from public repositories. We only used publicly available APIs to obtain the data, so that others can reproduce the results / build on top of the data extraction pipeline we built: https://github.com/github/CodeSearchNet/tree/master/function_parser https://github.com/github/CodeSearchNet/tree/master/function...
- throwaway744678 7y ago
  What about code from public repository that don't have some kind of open source licence? Won't you be running into a copyright issue somewhere?
  - mabrocks 7y ago
    We only used code from repos that github has marked as using a non-copyleft open source license (i.e., we had an explicit license whitelist and used only repos matching that).
    - gmueckl 7y ago
      So how do you handle repositories that apply different licenses to different subsections of code? E.g. what haopens when an optional helper library is in a subdirectory and licensed differently?
- jacquesm 7y ago
  Thank you for responding. So, then the next question is whether or not the code found in public repositories is also representative for the code found in private ones. I can see arguments for it going either way (better, worse), which might have substantial impact on the applicability of the models for code from the class that it never saw.

marmaduke 7y ago

thanks for the dataset release! So many questions. Have you thought about any approaches to assess quality/accuracy of comments by using the data itself? One could assume a distribution of accuracy and look for correlated metrics on the comment? Other information such as author or project could be helpful if present, by simplying regularizing comment quality over author/project? If GitHub would be a primary beneficiary of any techniques developed with this dataset, would they volunteer any compute resources toward running models?

mallamanis 7y ago
[I'm one of the Microsoft Research people who worked on this] Thanks for your questions! We have thought of many heuristics but we didn't want to constrain the dataset release on some heuristic that we picked, possibly ruining the dataset. Participants in the challenge should feel free to apply additional filters as they see fit. For example, this [1] work could be useful as a filtering method. Unfortunately, we do not have the budget to provide any compute resources to help with running the models at this time. Note that any techniques developed with this dataset will be owned by those who develop them and it's up to them how/if they will make them available/open-source. [1] https://arxiv.org/abs/1806.04616 https://arxiv.org/abs/1806.04616

flohofwoe 7y ago

I know this sounds silly because code on github is visible to everybody anyway and that's a good thing, but I would appreciate a way to opt out my own code from such automated data gathering programs for machine learning purposes. Something like a robots.txt for github projects. Not that anybody would really care, only to make my intent clear that I don't support this sort of mass data gathering nonsense.

kogir 7y ago
You opt out by using a restrictive license or making your repo private.
- flohofwoe 7y ago
  Using a restrictive license seems a bit extreme when the whole point of a public github repo is to make the code available for other programmers. I consider this mass data gathering use for machine learning an edge case which should not harm the 'legitimate' users of the repository.
  - bpt3 7y ago
    Why do you see this effort as not "legitimate"?
zelly 7y ago
Self-host your own Git browser. Gitlab or Cgit is still good which is still used by the biggest project in the world (linux kernel).

tallanvor 7y ago

Why are you hosting the data in AWS instead of on Azure?

deleted 7y ago

[deleted]

ArtWomb 7y ago

Absolute game changer. We are witnessing the birth of "Big Code" From the paper, it's not really clear how the resulting binary or intermediate representation fits into the picture? Wouldn't a robust model have to exploit features of the underlying computer architecture as well? Besides the possible applications mentioned such as recommendation systems and code generation. I am wondering about automatic code annotation and documentation. With the eventual goal of creating a "code tutor" that would assist students in real-time as they type. I think what's interesting about this is that the most talented CS grads tend to have a high correlation with deep understanding not only of the history of digital logic design, but also math, physics etc. Fantastic work ;)