5 ms·
[I'm one of the Microsoft Research people who worked on this] There wasn't a technical reason (unless you count lazyness as a technical reason) -- we simply ha
by mabrocks 7y ago
[I'm one of the Microsoft Research people who worked on this]
There wasn't a technical reason (unless you count lazyness as a technical reason) -- we simply had infrastructure for Python specifically lying around from past research projects, which we initially reused.
After we got Nat's feedback, we redid our data processing pipeline completely to be based TreeSitter (which wasn't around when we started thinking about parsing Python), which makes it much easier to scale to the number of programming languages on GitHub.
- Dowwie 7y agoIn other words, the system is ready to be trained for languages not on the initial list, without requiring additional development?
- mabrocks 7y agoYou would need to extend the data-processing pipeline for the new language, which in the best case only requires to adapt the standard wrapper around the Tree Sitter parser. The wrapper needs to take care of language-specific details (e.g., where to find the documentation for a function, what methods should be filtered out, etc.) but can be fairly small. See https://github.com/github/CodeSearchNet/blob/master/function_parser/function_parser/parsers/ https://github.com/github/CodeSearchNet/blob/master/function... for examples.