6 ms·
Show HN: Python Source Code Refactoring Toolkit via AST
- TekMol 5y agoThe one thing I hope for when it comes to Python is that PyPy becomes the default runtime everywhere. CPython is so unbelievably slow when compared to languages like PHP and Javascript. When using PyPy, I keep encountering road bumps because many tools are still expecting that one uses CPython. Dear world - please accelerate the conversion to PyPy!
- treesciencebot 5y agoAuthor of this project here (and also a CPython core developer FWIW), for many aspects (including the AST) PyPy follows CPython closely even on implementation details so `refactor` would probably work out of the box once PyPy releases 3.9 (the latest stable version for PyPy is 3.7, and 3.8 is on the way!).
- keithasaurus 5y agoI think it's more likely that people start using mypyc for performance-critical code. https://mypyc.readthedocs.io/en/latest/ https://mypyc.readthedocs.io/en/latest/
- formerly_proven 5y agoI assume the performance profile is similar to compiling regular Python code with Cython (which it can do). There is a decent but not world-changing speed-up; this makes sense, because pretty much the only thing you are removing from the equation is the instruction and stack overhead of the interpreter, you still have to perform all the equivalent work that CPython does, otherwise you would end up with different semantics. And this work is substantial. In contrast, a tracing JIT can dynamically elide most of this work without changing the semantics.
- xapata 5y agoAfter switching to Cython, you can add Cython syntax, like type definitions, to gain big speed improvements.
- rattray 5y agoWow, this looks exciting. Looks like Cython without any extra work/syntax, for those using mypy. I wish their readme gave some kind of estimate of how the speedups might compare to Cython and PyPy. But it's currently alpha, so there may be big differences by the time it's ready for production use. Similar approach to the Sorbet Compiler for Ruby (albeit targeting C instead of llvm... I wonder how that might impact optimizability).
- heavyset_go 5y agoProjects that depend on CPython C extensions should consider migrating to HPy[1] for C extension compatibility across Python implementations. [1] https://github.com/hpyproject/hpy https://github.com/hpyproject/hpy
- sitkack 5y agoThis looks interesting, but it also looks ambitious and not anywhere ready. Everyone would be well served for extensions to not be aware of Python internals and expose a clean handle based api to all users. For this, cffi is sufficient and already available across nearly all Python implementations. https://cffi.readthedocs.io/en/latest/ https://cffi.readthedocs.io/en/latest/ I do agree that a standardized Python should have an object model that is a clean public contract, but given that Python is CPython and there is no standardization, that the library runtime is married to the core also makes this incredibly difficult politically.
- hsbauauvhabzb 5y agois performance even an issue either the tool? I regularly see people complain about python speed but normally find it’s algorithmic as opposed to interpreter imposed..
- milkbikis 5y agoNice, I've always thought Python could do better by adopting source transformation tools like babel. I implemented something similar using lib2to3, which preserves whitespace and comments more accurately (than the ast module): https://github.com/banga/prefactor https://github.com/banga/prefactor
- treesciencebot 5y agoYes! I love to do transformations with lib2to3, but unfortunately it is now deprecated. As I've stated in the README, for complex refactorings I also would personally prefer working with CST (probably through parso instead of regular lib2to3), but for refactoring small fragments I wasn't able to observe major differences in formatting in big codebases. Thanks for sharing this btw!
- milkbikis 5y agoAh I didn’t know it was deprecated, but good to see new work happening in this area!
- graton 5y agoA similar type project. Though I haven't seen much activity recently: https://github.com/facebookincubator/Bowler https://github.com/facebookincubator/Bowler
- treesciencebot 5y agoI haven't used bowler, but from what I can see it is using lib2to3 (through fissix), which can't parse newer Python code (parenthesized context managers, new pattern matching statement etc.) due to it being using a LL(1) parser. The regular ast on the other hand is always compatible with the current grammar, since that is what CPython internally consumes. It is also more handy to work with since majority of the linters already using it, so it would be very easy to port rules.
- jreese 5y ago(Author of Bowler/maintainer of fissix) The main concern with using the ast module from stdlib is that you lose all formatting/whitespace/comments in your syntax tree, so writing any changes back to the original sources requires doing a lot more extra work to preserve the original formatting, put back comments, etc. This is entire point of lib2to3/fissix and LibCST, allowing large scale CST manipulation while preserving all of those comments and formatting. We do recognize the limitations of lib2to3/fissix, though, so there have been some backburner plans to move Bowler onto LibCST, as well as building a PEG-based parser for LibCST specifically to enable support for 3.10 and future syntax/grammar changes. But of course, this is very difficult to give any ETA or target for release.
- hsbauauvhabzb 5y agoI’ve been thinking about a similar problem I have on a regular basis: I audit a lot of code in many languages, sometimes I’m not even able to build solutions. my toolset pretty much includes git and rudimentary regex. A recent example is a PHP function which is insecure when it’s used at least 3 times in a string concat. Any suggestion on an AST ‘framework’ that would help me parse code easier? Language specific or generic, even if it only sort-of fits (I don’t even know what I want). Automated Code analysers exist, but I want something manual.
- jay-aye-see-key 5y agoThis is not a problem I'd had, but it sounds like something tree-sitter might be able to help with. I've been using tree-sitter for syntax highlighting, simple refactors, and custom text objects in NeoVim recently (and love it, thanks NeoVim devs!). Have a play with the syntax tree it generates https://tree-sitter.github.io/tree-sitter/playground https://tree-sitter.github.io/tree-sitter/playground (it can generate a tree for most languages, not just the ones in this playground).
- chrisphilip 5y agoI wrote a library that takes a language specification (BNF, ABNF, etc) into a custom AST in Python. Then you can define a set of visitors for the custom AST to transform the tree into whatever you want. I implemented some checks integrated into the Python type system to type-check the visitors and I've used the library to do some non-trivial manipulations before. It's mostly a pet project and I just wanted to share since it could maybe at least inspire something. https://github.com/chrisphilip322/prosodia https://github.com/chrisphilip322/prosodia
- woile 5y agoMaybe something like this: https://semgrep.dev/ https://semgrep.dev/ I found it last week. It may be able to detect a potential issue in multiple languages, but not sure if it supports refactoring.
- dreamer7 5y agoWould this be a good tool to attempt simple transpiling of typescript code to python? For my use-cases, not much changes between the two - 1. Replace opening and closing braces with colon. 2. Remove semi colons 3. Replace function with def 4. Remove type definitions 5. Replace array manipulation functions with python equivalents. 6. Remove tokens like const, let and var
- treesciencebot 5y agoIt would probably be better if you can just write a typescript parser, and then you can transpile type script AST directly to Python AST and use ast.unparse() without any additional libraries.
- dreamer7 5y agoWould this be a good tool to attempt simple transpiling of typescript code to python? For my use-cases, not much changes between the two - 1. Replace opening and closing braces with colon. 2. Remove semi colons 3. Replace function with def 4. Remove type definitions 5. Replace array manipulation functions with python equivalents. 6. Remove tokens like const, let and var
- erezsh 5y agoIt's a good start, but there are many other differences. a[i] can mean getattr (but usually not), and can throw a KeyError. Similarly, a.b may return null if it doesn't exist, while in Python it will throw an AttributeError. You might need to add uses of global or nonlocal. And there are a lot of other small details. However, I think it will be much easier than translating the other way around :)
- boxed 5y agoYou should check out parso which is a fully roundtrip-able AST. That way your tool won't throw away comments and formatting.
- treesciencebot 5y agoThanks for the suggestion though just want to clarify, I am also one of the maintainers of parso :-) (and also the ast module in the standard library)
- treesciencebot 5y agoExtra clarification: I'm well aware the concerns about tokens vs CST vs AST, that is why I decided to write down a library that does refactors on small code fragments rather than just unparse()ing everything. CST is definitely more sound in terms of preserving these details but dealing with it much more complicated than the regular AST. Refactor's README these also mentions other libraries for complex refactors: parso, LibCST, FixIT
- bmc7505 5y agoI wonder if something like this exists for other languages? Most rewriting/refactoring tools make too many assumptions about the surrounding project. Something simple which supports pattern matching against a CST subtree and invertible parsing/unparsing would be the ideal polyglot macro-system. Suppose I want to find and rename all variables with a certain type or rewrite an expression of a certain shape in multiple languages at once (e.g. Java/JavaScript/C/C++). Something between a regular expressions (too brittle) and a full parser would be great a great solution to have for this use case.
- erezsh 5y agoI actually have a prototype of something like that, built on top of Lark. It's still far from being complete or user friendly, but I'm already successfully transpiling a subset of Python to Javascript. It still needs a lot of handholding, but at the same time I get to use gems like - ... = TemplateTranslator({ 'isinstance($a, str)': "typeof($a) == 'string'", 'isinstance($a, bool)': "typeof($a) == 'boolean'", 'len($a)': '$a.length', 'str($a)': '$a.toString()', 'getattr($a, $b)': '$a and $a[$b]', 'getattr($a, $b, $c)': '($a and $a[$b]) or $c', 'hasattr($a, $b)': '$b in $a', '$a.join($b)': '$b.join($a)', ... })
- bmc7505 5y agoNice! I like the syntax, this reminds me of the structural search and replace feature from IntelliJ IDEA. Is this the project you were referring to? https://github.com/erezsh/py2js https://github.com/erezsh/py2js