7 ms·
I have a question and then a general vent 1. Does anyone know the latest update on NumPyPy? PyPy for me is just not a usable proposition because I heavily use
by boultonmark 9y ago
I have a question and then a general vent
1. Does anyone know the latest update on NumPyPy? PyPy for me is just not a usable proposition because I heavily use Numpy (and Scipy et al). So I am forced to use slow Python + fast Numpy or slow Numpy + fast Python. Very saddening. The C-Extension is just so off the pace, NumPyPy was meant to solve that quandry.
And I know some smart Alec will trot out the usual 'downshift into C' line that everyone (including Guido) use as the final goto solution for performance but that is simply a disgrace in 2017. Even JavaScript is fast. Why can I not choose to write Python and it be fast?? And yet Python 3 is getting slower. Don't agree?
Look at these benchmarks of Python heaps written in Python (not using the C based builtin heapq) https://github.com/MikeMirzayanov/binary-heap-benchmark https://github.com/MikeMirzayanov/binary-heap-benchmark Python generally is off the pace but Python 3 is about twice as slow as 2 and miles off JavaScript.
But PyPy is proof that Python can be fast. It makes quote/unquote "Pure Python" within striking distance of Go and and when I run that test suit on PyPy, its similar to the Node.js score. Why does this matter?
Because I want to write bloody Python not C.
And it is so tantalisingly close - look at a blog post like: https://dnshane.wordpress.com/2017/02/14/benchmarking-python-heaps/ https://dnshane.wordpress.com/2017/02/14/benchmarking-python... The performance of the Fibonacci Heap that someone wrote in quote/unquote "Pure Python", when run in CPython can never compete with HeapQ (the C based builtin lib), but on PyPy it can. Fast code written in Python. So what are the problems holding back PyPy? I think possibly money and number of devs working on stuff. Javascript had Mozilla, Google, Microsoft and Apple in a browser war + loads of open source input.
But is the biggest stumbling block not Guido himself and the core Python devs? Do they just philosophically not agree with PyPy or is it just disinterest?
Well whatever it is, it is heart-breaking to want to write fast code in my favourite language and leverage all its power including Numpy/Scipy etc and not be able to. And yes my use-case is perhaps quite unique, a very CPU intensive service that ideally computes and returns a real-time calculation (that includes 500k function calls) in 10-50ms.
But getting fast Numpy in the PyPy mix (i.e all the speed of the JIT + no worse Numpy) would be a HUGE step forward for me in PyPy adoption. What is the latest? How can I help?
- mkj 9y agoI can't answer how numpypy is going, but Numba works pretty well for me to write fast numeric code in python. A bit restricted language and installing llvm is a bit of a hassle, but overall it's great.
- ant6n 9y agoThat's CPython and the reason you're not noticing its slow is probably because most of your time is spent computing inside compiled library functions.
- maxerickson 9y agoNumba is a Python compiler: http://numba.pydata.org/ http://numba.pydata.org/ So the fast functions they are using are likely compiled, they aren't necessarily library functions.
- mattbillenstein 9y agoI think the core devs are doing a fine job with CPython as the reference implementation and developing PyPy takes a different set of expertise; JIT and compilers specifically. I think sponsorship of PyPy would be welcome -- but it seems non-obvious where that would come from. Javascript has the fortune of being the language that drives a very important platform -- and Chrome has been a particularly strategic investment for Google to have more control over the web than it ever has before. Java has Android... Python unfortunately doesn't have that sort of standing in any area that I'm aware of.
- red2awn 9y ago> Do they just philosophically not agree with PyPy or is it just disinterest? I think Guido wants the reference implementation to have a simple and straightforward codebase, a JIT is anything but that.
- boultonmark 9y agoAnd that would be fine, and my mind goes to similar examples like Lua where the reference and the JIT versions co-exist but PyPy has not had the impact that LuaJIT has had (for example) on the LUA community.
- petters 9y agoI agree writing C extensions is not a solution. It is evidently too hard. Even the standard pickle library in Python 3 has a memory corruption bug. http://bugs.python.org/issue23655 http://bugs.python.org/issue23655
- nimish 9y agoI desperately want numpy, pandas and scipy on pypy3 Then I can ditch Cpython and its stockholm syndrome slowness forever
- pjmlp 9y agoI get the feeling that CPython developers don't care that much about PyPy, I might be wrong though. Hence why I am looking forward to Julia making inroads into Python's domain, as that might be the butterfly effect that triggers PyPy adoption.
- orf 9y agoNewsflash: code like this[1] will never be fast in CPython, and if you write a lot of code like that and are sad when it's slow then you need a different language, especially if you expect it to be as fast as a JIT compiled language like js on v8. Or use something like Cython. That benchmark is pretty meaningless anyway, IMO. Here are some halfway decent, official and up to date benchmarks comparing python 2 and python 3[2]. Python 3 is slower in some areas, noticilby startup time, but it's not all doom and gloom. It's faster in a lot of places. And productivity is hard to benchmark, but IMO py3 is way better in the area. 1. https://github.com/MikeMirzayanov/binary-heap-benchmark/blob/master/python3/heap.py https://github.com/MikeMirzayanov/binary-heap-benchmark/blob... 2. https://speed.python.org/comparison/?exe=12%2BL%2B3.6%2C12%2BL%2B2.7&ben=616%2C617%2C618%2C619%2C620%2C621%2C622%2C623%2C624%2C625%2C626%2C627%2C628%2C629%2C630%2C631%2C632%2C680%2C633%2C634%2C635%2C636%2C637%2C638%2C639%2C640%2C641%2C642%2C643%2C644%2C645%2C646%2C647%2C648%2C681%2C649%2C650%2C651%2C652%2C653%2C654%2C655%2C656%2C657%2C658%2C659%2C660%2C661%2C682%2C662%2C663%2C664%2C665%2C666%2C667%2C669%2C668%2C670%2C671%2C672%2C673%2C674%2C675%2C678%2C677%2C676%2C679&env=1&hor=true&bas=none&chart=normal+bars https://speed.python.org/comparison/?exe=12%2BL%2B3.6%2C12%2...
- boultonmark 9y agoAh the 'smart alec' has appeared. I'm not stupid, I know code like that won't be faster in Python but PyPy shows that it can be a hell of a lot faster than CPython and right up there with Node.js and the travesty is that CPython is so far off the pace and getting slower
- lclarkmichalek 9y agoIt's not really a travesty. CPython just isn't designed for that kind of workload, and that's fine, because not many people use it to do that kind of thing.
- dr_zoidberg 9y agoPython 3.6.1 is a lot faster than 3.3.2 which was used back when the heap benchmark was done. In my system, 2.7.10 vs 3.6.1: C:\Test\PyBench>py -2 test.py Done in 1188.308127 C:\Test\PyBench>py -3 test.py Done in 1454.897614 Please bring up to date benchmarks to the discussion, and stop complaining about old problems. Note: adjusted I the workload to be 1000 less iterations, to get the results fast for this comment, so this numbers aren't comparable to the list in the github repo. But even if I hadn't done that, they wouldn't be because I ran these in my system.
- fijal 9y agoin short - funding. If we can find someone who wants fast numpy AND fast python under the same hood, we can combine the approaches of cpyext and numpypy and make it fast. The project is just too big to do on spare time. I've been trying to find some funding for that for quite a while, but I haven't been able to find any sizable backer just yet. Cheers, Maciej Fijalkowski
- boultonmark 9y agoMaciej, how much would that require, ballpark? I think this is something there would be massive support for. My company would support it
- bastawhiz 9y agoHow much does it cost to pay a knowledgeable engineer for a few years? Probably the better part of a million dollars, at least.
- boultonmark 9y agoIs your name Maciej?
- surye 9y agoIs this a private correspondence?
- fijal 9y agoHi I would say, ballpark, $100k. Email me at fijal at baroquesoftware.com EDIT: the actual figure depends precisely on what you want, which parts of the stack have to be exactly how fast etc.
- nimish 9y agoHow do I get involved?
- Redoubts 9y ago
- cwyers 9y agoEven Javascript is fast? Microsoft, Apple, Google and Mozilla have all invested heavily in making that happen.
- mastazi 9y ago- Statement: "Python 3 is getting slower" (tense: present continuous). - Proof: a benchmark based on Python 3.3 (Python 3.3 was released in 2012).
- fnbr 9y agoWhat are you currently using to solve that problem? I've ran into that problem too, and had to use C-extensions to make my code faster, which isn't ideal.
- flavio81 9y agoI understand that Numpy is fast in CPython because it relies on high speed code being done in C. What you could do is much simpler --- : Split your Python application in two parts: 1. Keep your functions that make heavy use of Numpy and Scipy under CPython; expose your algorithms/functions as a web service/REST service/etc running under CPython. 2. The rest of the application, which of course needs to call the functions in (1), can be written in PyPy and call the web service in (1). This is where you would put the general-purpose stuff like web, graphics, database access, and of course all symbolic manipulations that do not require Numpy/Scipy. I think this is a simple, workable solution.
- smortaz 9y agoWe [msft python team] tried to get PyPy some funding, but it didn't go very far. I'll keep trying. We've also started this project to enable jitting for CPython: https://github.com/Microsoft/Pyjion https://github.com/Microsoft/Pyjion The core of it is there and it's been accepted into the main trunk, but we haven't had the cycles to do quality codegen yet.
- stuaxo 9y agoPyjion looks good, once it is easy to build in Linux it will probably attract some users and even possibly some extra developers.
- brachi 9y agoYou can read this great article just released about the Python 2017 language summit: "Keeping Python Competitive" [1]. There you can read opinions by many core developers. Pypy is also discussed [1] https://lwn.net/Articles/723949/ https://lwn.net/Articles/723949/
- deleted 9y ago[deleted]
- gshulegaard 9y ago> And I know some smart Alec will trot out the usual 'downshift into C' line that everyone (including Guido) use as the final goto solution for performance but that is simply a disgrace in 2017. Easy gluing of other languages together has long been something I considered a strength...but I suppose to each their own. > Why can I not choose to write Python and it be fast?? Well there are lots of reasons...including implementation issues and I don't know them all...but I think Python has a very clear productivity niche. Personally, I am ok with Python trading performance for productivity. For the most part, I haven't had Python be so much of a bottleneck that writing a very small part of logic to be performant hasn't solved my use case. > And yet Python 3 is getting slower. Don't agree? Yeah I don't agree...that benchmark uses Python 3.3. The corner on Python 3 performance over Python 2 started turning around 3.4. Perhaps a talk from this years PyCon would help illustrate: https://www.youtube.com/watch?v=d65dCD3VH9Q https://www.youtube.com/watch?v=d65dCD3VH9Q > But PyPy is proof that Python can be fast. Indeed, I would even say that Cython is even more proof that there are frontiers of performance that could be explored. But with PyPy (as with Cython) their are sacrifices you have to make. Personally, I think the most promising performance improvement that is tantalizingly close for me is Larry Hasting's Gilectomy project: > https://www.youtube.com/watch?v=pLqv11ScGsQ https://www.youtube.com/watch?v=pLqv11ScGsQ But at the same time, I am not sure that Python ever needs to be fast running in CPython. With `WASM` perhaps it is better to just compile Python. I don't know, performance in Python has always been a mixed bag...but personally I think it doesn't get much focus because it doesn't really serve Python's target niche. I don't know if there ever will be (or should) be 1 language to do everything...and as it is Python is a good "productivity" focused language to have in your toolbox so-to-speak.
- neutronicus 9y agoUnlike you, Python is _not_ my favorite language, but the matplotlib lock-in is real. Hopefully a matplotlib-equivalent will materialize on Clojure (where Linear Algebra is plenty fast and the language itself is fast-enough out of the box) so I can be done with Python forever.
- takeda 9y agoIf your code looks like this: https://github.com/MikeMirzayanov/binary-heap-benchmark/blob/master/python3/heap.py https://github.com/MikeMirzayanov/binary-heap-benchmark/blob... Then you probably should not use Python, python is more of a glue language which you should strive to make your program looking like a business logic, in real word to solve this problem you would write code such as this: import time if __name__ == "__main__": start = time.clock() N = 10000000 h = list(range(N)) h.sort() for i, v in enumerate(h): assert(i == v) print("Done in %f" % ((time.clock() - start) * 1000)) $ python3.6 heap.py Done in 2389.877000 Or if heap needs to be used: import time from heapq import heapify, heappop if __name__ == "__main__": start = time.clock() N = 10000000 h = list(range(N)) heapify(h) for i in range(len(h)): assert(i == heappop(h)) print("Done in %f" % ((time.clock() - start) * 1000)) $ python3.6 heap.py Done in 10716.348000 Micro benchmarks are silly because you'll never do those things in real code.