5 ms·

The idea is that most of the regular numpy (for CPython) is not written in Python anyway. It is actually written in C. For PyPy, they will be writing major port

The idea is that most of the regular numpy (for CPython) is not written in Python anyway. It is actually written in C. For PyPy, they will be writing major portions of it in RPython instead of C, and RPython is way more high level than C.

andrewcooke 15y ago
i know numpy is largely written in c. i am not sure whether i am incompetent at explaining myself, or if people simply are not reading what i post. my point, again, is that pypy is supposed to give speed comparable to c (particularly in tight loops that don't have changes of type, which you would imagine numpy to be) without needing to write in either c or rpython - you can just use python. and this is confirmed by the poster on that thread, who said that numpy was exceptional, and that other people should not need to use rpython... my original comment, then, was that perhaps whatever was unusual could itself be abstracted out and made available at the python level. it's really not that hard to understand, is it? :o(
- wtallis 15y ago
  The "unusual code" portion of numpy is the entirety of numpy. It's BLAS for Python. Once it's well-optimized, basically any other numerical code or libraries for python (things like scipy and probably large portions of PIL, and applications built on them) should do reasonably well written in pure python running on PyPy. The JIT in PyPy should be able to offer good performance for non-numerical, dynamic, branchy code, so you will very rarely have to drop down to a lower-level language (C or RPython) except for FFI. JITs can help with tight loops that can't be identified and optimized at compile time, but that's not what Numpy is composed of, and code using numpy shouldn't have to go through several iterations before the Numpy routines get identified as hotspots and optimized. Numpy implements algorithms that are designed to be easy to statically compile into machine language code that makes efficient use of memory, FPU, and cache resources.
  - andrewcooke 15y ago
    i still don't understand why the jit can't also do a decent job on "easy" code. i can see that sse and, as you mention, the need for warm up would be an issue. but if that is the case why not include ways to support sse and flag that assumptions about types are reliable in python? (or, if that's just too hard, make it easy to mix rpython and python - perhaps i am mistaken here but last time i looked that seemed to be completely undocumented and apparently hard). it's often the case in numerical code that there's some algorithm that you can't convert into standard matrix routines, but you still want it to run fast. when you're using BLAS + fortran (or c) that's not a problem. but here it will be. sure, that was also an issue with numpy - but why can't pypy be better than old numpy? [i am not trying to ask for the technically impossible here - i'm saying that numpy is not so amazingly different that rpython isn't going to be necessary for other projects. either make rpython accessible, or do numpy in some other way - what bugs me is the "we can use rpython because we're special, but you can't because you don't need the power" attitude]

andrewcooke 15y ago

i know numpy is largely written in c. i am not sure whether i am incompetent at explaining myself, or if people simply are not reading what i post. my point, again, is that pypy is supposed to give speed comparable to c (particularly in tight loops that don't have changes of type, which you would imagine numpy to be) without needing to write in either c or rpython - you can just use python. and this is confirmed by the poster on that thread, who said that numpy was exceptional, and that other people should not need to use rpython... my original comment, then, was that perhaps whatever was unusual could itself be abstracted out and made available at the python level. it's really not that hard to understand, is it? :o(

wtallis 15y ago
The "unusual code" portion of numpy is the entirety of numpy. It's BLAS for Python. Once it's well-optimized, basically any other numerical code or libraries for python (things like scipy and probably large portions of PIL, and applications built on them) should do reasonably well written in pure python running on PyPy. The JIT in PyPy should be able to offer good performance for non-numerical, dynamic, branchy code, so you will very rarely have to drop down to a lower-level language (C or RPython) except for FFI. JITs can help with tight loops that can't be identified and optimized at compile time, but that's not what Numpy is composed of, and code using numpy shouldn't have to go through several iterations before the Numpy routines get identified as hotspots and optimized. Numpy implements algorithms that are designed to be easy to statically compile into machine language code that makes efficient use of memory, FPU, and cache resources.
- andrewcooke 15y ago
  i still don't understand why the jit can't also do a decent job on "easy" code. i can see that sse and, as you mention, the need for warm up would be an issue. but if that is the case why not include ways to support sse and flag that assumptions about types are reliable in python? (or, if that's just too hard, make it easy to mix rpython and python - perhaps i am mistaken here but last time i looked that seemed to be completely undocumented and apparently hard). it's often the case in numerical code that there's some algorithm that you can't convert into standard matrix routines, but you still want it to run fast. when you're using BLAS + fortran (or c) that's not a problem. but here it will be. sure, that was also an issue with numpy - but why can't pypy be better than old numpy? [i am not trying to ask for the technically impossible here - i'm saying that numpy is not so amazingly different that rpython isn't going to be necessary for other projects. either make rpython accessible, or do numpy in some other way - what bugs me is the "we can use rpython because we're special, but you can't because you don't need the power" attitude]