20 ms·

I'm learning Python. Do someone knows a resource to learn the well-known tricks for "fast" Python? A book, a website, a cheat sheet or even a MOOC or part of a

I'm learning Python. Do someone knows a resource to learn the well-known tricks for "fast" Python? A book, a website, a cheat sheet or even a MOOC or part of a MOOC?

stefanos82 4y ago
Fluent Python; go for it!
- FrenchAmerican 4y ago
  Thanks! I appreciate, really.
  - stefanos82 4y ago
    I've just noticed it did not include my URL in original reply...weird. Anyway, here it is, this is the second edition of the book https://www.amazon.com/dp/1492056359 https://www.amazon.com/dp/1492056359 I read the first one at work a few years ago and was fascinated with all the goodies it shared with us. In my humble opinion, it's a must read by all Python developers, experienced or not; everyone will learn something new just by reading it.
marcosdumay 4y ago
I would advise against optimizing your Python programs for speed. Fighting against a language's nature never leads to good results, it's much easier and better to use a faster language if (or where) you need the speed. This is even more relevant if you are still learning the language. Focus on learning it, and leave arcane always changing implementation details for after you know it well.
- detaro 4y ago
  Or rather, focus on the big things. Algorithmic complexity etc. Also for all languages where you want to do performance work: Learn to use a profiler, so you can find the things that matter. And then selectively look what you can improve there. Even in Python, some hacks or slightly unergonomic patterns in a really hot loop can be worth a lot.
- FrenchAmerican 4y ago
  OK, thanks, I will take that into account.
ThreePinkApples 4y ago
There is currently a "Faster CPython"[1] project going on that seek to improve the speed of Python. With Python 3.11 later this year there will be several speed improvements targeting common uses of Python, which might make some of these existing speedup tricks irrelevant. Or at least less relevant. [1] https://docs.python.org/3.11/whatsnew/3.11.html#faster-cpython https://docs.python.org/3.11/whatsnew/3.11.html#faster-cpyth...
TheDesolate0 4y ago
BiteCode_dev 4y ago
There are plenty of things for which python is not slow, but people using python don't know a lot about programming, and end up with a slow result. A couple of advises: - the right algo will go a long, long way. - know your data structures. E.G: assigning to a slice is ridiculously fast (even while unpacking), memory views may save a lot on byte heavy workloads, heapq and deque are underrated, etc. Also check out https://wiki.python.org/moin/TimeComplexity https://wiki.python.org/moin/TimeComplexity for big O notations on python builtin types common operations to understand what you pay for. - know the stdlib. collections, itertools and functools all contain incredible gems. - delegate. Python is a fantastic glue language, use it for what it's good at. Your database, your numpy arrays, your cache are all amazing are what they do, no matter the language. Let them do the heavy work. - don't kill good perfs by ignorance. I regularly see people casting a generator, iterating on a dataframe, calling readlines() on a file or doing something else that is destroying the otherwise excellent perfs of their program. - know the ecosystem. There are some very good fast libs out there: diskcache, sortedcontainer, scipy, uvloop... - use threads to avoid blocking a GUI, multiprocess to share work between CPU and asyncio to speed up network operations. Each tool has a sweet spot. But threads are underrated, they work well for a couple of hundred parallel network operations, and most C libs will actually release the GIL, so they can use several CPU more often than you'd thin. Also, use pools if you can, shared_memory in 3.8 or mmap. - sometime the dirty solution is just faster, like subprocessing to ffmpeg. - The more recent, the slower. Sure, I love statistics, pathlib and dataclasses. In a regular code they are great. On a bottleneck however, they are very slow. - the array module is not supposed to speed up the code, only save memory. But sometimes it does. - printing to the terminal is limiting. Sometime your program is doing fine, the display is preventing it to go faster. At least check the flush. - comprehensions are faster than alternatives. - pre-allocating lists and dicts can help. - measure. The austin profiler is your friend. - rewriting the hot path in a faster language is likely more interesting than writing the whole program in it. A bit of nim or rust is easy to call from python. - some python distribution are faster than others. I don't mean just pypy, but also regular cpyhon that have been built specially for some architectures. E.G: https://www.intel.com/content/www/us/en/developer/tools/oneapi/distribution-for-python.html https://www.intel.com/content/www/us/en/developer/tools/onea... - nuitka code compilation can speed up the code by a factor of 4, it's nice, especially for startup.
blip54321 4y ago
My 5-minute version: - Get the overall system structure in Python. Get the architectural design right, and the big-O stuff right. - If there are bottlenecks, re-code those directly in C, cython, or similar. Or better yet, find libraries. Python is great for expressing high-level operations and system design. It is also very easy to integrate with native code. I've never had much happiness in optimizing Python itself beyond that. Broadly speaking, code falls into three categories: - Most code: Instant. Performance doesn't matter. Python - Some code: Big-O(lifespan of the universe). Not worth building. - Narrow slice of stuff in between: Don't do in Python, but use Python as the glue.

I'm learning Python. Do someone knows a resource to learn the well-known tricks for "fast" Python? A book, a website, a cheat sheet or even a MOOC or part of a MOOC?

stefanos82 4y ago

Fluent Python; go for it!

FrenchAmerican 4y ago
Thanks! I appreciate, really.
- stefanos82 4y ago
  I've just noticed it did not include my URL in original reply...weird. Anyway, here it is, this is the second edition of the book https://www.amazon.com/dp/1492056359 https://www.amazon.com/dp/1492056359 I read the first one at work a few years ago and was fascinated with all the goodies it shared with us. In my humble opinion, it's a must read by all Python developers, experienced or not; everyone will learn something new just by reading it.

marcosdumay 4y ago

I would advise against optimizing your Python programs for speed. Fighting against a language's nature never leads to good results, it's much easier and better to use a faster language if (or where) you need the speed. This is even more relevant if you are still learning the language. Focus on learning it, and leave arcane always changing implementation details for after you know it well.

detaro 4y ago
Or rather, focus on the big things. Algorithmic complexity etc. Also for all languages where you want to do performance work: Learn to use a profiler, so you can find the things that matter. And then selectively look what you can improve there. Even in Python, some hacks or slightly unergonomic patterns in a really hot loop can be worth a lot.
FrenchAmerican 4y ago
OK, thanks, I will take that into account.

ThreePinkApples 4y ago

There is currently a "Faster CPython"[1] project going on that seek to improve the speed of Python. With Python 3.11 later this year there will be several speed improvements targeting common uses of Python, which might make some of these existing speedup tricks irrelevant. Or at least less relevant. [1] https://docs.python.org/3.11/whatsnew/3.11.html#faster-cpython https://docs.python.org/3.11/whatsnew/3.11.html#faster-cpyth...

TheDesolate0 4y ago

BiteCode_dev 4y ago

There are plenty of things for which python is not slow, but people using python don't know a lot about programming, and end up with a slow result. A couple of advises: - the right algo will go a long, long way. - know your data structures. E.G: assigning to a slice is ridiculously fast (even while unpacking), memory views may save a lot on byte heavy workloads, heapq and deque are underrated, etc. Also check out https://wiki.python.org/moin/TimeComplexity https://wiki.python.org/moin/TimeComplexity for big O notations on python builtin types common operations to understand what you pay for. - know the stdlib. collections, itertools and functools all contain incredible gems. - delegate. Python is a fantastic glue language, use it for what it's good at. Your database, your numpy arrays, your cache are all amazing are what they do, no matter the language. Let them do the heavy work. - don't kill good perfs by ignorance. I regularly see people casting a generator, iterating on a dataframe, calling readlines() on a file or doing something else that is destroying the otherwise excellent perfs of their program. - know the ecosystem. There are some very good fast libs out there: diskcache, sortedcontainer, scipy, uvloop... - use threads to avoid blocking a GUI, multiprocess to share work between CPU and asyncio to speed up network operations. Each tool has a sweet spot. But threads are underrated, they work well for a couple of hundred parallel network operations, and most C libs will actually release the GIL, so they can use several CPU more often than you'd thin. Also, use pools if you can, shared_memory in 3.8 or mmap. - sometime the dirty solution is just faster, like subprocessing to ffmpeg. - The more recent, the slower. Sure, I love statistics, pathlib and dataclasses. In a regular code they are great. On a bottleneck however, they are very slow. - the array module is not supposed to speed up the code, only save memory. But sometimes it does. - printing to the terminal is limiting. Sometime your program is doing fine, the display is preventing it to go faster. At least check the flush. - comprehensions are faster than alternatives. - pre-allocating lists and dicts can help. - measure. The austin profiler is your friend. - rewriting the hot path in a faster language is likely more interesting than writing the whole program in it. A bit of nim or rust is easy to call from python. - some python distribution are faster than others. I don't mean just pypy, but also regular cpyhon that have been built specially for some architectures. E.G: https://www.intel.com/content/www/us/en/developer/tools/oneapi/distribution-for-python.html https://www.intel.com/content/www/us/en/developer/tools/onea... - nuitka code compilation can speed up the code by a factor of 4, it's nice, especially for startup.

blip54321 4y ago

My 5-minute version: - Get the overall system structure in Python. Get the architectural design right, and the big-O stuff right. - If there are bottlenecks, re-code those directly in C, cython, or similar. Or better yet, find libraries. Python is great for expressing high-level operations and system design. It is also very easy to integrate with native code. I've never had much happiness in optimizing Python itself beyond that. Broadly speaking, code falls into three categories: - Most code: Instant. Performance doesn't matter. Python - Some code: Big-O(lifespan of the universe). Not worth building. - Narrow slice of stuff in between: Don't do in Python, but use Python as the glue.