7 ms·

Can anyone elin (explain like I am noob), why should the Gil be removed? Why need concurrency in normal python scripts let’s days for web scraping or machine le

Can anyone elin (explain like I am noob), why should the Gil be removed? Why need concurrency in normal python scripts let’s days for web scraping or machine learning ? I don’t really get it.

josefx 4y ago
Going by the paper it also improves performance even for single threaded code[1]. The GIL is just dated design, there isn't anything good about it. [1]https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsDFosB5e6BfnXLlejd9l0/edit#heading=h.6i2dwx2chdq9 https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsD... > Why need concurrency in normal python scripts let’s days for web scraping or machine learning ? I had a few scripts where I had to process over a hundred files, just running that in parallel can reduce a job that takes minutes to seconds.
- ptx 4y ago
  As some other commenters have mentioned, the GIL removal branch also makes some unrelated optimizations and the performance improvement comes from those rather than from the GIL removal itself, as I understand it. From your link: "Stripping out some of the GIL-removal related changes would result in even faster performance on the single-threaded pyperformance benchmark suite. [...] The resulting interpreter is about 9% faster than the no-GIL proof-of-concept (or ~19% faster than CPython 3.9.0a3). That 9% difference between the “nogil” interpreter and the stripped-down “nogil” interpreter can be thought of as the “cost” of the major GIL-removal changes." So it seems removing the GIL has a negative impact on single-threaded code, with the version that has both the GIL and the unrelated optimizations being 9% faster.
- sshlocalhost98 4y ago
  Thank you sir/ma’am for the explanation! The Google docs was an excellent resource.
coldtea 4y ago
>Can anyone elin (...) Why need concurrency in normal python scripts let’s days for web scraping or machine learning ? I don’t really get it. That's not a noob question. That's a loaded question, from a perspective of knowing what people use Python for. People use Python for all kinds of things, and a lot of them would be faster if they were able to take advantage of multithreading without the GIL.
- ModernMech 4y ago
  True but they would also be faster (probably more so) if Python had a jit compiler. Just seems that people want Python to be something other than it is. I guess the assumption is that it’s easier to remove the GIL or jit Python than the alternative, which would be to port whatever libraries to another ecosystem. Maybe a safe bet although there’s no deadline for GIL removal and that’s been a goal for decades.
  - coldtea 4y ago
    >Just seems that people want Python to be something other than it is. Well, if they didn't, we'd still be stuck with Python 1 or 0.1. Why is the GIL suddenly where Python should keep "being what it is", and not any of those tons of changes, from 0.1 to 3.10? Especially since the removal of the GIL doesn't change any spirit/essence of Python - just makes it faster. Python wasn't conceived as "having a GIL" being some essential part of it, it was just a bad tradeoff for implementation convenience made back in the day where common multi-core machines were 20 years in the future...
- sshlocalhost98 4y ago
  Thank you sir/ma’am for the explanation!
formerly_proven 4y ago
Python is not actually a single-purpose language for calling numpy or tf.
- disgruntledphd2 4y ago
  Yeah, I hear it can call pandas too \s
- WorldMaker 4y ago
  In that case, maybe Python should just directly transpile to FORTRAN and save some C bridges? Python's syntax does resemble FORTRAN some days.

Can anyone elin (explain like I am noob), why should the Gil be removed? Why need concurrency in normal python scripts let’s days for web scraping or machine learning ? I don’t really get it.

josefx 4y ago

Going by the paper it also improves performance even for single threaded code[1]. The GIL is just dated design, there isn't anything good about it. [1]https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsDFosB5e6BfnXLlejd9l0/edit#heading=h.6i2dwx2chdq9 https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsD... > Why need concurrency in normal python scripts let’s days for web scraping or machine learning ? I had a few scripts where I had to process over a hundred files, just running that in parallel can reduce a job that takes minutes to seconds.

ptx 4y ago
As some other commenters have mentioned, the GIL removal branch also makes some unrelated optimizations and the performance improvement comes from those rather than from the GIL removal itself, as I understand it. From your link: "Stripping out some of the GIL-removal related changes would result in even faster performance on the single-threaded pyperformance benchmark suite. [...] The resulting interpreter is about 9% faster than the no-GIL proof-of-concept (or ~19% faster than CPython 3.9.0a3). That 9% difference between the “nogil” interpreter and the stripped-down “nogil” interpreter can be thought of as the “cost” of the major GIL-removal changes." So it seems removing the GIL has a negative impact on single-threaded code, with the version that has both the GIL and the unrelated optimizations being 9% faster.
sshlocalhost98 4y ago
Thank you sir/ma’am for the explanation! The Google docs was an excellent resource.

coldtea 4y ago

>Can anyone elin (...) Why need concurrency in normal python scripts let’s days for web scraping or machine learning ? I don’t really get it. That's not a noob question. That's a loaded question, from a perspective of knowing what people use Python for. People use Python for all kinds of things, and a lot of them would be faster if they were able to take advantage of multithreading without the GIL.

ModernMech 4y ago
True but they would also be faster (probably more so) if Python had a jit compiler. Just seems that people want Python to be something other than it is. I guess the assumption is that it’s easier to remove the GIL or jit Python than the alternative, which would be to port whatever libraries to another ecosystem. Maybe a safe bet although there’s no deadline for GIL removal and that’s been a goal for decades.
- coldtea 4y ago
  >Just seems that people want Python to be something other than it is. Well, if they didn't, we'd still be stuck with Python 1 or 0.1. Why is the GIL suddenly where Python should keep "being what it is", and not any of those tons of changes, from 0.1 to 3.10? Especially since the removal of the GIL doesn't change any spirit/essence of Python - just makes it faster. Python wasn't conceived as "having a GIL" being some essential part of it, it was just a bad tradeoff for implementation convenience made back in the day where common multi-core machines were 20 years in the future...
sshlocalhost98 4y ago
Thank you sir/ma’am for the explanation!

formerly_proven 4y ago

Python is not actually a single-purpose language for calling numpy or tf.

disgruntledphd2 4y ago
Yeah, I hear it can call pandas too \s
WorldMaker 4y ago
In that case, maybe Python should just directly transpile to FORTRAN and save some C bridges? Python's syntax does resemble FORTRAN some days.