5 ms·

This seems like such a significant problem for the language and has been around since the beginning of time. Considering the amount of value that Python adds to

This seems like such a significant problem for the language and has been around since the beginning of time. Considering the amount of value that Python adds to companies and individuals around the world, is there a reason that institutions or someone with the means hasn't funded a project to "solve" the GIL problem?

snissn 6y ago
I think it's just not that big of a problem / can be worked around. I had a multi terabyte of RAM across 1000s of cores python product and we just used one process per core
davvid 6y ago
Google threw money at the problem and that didn't help. Lookup the history behind the "Unladen Swallow" project.
- pansa2 6y ago
  > Google threw money at the problem Not really, especially compared to the amount of money they threw at JavaScript. IIRC Unladen Swallow was just a couple of interns.
- jashmatthews 6y ago
  Unladen Swallow wasn’t actually a serious attempt like V8 or anything and a large part of why it failed was that LLVM wasn’t ready and even when it was, it was ill suited to compiling a highly dynamic language like Python.
acdha 6y ago
It's far more common to have Python programs be I/O bound, and when they are CPU bound it's often not due to GIL contention (remember that C extensions can drop the GIL before doing something lengthy). It would be nice if the GIL was gone but a fair fraction of Python developers would not notice much difference.
- pdonis 6y ago
  > It's far more common to have Python programs be I/O bound But part of the reason for that is that Python programmers know there's no point in trying to run multiple CPU-bound tasks in the same Python process, so they don't try. > C extensions can drop the GIL before doing something lengthy Yes, but they're still limited in what they can do--as soon as they call back into a Python bytecode they're GIL-bound again. And if you depend on C extensions any time you need CPU bound concurrent tasks, you're giving up a lot of the advantages of using Python in the first place.
  - acdha 6y ago
    It’s part of the reason but not all or, I suspect, even most: the number of things which people need to compute with multi threading but without much I/O is not an especially large fraction of what people use pure Python for. If you need raw CPU speed for arbitrary code, it’s not the language most people would pick. The exceptions also tend to have existing high-quality extensions (crypto, compression, image processing, etc.) so while it’s technically true that you’re giving up Python most people aren’t doing that personally - they’re just calling Pillow or numpy - or they’re using it for a tiny fraction of the total program. Frequently this ends up being the same speed or even faster than using other languages because most people are either using the same C libraries or learning just how many optimizations their simple implementation lacked. Again, it’s not hard to come up with things where the GIL is inarguably a bottleneck but it comes up a lot more in debates than real-life in my experience.
    - andreareina 6y ago
      I'm one of those with embarrassingly-parallel cpu-bound workloads. The multiprocessing module works, but the extra bookkeeping and plumbing over an actually-parallel-multithreading implementation is a pain in the butt. That said, the multithreading speedup is both faster and easier than porting to another language.
      - acdha 6y ago
mattip 6y ago
PyPy put up a prototype and proposal to deliver a GILless python for $100,000. No one took us up on the offer. It is still open. https://morepypy.blogspot.com/2017/08/lets-remove-global-interpreter-lock.html https://morepypy.blogspot.com/2017/08/lets-remove-global-int...
- jashmatthews 6y ago
  I find it really sad that no company will fund this. One of the huge python deployments would recoup this in hosting quickly. Instead we’re all just burning more money on AWS.
andreareina 6y ago
I think the big problem is all the code out there that depends on the GIL. A hypothetical python4 that removed the GIL would probably cause issues of the same general nature as the 2 -> 3 migration, though at a reduced magnitude.

snissn 6y ago

I think it's just not that big of a problem / can be worked around. I had a multi terabyte of RAM across 1000s of cores python product and we just used one process per core

davvid 6y ago

Google threw money at the problem and that didn't help. Lookup the history behind the "Unladen Swallow" project.

pansa2 6y ago
> Google threw money at the problem Not really, especially compared to the amount of money they threw at JavaScript. IIRC Unladen Swallow was just a couple of interns.
jashmatthews 6y ago
Unladen Swallow wasn’t actually a serious attempt like V8 or anything and a large part of why it failed was that LLVM wasn’t ready and even when it was, it was ill suited to compiling a highly dynamic language like Python.

acdha 6y ago

It's far more common to have Python programs be I/O bound, and when they are CPU bound it's often not due to GIL contention (remember that C extensions can drop the GIL before doing something lengthy). It would be nice if the GIL was gone but a fair fraction of Python developers would not notice much difference.

pdonis 6y ago
> It's far more common to have Python programs be I/O bound But part of the reason for that is that Python programmers know there's no point in trying to run multiple CPU-bound tasks in the same Python process, so they don't try. > C extensions can drop the GIL before doing something lengthy Yes, but they're still limited in what they can do--as soon as they call back into a Python bytecode they're GIL-bound again. And if you depend on C extensions any time you need CPU bound concurrent tasks, you're giving up a lot of the advantages of using Python in the first place.
- acdha 6y ago
  It’s part of the reason but not all or, I suspect, even most: the number of things which people need to compute with multi threading but without much I/O is not an especially large fraction of what people use pure Python for. If you need raw CPU speed for arbitrary code, it’s not the language most people would pick. The exceptions also tend to have existing high-quality extensions (crypto, compression, image processing, etc.) so while it’s technically true that you’re giving up Python most people aren’t doing that personally - they’re just calling Pillow or numpy - or they’re using it for a tiny fraction of the total program. Frequently this ends up being the same speed or even faster than using other languages because most people are either using the same C libraries or learning just how many optimizations their simple implementation lacked. Again, it’s not hard to come up with things where the GIL is inarguably a bottleneck but it comes up a lot more in debates than real-life in my experience.
  - andreareina 6y ago
    I'm one of those with embarrassingly-parallel cpu-bound workloads. The multiprocessing module works, but the extra bookkeeping and plumbing over an actually-parallel-multithreading implementation is a pain in the butt. That said, the multithreading speedup is both faster and easier than porting to another language.
    - acdha 6y ago

mattip 6y ago

PyPy put up a prototype and proposal to deliver a GILless python for $100,000. No one took us up on the offer. It is still open. https://morepypy.blogspot.com/2017/08/lets-remove-global-interpreter-lock.html https://morepypy.blogspot.com/2017/08/lets-remove-global-int...

jashmatthews 6y ago
I find it really sad that no company will fund this. One of the huge python deployments would recoup this in hosting quickly. Instead we’re all just burning more money on AWS.

andreareina 6y ago

I think the big problem is all the code out there that depends on the GIL. A hypothetical python4 that removed the GIL would probably cause issues of the same general nature as the 2 -> 3 migration, though at a reduced magnitude.