5 ms·

Although this is nice, the problems with the GIL are often blown out of proportion: people stating that you couldn't do efficient (compute-bounded) multi-proces

Although this is nice, the problems with the GIL are often blown out of proportion: people stating that you couldn't do efficient (compute-bounded) multi-processing, which was never the case as the `multiprocessing` module works just fine.

liuliu 3y ago
`multiprocessing` works fine for serving HTTP requests or do some other subset of embarrassingly-parallel problems.
- skrause 3y ago
  > `multiprocessing` works fine for serving HTTP requests Not if you use Windows, then it's a mess. I have a suspicion that people who say that the multiprocessing works just fine never had to seriously use Python on Windows.
  - ptx 3y ago
    Why is it a mess? What's wrong with it on Windows?
    - skrause 3y ago
      * A lack of fork() makes starting new processes slow. * All Python webservers that somewhat support multiprocessing on Windows disable the IOCP asyncio event loop when using more than one process (because it breaks in random ways), so you're left with the slower select() event loop which doesn't support more than 512 connections.
    - colatkinson 3y ago
      Adding on to the other comment, multiprocessing is also kinda broken on Linux/Mac. 1. Because global objects are refcounted, CoW effectively isn't a thing on Linux. They did add a way to avoid this [0], but you have to manually call it once your main imports are done. 2. On Mac, turns out a lot of the system libs aren't actually fork-safe [1]. Since these get imported inadvertently all the time, Python on Mac actually uses `spawn` [2] -- so it's roughly as slow as on Windows. I haven't worked in Python in a couple years, but handling concurrency while supporting the major OSes was a goddamn mess and a half. [0]: https://docs.python.org/3.12/library/gc.html#gc.freeze https://docs.python.org/3.12/library/gc.html#gc.freeze [1]: https://bugs.python.org/issue33725 https://bugs.python.org/issue33725 [2]: https://docs.python.org/3.12/library/multiprocessing.html#contexts-and-start-methods https://docs.python.org/3.12/library/multiprocessing.html#co...
      - deleted 3y ago
        [deleted]
      - fulafel 3y ago
        Re (1), are there publicly documented cases with numbers on observed slowdowns with it? I see this mentioned from time to time, but intuitively you'd think this wouldn't pose a big slowdown since the system builtin objects would have been allocated at the same time (startup) and densely located on smaller nr of pages. I guess if you have a lot of global state in your app it could be more significant. Would also be interesting to see a benchmark using hugepages, you'd think this could solve remaining perf problems if they were due to large number of independent CoW page faults.
        fulafel 3y ago
        Replying to my self: it seems one poster case was Instagram and their very large Django app: https://bugs.python.org/issue40255#msg366835 https://bugs.python.org/issue40255#msg366835
  - rmbyrro 3y ago
    Probably a very small minority of Python codebases run on Windows, no? That's my impression. It would explain why so many people are unaware of multiprocessing issues on Windows. I've never ran any serious Python code on windows...
kroolik 3y ago
Managing processes is more annoying than threads, though. Incl. data passing and so forth.
- pillusmany 3y ago
  The "ray" library makes running python code on multi core and clusters very easy.
  - smcl 3y ago
    Interesting - looking at their homepage they seem to lean heavily into the idea that it's for optimising AI/ML work, not multi-process generally.
    - pillusmany 3y ago
      You can use just ray.core to do multi process. You can do whatever you want in the workers, I parse JSONs and write to sqlite files.
  - kroolik 3y ago
    Although its great the library helps with multicore Python, the existence of such package shouldnt be an excuse not to improve the state of things in std python
vita7777777 3y ago
On the other hand, this particular argument also gets overused. Not all compute-bounded parallel workloads are easily solved by dropping into multiprocessing. When you need to share non-trivial data structures between the processes you may quickly run into un/marshalling issues and inefficiency.
ynik 3y ago
multiprocessing only works fine when you're working on problems that don't require 10+ GB of memory per process. Once you have significant memory usage, you really need to find a way to share that memory across multiple CPU cores. For non-trivial data structures partly implemented in C++ (as optimization, because pure python would be too slow), that means messing with allocators and shared memory. Such GIL-workarounds have easily cost our company several man-years of engineer time, and we still have a bunch of embarrassingly parallel stuff that we still cannot parallelize due to GIL and not yet supporting shared memory allocation for that stuff. Once the Python ecosystem supports either subinterpreters or nogil, we'll happily migrate to those and get rid of our hacky interprocess code. Subinterpreters with independent GILs, released with 3.12, theoretically solve our problems but practically are not yet usable, as none of Cython/pybind11/nanobind support them yet. In comparison, nogil feels like it'll be easier to support.
- pillusmany 3y ago
  "Ray" can share python objects memory between processes. It's also much easier to use than multi processing.
  - ptx 3y ago
    How does that work? I'm not familiar with Ray, but I'm assuming you might be referring to actors [1]? Isn't that basically the same idea as multiprocessing's Managers [2], which also allow client processes to manipulate a remote object through message-passing? (See also DCOM.) [1] https://docs.ray.io/en/latest/ray-core/walkthrough.html#calling-an-actor https://docs.ray.io/en/latest/ray-core/walkthrough.html#call... [2] https://docs.python.org/3/library/multiprocessing.html#managers https://docs.python.org/3/library/multiprocessing.html#manag...
    - pillusmany 3y ago
      Shared memory: https://docs.ray.io/en/latest/ray-core/objects.html https://docs.ray.io/en/latest/ray-core/objects.html
jcranmer 3y ago
> as the `multiprocessing` module works just fine. Something that tripped me up when I last did `multiprocessing` was that communication between the processes requires marshaling all the data into a binary format to be unmarshaled on the other side; if you're dealing with 100s of MB of data or more, that can be quite some significant expense.

liuliu 3y ago

`multiprocessing` works fine for serving HTTP requests or do some other subset of embarrassingly-parallel problems.

skrause 3y ago
> `multiprocessing` works fine for serving HTTP requests Not if you use Windows, then it's a mess. I have a suspicion that people who say that the multiprocessing works just fine never had to seriously use Python on Windows.
- ptx 3y ago
  Why is it a mess? What's wrong with it on Windows?
  - skrause 3y ago
    * A lack of fork() makes starting new processes slow. * All Python webservers that somewhat support multiprocessing on Windows disable the IOCP asyncio event loop when using more than one process (because it breaks in random ways), so you're left with the slower select() event loop which doesn't support more than 512 connections.
  - colatkinson 3y ago
    Adding on to the other comment, multiprocessing is also kinda broken on Linux/Mac. 1. Because global objects are refcounted, CoW effectively isn't a thing on Linux. They did add a way to avoid this [0], but you have to manually call it once your main imports are done. 2. On Mac, turns out a lot of the system libs aren't actually fork-safe [1]. Since these get imported inadvertently all the time, Python on Mac actually uses `spawn` [2] -- so it's roughly as slow as on Windows. I haven't worked in Python in a couple years, but handling concurrency while supporting the major OSes was a goddamn mess and a half. [0]: https://docs.python.org/3.12/library/gc.html#gc.freeze https://docs.python.org/3.12/library/gc.html#gc.freeze [1]: https://bugs.python.org/issue33725 https://bugs.python.org/issue33725 [2]: https://docs.python.org/3.12/library/multiprocessing.html#contexts-and-start-methods https://docs.python.org/3.12/library/multiprocessing.html#co...
    - deleted 3y ago
      [deleted]
    - fulafel 3y ago
      Re (1), are there publicly documented cases with numbers on observed slowdowns with it? I see this mentioned from time to time, but intuitively you'd think this wouldn't pose a big slowdown since the system builtin objects would have been allocated at the same time (startup) and densely located on smaller nr of pages. I guess if you have a lot of global state in your app it could be more significant. Would also be interesting to see a benchmark using hugepages, you'd think this could solve remaining perf problems if they were due to large number of independent CoW page faults.
      - fulafel 3y ago
        Replying to my self: it seems one poster case was Instagram and their very large Django app: https://bugs.python.org/issue40255#msg366835 https://bugs.python.org/issue40255#msg366835
- rmbyrro 3y ago
  Probably a very small minority of Python codebases run on Windows, no? That's my impression. It would explain why so many people are unaware of multiprocessing issues on Windows. I've never ran any serious Python code on windows...

kroolik 3y ago

Managing processes is more annoying than threads, though. Incl. data passing and so forth.

pillusmany 3y ago
The "ray" library makes running python code on multi core and clusters very easy.
- smcl 3y ago
  Interesting - looking at their homepage they seem to lean heavily into the idea that it's for optimising AI/ML work, not multi-process generally.
  - pillusmany 3y ago
    You can use just ray.core to do multi process. You can do whatever you want in the workers, I parse JSONs and write to sqlite files.
- kroolik 3y ago
  Although its great the library helps with multicore Python, the existence of such package shouldnt be an excuse not to improve the state of things in std python

vita7777777 3y ago

On the other hand, this particular argument also gets overused. Not all compute-bounded parallel workloads are easily solved by dropping into multiprocessing. When you need to share non-trivial data structures between the processes you may quickly run into un/marshalling issues and inefficiency.

ynik 3y ago

multiprocessing only works fine when you're working on problems that don't require 10+ GB of memory per process. Once you have significant memory usage, you really need to find a way to share that memory across multiple CPU cores. For non-trivial data structures partly implemented in C++ (as optimization, because pure python would be too slow), that means messing with allocators and shared memory. Such GIL-workarounds have easily cost our company several man-years of engineer time, and we still have a bunch of embarrassingly parallel stuff that we still cannot parallelize due to GIL and not yet supporting shared memory allocation for that stuff. Once the Python ecosystem supports either subinterpreters or nogil, we'll happily migrate to those and get rid of our hacky interprocess code. Subinterpreters with independent GILs, released with 3.12, theoretically solve our problems but practically are not yet usable, as none of Cython/pybind11/nanobind support them yet. In comparison, nogil feels like it'll be easier to support.

pillusmany 3y ago
"Ray" can share python objects memory between processes. It's also much easier to use than multi processing.
- ptx 3y ago
  How does that work? I'm not familiar with Ray, but I'm assuming you might be referring to actors [1]? Isn't that basically the same idea as multiprocessing's Managers [2], which also allow client processes to manipulate a remote object through message-passing? (See also DCOM.) [1] https://docs.ray.io/en/latest/ray-core/walkthrough.html#calling-an-actor https://docs.ray.io/en/latest/ray-core/walkthrough.html#call... [2] https://docs.python.org/3/library/multiprocessing.html#managers https://docs.python.org/3/library/multiprocessing.html#manag...
  - pillusmany 3y ago
    Shared memory: https://docs.ray.io/en/latest/ray-core/objects.html https://docs.ray.io/en/latest/ray-core/objects.html

jcranmer 3y ago

> as the `multiprocessing` module works just fine. Something that tripped me up when I last did `multiprocessing` was that communication between the processes requires marshaling all the data into a binary format to be unmarshaled on the other side; if you're dealing with 100s of MB of data or more, that can be quite some significant expense.