5 ms·

If you have a non trivial application, multiprocessing just takes a lot of memory. Every child process that you create duplicates the parent memory. There are s

If you have a non trivial application, multiprocessing just takes a lot of memory. Every child process that you create duplicates the parent memory. There are some interesting hacks like gc.freeze that exploits the copy on write feature of forks to reduce memory, but ultimately you can just create a few hundred of processes compared to thousands of threads because of memory consumption.

coldtea 3y ago
>If you have a non trivial application, multiprocessing just takes a lot of memory. Every child process that you create duplicates the parent memory. Not really, unless you want to alter it. The OS uses copy on write behind the scenes for forked processes, so will use the same memory locations already loaded until/if you modify that. So parent memory isn't really duplicated. As for any new memory allocated by each child process, that's its own.
- sgeisenh 3y ago
  Unfortunately the generational GC modifies bits all over the heap, so you have to use some tricks to really leverage copy on write (as the commenter alludes to).
- whywhywhydude 3y ago
  Unfortunately, python garbage collector messes up copy on write. Here’s a blog from instagram on how they fixed it - https://instagram-engineering.com/copy-on-write-friendly-python-garbage-collection-ad6ed5233ddf https://instagram-engineering.com/copy-on-write-friendly-pyt...
- Kranar 3y ago
  Fork's copy on write does not mix well with garbage collection.
- zbentley 3y ago
  The situation is a bit more complicated than this. While it's usually not the case that child processes always duplicate parent memory, that does happen on certain platforms (MacOS and Windows) on some Pythons. Additionally, the situation regarding unexpected page dirtying of copy-on-write memory is nuanced as well, which some of the sibling comments allude to. I'll copy the tl;dr from another comment I've made nearby: There are three main ways to share data into your multiprocessing processes: 1. By sending that data to them with IPC/pickling/copying, e.g. via "multiprocessing.Pool.apply_async()", "multiprocessing.Queue.put()" or "concurrent.futures.ProcessPoolExecutor.submit()". 2. Copy-on-write via fork(2). In this mode, globally-visible data structures in Python that were created before your Pool/ProcessPoolExecutor are made accessible to code in child processes for (nearly) free, with no pickling, and no copying unless they are mutated in the child process. Two caveats here, which I've discussed in other comments on this thread: mutation may occur via garbage collection even if you don't explicitly change fork-shared data in Python[1]; and fork(2) is not used by default in multiprocessing on MacOS or Windows[2]. 3. Using explicit shared memory data structures provided by Multiprocessing[3][4]. 1. https://news.ycombinator.com/item?id=36940118 https://news.ycombinator.com/item?id=36940118 2. https://news.ycombinator.com/item?id=36941791 https://news.ycombinator.com/item?id=36941791 3. https://docs.python.org/3/library/multiprocessing.html#shared-ctypes-objects https://docs.python.org/3/library/multiprocessing.html#share... 4. https://docs.python.org/3/library/multiprocessing.shared_memory.html https://docs.python.org/3/library/multiprocessing.shared_mem...

coldtea 3y ago

>If you have a non trivial application, multiprocessing just takes a lot of memory. Every child process that you create duplicates the parent memory. Not really, unless you want to alter it. The OS uses copy on write behind the scenes for forked processes, so will use the same memory locations already loaded until/if you modify that. So parent memory isn't really duplicated. As for any new memory allocated by each child process, that's its own.

sgeisenh 3y ago
Unfortunately the generational GC modifies bits all over the heap, so you have to use some tricks to really leverage copy on write (as the commenter alludes to).
whywhywhydude 3y ago
Unfortunately, python garbage collector messes up copy on write. Here’s a blog from instagram on how they fixed it - https://instagram-engineering.com/copy-on-write-friendly-python-garbage-collection-ad6ed5233ddf https://instagram-engineering.com/copy-on-write-friendly-pyt...
Kranar 3y ago
Fork's copy on write does not mix well with garbage collection.
zbentley 3y ago
The situation is a bit more complicated than this. While it's usually not the case that child processes always duplicate parent memory, that does happen on certain platforms (MacOS and Windows) on some Pythons. Additionally, the situation regarding unexpected page dirtying of copy-on-write memory is nuanced as well, which some of the sibling comments allude to. I'll copy the tl;dr from another comment I've made nearby: There are three main ways to share data into your multiprocessing processes: 1. By sending that data to them with IPC/pickling/copying, e.g. via "multiprocessing.Pool.apply_async()", "multiprocessing.Queue.put()" or "concurrent.futures.ProcessPoolExecutor.submit()". 2. Copy-on-write via fork(2). In this mode, globally-visible data structures in Python that were created before your Pool/ProcessPoolExecutor are made accessible to code in child processes for (nearly) free, with no pickling, and no copying unless they are mutated in the child process. Two caveats here, which I've discussed in other comments on this thread: mutation may occur via garbage collection even if you don't explicitly change fork-shared data in Python[1]; and fork(2) is not used by default in multiprocessing on MacOS or Windows[2]. 3. Using explicit shared memory data structures provided by Multiprocessing[3][4]. 1. https://news.ycombinator.com/item?id=36940118 https://news.ycombinator.com/item?id=36940118 2. https://news.ycombinator.com/item?id=36941791 https://news.ycombinator.com/item?id=36941791 3. https://docs.python.org/3/library/multiprocessing.html#shared-ctypes-objects https://docs.python.org/3/library/multiprocessing.html#share... 4. https://docs.python.org/3/library/multiprocessing.shared_memory.html https://docs.python.org/3/library/multiprocessing.shared_mem...