5 ms·
Adding on to the other comment, multiprocessing is also kinda broken on Linux/Mac. 1. Because global objects are refcounted, CoW effectively isn't a thing on L
by colatkinson 3y ago
Adding on to the other comment, multiprocessing is also kinda broken on Linux/Mac.
1. Because global objects are refcounted, CoW effectively isn't a thing on Linux. They did add a way to avoid this [0], but you have to manually call it once your main imports are done.
2. On Mac, turns out a lot of the system libs aren't actually fork-safe [1]. Since these get imported inadvertently all the time, Python on Mac actually uses `spawn` [2] -- so it's roughly as slow as on Windows.
I haven't worked in Python in a couple years, but handling concurrency while supporting the major OSes was a goddamn mess and a half.
[0]: https://docs.python.org/3.12/library/gc.html#gc.freeze https://docs.python.org/3.12/library/gc.html#gc.freeze
[1]: https://bugs.python.org/issue33725 https://bugs.python.org/issue33725
[2]: https://docs.python.org/3.12/library/multiprocessing.html#contexts-and-start-methods https://docs.python.org/3.12/library/multiprocessing.html#co...
- deleted 3y ago[deleted]
- fulafel 3y agoRe (1), are there publicly documented cases with numbers on observed slowdowns with it? I see this mentioned from time to time, but intuitively you'd think this wouldn't pose a big slowdown since the system builtin objects would have been allocated at the same time (startup) and densely located on smaller nr of pages. I guess if you have a lot of global state in your app it could be more significant. Would also be interesting to see a benchmark using hugepages, you'd think this could solve remaining perf problems if they were due to large number of independent CoW page faults.
- fulafel 3y agoReplying to my self: it seems one poster case was Instagram and their very large Django app: https://bugs.python.org/issue40255#msg366835 https://bugs.python.org/issue40255#msg366835