9 ms·
Reverting the incremental GC in Python 3.14 and 3.15
- davidkwast 4mo ago"Python 3.14 shipped with a new incremental garbage collector. However, we’ve had a number of reports of significant memory pressure in production environments. We’ve decided to revert it in both 3.14 and 3.15, and go back to the generational GC from 3.13." Sounds the right move for me
- winrid 4mo agoThe main benefit of python to me is that while slow, it's predictable. I do think they're going to get a lot more resistance to adding JITs, moving GCs, etc. it will become java with a million knobs to tune. If people want a JIT'd python just use pypy, right?
- sigmoid10 4mo agoAnd if people want python with java, there's always Jython.
- brokensegue 4mo agojython has been basically unmaintained for quite some time
- sigmoid10 4mo agoWell, they never made the jump to Python 3. But shipping 2.7 interpreters in 2024 was quite an achievement on its own. So their users already know this pain. And from my experience in academia, python 2.7 and java 8 will probably be used for another 20 years before the last machine running that stuff burns out.
- arikrahman 4mo agoJython is unmaintained, I'd recommend Clojure. Use python libraries and code while seamlessly targeting the JVM.
- brian_herman 4mo agoGraal vm has support for python 3 unfortunately it’s funded by oracle.
- wavemode 4mo agoIf it makes you feel any better (it probably doesn't), the development of OpenJDK and the Java language itself is also mostly funded by Oracle
- pjmlp 4mo agoJava is funded by Oracle, all of it. People parrot to use OpenJDK without understanding it is mostly Oracle employees working on it. And if you dislike Oracle, the other minor contributors are Red-Hat, IBM, SAP, Microsoft, Alibaba, Azul,... which for many HNers are the same.
- froh 4mo agojpype and graalpy are life. jython went EOL.with python 2 going EOL.
- stackskipton 4mo agoAs Python using SRE and supporting Python Flask apps, most of us would love JIT in Python assuming it pretty much drop in replacement. PyPy doesn't have the support it needs and is stuck on 3.11.
- graemep 4mo agoResistance from anyone who matters to the developers?
- pron 4mo agoJava lost almost all those knobs a while ago (I mean they're there, but you're better off relying on the defaults). The modern GCs have one or at most two knobs remaining, and even that will become unnecessary next year. As to predictablity, you get maximal pause time of well under 1ms for heaps up to 16TB.
- JackSlateur 4mo agoAs far as I know, java has 7 GC implementations, none of which are perfect, all of which have drawbacks Lately, they seems to work with CRIU, various heuristics, multi-stage in-process bytecode compilation .. Java is a mess, they are working hard to avoid fixing their issue (that nobody else have, so fixes are available)
- coldtea 4mo ago>As far as I know, java has 7 GC implementations, none of which are perfect, all of which have drawbacks Compared to Python's, all of them are beyond perfect. And 99.9% of the time you don't even need to use anything but the default.
- refulgentis 4mo ago> Compared to Python's, all of them are beyond perfect. I somehow understand the situation less after reading this. Is Python's GC bad, or are there cyclic reference issues? Is it possible to detect cyclic references perfectly? What does beyond perfect mean? If we have 7 and 0.1% of the time you need one of the 6 that is non-default, how do we choose? Is the understated version of "Compared to Python's, all of them are beyond perfect" "I think Java's are great"? If not, what about Python's impl makes it so lackluster to any of 7 of Java's?
- murderfs 4mo ago> Is Python's GC bad, or are there cyclic reference issues? Unless you're being pedantic and including reference counting without cycle detection as GC, if your GC has cyclic reference issues, your GC is bad. > Is it possible to detect cyclic references perfectly? Yes? That's the entire point of tracing GC. You have some set of root objects that you start with (globals, objects on thread stacks, etc.) and then you mark every object that's reachable from them. Anything that's not reachable is garbage, even if there are cycles within them.
- davidkwast 4mo agoIt is the same for me. Predicability is better than any optimization.
- zozbot234 4mo agoWhy not just use Go? It has a proper concurrent, non-moving GC that, AIUI, has not been associated with sudden memory spikes.
- brokencode 4mo agoFor a new project, teams can decide whether to use Go, but there are many millions of lines of existing Python servers out there. Not to mention that there are differences in ecosystem, familiarity, and ergonomics that may make a team want to stick with Python. “Just use Go” is not really actionable advice in most cases.
- LtWorf 4mo agoIt's a tradeoff. Go programs are extremely slow at starting up for example.
- bmitc 4mo agoThat doesn't matter for anything other then CLIs.
- AlecSchueler 4mo agoSome people are writing CLIs
- bmitc 4mo agoYes, of course, but I took the conversation to be centered around backend uses cases. What CLI is experiencing garbage collection issues like that in this discussion?
- CamouflagedKiwi 4mo agoWhat? Compared to Python they're like lightning. Typically milliseconds to the start of main() - admittedly they can be slowed down by init() nonsense and terrible generated protobuf code nonsense in deep dependency trees - but with a non-trivial Python program you can look forward to an order of magnitude more. There are techniques to help address that but (1) they're not idiomatic and (2) it still only mitigates it. I suppose Go programs are slower than the equivalent thing in C or C++, but I'm not sure that's a very relevant comparison in most cases today (how many new things being written would choose those languages).
- bmitc 4mo agoIn what way do you feel Python is predictable, especially in comparison to other languages one would build a backend system in? It's predictable vs Rust, C#, F#, Elixir, Go, etc.?
- CamouflagedKiwi 4mo agoPyPy is not looking healthy right now - it's several versions behind in support and, while it's not dead, it looks like it might be settling down for a rest. Obviously it's not easy to move the whole language of a big codebase, but I feel a lot of this stuff (fiddling with GC, JITing, type hints, and I'm dubious about the free-threading stuff) tries to take Python somewhere it isn't really good at, and if that's what you want, you really want a different language.
- bmitc 4mo agoWhy are people still building systems on top of a language that continually undergoes fundamental changes nearly 40 years after release? Is this not the strongest indication that this language is not well designed, it is unstable, and encounters many issues that flat out don't exist in other high level languages?
- mirashii 4mo agoWhat language that is actually used 40 years after release isn't undergoing big, fundamental changes? Java? Nope, you're getting a fundamental change in Valhalla C++? Nope, new language edition every few years with fundamental changes C? C23 has a number of fairly fundamental changes, expect more in the next language revision I think your sense of causality is backwards here. These languages are getting fundamental changes because they're being widely used. That is what motivates and drives the change. Languages with no users don't need to change.
- adrian_b 4mo agoAs you say, any widely used language gets fundamental changes from time to time. But most such languages handle much better the compatibility with legacy applications. Python is the main culprit in most cases when I see conflicts between various software packages that insist to use only a specific version of their dependencies. This is why I have to keep installed many versions of Python, and the Linux distribution that I use must take care to prevent interference between those Python versions.
- bmitc 4mo ago> Languages with no users don't need to change. That's fine, but that's clearly not what I'm talking about. Languages like F#, Elixir, etc. don't undergo fundamental changes. Yes, every language evolves. But for Python, we're talking about grafting literally fundamental stuff on top of a language not designed for any of these things. For example, if someone went and redesigned Python to solve its warts, you'd basically end up with F#.
- NooneAtAll3 4mo agoI'm genuinely surprised that python change was even possible without PEP
- metalliqaz 4mo agoIt's not a change to the language, it's a change to the cpython runtime
- ammar2 4mo agoPEPs aren't necessarily just for language changes, e.g https://peps.python.org/pep-0436/ https://peps.python.org/pep-0436/ which is largely a CPython implementation detail.
- NooneAtAll3 4mo agoproblem is that Python is so centralized, CPython is essentially a "reference implementation"
- Fizzadar 4mo agoExactly! Would like to understand more how that came about. PEP exists for a reason.
- AdamN 4mo agoYeah it seems like a miss. I guess the thinking was that it wasn't developer-facing and just an internal optimization. But of course any change to garbage collection will change the memory and cpu dynamics of the process in a material way.
- giancarlostoro 4mo agoMakes ya miss having a BDFL. Dang I didn't realize he's 70 now. https://en.wikipedia.org/wiki/Guido_van_Rossum https://en.wikipedia.org/wiki/Guido_van_Rossum
- zitterbewegung 4mo agoI wouldn’t recommend running the latest Python in prod. Honestly 3.x.7 releases are the most mature .
- askllk 4mo agoAll these issues were known in previous attempts for removing the GIL. But if Instagram/Meta want it, everyone stands to attention and finds out the obvious problems years later. Kind of like in geopolitics. I hope Meta switches Instagram to PHP/Hack so they leave Python alone.
- simonw 4mo agoThe no-GIL work (free-threading) is unrelated to this incremental GC work. Free-threading actually uses its own, separate GC: https://labs.quansight.org/blog/free-threaded-gc-3-14 https://labs.quansight.org/blog/free-threaded-gc-3-14
- brianwawok 4mo agoIn the world of AI written code, Python just doesn’t make sense. Converted about 100k lines in the last few months to golang and the performance is life changing. Curious if we will see global Python adoption fall by 75% or more in the next few years.
- backwardation_b 4mo agonothing about the performance characteristics of python changed with AI so why would you use python over golang if performance is a requirement/bottleneck? Trying to understand the reasoning as to me golang and python are equally simple to write and understand.
- phainopepla2 4mo agoRegardless of whether golang and python are actually equally simple, python certainly has the reputation of being easier to write and read than almost any other language. That is a big part of its popularity.
- Yokohiii 4mo agoIf language X is a persons comfort zone, that person will often default to it. Python is certainly more widespread then go. Also, even if it looks like that to you, there are still people that write code with their own hands.
- 4mo ago
- bhouston 4mo ago.NET seems to have regularly changed the garbage collector over the years and I do not remember any similar surprises in production. I wonder why they have had better experience? I thought that by now dynamic garbage collection was a known quantity so that making changes, outside of out right bugs, is fairly safe and predictable?
- stackskipton 4mo agoOne thing Microsoft does really well is eating its own dogfood and Microsoft feeds a ton of .Net dogs. So any change to GC starts with massive .Net MSFT code base so they get extremely good telemetry back about any downsides and might be able to fix it in time.
- pjmlp 4mo agoDid really well, unfortunately. There is almost no dog fooding on Windows development since version 8, Typescript team rather rewrite the compiler in Go, Azure has plenty of Go, Rust and Java projects alongside .NET.
- stackskipton 4mo agoMicrosoft does use Go/Rest/Java in places but they still have a ton of .Net. Windows Development is not "We are not dogfooding", it's that incentives are misaligned with customer wants. .Net team incentives are aligned with customer wants, provide a language that is highly performant and easy enough to write.
- pjmlp 4mo agoOh, they really don't dogfood Windows development any longer, regardless of the incentives. I have my WinRT 8, UAP 8.1, UWP 10, Project Reunion, .NET Native, C++/CX, C++/WinRT, XAML Islands, XAML Direct, WinUI 2.0, WinUi 3.0, WinAppSDK and what not scars to prove how they aren't dog fooding any piece of it in any meaningful manner. Heck they keep talking about C++ support in WinUI 3, as if the team hasn't left the project and is now playing with Rust instead. They managed that plenty of early WinRT advocates became their hardest critics, while not believing anything else they put out, like now this Windows K2 project.
- sega_sai 4mo agoI think reverting is not problem per se, but releasing a highly problematic version without proper testing in such an essential component is.
- LaFolle 4mo agoYeah they noted that it went without PEP. Looks like a PEP will come now if it maintains at par perf.
- alexzhaosheng 4mo ago[dead]
- lukassbrad 4mo ago[flagged]
- athoscouto 4mo agoWe've been impacted by this. I migrated our services to Python 3.14 so we could attach profilers during runtime. A couple of services looked like they had a memory leak. Memory was continuously increasing over time. Thanks to Python 3.14, we were able to use memray to understand what was going on. Those services were recreating HTTP clients (aiohttp) for every inbound request, and memory allocated by the downstream SSL lib was growing faster than it was being released. We ended up rolling back to 3.13, which fixed the issue. I'll try again with 3.14.5.
- LaFolle 4mo agoOn profilers - profiling will come in 3.15, are you referring to remote exec? It is a great feature I am very exited about, at the same time afraid that the company won’t allow ptrace capability in prod.
- athoscouto 4mo agoyes. remote exec allows me to attach profilers (e.g. memray) directly into a running process. i'm also excited about the upcoming statistical (cpu) profiler from 3.15
- nas 4mo agoIf you are using "httpx", it's likely caused by a reference cycle. I made a PR to fix it but the maintainers haven't applied it. :-( https://github.com/encode/httpx/pull/3733 https://github.com/encode/httpx/pull/3733 The reference cycle httpx creates is kind of a worst-case scenario for the incremental GC issue. Both the generational (3.13 and older) and incremental GC are triggered by the net new "container" objects (objects that have references to others, like lists and not like ints and floats). The short summary is that you need to create more container objects before the incremental GC triggers. In the case of the httpx reference cycle, you have a relatively small number of container objects hanging on to a lot of memory, due the SSL context data (which is a big memory hog). Reverting back to the generational GC was the wise thing to do, even though it's a bit scary to do in a bugfix release. The incremental GC works for most people but in the minority of cases it doesn't, it uses quite a lot more memory. I'm pretty sure with some additional tuning, the incremental GC would be fine too but it just didn't get that tuning. The generational GC has literal decades of real-world use (Guido merged my patch on Jun 2000, Tim Peters did a bunch of tuning after that to optimize it).
- emil-lp 4mo agoIf I understand correctly, this is one of the changes that caused the regression: https://github.com/python/cpython/pull/117120 https://github.com/python/cpython/pull/117120
- __loam 4mo agoPython is such a mess.
- nodesocket 4mo agoIf using containers I believe this change was pushed in image python:3.14.5-slim-trixie
- sathishmg 4mo ago[flagged]
- hpcgroup 4mo ago[flagged]
- insumanth 4mo agoThis is the first time I came across a change (a big one) that was implemented without passing through PEP. I thought it was standard.
- emmanuelsemugga 4mo agoThis is so salient to put under consideration deeply
- luodaint 4mo agoThe problem is that you create an HTTP client for each incoming request. In other words, you recreate SSL context and cause reference cycles with each request. From the memory point of view, the program seems to have a memory leak. However, the solution to the issue lies at the application level: just create your client once and reuse it for each subsequent request. One of heuristics to find memory leaks can be stated as follows: if you instantiate any HTTP or connection objects inside the request handler, it's likely that you've made a mistake.