12 ms·

Why Is the Migration to Python 3 Taking So Long?

202 points by josep2 7y ago

digitalsushi 7y ago
I never knew if this was a cynic's answer or truthful, but I was told by my manager at one point that RedHat's OS is superglued to python2 and spends a lot of money to keep python2 in good working order. I highly expect it's a cynic's response and please read it as such until someone in-the-know can retort my post.
- maweki 7y ago
  A lot of tooling (yum, for example) was written in python. This all takes a lot of time to port, especially gtk/gobject stuff, but nowadays this is all written for python3 or JavaScript or Vala.
- navigatr 7y ago
  Red Hat has long support contracts for their server OSes that shipped with Python 2 when it was still kosher to do so. That means they'll patch Python 2 should vulnerabilities be found on their OS.
  - ianai 7y ago
    It’s a pain from the outside looking into their business. Their contracts, though, are their business.
    - int_19h 7y ago
      Those support contracts aren't cheap, though. I don't think they're getting the short end of the stick here. If anything, the companies that run lots of Python 2 code in the house that haven't migrated yet - which is not uncommon in the enterprise - have the incentive to run that code on RedHat, if they aren't doing so already. I wouldn't even be surprised if they provide support contracts specifically for Python 2 on their newer OSes. ActiveState announced that they'll continue providing paid support for Python 2 on all platforms, so some people at least feel like there's a business model there... https://www.activestate.com/company/press/press-releases/activestate-offers-extended-support-for-python-2-beyond-eol/ https://www.activestate.com/company/press/press-releases/act...
- geofft 7y ago
  I don't think that's true, or at least it's no longer true: RHEL 8 does not ship with Python 2 by default. https://developers.redhat.com/blog/2018/11/14/python-in-rhel-8/ https://developers.redhat.com/blog/2018/11/14/python-in-rhel... Red Hat software that depends on Python, like yum, is in Python 3. I think it is true (as of pretty recently) that Red Hat is the only company employing a Python core dev to work on Python core dev stuff full time (see https://discuss.python.org/t/official-list-of-core-developers/924/2 https://discuss.python.org/t/official-list-of-core-developer...). But the core dev team is focused on Python 3, so that isn't a sign of Red Hat's Python 2 commitment either.
- hathawsh 7y ago
  See the RHEL release schedule: https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux#Version_history_and_timeline https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux#Versi... RHEL 6/7 and Centos 6/7 will support Python 2 until at least mid-2024.
- mkesper 7y ago
  There was no way updating the system Python in Rh/CentOS6 from 2.6 to 2.7 as that broke system scripts. You could only use 2.7 in non-standard paths (or Python 3).
SQueeeeeL 7y ago
Python, unlike a C executable, could hypothetically stop working tomorrow. Researchers who have code that works/businesses don't give a shit about the esoteric differences between py2 and py3, they just want their stuff to keep working. This is similar to the banks still running Cobal backends, I don't know why everyone cares, I guarantee multiple servers out there are running executables that couldn't be rebuilt, but those don't curse you out on terminal every time I turn one on
- sansnomme 7y ago
  Because software that isn't protected by walls of legislation value features and the end user UX. i.e. "evergreen". Your Nokia N900 running Meego may be sufficient for certain subsets of HN readers but your average user wants Material Design and cannot care less about ideological purity or "if it ain't broke don't fix it".
  - skykooler 7y ago
    Meego was never stable on the N900. Most people who are still using one are running Maemo.
  - bityard 7y ago
    > but your average user wants Material Design No, users most definitely do not care about Material Design. They only care about being able to quickly do the task the app or web site claims to allow them to do.
  - FpUser 7y ago
    "average user wants Material Design" - is that a fact or just a wishful thinking from people trying to push said design? EDIT: Personally for me while I find for example older windows interfaces ugly they were very consistent and functional. In modern designs I sometimes could hardly find what is clickable/actionable. It is not interface working for me but the other way around
- weff_ 7y ago
  At the end of the day, it boils down to dependency management, wouldn't you say? That is, Python2 could be running forever if the environment is maintained. In the same vein, a messed-up OS or libs can stop a C app in its tracks.
paulie_a 7y ago
Simple answer. Updating legacy code. If it works now why upgrade? Stability is more important than some new features of the language. Granted that has its downfalls via security issues but stability wins If you have a hole it's hard to dig yourself out of it. This is why I prefer modular apps instead of monolithic codebases. You can upgrade piece by piece. Otherwise it's all or nothing and dangerous
downerending 7y ago
Because there's been not enough carrot and too much stick. The only real killer feature of Python3 is the async programming model. Unfortunately, the standard library version is numbingly complex. (Curio is far easier to follow, but doesn't appear to have a future.) On the down side, switching to Unicode strings is a major hurdle. It mostly "just works", but when it doesn't, it can be difficult to see what's going on. Probably most programmers don't really understand all of the ins and outs. And on top of that, you get weird bugs like this one, which apparently is simply never going to be fixed. https://github.com/pallets/click/issues/1212 https://github.com/pallets/click/issues/1212
- semiotagonal 7y ago
  I wonder how many people feel that migrating to Python3 would have been worth doing in the absence of being forced to do so. Dropbox invested three years of work, actually hired Python's creator, and are still not done. What are they getting out of it that they wouldn't have gotten if Python2 simply had been maintained?
  - crustacean 7y ago
    This is such a good point. Take SQL. It has survived because it is well designed, and changes so little and so slowly, and it’s obvious what SQL is and isn’t meant for. Amateur programmers can port SQL queries between database systems semi-painlessly. In the right environment a query can survive with small edits for YEARS. Who wants to break old SQL? Nobody.
    - skybrian 7y ago
      That's something of an illusion. The flaws in SQL often aren't noticed because you usually pick one database vendor and stick with them. There are plenty of differences between vendors, but you don't usually have to support them at the same time, and you don't notice for simple cases. But changing database vendors for a company can be a big deal, as bad as going from Python 2 to 3.
pmoriarty 7y ago
Another data point: According to my highly unscientific survey of the packages in Gentoo's package repo, there are roughly: - 2500 packages that work with Python 2 or 3 - 1350 packages that work with Python 2 only - 350 that work with Python 3 only My methodology: http://dpaste.com/1M0TCV7 http://dpaste.com/1M0TCV7
- roland35 7y ago
  That is interesting data. I would imagine the most widely used packages are compatible with both, but that data is probably harder to get!
- stevesimmons 7y ago
  I bet many of the Py2-only ones are old legacy packages that have been superceded by newer better options.
- girst 7y ago
  and yet another (fedora's): of 3414 packages total: - 3122 Python 3 only - 88 Dual support - 8 Py2 leaf (standalone packages; may be dropped) - 77 Not ported (will be dropped unless ported) - 100 Blocked (require 1 or more "not ported" packages) - 18 Legacy (will be dropped) https://fedora.portingdb.xyz https://fedora.portingdb.xyz note that py3only/dualsupport only reflects how it is packaged in fedora, not what upstream provides.
m4r35n357 7y ago
It _is_ happening, but moaning about it won't make it happen any more quickly.
- eej71 7y ago
  I think the intent is less moaning about it - and more learning about the blocking points so they can be better surmounted. At the very least, perhaps the blocking points can be avoided in the future. Those who do not learn from history... etc etc.
alexhutcheson 7y ago
[copying comment from an older HN thread, not speaking on behalf of any employer, opinions my own] I think many people underestimate the challenge that the 2 to 3 migration presents for large enterprises. The core issue is that even though the migration for any given module is normally really easy, the total effort required to migrate is still essentially O(n) in module count/file count, because even with current tooling you still need to have an engineer look at every module to do the change safely. Even if it only takes ~5 minutes per module to make the changes and validate that it works correctly, this becomes a giant undertaking when you have tens of thousands of files to migrate. The fact that it takes a long time also creates other problems. Your business isn't going to hit "pause" on other development, so there will be changes constantly introduced into modules you've already "swept". It's going to be hard to make sure 100% of your engineers and code reviewers are knowledgeable about the specific requirements to make sure the code works in both 2 and 3, so you would really like some automated safeguards to make sure they don't introduce anything that won't work in 3. Pylint helps with this, but won't catch everything. Unit tests are obviously essential, but: 1. Even a well-tested project won't have tests that cover 100% of code paths and behavior. 2. You're stuck running the tests on both python2 and python3 for the duration of the migration, which doubles the resource (compute, memory, etc.) cost of your Python CI and regression testing infrastructure for the duration of the migration. Most big companies have passionate Python advocates who really want to be on Python 3, but the scale of the problem and the lack of tooling to tackle it with a sub-O(n) amount of effort make the overall project risky and expensive for the business.
- war1025 7y ago
  As a member of a team hopefully nearing the end of the python3 migration, this is exactly correct. The unicode switch is a nightmare in terms of having to go through and double/triple check everything and still get it wrong half the time. Particularly when it comes to moving data over the network. The big selling point for Python3 finally came with the built-in async support, but we've been using Twisted for a decade, which works nearly identically, so even that wasn't a huge draw for us. Further, many of our dependencies were python2-only up until the last year or two. Really the only reason we're going through the effort right now is that Python2 is rapidly approaching End of life.
lasermike026 7y ago
Management wants new features not porting. They will only port when they absolutely have to.
- war1025 7y ago
  Even if management wanted porting, the python 2 -> 3 migration path is very painful in the details, while on the surface not having a lot to offer at the other end in terms of new capability.
  - lasermike026 7y ago
    Roger that.
- compiler-guy 7y ago
  As well they should. No one uses Dropbox today that didn't a couple of years ago because it is using Python 3 instead of Python 2. The migration is financially negative in the short term, and very clearly so. It might be financially positive over the long term (due to easier maintenance and higher performance), but that is definitely maybe. Especially for an app that is otherwise very stable.
MaulingMonkey 7y ago
We're still waiting for Maya to switch. It's still using Python 2, and Autodesk keeps putting off the transition, partially because it'll break the scripts of all their downstream users.
- pfranz 7y ago
  I think that's partially correct. If you follow https://vfxplatform.com/ https://vfxplatform.com/ it says Python3 was pushed from 2019 to 2020 to allow everyone to upgrade Qt and PySide (which are prerequisites). The reason it was deprioritized in previous years were similar transitions like C++14, gcc, etc. I've been meaning to dig into Maya, Houdini, Nuke's Python 3 transition plans. I know Houdini will offer a Python 3 option with Houdini 18 (shipping in the next month or so). I don't think the reason was because of downstream users. Python 3 was an inevitable change. Previously, they swapped out PyQt for PySide which wasn't a forced change, but required everyone to update their Python scripts.
upofadown 7y ago
Py3 is for all practical purposes a language fork of Py2. So it doesn't really make sense to talk of a "migration". If Py2 becomes unworkable somehow then people will rewrite stuff. Some of it might even be in Py3. Considering all the stuff that is written in Py2 I really don't see it being out and out abandoned. That wouldn't really make any sense. With computer languages stuff never goes away.
- carapace 7y ago
  Py2 is the new Fortran, I like to say.
  - sigjuice 7y ago
    And they both use actual Fortran :) $ python3 Python 3.7.4 (default, Sep 7 2019, 18:27:02) [Clang 10.0.1 (clang-1001.0.46.4)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> [2]+ Stopped python3 $ lsof -c Python | sed -n "/fortran/s/$USER/<redacted>/gp" Python 35190 <redacted> txt REG 1,4 1550456 12887664541 /Users/<redacted>/Library/Python/3.7/lib/python/site-packages/numpy/.dylibs/libgfortran.3.dylib
  - int_19h 7y ago
    Py2 is the new Fortran 77. It was good in its time, and great things were done in it that are still around... but let's move onto F90 already.
m45t3r 7y ago
Any non-trivial piece of code will probably be a pain in the ass to port for a more recent version of the language, and the level of PITA depends on the size of the project X size of the changes in the language. Case in point, I worked in a project using Ruby. When we migrated from Ruby 2.4.0 to 2.4.6 (yeah, a minor upgrade), it broke spectacularly. Trying multiple Ruby versions, the change was actually introduced in Ruby 2.4.1. After some investigation, a change in Net::HTTP library from stdlib had a change that broke a dependency from a dependency. The fix was just a line of code (we just need to change the adapter used for HTTP communication), however it was two days of work for a minor upgrade. My current job tried to migrate from Java 8 to Java 11. It also broke multiple services. This one is still in progress, months later. Python 2 to Python 3 is bigger than both of those version changes (however it is equivalent to Ruby 1.8 to 1.9 changes), so yeah, it does take more time. And like some projects that are forever running Ruby 1.8 or Java 8 (or even worse, Java 6), we will have projects forever running Python 2 too.
linsomniac 7y ago
What do you mean taking so long? The original target date, proposed by Guido, was Jan 1, 3000. Looks like we're 980 years ahead of schedule! :-) I say this in part because comedy, but also because it was anticipated to be a long project. It was originally called "Python 3000".
Animats 7y ago
First off. Python 2.6 and 2.7 supported Unicode just fine. I had a large all-Unicode system in Python 2.6. You had to write u'word" to get a Unicode constant, and use a "unicode(s)" function here and there. Also, the part that remained "compatible" was that "str" remained an array of bytes, even though there was also a type "bytes" and a "bytearray". Early Python 3 was hell for conversion. The syntax was changed for no good reason. u'word" became illegal. (That later went back in.) The "2 to 3 converter" was a joke. I didn't have the "print statement problem" because my code called a logging function for all debug output. Many of the P3 libraries didn't work. (The all-Python MySQL connector failed the first time I tried to do a bulk load bigger than a megabyte, indicating that nobody was using it.) It took years before the libraries were cleaned up. Python 3 got some really weird features, such as type declarations that don't do anything. I can see having type declarations, especially for parameters, but they need to be used both for checking and optimization. CPython boxes everything, which is terrible for numerics and is why most serious math has to be done in C libraries. My comment on that was "Stop him before he kills again."
- wnissen 7y ago
  They still did stuff that makes no sense, to me, like not supporting `iteritems` on dictionaries and `xrange` on sequences. Or using `str` to mean bytes? That would have been possible to make backwards compatible until Python 4. At this point, does anyone think there will be a Python 4?
  - kevin_thibedeau 7y ago
    You're expected to clean that stuff up with 2to3. The whole idea is to not leave band-aids in place.
- viraptor 7y ago
  > Python 2.6 and 2.7 supported Unicode just fine It did, but in a way that chainsaws support sculpting just fine. Technically possible. Very advanced people will know how to handle it. Everybody else is just going to injure themselves randomly. Most people writing py2 got the text/binary processing working on accident. Things appear to work until you throw actual Unicode into parameters and then nobody knows what happens. There's a number of "what does this decoding exception mean" questions on stackoverflow every day. They're often actual bugs people could ignore before. Now they're told immediately and I believe that's better.
goatinaboat 7y ago
The 2to3 tool should add .decode(‘utf-8’) to every string manipulation, even better Python 3 should have a flag to make that the behaviour and even better that should default to on. So much effort wasted doing this in a large codebase. And what do you get for it? It’s just not worth it. Nobody actually needs Python 3, it was foisted on them by the developers. What everyone really wanted was Python 2.8.
- kstrauser 7y ago
  Speak for yourself. After using Python 3, I can't stand touching Python 2 codebases. Is a given str object an actual text string that's been decoded into a valid charset, or is it an array of bytes fresh from the network / database / file? Who knows, unless you trace back to its point of origin. I personally love that Python 3 says "this is text, and this is some binary data I got somewhere, and they are not the same thing".
- joshuamorton 7y ago
  Please no! Text is text. Bytes is bytes. Convert to the correct form on input boundaries, and that's it. Don't switch back and forth internally, it's overly complicated, error prone, and slower.
  - test7777 7y ago
    unix pipes (stdin, stdout) are bytes, files are bytes, filenames are bytes. yet, for some reason python3 thinks al of those are text. its not the coders that are wrong, it is the language.
    - kstrauser 7y ago
      No, it's Python 2 that thinks those are all text. Python 3 makes you explicitly say "convert this stream of bytes into a text encoded string", and from then on it's a str object. Python 2 was happy to (try to) let you call text methods on a JPEG. Python 3 draws an appropriate distinction between the two.
mbparks 7y ago
In my organization, business needs overrule technical details. Management didn't want to spend the resources until forced to do so, rather focus on revenue-generating work. Can see it both ways I guess. We have recently started upgrading our codebase to support 3.X
twblalock 7y ago
I recently went through a fairly large upgrade from JDK8 to JDK 11 and it was a bit of a pain -- lots of dependencies to update, etc. But very few code changes were required, and the static type system made it pretty clear when the codebase was broken -- it just wouldn't build. It still took my team several weeks. Migrating from Python 2 to Python 3 is way worse than that -- code changes are required, and because Python is a dynamic language you may not notice bugs until you actually run the code (or even worse, until after you release it to production and some code branch that is rarely invoked somehow gets called...). In other words, the tooling and the type system are not confidence-inspiring and it's really hard to verify that you migrated without breaking stuff.
- LIV2 7y ago
  Will this will make people less likely to use Python in future for some projects? I'm no developer or manager but I figure there'd be a thought in the back of my head thinking "what if we need to rewrite this all again for the next major release"
  - xemdetia 7y ago
    I don't think it will make people less likely to use Python overall, but for certain categories of projects there are some things that are just done better in other languages. Both Java and Go for example have specific programming language constructs and contracts to ensure that codebases can move from version to version easily and I do not feel like there is a dynamic/untyped language that really provides that same level of stability. The ones that do get close to that are the ones that are extremely small in scope and are not in the same class of batteries included language, and because of their small scope they have a general limit of change. At a certain point this sort of compatibility/forward motion of a codebase through big language revisions is something that has to be designed as part of the language in either being able to break it down into small enough chunks to chew through in pieces (updating a submodule with the updated language without affecting anything else), completely transparent to the code being run through it (this happens for compilers for C for different standards), or to have a version to version automated rewriting mechanism that is so reliable the outcome of the automated tool is not in question (tools like Go's gofmt). Python in my opinion only has partial solutions to all of those answers so it turns into a lot of hand work. So while there are other languages that may do other things better there are still a class of programs that are very effective to write in Python, and that's plenty enough reason to keep it around. Do not forget that Python 2 was released in 2000 and Python 3 was released almost a decade later. The general time scale makes worrying about the next release for many people, but for people who do they start considering other languages because that's important to them.
jdhawk 7y ago
The same reason there is still a ton of legacy PHP 5.6 code. Migration in interpreted languages that implement major breaking changes is really tedious.
recursivecaveat 7y ago
The simple reason is that there was no compelling feature to reward you for upgrading. You'd spend a tremendous amount of effort for dubious return and (until recently) a smaller ecosystem. 1. Unicode support was actually an anti-feature for most existing code. If you're writing a simple script you prefer 'garbage-in, garbage-out' unicode rather than scattering casts everywhere to watch it randomly explode when an invalid byte sneaks in. If you did have a big user-facing application that cared about unicode, then the conversion was incredibly painful for you because you were a real user of the old style. 2. Minor nice-to-haves like print-function, float division, and lazy ranges just hide landmines in the conversion while providing minimal benefit. In the latest py3 versions we've finally gotten some sugar to tempt people over: asyncio, f-strings, dataclasses, and type annotations. Still not exactly compelling, but at least something to encourage the average Joe to put in all the effort.
- takeda 7y ago
  > Unicode support was actually an anti-feature for most existing code. If you're writing a simple script you prefer 'garbage-in, garbage-out' unicode rather than scattering casts everywhere to watch it randomly explode when an invalid byte sneaks in. If you did have a big user-facing application that cared about unicode, then the conversion was incredibly painful for you because you were a real user of the old style. Actually that's the behavior of python 2, it works fine, until you send invalid characters then it blows up. In python 3 it always blows up when you mix bytes with text so you can catch the issue early on. > In the latest py3 versions we've finally gotten some sugar to tempt people over: asyncio, f-strings, dataclasses, and type annotations. Still not exactly compelling, but at least something to encourage the average Joe to put in all the effort. That's because until 2015 all python 2.7 features were from python 3. Python 2.7 was basically python 3 without the incompatible changes. After they stopped backporting features in 2015. Suddenly python 3 started looking more attractive.
  - pmontra 7y ago
    > In python 3 it always blows up when you mix bytes with text so you can catch the issue early on. This is definitely the case. I've been wrestling with bytes and strings all the time during the port of a Django application to Python 3 for a costumer. I can see myself encoding and decoding response bodies and JSON for the time being. For reasons I didn't investigate I don't have to do that with projects in Ruby and Elixir. It seems everything is a string there and yet they work.
KaiserPro 7y ago
I now work on a python 3.6 codebase. Something like 100k lines of code. in practice 80% of the code was written for 3.5. However, barring speed improvements, there isn't much to offer, apart from unicode, f strings and annotations. If python 3 had proper multithreading, that might have been worth breaking backwards compatibility for.
neilobremski 7y ago
The upward migration is imminent for ALL CONNECTED applications and not just Python 2 to 3. An issue I've seen with PIP (that must be relevant to other platforms as well) is that version-locked packages are for software that can no longer communicate with the actual SaaS/API because THAT layer has changed. It's really THIS that is forcing conversion of Python 2.7 to 3 because API vendors will stop supporting old software while they continue breaking their own interfaces. The alternative is the end user or FOSS picking up the slack but that's only going to happen for SOME of the API's. In the end it will be cheaper (albeit still painful) for companies to upgrade their code to Python 3. I have a lot of Python 2.7 code that I wrote years ago which has been running smoothly and my team is generally going to rewrite rather than "convert" because I really don't trust conversions. I'd rather see all bugs upfront rather than hidden in the fog.
test7777 7y ago
thats nothing, the migration from c to c++ has been going on even longer!
JohnFen 7y ago
I haven't migrated to Python 3 simply because I see no reason to go the the pain and hassle of porting my existing Python code. Python 3 doesn't give me anything that I want or need badly enough to put that much work into it.
munherty 7y ago
This is like why are hedge funds still using excel to model. Also why is SAS stilll used
klyrs 7y ago
In my experience, the biggest issue that I face is "what do you mean by Python 3?" I count 4 minor versions which aren't fundamentally broken, and I encounter them all on a regular basis. A lot of my code is performance critical, and, for example, I'm still salty about dictionary operations taking O(log(n)). But the proliferation of active minor versions makes it very difficult to write portable, performant code. It's become a sticky wicket. I want to migrate to Python 3 (and, by and large, I have in most of my projects). But what version do I target? Will my dependencies make the same choice? Or does "migration" turn into a sisyphean task? It's becoming burdensome enough that I'm contemplating abandoning the language for something more stable.
- user5994461 7y ago
  If it helps you, you can consider that no version below 3.5 is worth thinking of. It's the edge when python added back enough features to ease the migration and many libraries started being ported. Current version is 3.7. If you expect your migration work to take a year, you should consider going for 3.7 and above only, because the previous minor versions will be dropped by the time you're done.
  - klyrs 7y ago
    Thank you (and it's good to say so if another reader doesn't know), but yeah, the 4 versions I was referring to were 3.5-3.8... but my point is that it's now a perpetually moving target. And fwiw "3.7 is the current version" doesn't help my users.
    - user5994461 7y ago
      I think pip can select the version automatically as long as you provide the right metadata with the package. Shouldn't matter much really. Maybe write in the README that the package is only tested on 3.7 and above so users can be aware of that and check. Bet they struggle to figure out what version to upgrade to as well.
      - 7y ago
UptownMusic 7y ago
Simple. Python 3 is theoretically better but not in fact better, at least for most applications of Python.
- lkbm 7y ago
  In terms of feature-richness, I'd choose Python 3 over 2.7. Pre-3.6 it was slower, but at this point it's more memory efficient and generally faster, so it is in fact better in basically all applications. The problem we have where I work is some very clever 2.7 code that isn't easy to redo in Python 3. For any new project I do, I use Python 3.
at_a_remove 7y ago
In my current job, I can point out something that might be a contributing factor: I work in an industry where there is basically one 800lb gorilla of a vendor. They update rarely, because their product is a mission-critical, life-or-death sort of thing. Their current product is heavily, heavily integrated with x.y.z version of software from a different vendor in a different segment, but also weighing in at 800lb. Yes, they specify x.y.z, not just x or even x.y. That software comes bundled with a Python 2.7.5 distribution. Imagine my woes trying to get pip running, which unhelpfully suggests I upgrade Python. Cannot seem to find any other path to even get pip going because of what I call the "lol just upgrade n00b" factor. Perhaps that information once existed but I cannot find it. So, I am stuck on this version because of some pretty tight integration, at a couple of removes. I think the vendor-linkage can cause some "drag" that folks who work in a greenfield environment might not be thinking about. It can be unfortunate but there it is.
- user5994461 7y ago
  Surprised it's including pip, if it's really this old or tightly coupled, it's common to not have pip at all, or the old pip version simply can't run. If it can help you. The trick I use is to install a normal python 2.7 interpreter with pip. Then you can use it to install software to any directory, including the one from the other application. There are flags to specify what to install, from where to where, internet or not, something like pip install packagename --target=/to/app/lib
  - tingletech 7y ago
    I came across a system where pip had broken the other day. easy_install to the rescue!
kissgyorgy 7y ago
I honestly don't care as long as I don't have to deal with Python 2 code bases anymore. The important point is that all of the popular open source libraries and frameworks are ported.
cygned 7y ago
> most large organizations, outside of the hype cycle of technical news posts, move much more slowly than the press or blogs would have you think That’s the reason I am so upset with today’s JavaScript ecosystem - things move so fast that good technology is being deprecated and changed constantly which breaks all kinds of things in other places.
m463 7y ago
For me, it was that python on macos was 2.7
guardiangod 7y ago
>Why Is the Migration to Python 3 Taking So Long? For the same reason why migration to IPv6 is taking so long. Both technologies don't solve immediate problems end users are facing. Instead they solve 'nice to fix' problems that few people care about.
leetrout 7y ago
Or, in the case of my employer (and new ecosystem for me), we are using Jython which AFAIK has no clear plans or path to support Python 3 at this time. Iron Python has started work to support 3 but it's still in development and seems to be a ways off.
drdeadringer 7y ago
About 10 years ago I started learning Python. I double-timed across both versions. I decided to land on v3 for good. I haven't looked back. Granted that Big Corp cannot have such flexible actions. I reserve sympathy.
sigjuice 7y ago
Are there any examples of rewrites of large code bases from one language to another?
- yjftsjthsd-h 7y ago
  Mercurial, perhaps?
just_myles 7y ago
Personally, I don't see the need to migrate entire code bases if there is no need. I think that much is obvious. Perhaps the focus should be put onto pivoting instead. That way it doesn't leave your dev team on the hook to go back and change existing code bases in python 2.
Groxx 7y ago
Because it's not just syntactic changes, it's implied-semantic changes too. You can't mechanically transform a project and know that it'll work. And you can't do it gradually, so it's all-or-nothing. (yes, "six" exists, but you still execute one way or another) And you'll have to change the versions of all your libraries, which is not usually a smooth experience in the Python ecosystem. (this is another place where it's "all or nothing", since six can't help you if your dependencies don't all use it + use it correctly) --- It's a huge risk with huge cost for already-working, running code. For new stuff, sure, write it in 3, but 2.7 works fine and has the added benefit of being very well understood by this point.
- mixmastamyk 7y ago
  You can definitely do it gradually with a few bumps but there are categories of large apps that are more difficult than average.
qwerty456127 7y ago
What are some Python 2 features that make it hard to transpile to Python 3 automatically?
NelsonMinar 7y ago
The key inflection point was around Python 3.3 when enough bridging technologies and tools came along to either migrate code or else write code that supported both languages. Things like adding the u'string' syntax to Python 2, the creation of six.py, all the various features in the future package. That gave a much smoother transition path and enabled crucial libraries to work in Python 3, which then let everyone else migrate too.
musicale 7y ago
Because they broke backward compatibility in very annoying ways without providing a fallback mechanism. I still haven't forgiven them for killing the print statement, which could have peacefully coexisted with a print() function.
- mixmastamyk 7y ago
  Bugged me for a while five years ago. Then I moved everything to logging and use an editor snippet: pr<TAB> —> print(‘foo:’, %cursor%)
- akx 7y ago
  Having two ways to do the same thing is against The Zen of Python.
xvilka 7y ago
Find a few remote code executions in Python 2 after Jan 1 2020, and migration will be faster.
soyiuz 7y ago
In the academic data science-y world, the transition has passed the hump. In other words, where I was reluctant to switch a few years ago because some of my basic tooling was still in the 2 land, today the critical mass of important libraries has been ported and updated. Most of the cutting-edge docs and tutorials (Google's GPT2 for example) default to python3.
izolate 7y ago
As somebody who only occasionally uses Python, the fact that the default `python` binary on my system resolves to Python 2.x and I need to specify `python3` to invoke 3.x, means I am quite often mistakenly using Python 2 instead of 3. How can we expect Python 3 to become the default if Python 2 still asserts such dominance?
- jonfw 7y ago
  That's not a python thing- that most likely has to do with your package manager. In my archlinux installation, python resolves to 3, and I have to use python2 if I want 2
  - izolate 7y ago
    Well, PEP 394 suggested it be this way, so Python is also a bit complicit.
    - bscphil 7y ago
      As the history section of that PEP notes, it was actually written in part as a response to distributions like Arch changing the default Python to Python3. Although it's true that the PEP originally said that python should point to Python2 (it doesn't any more), it also said that code requiring a specific version of Python shouldn't assume anything about the system default Python, instead using either python2 or python3 - and requests that distributors of Python provide these commands. This suggestion was adopted by nearly everyone, including Arch Linux. If you're writing a script that calls Python, you probably still want python3 as your command for compatibility across all current distributions. In other words, the intention of the PEP was not to ingrain Python 2 as the "default" Python, it was to get people to stop making assumptions about what Python version they would get and use the python2 and python3 commands instead. I do agree that it has had that effect.
      - int_19h 7y ago
        The original recommendation had to be made, because at the time, there was a lot of software from pre-Py3 days that would have #!/usr/bin/env python, and that was broken badly if it was pointing at Python 3. They have changed the PEP now that Python 2 is reaching EOL, since it wouldn't make sense to demand "python" to be an alias for an unsupported version.
jeltsin1234 7y ago
The thing is python3 IS a better language. Unicode is very important for me, as i deal with a multitude of languages. For a good comparison, see how PHP failed unicode with PHP6 and we still deal with an insanity that is the mb_ functions. In python this is a non issue, and its a very nice language to work with.

202 points by josep2 7y ago

I never knew if this was a cynic's answer or truthful, but I was told by my manager at one point that RedHat's OS is superglued to python2 and spends a lot of money to keep python2 in good working order. I highly expect it's a cynic's response and please read it as such until someone in-the-know can retort my post.

maweki 7y ago
A lot of tooling (yum, for example) was written in python. This all takes a lot of time to port, especially gtk/gobject stuff, but nowadays this is all written for python3 or JavaScript or Vala.
navigatr 7y ago
Red Hat has long support contracts for their server OSes that shipped with Python 2 when it was still kosher to do so. That means they'll patch Python 2 should vulnerabilities be found on their OS.
- ianai 7y ago
  It’s a pain from the outside looking into their business. Their contracts, though, are their business.
  - int_19h 7y ago
    Those support contracts aren't cheap, though. I don't think they're getting the short end of the stick here. If anything, the companies that run lots of Python 2 code in the house that haven't migrated yet - which is not uncommon in the enterprise - have the incentive to run that code on RedHat, if they aren't doing so already. I wouldn't even be surprised if they provide support contracts specifically for Python 2 on their newer OSes. ActiveState announced that they'll continue providing paid support for Python 2 on all platforms, so some people at least feel like there's a business model there... https://www.activestate.com/company/press/press-releases/activestate-offers-extended-support-for-python-2-beyond-eol/ https://www.activestate.com/company/press/press-releases/act...
geofft 7y ago
I don't think that's true, or at least it's no longer true: RHEL 8 does not ship with Python 2 by default. https://developers.redhat.com/blog/2018/11/14/python-in-rhel-8/ https://developers.redhat.com/blog/2018/11/14/python-in-rhel... Red Hat software that depends on Python, like yum, is in Python 3. I think it is true (as of pretty recently) that Red Hat is the only company employing a Python core dev to work on Python core dev stuff full time (see https://discuss.python.org/t/official-list-of-core-developers/924/2 https://discuss.python.org/t/official-list-of-core-developer...). But the core dev team is focused on Python 3, so that isn't a sign of Red Hat's Python 2 commitment either.
hathawsh 7y ago
See the RHEL release schedule: https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux#Version_history_and_timeline https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux#Versi... RHEL 6/7 and Centos 6/7 will support Python 2 until at least mid-2024.
mkesper 7y ago
There was no way updating the system Python in Rh/CentOS6 from 2.6 to 2.7 as that broke system scripts. You could only use 2.7 in non-standard paths (or Python 3).

SQueeeeeL 7y ago

Python, unlike a C executable, could hypothetically stop working tomorrow. Researchers who have code that works/businesses don't give a shit about the esoteric differences between py2 and py3, they just want their stuff to keep working. This is similar to the banks still running Cobal backends, I don't know why everyone cares, I guarantee multiple servers out there are running executables that couldn't be rebuilt, but those don't curse you out on terminal every time I turn one on

sansnomme 7y ago
Because software that isn't protected by walls of legislation value features and the end user UX. i.e. "evergreen". Your Nokia N900 running Meego may be sufficient for certain subsets of HN readers but your average user wants Material Design and cannot care less about ideological purity or "if it ain't broke don't fix it".
- skykooler 7y ago
  Meego was never stable on the N900. Most people who are still using one are running Maemo.
- bityard 7y ago
  > but your average user wants Material Design No, users most definitely do not care about Material Design. They only care about being able to quickly do the task the app or web site claims to allow them to do.
- FpUser 7y ago
  "average user wants Material Design" - is that a fact or just a wishful thinking from people trying to push said design? EDIT: Personally for me while I find for example older windows interfaces ugly they were very consistent and functional. In modern designs I sometimes could hardly find what is clickable/actionable. It is not interface working for me but the other way around
weff_ 7y ago
At the end of the day, it boils down to dependency management, wouldn't you say? That is, Python2 could be running forever if the environment is maintained. In the same vein, a messed-up OS or libs can stop a C app in its tracks.

paulie_a 7y ago

Simple answer. Updating legacy code. If it works now why upgrade? Stability is more important than some new features of the language. Granted that has its downfalls via security issues but stability wins If you have a hole it's hard to dig yourself out of it. This is why I prefer modular apps instead of monolithic codebases. You can upgrade piece by piece. Otherwise it's all or nothing and dangerous

downerending 7y ago

Because there's been not enough carrot and too much stick. The only real killer feature of Python3 is the async programming model. Unfortunately, the standard library version is numbingly complex. (Curio is far easier to follow, but doesn't appear to have a future.) On the down side, switching to Unicode strings is a major hurdle. It mostly "just works", but when it doesn't, it can be difficult to see what's going on. Probably most programmers don't really understand all of the ins and outs. And on top of that, you get weird bugs like this one, which apparently is simply never going to be fixed. https://github.com/pallets/click/issues/1212 https://github.com/pallets/click/issues/1212

semiotagonal 7y ago
I wonder how many people feel that migrating to Python3 would have been worth doing in the absence of being forced to do so. Dropbox invested three years of work, actually hired Python's creator, and are still not done. What are they getting out of it that they wouldn't have gotten if Python2 simply had been maintained?
- crustacean 7y ago
  This is such a good point. Take SQL. It has survived because it is well designed, and changes so little and so slowly, and it’s obvious what SQL is and isn’t meant for. Amateur programmers can port SQL queries between database systems semi-painlessly. In the right environment a query can survive with small edits for YEARS. Who wants to break old SQL? Nobody.
  - skybrian 7y ago
    That's something of an illusion. The flaws in SQL often aren't noticed because you usually pick one database vendor and stick with them. There are plenty of differences between vendors, but you don't usually have to support them at the same time, and you don't notice for simple cases. But changing database vendors for a company can be a big deal, as bad as going from Python 2 to 3.

pmoriarty 7y ago

Another data point: According to my highly unscientific survey of the packages in Gentoo's package repo, there are roughly: - 2500 packages that work with Python 2 or 3 - 1350 packages that work with Python 2 only - 350 that work with Python 3 only My methodology: http://dpaste.com/1M0TCV7 http://dpaste.com/1M0TCV7

roland35 7y ago
That is interesting data. I would imagine the most widely used packages are compatible with both, but that data is probably harder to get!
stevesimmons 7y ago
I bet many of the Py2-only ones are old legacy packages that have been superceded by newer better options.
girst 7y ago
and yet another (fedora's): of 3414 packages total: - 3122 Python 3 only - 88 Dual support - 8 Py2 leaf (standalone packages; may be dropped) - 77 Not ported (will be dropped unless ported) - 100 Blocked (require 1 or more "not ported" packages) - 18 Legacy (will be dropped) https://fedora.portingdb.xyz https://fedora.portingdb.xyz note that py3only/dualsupport only reflects how it is packaged in fedora, not what upstream provides.

m4r35n357 7y ago

It _is_ happening, but moaning about it won't make it happen any more quickly.

eej71 7y ago
I think the intent is less moaning about it - and more learning about the blocking points so they can be better surmounted. At the very least, perhaps the blocking points can be avoided in the future. Those who do not learn from history... etc etc.

alexhutcheson 7y ago

[copying comment from an older HN thread, not speaking on behalf of any employer, opinions my own] I think many people underestimate the challenge that the 2 to 3 migration presents for large enterprises. The core issue is that even though the migration for any given module is normally really easy, the total effort required to migrate is still essentially O(n) in module count/file count, because even with current tooling you still need to have an engineer look at every module to do the change safely. Even if it only takes ~5 minutes per module to make the changes and validate that it works correctly, this becomes a giant undertaking when you have tens of thousands of files to migrate. The fact that it takes a long time also creates other problems. Your business isn't going to hit "pause" on other development, so there will be changes constantly introduced into modules you've already "swept". It's going to be hard to make sure 100% of your engineers and code reviewers are knowledgeable about the specific requirements to make sure the code works in both 2 and 3, so you would really like some automated safeguards to make sure they don't introduce anything that won't work in 3. Pylint helps with this, but won't catch everything. Unit tests are obviously essential, but: 1. Even a well-tested project won't have tests that cover 100% of code paths and behavior. 2. You're stuck running the tests on both python2 and python3 for the duration of the migration, which doubles the resource (compute, memory, etc.) cost of your Python CI and regression testing infrastructure for the duration of the migration. Most big companies have passionate Python advocates who really want to be on Python 3, but the scale of the problem and the lack of tooling to tackle it with a sub-O(n) amount of effort make the overall project risky and expensive for the business.

war1025 7y ago
As a member of a team hopefully nearing the end of the python3 migration, this is exactly correct. The unicode switch is a nightmare in terms of having to go through and double/triple check everything and still get it wrong half the time. Particularly when it comes to moving data over the network. The big selling point for Python3 finally came with the built-in async support, but we've been using Twisted for a decade, which works nearly identically, so even that wasn't a huge draw for us. Further, many of our dependencies were python2-only up until the last year or two. Really the only reason we're going through the effort right now is that Python2 is rapidly approaching End of life.

lasermike026 7y ago

Management wants new features not porting. They will only port when they absolutely have to.

war1025 7y ago
Even if management wanted porting, the python 2 -> 3 migration path is very painful in the details, while on the surface not having a lot to offer at the other end in terms of new capability.
- lasermike026 7y ago
  Roger that.
compiler-guy 7y ago
As well they should. No one uses Dropbox today that didn't a couple of years ago because it is using Python 3 instead of Python 2. The migration is financially negative in the short term, and very clearly so. It might be financially positive over the long term (due to easier maintenance and higher performance), but that is definitely maybe. Especially for an app that is otherwise very stable.

MaulingMonkey 7y ago

We're still waiting for Maya to switch. It's still using Python 2, and Autodesk keeps putting off the transition, partially because it'll break the scripts of all their downstream users.

pfranz 7y ago
I think that's partially correct. If you follow https://vfxplatform.com/ https://vfxplatform.com/ it says Python3 was pushed from 2019 to 2020 to allow everyone to upgrade Qt and PySide (which are prerequisites). The reason it was deprioritized in previous years were similar transitions like C++14, gcc, etc. I've been meaning to dig into Maya, Houdini, Nuke's Python 3 transition plans. I know Houdini will offer a Python 3 option with Houdini 18 (shipping in the next month or so). I don't think the reason was because of downstream users. Python 3 was an inevitable change. Previously, they swapped out PyQt for PySide which wasn't a forced change, but required everyone to update their Python scripts.

upofadown 7y ago

Py3 is for all practical purposes a language fork of Py2. So it doesn't really make sense to talk of a "migration". If Py2 becomes unworkable somehow then people will rewrite stuff. Some of it might even be in Py3. Considering all the stuff that is written in Py2 I really don't see it being out and out abandoned. That wouldn't really make any sense. With computer languages stuff never goes away.

carapace 7y ago
Py2 is the new Fortran, I like to say.
- sigjuice 7y ago
  And they both use actual Fortran :) $ python3 Python 3.7.4 (default, Sep 7 2019, 18:27:02) [Clang 10.0.1 (clang-1001.0.46.4)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> [2]+ Stopped python3 $ lsof -c Python | sed -n "/fortran/s/$USER/<redacted>/gp" Python 35190 <redacted> txt REG 1,4 1550456 12887664541 /Users/<redacted>/Library/Python/3.7/lib/python/site-packages/numpy/.dylibs/libgfortran.3.dylib
- int_19h 7y ago
  Py2 is the new Fortran 77. It was good in its time, and great things were done in it that are still around... but let's move onto F90 already.

m45t3r 7y ago

Any non-trivial piece of code will probably be a pain in the ass to port for a more recent version of the language, and the level of PITA depends on the size of the project X size of the changes in the language. Case in point, I worked in a project using Ruby. When we migrated from Ruby 2.4.0 to 2.4.6 (yeah, a minor upgrade), it broke spectacularly. Trying multiple Ruby versions, the change was actually introduced in Ruby 2.4.1. After some investigation, a change in Net::HTTP library from stdlib had a change that broke a dependency from a dependency. The fix was just a line of code (we just need to change the adapter used for HTTP communication), however it was two days of work for a minor upgrade. My current job tried to migrate from Java 8 to Java 11. It also broke multiple services. This one is still in progress, months later. Python 2 to Python 3 is bigger than both of those version changes (however it is equivalent to Ruby 1.8 to 1.9 changes), so yeah, it does take more time. And like some projects that are forever running Ruby 1.8 or Java 8 (or even worse, Java 6), we will have projects forever running Python 2 too.

linsomniac 7y ago

What do you mean taking so long? The original target date, proposed by Guido, was Jan 1, 3000. Looks like we're 980 years ahead of schedule! :-) I say this in part because comedy, but also because it was anticipated to be a long project. It was originally called "Python 3000".

Animats 7y ago

First off. Python 2.6 and 2.7 supported Unicode just fine. I had a large all-Unicode system in Python 2.6. You had to write u'word" to get a Unicode constant, and use a "unicode(s)" function here and there. Also, the part that remained "compatible" was that "str" remained an array of bytes, even though there was also a type "bytes" and a "bytearray". Early Python 3 was hell for conversion. The syntax was changed for no good reason. u'word" became illegal. (That later went back in.) The "2 to 3 converter" was a joke. I didn't have the "print statement problem" because my code called a logging function for all debug output. Many of the P3 libraries didn't work. (The all-Python MySQL connector failed the first time I tried to do a bulk load bigger than a megabyte, indicating that nobody was using it.) It took years before the libraries were cleaned up. Python 3 got some really weird features, such as type declarations that don't do anything. I can see having type declarations, especially for parameters, but they need to be used both for checking and optimization. CPython boxes everything, which is terrible for numerics and is why most serious math has to be done in C libraries. My comment on that was "Stop him before he kills again."

wnissen 7y ago
They still did stuff that makes no sense, to me, like not supporting `iteritems` on dictionaries and `xrange` on sequences. Or using `str` to mean bytes? That would have been possible to make backwards compatible until Python 4. At this point, does anyone think there will be a Python 4?
- kevin_thibedeau 7y ago
  You're expected to clean that stuff up with 2to3. The whole idea is to not leave band-aids in place.
viraptor 7y ago
> Python 2.6 and 2.7 supported Unicode just fine It did, but in a way that chainsaws support sculpting just fine. Technically possible. Very advanced people will know how to handle it. Everybody else is just going to injure themselves randomly. Most people writing py2 got the text/binary processing working on accident. Things appear to work until you throw actual Unicode into parameters and then nobody knows what happens. There's a number of "what does this decoding exception mean" questions on stackoverflow every day. They're often actual bugs people could ignore before. Now they're told immediately and I believe that's better.

goatinaboat 7y ago

The 2to3 tool should add .decode(‘utf-8’) to every string manipulation, even better Python 3 should have a flag to make that the behaviour and even better that should default to on. So much effort wasted doing this in a large codebase. And what do you get for it? It’s just not worth it. Nobody actually needs Python 3, it was foisted on them by the developers. What everyone really wanted was Python 2.8.

kstrauser 7y ago
Speak for yourself. After using Python 3, I can't stand touching Python 2 codebases. Is a given str object an actual text string that's been decoded into a valid charset, or is it an array of bytes fresh from the network / database / file? Who knows, unless you trace back to its point of origin. I personally love that Python 3 says "this is text, and this is some binary data I got somewhere, and they are not the same thing".
joshuamorton 7y ago
Please no! Text is text. Bytes is bytes. Convert to the correct form on input boundaries, and that's it. Don't switch back and forth internally, it's overly complicated, error prone, and slower.
- test7777 7y ago
  unix pipes (stdin, stdout) are bytes, files are bytes, filenames are bytes. yet, for some reason python3 thinks al of those are text. its not the coders that are wrong, it is the language.
  - kstrauser 7y ago
    No, it's Python 2 that thinks those are all text. Python 3 makes you explicitly say "convert this stream of bytes into a text encoded string", and from then on it's a str object. Python 2 was happy to (try to) let you call text methods on a JPEG. Python 3 draws an appropriate distinction between the two.

mbparks 7y ago

In my organization, business needs overrule technical details. Management didn't want to spend the resources until forced to do so, rather focus on revenue-generating work. Can see it both ways I guess. We have recently started upgrading our codebase to support 3.X

twblalock 7y ago

I recently went through a fairly large upgrade from JDK8 to JDK 11 and it was a bit of a pain -- lots of dependencies to update, etc. But very few code changes were required, and the static type system made it pretty clear when the codebase was broken -- it just wouldn't build. It still took my team several weeks. Migrating from Python 2 to Python 3 is way worse than that -- code changes are required, and because Python is a dynamic language you may not notice bugs until you actually run the code (or even worse, until after you release it to production and some code branch that is rarely invoked somehow gets called...). In other words, the tooling and the type system are not confidence-inspiring and it's really hard to verify that you migrated without breaking stuff.

LIV2 7y ago
Will this will make people less likely to use Python in future for some projects? I'm no developer or manager but I figure there'd be a thought in the back of my head thinking "what if we need to rewrite this all again for the next major release"
- xemdetia 7y ago
  I don't think it will make people less likely to use Python overall, but for certain categories of projects there are some things that are just done better in other languages. Both Java and Go for example have specific programming language constructs and contracts to ensure that codebases can move from version to version easily and I do not feel like there is a dynamic/untyped language that really provides that same level of stability. The ones that do get close to that are the ones that are extremely small in scope and are not in the same class of batteries included language, and because of their small scope they have a general limit of change. At a certain point this sort of compatibility/forward motion of a codebase through big language revisions is something that has to be designed as part of the language in either being able to break it down into small enough chunks to chew through in pieces (updating a submodule with the updated language without affecting anything else), completely transparent to the code being run through it (this happens for compilers for C for different standards), or to have a version to version automated rewriting mechanism that is so reliable the outcome of the automated tool is not in question (tools like Go's gofmt). Python in my opinion only has partial solutions to all of those answers so it turns into a lot of hand work. So while there are other languages that may do other things better there are still a class of programs that are very effective to write in Python, and that's plenty enough reason to keep it around. Do not forget that Python 2 was released in 2000 and Python 3 was released almost a decade later. The general time scale makes worrying about the next release for many people, but for people who do they start considering other languages because that's important to them.

jdhawk 7y ago

The same reason there is still a ton of legacy PHP 5.6 code. Migration in interpreted languages that implement major breaking changes is really tedious.

recursivecaveat 7y ago

The simple reason is that there was no compelling feature to reward you for upgrading. You'd spend a tremendous amount of effort for dubious return and (until recently) a smaller ecosystem. 1. Unicode support was actually an anti-feature for most existing code. If you're writing a simple script you prefer 'garbage-in, garbage-out' unicode rather than scattering casts everywhere to watch it randomly explode when an invalid byte sneaks in. If you did have a big user-facing application that cared about unicode, then the conversion was incredibly painful for you because you were a real user of the old style. 2. Minor nice-to-haves like print-function, float division, and lazy ranges just hide landmines in the conversion while providing minimal benefit. In the latest py3 versions we've finally gotten some sugar to tempt people over: asyncio, f-strings, dataclasses, and type annotations. Still not exactly compelling, but at least something to encourage the average Joe to put in all the effort.

takeda 7y ago
> Unicode support was actually an anti-feature for most existing code. If you're writing a simple script you prefer 'garbage-in, garbage-out' unicode rather than scattering casts everywhere to watch it randomly explode when an invalid byte sneaks in. If you did have a big user-facing application that cared about unicode, then the conversion was incredibly painful for you because you were a real user of the old style. Actually that's the behavior of python 2, it works fine, until you send invalid characters then it blows up. In python 3 it always blows up when you mix bytes with text so you can catch the issue early on. > In the latest py3 versions we've finally gotten some sugar to tempt people over: asyncio, f-strings, dataclasses, and type annotations. Still not exactly compelling, but at least something to encourage the average Joe to put in all the effort. That's because until 2015 all python 2.7 features were from python 3. Python 2.7 was basically python 3 without the incompatible changes. After they stopped backporting features in 2015. Suddenly python 3 started looking more attractive.
- pmontra 7y ago
  > In python 3 it always blows up when you mix bytes with text so you can catch the issue early on. This is definitely the case. I've been wrestling with bytes and strings all the time during the port of a Django application to Python 3 for a costumer. I can see myself encoding and decoding response bodies and JSON for the time being. For reasons I didn't investigate I don't have to do that with projects in Ruby and Elixir. It seems everything is a string there and yet they work.

KaiserPro 7y ago

I now work on a python 3.6 codebase. Something like 100k lines of code. in practice 80% of the code was written for 3.5. However, barring speed improvements, there isn't much to offer, apart from unicode, f strings and annotations. If python 3 had proper multithreading, that might have been worth breaking backwards compatibility for.

neilobremski 7y ago

The upward migration is imminent for ALL CONNECTED applications and not just Python 2 to 3. An issue I've seen with PIP (that must be relevant to other platforms as well) is that version-locked packages are for software that can no longer communicate with the actual SaaS/API because THAT layer has changed. It's really THIS that is forcing conversion of Python 2.7 to 3 because API vendors will stop supporting old software while they continue breaking their own interfaces. The alternative is the end user or FOSS picking up the slack but that's only going to happen for SOME of the API's. In the end it will be cheaper (albeit still painful) for companies to upgrade their code to Python 3. I have a lot of Python 2.7 code that I wrote years ago which has been running smoothly and my team is generally going to rewrite rather than "convert" because I really don't trust conversions. I'd rather see all bugs upfront rather than hidden in the fog.

test7777 7y ago

thats nothing, the migration from c to c++ has been going on even longer!

JohnFen 7y ago

I haven't migrated to Python 3 simply because I see no reason to go the the pain and hassle of porting my existing Python code. Python 3 doesn't give me anything that I want or need badly enough to put that much work into it.

munherty 7y ago

This is like why are hedge funds still using excel to model. Also why is SAS stilll used

klyrs 7y ago

In my experience, the biggest issue that I face is "what do you mean by Python 3?" I count 4 minor versions which aren't fundamentally broken, and I encounter them all on a regular basis. A lot of my code is performance critical, and, for example, I'm still salty about dictionary operations taking O(log(n)). But the proliferation of active minor versions makes it very difficult to write portable, performant code. It's become a sticky wicket. I want to migrate to Python 3 (and, by and large, I have in most of my projects). But what version do I target? Will my dependencies make the same choice? Or does "migration" turn into a sisyphean task? It's becoming burdensome enough that I'm contemplating abandoning the language for something more stable.

user5994461 7y ago
If it helps you, you can consider that no version below 3.5 is worth thinking of. It's the edge when python added back enough features to ease the migration and many libraries started being ported. Current version is 3.7. If you expect your migration work to take a year, you should consider going for 3.7 and above only, because the previous minor versions will be dropped by the time you're done.
- klyrs 7y ago
  Thank you (and it's good to say so if another reader doesn't know), but yeah, the 4 versions I was referring to were 3.5-3.8... but my point is that it's now a perpetually moving target. And fwiw "3.7 is the current version" doesn't help my users.
  - user5994461 7y ago
    I think pip can select the version automatically as long as you provide the right metadata with the package. Shouldn't matter much really. Maybe write in the README that the package is only tested on 3.7 and above so users can be aware of that and check. Bet they struggle to figure out what version to upgrade to as well.
    - 7y ago

UptownMusic 7y ago

Simple. Python 3 is theoretically better but not in fact better, at least for most applications of Python.

lkbm 7y ago
In terms of feature-richness, I'd choose Python 3 over 2.7. Pre-3.6 it was slower, but at this point it's more memory efficient and generally faster, so it is in fact better in basically all applications. The problem we have where I work is some very clever 2.7 code that isn't easy to redo in Python 3. For any new project I do, I use Python 3.

at_a_remove 7y ago

In my current job, I can point out something that might be a contributing factor: I work in an industry where there is basically one 800lb gorilla of a vendor. They update rarely, because their product is a mission-critical, life-or-death sort of thing. Their current product is heavily, heavily integrated with x.y.z version of software from a different vendor in a different segment, but also weighing in at 800lb. Yes, they specify x.y.z, not just x or even x.y. That software comes bundled with a Python 2.7.5 distribution. Imagine my woes trying to get pip running, which unhelpfully suggests I upgrade Python. Cannot seem to find any other path to even get pip going because of what I call the "lol just upgrade n00b" factor. Perhaps that information once existed but I cannot find it. So, I am stuck on this version because of some pretty tight integration, at a couple of removes. I think the vendor-linkage can cause some "drag" that folks who work in a greenfield environment might not be thinking about. It can be unfortunate but there it is.

user5994461 7y ago
Surprised it's including pip, if it's really this old or tightly coupled, it's common to not have pip at all, or the old pip version simply can't run. If it can help you. The trick I use is to install a normal python 2.7 interpreter with pip. Then you can use it to install software to any directory, including the one from the other application. There are flags to specify what to install, from where to where, internet or not, something like pip install packagename --target=/to/app/lib
- tingletech 7y ago
  I came across a system where pip had broken the other day. easy_install to the rescue!

kissgyorgy 7y ago

I honestly don't care as long as I don't have to deal with Python 2 code bases anymore. The important point is that all of the popular open source libraries and frameworks are ported.

cygned 7y ago

> most large organizations, outside of the hype cycle of technical news posts, move much more slowly than the press or blogs would have you think That’s the reason I am so upset with today’s JavaScript ecosystem - things move so fast that good technology is being deprecated and changed constantly which breaks all kinds of things in other places.

m463 7y ago

For me, it was that python on macos was 2.7

guardiangod 7y ago

>Why Is the Migration to Python 3 Taking So Long? For the same reason why migration to IPv6 is taking so long. Both technologies don't solve immediate problems end users are facing. Instead they solve 'nice to fix' problems that few people care about.

leetrout 7y ago

Or, in the case of my employer (and new ecosystem for me), we are using Jython which AFAIK has no clear plans or path to support Python 3 at this time. Iron Python has started work to support 3 but it's still in development and seems to be a ways off.

drdeadringer 7y ago

About 10 years ago I started learning Python. I double-timed across both versions. I decided to land on v3 for good. I haven't looked back. Granted that Big Corp cannot have such flexible actions. I reserve sympathy.

sigjuice 7y ago

Are there any examples of rewrites of large code bases from one language to another?

yjftsjthsd-h 7y ago
Mercurial, perhaps?

just_myles 7y ago

Personally, I don't see the need to migrate entire code bases if there is no need. I think that much is obvious. Perhaps the focus should be put onto pivoting instead. That way it doesn't leave your dev team on the hook to go back and change existing code bases in python 2.

Groxx 7y ago

Because it's not just syntactic changes, it's implied-semantic changes too. You can't mechanically transform a project and know that it'll work. And you can't do it gradually, so it's all-or-nothing. (yes, "six" exists, but you still execute one way or another) And you'll have to change the versions of all your libraries, which is not usually a smooth experience in the Python ecosystem. (this is another place where it's "all or nothing", since six can't help you if your dependencies don't all use it + use it correctly) --- It's a huge risk with huge cost for already-working, running code. For new stuff, sure, write it in 3, but 2.7 works fine and has the added benefit of being very well understood by this point.

mixmastamyk 7y ago
You can definitely do it gradually with a few bumps but there are categories of large apps that are more difficult than average.

qwerty456127 7y ago

What are some Python 2 features that make it hard to transpile to Python 3 automatically?

NelsonMinar 7y ago

The key inflection point was around Python 3.3 when enough bridging technologies and tools came along to either migrate code or else write code that supported both languages. Things like adding the u'string' syntax to Python 2, the creation of six.py, all the various features in the future package. That gave a much smoother transition path and enabled crucial libraries to work in Python 3, which then let everyone else migrate too.

musicale 7y ago

Because they broke backward compatibility in very annoying ways without providing a fallback mechanism. I still haven't forgiven them for killing the print statement, which could have peacefully coexisted with a print() function.

mixmastamyk 7y ago
Bugged me for a while five years ago. Then I moved everything to logging and use an editor snippet: pr<TAB> —> print(‘foo:’, %cursor%)
akx 7y ago
Having two ways to do the same thing is against The Zen of Python.

xvilka 7y ago

Find a few remote code executions in Python 2 after Jan 1 2020, and migration will be faster.

soyiuz 7y ago

In the academic data science-y world, the transition has passed the hump. In other words, where I was reluctant to switch a few years ago because some of my basic tooling was still in the 2 land, today the critical mass of important libraries has been ported and updated. Most of the cutting-edge docs and tutorials (Google's GPT2 for example) default to python3.

izolate 7y ago

As somebody who only occasionally uses Python, the fact that the default `python` binary on my system resolves to Python 2.x and I need to specify `python3` to invoke 3.x, means I am quite often mistakenly using Python 2 instead of 3. How can we expect Python 3 to become the default if Python 2 still asserts such dominance?

jonfw 7y ago
That's not a python thing- that most likely has to do with your package manager. In my archlinux installation, python resolves to 3, and I have to use python2 if I want 2
- izolate 7y ago
  Well, PEP 394 suggested it be this way, so Python is also a bit complicit.
  - bscphil 7y ago
    As the history section of that PEP notes, it was actually written in part as a response to distributions like Arch changing the default Python to Python3. Although it's true that the PEP originally said that python should point to Python2 (it doesn't any more), it also said that code requiring a specific version of Python shouldn't assume anything about the system default Python, instead using either python2 or python3 - and requests that distributors of Python provide these commands. This suggestion was adopted by nearly everyone, including Arch Linux. If you're writing a script that calls Python, you probably still want python3 as your command for compatibility across all current distributions. In other words, the intention of the PEP was not to ingrain Python 2 as the "default" Python, it was to get people to stop making assumptions about what Python version they would get and use the python2 and python3 commands instead. I do agree that it has had that effect.
    - int_19h 7y ago
      The original recommendation had to be made, because at the time, there was a lot of software from pre-Py3 days that would have #!/usr/bin/env python, and that was broken badly if it was pointing at Python 3. They have changed the PEP now that Python 2 is reaching EOL, since it wouldn't make sense to demand "python" to be an alias for an unsupported version.

jeltsin1234 7y ago

The thing is python3 IS a better language. Unicode is very important for me, as i deal with a multitude of languages. For a good comparison, see how PHP failed unicode with PHP6 and we still deal with an insanity that is the mb_ functions. In python this is a non issue, and its a very nice language to work with.