7 ms·
How we rolled out one of the largest Python 3 migrations
- rawoke083600 8y agoI would have thought that Go would be perfect for this sort of system-level-like programming ?
- kenhwang 8y agoLooks like they already have Rust/C++/obj-c in use, which all fill that purpose a bit better than Go would.
- basic1 8y agoGo wasn't a thing in 2007.
- mrgordon 8y agoGo wasn't created until 2009 and Dropbox already had tons of Python code by then as well as extremely accomplished Python programmers (they hired the creator of Python only a few years later). I also don't think Go would have allowed the tight integration with, for example, OS X where Dropbox actually superimposes their icons onto your Finder icons. As far as I understand, it was only because of smart programming and use of OS X's Python / Ruby bridge functionality that they were able to do it.
- Longhanks 8y agoThere’s an official API since 2014 (Yosemite): https://developer.apple.com/library/archive/documentation/General/Conceptual/ExtensibilityPG/Finder.html https://developer.apple.com/library/archive/documentation/Ge...
- NegativeLatency 8y agoDropbox predates that api though. They used to hack into Finder.app's api IIRC.
- Longhanks 8y agoWhich is part of why that extension point was introduced :) https://twitter.com/nickjshearer/status/608833134902140928 https://twitter.com/nickjshearer/status/608833134902140928
- crdoconnor 8y agoGo code is about ~30% more verbose than equivalent python.
- azhenley 8y agoWow, I wouldn't have expected that. Do you have a reference for the 30%? I'd like to see what is going on to cause that big of a difference.
- SEJeff 8y agoEvery other 5 lines being the equiv of: if err != nil { ... error handling ... } Python just uses exceptions, and they bubble up. No need for this excessively verbose error checking.
- SEJeff 8y agoI'm not sure why it is downvoted. This is true. I write both Golang and Python as part of my day job. I love both languages, but this is definitely the truth. Go is more verbose. It also gives you the wonder of the compiler telling you about doing stupid things. That doesn't make it a worse language.
- samfisher83 8y agoDoes go not have exception handling?
- lacksconfidence 8y agogo has something in the same general area of error handling called panic/recover. It's not exceptions in the normal sense, and devs expect nothing should panic across api boundaries.
- SEJeff 8y agoGo does not have exception handling at all. It is a proposal they're seriously considering for go2: https://go.googlesource.com/proposal/+/master/design/go2draft.md https://go.googlesource.com/proposal/+/master/design/go2draf... See the bits about error handling and error values.
- ChrisSD 8y agoGo isn't good at the system level. I.e. calls to C APIs are expensive, etc.
- JustSomeNobody 8y agoWait, I thought Go was designed as a system language.
- xapata 8y agoGo was designed for concurrency (and the docs take care to distinguish that from parallelism), fast compilation, and easy deployment.
- floatboth 8y agoOnly in the Google sense i.e. "for building big systems… like servers". Not as in "close to your operating system" or "you should write an operating system in it".
- rrdharan 8y agoDropbox uses Go (https://github.com/dropbox/godropbox https://github.com/dropbox/godropbox), Rust (e.g. https://github.com/dropbox/rust-brotli https://github.com/dropbox/rust-brotli) and Python on the server-side, and both Rust (https://github.com/dropbox/finderinfo-rust https://github.com/dropbox/finderinfo-rust) and Python on the client-side. As other commenters have noted, a lot of their Python use for large scale systems was an artifact of history and available choices at the time, but from my experience during my time there and following as an outside observer since leaving, they seem to make reasonable infrastructure and language decisions for their core product.
- bdr 8y agoWhy are people downvoting this? It’s a fine question to ask. I think people have been downvoting questions more lately, and it’s unfortunate.
- rawoke083600 8y agoSo much downvoting for a honest question :/
- kryptiskt 8y agoThere are few comments worser on HN than "why didn't you implement in [rival language]?" as if [rival language] was a default choice that could only be deviated from if it was extensively motivated. The claim that Go is "perfect" doesn't make things better.
- thijsvandien 8y agoInteresting write-up, though it leaves me terrified how a relatively small, more or less single purpose application like the Dropbox client has over 1 million lines of code in Python alone.
- larrik 8y agoDropbox doesn't feel that small or "single purpose" to me anymore.
- thijsvandien 8y agoWhat does the Dropbox client do for me other than syncing files and exposing a bit of the online functionality such as generating share links? (Serious question...)
- iscrewyou 8y agoWe can simply start by asking what does “syncing files” include? Watching files. Keeping backup of files. Keeping conflicts resolved. Watching Selective Sync files and folders. Watching Smart Sync files and folders. Notifications for synced files. Etc. etc. There’s way more the client does than what I mention.
- thijsvandien 8y agoI agree that syncing is far from trivial, but that doesn't change it's a single purpose.
- kiriakasis 8y agoNot to build a straw man but by this logic also a browser is a single purpose application
- paulddraper 8y agoA rocket ship is also "single purpose."
- amanzi 8y agoThey forgot step 1 - hire the creator of Python... But seriously, that's an interesting post.
- amelius 8y agoGuido van Rossem recently stepped down from leading the Python project. https://news.ycombinator.com/item?id=17515492 https://news.ycombinator.com/item?id=17515492
- romed 8y ago
- chrisseaton 8y agoWhat happened to their new implementation of Python they were building? I know it was cancelled, but not really why.
- phonon 8y agoProbably because it was 2.7 only, and only 10% faster on Dropbox's code. https://blog.pyston.org/2017/01/31/pyston-0-6-1-released-and-future-plans/ https://blog.pyston.org/2017/01/31/pyston-0-6-1-released-and...
- chrisseaton 8y agoI'm surprised 10% wasn't enough - 10% bottom-line improvement in programming language implementation is normally massive. Twitter is singing from the roof-tops about 10% improvement in Java performance from the new Graal JIT compiler.
- ummonk 8y agoI would imagine that for code that is performance sensitive enough that a 10% improvement matters, they would be porting to a language with saner performance instead?
- phonon 8y agoPyPy is already way better than that, and has a Python 3.5 implementation.
- pmart123 8y agoI'm by no means an expert on the subject, but isn't there an advantage to having a JIT compiler use LLVM? On another note, I believe one thing that has been problematic for pypy adoption is that it does not automatically work with C extensions or Cython, and generally if someone already had performance issues with CPython, they would have written some C/Cython extensions?
- dwheeler 8y agoIf you can't get it directly (I can't), there is an archive available at https://web.archive.org/web/20180925210033/https://blogs.dropbox.com/tech/2018/09/how-we-rolled-out-one-of-the-largest-python-3-migrations-ever/ https://web.archive.org/web/20180925210033/https://blogs.dro...
- yieFiseh 8y agoBut their Debian packages still depend on Python 2.
- mr_toad 8y ago> However, rather than use the native toolchains directly, such as Xcode for macOS, we delegated the creation of platform-compliant binaries to py2exe for Windows, py2app for macOS, and bbfreeze for Linux. I wish the authors of more Python tools would deploy standalone applications. I do not like having to maintain various sets of Python installers/package managers (because every Python tool seems to use a different installer). Especially on cloud servers that often lack a whole set of dependencies that Python developers just seem to take for granted. I’m not a Python developer. I don’t have the time or the inclination to repackage various tools and untangle dependencies. Given the choice between trying to figure out how to get multiple Python tools to behave together, or using another tool, I’ll almost always choose an alternative.
- cjhanks 8y agoThis is possible and actually pretty trivial. You install all 3rd party dependencies into some directory. The command line entry point is then a simple BASH script which sets the PYTHONPATH to the appropriate installation location and then does the appropriate exec call. You then have a functionally portable python installation.
- orf 8y agoIsn't this just a hacky virtualenv? 'python3 -mvenv dir && ./dir/bin/pip install requirements'?
- jofer 8y agoThere are a lot of minor differences, but the biggest difference is that you're able to be completely independent of the system python install. You bundle a complete python interpreter, all libraries needed, etc. The user doesn't need to have python installed at all, and if they have 2.x instead of 3.x or 3.3 when you're expecting features that are only present in >= 3.5, it's no issue. This may sound trivial, but it's a _huge_ deal, particularly when you need to deploy something that runs on multiple different OSes and versions of OSes. Other than that, the "directory full of libs, binaries, and code" approach is a lot easier to package into something that will work well with the native package manager (e.g. an .msi for windows, etc).
- vemv 8y agoI hope this kills (or helps killing) the 100+ thread count I have always seen in macOS. It surpasses any other thread count from far more important/sophisticated processes. I'd say that's a waste (if not abuse) of the system's resources and scheduling system.
- Born_Again 8y agoGreat write-up and those two graphs are interesting. It's cool to learn how different companies treat their beta users. I wish this article touched upon more of the technically difficulties with switching from Python 2 to 3 too. A̵l̵s̵o̵ ̵a̵ ̵s̵m̵a̵l̵l̵ ̵h̵e̵a̵d̵s̵-̵u̵p̵:̵ ̵a̵t̵ ̵t̵h̵e̵ ̵e̵n̵d̵ ̵o̵f̵ ̵t̵h̵e̵ ̵f̵i̵r̵s̵t̵ ̵p̵a̵r̵a̵g̵r̵a̵p̵h̵,̵ ̵"̵v̵e̵n̵e̵r̵a̵b̵l̵e̵"̵ ̵s̵h̵o̵u̵l̵d̵ ̵b̵e̵ ̵"̵v̵u̵l̵n̵e̵r̵a̵b̵l̵e̵"̵ ̵(̵u̵n̵l̵e̵s̵s̵ ̵y̵o̵u̵ ̵m̵e̵a̵n̵ ̵w̵e̵ ̵s̵h̵o̵u̵l̵d̵ ̵l̵o̵o̵k̵ ̵b̵a̵c̵k̵ ̵a̵t̵ ̵t̵h̵e̵ ̵s̵a̵c̵r̵e̵d̵ ̵p̵y̵w̵i̵n̵3̵2̵ ̵l̵i̵b̵r̵a̵r̵y̵ ̵w̵i̵t̵h̵ ̵h̵o̵n̵o̵r̵)̵.̵
- pzone 8y agoVenerable is correct here.
- Born_Again 8y agoYou're right, I haven't seen it used outside of a religious context before.
- makecheck 8y agoIt’s funny, after Mojave was released recently I thought it might finally have a python3 installed, even if it’s not the default. Nope, still Python 2.7. This is good for me since I deployed a Python-dependent app under the assumption that the system Python would be stable and reliable. It allows relatively complex things to be achieved with a tiny download package. I’ve been prepared to adopt Python 3 for awhile but it just isn’t necessary when using system defaults.
- Longhanks 8y agoIt'll be very interesting to see what happens with the next macOS when Python 2 will be EOL (which is roughly 3 months after its release). The upgrade to Python 3 is long overdue.
- CJefferson 8y agoPython being EOL sounds scary but actually won't matter to Apple. They already apply custom patches, they can carry on running python 2.7 forever, with minor bug fixes where really required.
- guitarbill 8y ago> they can carry on running python 2.7 forever would they do this? does anything macOS internal depend on python 2.7?
- varenc 8y agothe `xattr` terminal command relies on the system installed Python, though this seems to be the only example. If you look at the source in `/usr/bin/xattr` it does some work to deal with different versions of Python. All the work ultimately gets handled by the xattr module preinstalled with the system Python. This module has Apple's copyright in it and is different than the `xattr` module on pypi. Wonder how this Python one-off in macOS came to be.
- madrox 8y agoAs a python engineer still living in 2.7, it's great to see major codebases making the move. I know I have a similar experience coming in my future, and I appreciate hearing what seems like more or less a success story come from it.
- sandGorgon 8y agoIs there any sample code that illustrates this kind of python embedding versus one of the freezer scripts ? We used to build desktop software using pyqt and freeze it. I wonder how that entire toolchain looks like in this new way.
- jnwatson 8y agoSee https://docs.python.org/3.7/extending/embedding.html https://docs.python.org/3.7/extending/embedding.html. The idea behind embedding is you might have a Python shell in a larger app. But you can also use it to tightly control the execution of the interpreter.
- wiz21c 8y agoFTA : >>> On the surface, the application would more closely resemble what the platform expects, while behind various libraries, teams would have more flexibility to use their choice of programming language or tooling. I'm always fascinated by how the implementation of the core principles of an application is dictated by factors alien to it, such as OS, company organisation, etc. Therefore, the job of coding is often a small part compared to the amounts of trivialities, project management decisions, customer's ideas, corporate policies, etc. Although my soul is a coder's one, I always realize how much coding is just a small part of what I call application development.
- drej 8y agoHere's a nice overview of how Facebook migrated their codebase to Python 3. While it's different in nature (server side vs. client side), it's rather interesting. https://www.youtube.com/watch?v=H4SS9yVWJYA https://www.youtube.com/watch?v=H4SS9yVWJYA
- d__k 8y agoA guide to migrating to Python 3 (with pleasure :) https://github.com/arogozhnikov/python3_with_pleasure https://github.com/arogozhnikov/python3_with_pleasure
- peter_retief 8y agoI hope the people who told me that python wasn't good for large rollouts read this
- j7ake 8y agoAnyone curious to estimate the cost for Dropbox to migrate to Python 3?
- xchaotic 8y agoand also calculating the cost savings if any. On a technical level the whole exercise seems to lead to a slightly more elegant, consistent code base, but it's still a long way from earning or saving any actual money.