7 ms·
Async I/O for Python 3
- kzahel 14y agoDoes anyone have a link to the video? Or for various PyCon 2013 videos in general?
- wting 14y agoAFAIK, all PyCon 2013 talks were recorded and are being processed. Once they're finished the videos will be uploaded here: http://pyvideo.org/category/33/pycon-us-2013 http://pyvideo.org/category/33/pycon-us-2013
- pekk 14y agoStill puzzled why the primary inspiration for this is Twisted, which has some really ugly APIs.
- timc3 14y agoSome people stick with what they know. Twisted is horrible to work with though
- zubinmehta 14y agothat rhymed!
- ceronman 14y agoTwisted with inlineCallbacks is actually quite nice IMO. It's very similar to C# async/await.
- jlgreco 14y agoTwisted's inlineCallbacks singlehandedly turn Twisted in my mind from an abomination into something that is a joy to work with. In lei of a more hands off Erlang/Go approach, I am convinced that style is the only way to go.
- stock_toaster 14y agoinlineCallbacks made twisted palatable. It didn't make it good.
- lucian1900 14y agoTwisted has excellent APIs. Its Transport and Protocol abstractions in particular are extremely handy.
- timc3 14y agoIf Guido still worked at Google would this have been hosted on google docs and I would have been able to read it without Office?
- pekk 14y agoI am reading it in the browser on Linux, so I'm pretty sure Office isn't required
- timc3 14y agoDoesn't work on this ipad. And I hate using a laptop in the bath.
- jurre 14y agoI'm reading it in the browser in Chrome on OSX, you don't need office!
- benatkin 14y agoIt's the dark side of dogfooding.
- hosay123 14y agoDid you even click the link? Renders in the browser here.
- rdtsc 14y agoSigh, another Async framework. Yes it has nice features such can replace the reactor/hub thing. Has futures/promises/deferreds. That has all been done before in Twisted. Yields are cute and there was monocle, I wouldn't say it exactly took off : https://github.com/saucelabs/monocle https://github.com/saucelabs/monocle Twisted has inlineCallbacks that use yields as well. Just import Twisted into stdlib then and use that. I am surprised that gevent was dismissed. Ok, there is also eventlet, if someone doesn't like gevent. Monkey patching is scary? Is it really _that_ scary? Most sane and portable IO code probably runs on that today. Why? Because there is no need to create a parallel world of libraries. Write a test does it pass? Does it handle your use case? I'll take not knowing if my green threads switch tasks and add another green thread lock, instead of doubling my code size with yields, callbacks and futures. Let's talk about Twisted (sorry couldn't resist WAT video reference). I remember for years searching for parallel libraries to parse exotic protocols. Regular Python library is there, but no, can't use that, sorry. Gotta go find or write one that returns Deferreds. You add a single Twisted module in your code, good luck! -- it ripples all the way to the top through your API and you are doomed being locked into the Twisted world forever. When gevent and eventlet came around it was like a breath of fresh air. This is what sane concurrent IO looks like in Python: http://eventlet.net/doc/examples.html http://eventlet.net/doc/examples.html My fear is that many will just say fuck it, I'll just use Go/Rust/Erlang for IO bound concurrent problems. It is nice having a benevolent dictator, except when he goes a little crazy, then dictatorship doesn't sounds so much fun anymore.
- pjscott 14y agoYES. Thank you for saying this. Some of the async code that I maintain uses Twisted and some of it uses Eventlet, and the difference between them is night and day. The code using Eventlet is so much cleaner, so much easier to maintain, and (oddly enough) so much less magical than the Twisted stuff. This was written by the same people, and they're all really good programmers, so the obvious confounding variables are not an issue here. Eventlet and Gevent are just so much better. Worried about monkey-patching? Then only monkey-patch the parts you need to be asynchronous. Worried about magic that you don't understand? Have a look at the code; the magic is actually pretty straightforward after you've paid a little attention to the man behind the curtain. If you're interested in async stuff for Python, I urge you to have a look at Eventlet or Gevent.
- deleted 14y ago[deleted]
- CoffeeDregs 14y agoPerhaps the video makes more clear the rationale. E.g. Possible solution: "Standardizing gevent solves all its problems". One of the responses: "I like to write clean code from scratch". Another: "I really like clean interfaces". So I'd prefer that the BDFL work with the gevent folks to get it cleaned up and integrated while adjusting it to expose a "clean interface". Perhaps the whole thing will make more sense once Guido provides more detail, but I'm underwhelmed and confused.
- ekimekim 14y agoI find that unlikely: Guido doesn't like gevent. Though yes, it's a nice thought.
- fzzzy 14y agoGuido has been resisting the stackless stack slicing assembly technique since I first learned about Python and Stackless Python in 1999. That's obviously never going to change.
- rdtsc 14y agoThat reminds me of one of those famous Roman Emperors that all is well and good as well as they make rational decisions, then eventually they turn senile or mad, and everyone realizes how dictatorship is not that much fun sometimes.
- fzzzy 14y agoFrom a certain perspective it is a rational decision. Because the CPython API relies so heavily on the C stack, either some platform-specific assembly is required to slice up the C stack to implement green threads, or the entire CPython API would have to be redesigned to not keep the Python stack state on the C stack. Way back in the day [1] the proposal for merging Stackless into mainline Python involved removing Python's stack state from the C stack. However there are complications with calling from C extensions back into Python that ultimately killed this approach. After this Stackless evolved to be a much less modified fork of the Python codebase with a bit of platform specific assembly that performed "stack slicing". Basically when a coro starts, the contents of the stack pointer register are recorded, and when a coro wishes to switch, the slice of the stack from the recorded stack pointer value to the current stack pointer value is copied off onto the heap. The stack pointer is then adjusted back down to the saved value and another task can run in that same stack space, or a stack slice that was stored on the heap previously can be copied back onto the stack and the stack pointer adjusted so that the task resumes where it left off. Then around 2005 the Stackless stack slicing assembly was ported into a CPython extension as part of py.lib. This was known as greenlet. Unfortunately all the original codespeak.net py.lib pages are 404 now, but here's a blog post from around that time that talks about it [2]. Finally the relevant parts of greenlet were extracted from py.lib into a standalone greenlet module, and eventlet, gevent, et cetera grew up around this packaging of the Stackless stack slicing code. So you see, using the Stackless strategy in mainline python would have either required breaking a bunch of existing C extensions and placing limitations on how C extensions could call back into Python, or custom low level stack slicing assembly that has to be maintained for each processor architecture. CPython does not contain any assembly, only portable C, so using greenlet in core would mean that CPython itself would become less portable. Generators, on the other hand, get around the issue of CPython's dependence on the C stack by unwinding both the C and Python stack on yield. The C and Python stack state is lost, but a program counter state is kept so that the next time the generator is called, execution resumes in the middle of the function instead of the beginning. There are problems with this approach; the previous stack state is lost, so stack traces have less information in them; the entire call stack must be unwound back up to the main loop instead of a deeply nested call being able to switch without the callers being aware that the switch is happening; and special syntax (yield or yield from) must be explicitly used to call out a switch. But at least generators don't require breaking changes to the CPython API or non-portable stack slicing assembly. So maybe now you can see why Guido prefers it. Myself, I decided that the advantages of transparent stack switching and interoperability outweighed the disadvantages of relying on non-portable stack slicing assembly. However Guido just sees things in a different light, and I understand his perspective. [1] http://www.python.org/dev/peps/pep-0219/ [2] http://agiletesting.blogspot.com/2005/07/py-lib-gems-greenlets-and-pyxml.html
- judah 14y ago>> "@coroutine / yield-from are very close to async / await in C# 5" Cool to see languages learning from one another.
- benatkin 14y agoWhy does Guido think this is general purpose enough to add to Python but that the scientific features to make it competitive with R aren't? Is he envious of node.js?
- cdavid 14y agoThe scientific community is not that interested in merging into the stdlib. Also, the main point of this is to allow for different async libs to find some common ground to stop the madness of having twisted-specific, tornado-specific, etc... The scientific community does not have this pb because everybody uses numpy.
- xradionut 14y agoGuido lets Enthought, Continuum and programmers of that ilk take care of the science side of Python.
- Demiurge 14y agoI think because scientific features are not fundamental tools of expression, while he is working a language that is trying to be the foundation (most general) for the more specific libraries or tools.
- coldtea 14y agoThe scientific features are only of interest to the (drum roll) scientific community. Async interests potentially everybody, including the server guys, the backend guys AND the scientific community. Not to mention that this is a few contained classes, whereas the scientific stuff is tons and tons of code to be included into Python, including lots of Fortran and C, that would more than triple the size of the standard library. Lastly, node.js? Lots of languages have a good story for async, from C# and Scala, to Go and Rust...
- Locke1689 14y agoAs Guido mentions, @coroutine/yield from is very similar to C#'s async implementation (with some differences like type safety). Since Guido has the barest of descriptions on how this works, you may find the C# async description useful. [1] [1] http://msdn.microsoft.com/en-us/library/vstudio/hh191443.aspx http://msdn.microsoft.com/en-us/library/vstudio/hh191443.asp...
- johnsoft 14y agoJust to check if I'm understanding the presentation right, will the implementation involve compiler magic to turn this: @coroutine def getresp(): s = socket() yield from loop.sock_connect(s, host, port) yield from loop.sock_sendall(s, b'xyzzy') data = yield from loop.sock_recv(s, 100) # ... into this, similar to how C# does it? (let's pretend multi-line lambdas exist for a minute) def getresp(): s = socket() loop.sock_connect(s, host, port).add_done_callback(lambda: loop.sock_sendall(s, b'xyzzy').add_done_callback(lambda: data = loop.sock_recv(s, 100).add_done_callback(lambda: # ... ) ) ) Or will the `yield from`s bubble up all the way to the event loop and avoid the need for that?
- ufo 14y agoI don't understand your question. From the implementation perspective Python doesn't rewrite things to continuation-passing-style but the end result should be the same.
- masklinn 14y agoPython does not do AST-rewriting at compilation, `yield` and `yield from` will handle stack reification for freezing and thawing of coroutines. So chains of `yield`s and `yield from`s will bubble to the event loop.
- rdtsc 14y agoNo magic there. It is Eventlet and Gevent have that magic. Here is how that looks: def getresp(): s = socket() s.connect((host,port)) s.sendall(s,b'xyzzy') data = s.recv(s,100) Compare that to any of the above. This is what is thrown away in favor of 'yield from' and @coroutine mess coupled with a completely parallel set of IO libraries.
- NDizzle 14y agoThis powerpoint viewer would be much nicer to use if I could hit spacebar to skip down a page.
- chris_mahan 14y agodropbox.com is blocked at work. Anyone have an alternate link? I can see it on my cell phone, but generally pptx don't display well on 4 in screens.
- VeejayRampay 14y agoReading that presentation, it seems that Python has way too many Asyncronous I/O libraries/frameworks on its hands (not to be inflammatory though, I see it as a chance). I really wonder why that is not the case in Ruby. I mean there are some, but there's mostly confidential and there doesn't seem to be much interest around them. Especially not to the point that the project leader would take a stab at it. Good on Python anyway, competition is good.
- ekimekim 14y agoA minor gripe (seperate from all my other gripes, which other people have already talked about): "...run code in another thread - sometimes there is no alternative - eg. getaddrinfo(), database connections" Just thought I'd mention that async-supporting DNS libs do exist (eg. gevent ships with C-ares), and in particular I've used async postgres database connections in both C and gevent. The code to gevent-ise psycopg2 connections is about 10 or 15 lines, iirc.
- masklinn 14y ago> The code to gevent-ise psycopg2 connections is about 10 or 15 lines, iirc. Because psycopg2 has supported async OOTB since 2.2 by exposing a pollable socket: http://initd.org/psycopg/docs/advanced.html#asynchronous-support http://initd.org/psycopg/docs/advanced.html#asynchronous-sup... There are limitations though, as noted by the docs: COPY and LOs don't work.
- opminion 14y agoWait, Guido is proposing implementing INTERCAL'S COME FROM? (as yield from)?
- csears 14y agoFor anyone curious, INTERCAL was originally a joke language which included a COMEFROM instruction that acted like GOTO in reverse: http://en.wikipedia.org/wiki/COMEFROM http://en.wikipedia.org/wiki/COMEFROM Python's "yield from" hands off execution to a sub-generator: http://www.python.org/dev/peps/pep-0380/ http://www.python.org/dev/peps/pep-0380/
- opminion 14y agoThis was not meant to be a silly joke, but a serious statement. The use of capitals is warranted by the language's syntax.
- nixarn 14y agoI think it's a great idea. I haven't tried Twisted and having to install some 3rd party component to get it working doesn't sound tempting, however being supported by default, does.