6 ms·

Zed Shaw's Advanced Network Architectures With ZeroMQ at Pycon 2011 [video]

75 points by atarashi 15y ago

JoachimSchipper 15y ago
Is there some text available somewhere (slides, perhaps)?
xtacy 15y ago
I guess it would've been better if the title were "Advanced Network Communication Architectures (or Patterns)" with ZeroMQ. Network Architectures sounds more hardware-ish and (at least to me) sounds like it's talking about the actual network topology... :-)
antidaily 15y ago
Is it Zed Shaw week on HN or something?
- pstuart 15y ago
  A submission like this is exactly what HN is about.
  - deleted 15y ago
    [deleted]
haberman 15y ago
Can't watch the video right away, but I'll just say that every time I read the ZeroMQ guide (http://zguide.zeromq.org/page:all http://zguide.zeromq.org/page:all) I am frustrated by the lack of any clear technical explanation of ZeroMQ's basic architecture. Today I gave it another shot, and after half an hour of trying to glean technical detail and skipping any parts that read like a like a comic book ("zap-pow-kaboom satori paradigm-shift moment"), I think I've learned the following: * ZMQ works by spawning a background thread that runs a poll() (or equivalent)-based async I/O loop. This thread is created when you create a ZMQ "context." * On top of a context you can create a ZMQ "socket", which may map to N underlying UNIX sockets (or a comparable transport), and which will transparently queue data to slow or unavailable receivers. * Over a socket you can send or receive "messages", which are length-delimited strings which are always delivered in full with the original length. Send/receive can be either blocking or non-blocking. * Contexts can be shared across threads, but sockets are not thread-safe. Communication between application threads and the ZMQ I/O thread is via a lock-free queue. The main features seem to be topology-agnostic programming (since you don't have to know what a socket is connected to to send/receive over it) and transparent queuing. In my opinion transparent queues can be problematic because they transparently use up memory that can be hard to account for. Topology-agnostic programming certainly seems interesting, but in my experience of distributed systems programming I never seem to need or want complex messaging topologies. I guess I don't really get what all the hype is about. Maybe I should watch the video.
- dochtman 15y ago
  I started using ZeroMQ at work a few weeks ago after reading the guide. The part that I like about ZeroMQ is that it's a simplifying abstraction, e.g. it's a fairly simple wrapper about sockets that makes life easier for me. I guess it's mostly like a library of patterns for socket usage, so it makes a request-response loop and pub-sub scenarios really easy to implement. The other thing that's great is that it's pretty much programming language agnostic, meaning that C++ code and Python code look fairly similar and can trivially be used together. As for the transparent queueing, there are simple options to control the size of the queue; it's even possible to spool the queue to disk as soon as the memory limit is reached (see setsockopt docs and look for HWM, for high water mark).
  - haberman 15y ago
    All of your reasons make sense to me, it's just that all the advocacy docs set ZMQ up to be this game-changing system. I can totally buy "a fairly simple wrapper about sockets that makes life easier for me." "world-saving superheros of the networking world" not so much.
    - dochtman 15y ago
      I don't know, the simple building blocks it provides can be combined in a bunch of ways that are really pretty cool. I think apenwarr would might call it a "simplifying assumption". It's hardly ever the complex things that are game-changing, right?
- tptacek 15y ago
  "Topology-agnostic programming" has been the norm on Wall Street for over a decade now; Tibco calls it "subject based" networking, I think? I also think it's kind of a misfeature, and that in the real world it tends to devolve to "hub and spoke client/server with lots more failure modes".
- ominous_prime 15y ago
  I don't think there's much to gain (at least right now) by using zmq in a simple single-client -> server setup. I read the testimonials, which got me excited, so I then tried to make use of zmq in a system I was working on. Using plain old sockets, I could whip up a connection in a few lines (OK, python helps here), and deal with all failure modes associated with them. Mapping this robust setup to zmq, required multiple layers, async requests, and a polling mechanism. I don't recall now, but I may have had to throw a router in there as well. I'm not dissing zmq, as it's a great technology, and solves some complex problems well, but it's not for everyone, and people without complex problems will more likely be confused by it than helped. The documentation is OK, but I think tries to abstract too much of what's going on, and just leaves you with a bunch of less-than-descriptive names for things that you have to take for granted. All the mamas, papas, and dealers analogies didn't help me at all.
runjake 15y ago
Previously (with comments): http://news.ycombinator.com/item?id=2319875 http://news.ycombinator.com/item?id=2319875

75 points by atarashi 15y ago

Is there some text available somewhere (slides, perhaps)?

xtacy 15y ago

I guess it would've been better if the title were "Advanced Network Communication Architectures (or Patterns)" with ZeroMQ. Network Architectures sounds more hardware-ish and (at least to me) sounds like it's talking about the actual network topology... :-)

antidaily 15y ago

Is it Zed Shaw week on HN or something?

pstuart 15y ago
A submission like this is exactly what HN is about.
- deleted 15y ago
  [deleted]

haberman 15y ago

Can't watch the video right away, but I'll just say that every time I read the ZeroMQ guide (http://zguide.zeromq.org/page:all http://zguide.zeromq.org/page:all) I am frustrated by the lack of any clear technical explanation of ZeroMQ's basic architecture. Today I gave it another shot, and after half an hour of trying to glean technical detail and skipping any parts that read like a like a comic book ("zap-pow-kaboom satori paradigm-shift moment"), I think I've learned the following: * ZMQ works by spawning a background thread that runs a poll() (or equivalent)-based async I/O loop. This thread is created when you create a ZMQ "context." * On top of a context you can create a ZMQ "socket", which may map to N underlying UNIX sockets (or a comparable transport), and which will transparently queue data to slow or unavailable receivers. * Over a socket you can send or receive "messages", which are length-delimited strings which are always delivered in full with the original length. Send/receive can be either blocking or non-blocking. * Contexts can be shared across threads, but sockets are not thread-safe. Communication between application threads and the ZMQ I/O thread is via a lock-free queue. The main features seem to be topology-agnostic programming (since you don't have to know what a socket is connected to to send/receive over it) and transparent queuing. In my opinion transparent queues can be problematic because they transparently use up memory that can be hard to account for. Topology-agnostic programming certainly seems interesting, but in my experience of distributed systems programming I never seem to need or want complex messaging topologies. I guess I don't really get what all the hype is about. Maybe I should watch the video.

dochtman 15y ago
I started using ZeroMQ at work a few weeks ago after reading the guide. The part that I like about ZeroMQ is that it's a simplifying abstraction, e.g. it's a fairly simple wrapper about sockets that makes life easier for me. I guess it's mostly like a library of patterns for socket usage, so it makes a request-response loop and pub-sub scenarios really easy to implement. The other thing that's great is that it's pretty much programming language agnostic, meaning that C++ code and Python code look fairly similar and can trivially be used together. As for the transparent queueing, there are simple options to control the size of the queue; it's even possible to spool the queue to disk as soon as the memory limit is reached (see setsockopt docs and look for HWM, for high water mark).
- haberman 15y ago
  All of your reasons make sense to me, it's just that all the advocacy docs set ZMQ up to be this game-changing system. I can totally buy "a fairly simple wrapper about sockets that makes life easier for me." "world-saving superheros of the networking world" not so much.
  - dochtman 15y ago
    I don't know, the simple building blocks it provides can be combined in a bunch of ways that are really pretty cool. I think apenwarr would might call it a "simplifying assumption". It's hardly ever the complex things that are game-changing, right?
tptacek 15y ago
"Topology-agnostic programming" has been the norm on Wall Street for over a decade now; Tibco calls it "subject based" networking, I think? I also think it's kind of a misfeature, and that in the real world it tends to devolve to "hub and spoke client/server with lots more failure modes".
ominous_prime 15y ago
I don't think there's much to gain (at least right now) by using zmq in a simple single-client -> server setup. I read the testimonials, which got me excited, so I then tried to make use of zmq in a system I was working on. Using plain old sockets, I could whip up a connection in a few lines (OK, python helps here), and deal with all failure modes associated with them. Mapping this robust setup to zmq, required multiple layers, async requests, and a polling mechanism. I don't recall now, but I may have had to throw a router in there as well. I'm not dissing zmq, as it's a great technology, and solves some complex problems well, but it's not for everyone, and people without complex problems will more likely be confused by it than helped. The documentation is OK, but I think tries to abstract too much of what's going on, and just leaves you with a bunch of less-than-descriptive names for things that you have to take for granted. All the mamas, papas, and dealers analogies didn't help me at all.

runjake 15y ago

Previously (with comments): http://news.ycombinator.com/item?id=2319875 http://news.ycombinator.com/item?id=2319875