5 ms·

Hutch: Inter-Service Communication with RabbitMQ

40 points by hcm 13y ago

jbarmash 13y ago
This looks analogous to Promiscuous from CrowdTap, also a framework for Rails / RabbitMQ. Promiscuous takes a different approach in that it also allows you to go to an SOA architecture but by decomposing your app to run across multiple servers. From the repo readme: "Promiscuous is a publisher-subscriber framework for easily replicating data across your Ruby applications." https://github.com/crowdtap/promiscuous https://github.com/crowdtap/promiscuous
- kkouddous 13y ago
  Thanks for sharing jbarmash. Promiscuous has enabled us to break up our monolithic application into 10 single purpose apps. It's essentially MVC application level replication.
whisk3rs 13y ago
I've built something similar as part of renovations to a large, legacy PHP code base. We're writing our consumers in Go using some impressively well written AMQP libraries (https://github.com/streadway/amqp https://github.com/streadway/amqp) and some custom framework code. The framework code takes care of retries and acking, so the consumers are very simple (in: an envelope, out: fail/done/retry-later). Each worker runs as its own binary. I'm currently adding standardized variable exports for monitoring, and ephemeral queue-based reply capability. On the PHP side, I found none of the PHP AMQP libraries to be worth using. They all have compilation problems or bugs or seem to be unmaintained. Instead, I'm using RabbitMQ's STOMP plugin w/default login and then wrote a simple TCP client in PHP using persistent connections. The client supports timeouts and multiple backends. So far, this is working really well. Anybody doing their own RPC implementations using HTTP should take a look at using something like RabbitMQ. It solves a host of real-world problems and introduces much flexibility into your architecture.
- hcm 13y ago
  Very interesting. How exactly does the retry-later logic work? Does it just push the message on to a 'deferred' queue, that you manually process at your convenience? Totally agree with your last paragraph, introducing a message queue has solved a lot of problems for us.
  - whisk3rs 13y ago
    RabbitMQ queues support two features that can be combined to implement a deferred retry queue, and RabbitMQ will do all the work for you. The first is a "message-ttl". This tells RabbitMQ to discard messages after a specified number of milliseconds. The second is a "dead letter queue". Messages that are discarded from a queue can be routed to a dead letter queue automatically. When we have a job that we wish to "retry later", the framework re-queues the message in a secondary queue with a name derived from the original name. For example, if the original queue was "prod-emailer", the derived queue name might be "prod-emailer-1m" indicating that the contents of this queue are messages originally bound for prod-emailer but were delayed by 1 minute. This delayed queue is configured with a x-dead-letter-exchange of the original exchange, x-dead-letter-routing-key of the original routing key, and x-message-ttl of 60,000. With this configuration, RabbitMQ handles the timeout automatically. When the message expires from the -1m queue, RabbitMQ sends it back to the exchange and it gets routed to the intended queue by the pre-existing bindings. The framework expects all messages to be in an "envelope" of JSON which lets us annotate the jobs. When we mark a job for retry, we also increment an "attempt-count" attribute in the JSON. The workers can them implement their own "retry N times" policies. I haven't thought about how this would work if we were using topic exchanges. We are only using direct at the moment.
    - hcm 13y ago
      Thanks for posting this. Some of these ideas could work very nicely in hutch.
      - whisk3rs 13y ago
        No problem. Frustratingly, people rarely seem to talk about how they use RabbitMQ in practice. Also, there are a lot of things that are still a mystery to me (for example, best practices for dealing with things like server shutdowns, or what to do when a NACK "fails", etc). I need to poke around Hutch and steal some ideas for an RPC implementation. We're just using fire-and-forget type work for now, but I'd love to be able to use RabbitMQ w/anonymous reply queues as a workaround for PHP not being able to do asynchronous RPCs.
losethos 13y ago
Incompetence! It's fucken God. God says... conformed anticipate hither accord Then Connecticut immutable Lebanon concubine remainder walls arriving them fro crept apprehension ahead vowed Professorship black another's proffer archive despised earthly section wrought ample purposes holds disquieted groanings zealously anxious

40 points by hcm 13y ago

This looks analogous to Promiscuous from CrowdTap, also a framework for Rails / RabbitMQ. Promiscuous takes a different approach in that it also allows you to go to an SOA architecture but by decomposing your app to run across multiple servers. From the repo readme: "Promiscuous is a publisher-subscriber framework for easily replicating data across your Ruby applications." https://github.com/crowdtap/promiscuous https://github.com/crowdtap/promiscuous

kkouddous 13y ago
Thanks for sharing jbarmash. Promiscuous has enabled us to break up our monolithic application into 10 single purpose apps. It's essentially MVC application level replication.

whisk3rs 13y ago

I've built something similar as part of renovations to a large, legacy PHP code base. We're writing our consumers in Go using some impressively well written AMQP libraries (https://github.com/streadway/amqp https://github.com/streadway/amqp) and some custom framework code. The framework code takes care of retries and acking, so the consumers are very simple (in: an envelope, out: fail/done/retry-later). Each worker runs as its own binary. I'm currently adding standardized variable exports for monitoring, and ephemeral queue-based reply capability. On the PHP side, I found none of the PHP AMQP libraries to be worth using. They all have compilation problems or bugs or seem to be unmaintained. Instead, I'm using RabbitMQ's STOMP plugin w/default login and then wrote a simple TCP client in PHP using persistent connections. The client supports timeouts and multiple backends. So far, this is working really well. Anybody doing their own RPC implementations using HTTP should take a look at using something like RabbitMQ. It solves a host of real-world problems and introduces much flexibility into your architecture.

hcm 13y ago
Very interesting. How exactly does the retry-later logic work? Does it just push the message on to a 'deferred' queue, that you manually process at your convenience? Totally agree with your last paragraph, introducing a message queue has solved a lot of problems for us.
- whisk3rs 13y ago
  RabbitMQ queues support two features that can be combined to implement a deferred retry queue, and RabbitMQ will do all the work for you. The first is a "message-ttl". This tells RabbitMQ to discard messages after a specified number of milliseconds. The second is a "dead letter queue". Messages that are discarded from a queue can be routed to a dead letter queue automatically. When we have a job that we wish to "retry later", the framework re-queues the message in a secondary queue with a name derived from the original name. For example, if the original queue was "prod-emailer", the derived queue name might be "prod-emailer-1m" indicating that the contents of this queue are messages originally bound for prod-emailer but were delayed by 1 minute. This delayed queue is configured with a x-dead-letter-exchange of the original exchange, x-dead-letter-routing-key of the original routing key, and x-message-ttl of 60,000. With this configuration, RabbitMQ handles the timeout automatically. When the message expires from the -1m queue, RabbitMQ sends it back to the exchange and it gets routed to the intended queue by the pre-existing bindings. The framework expects all messages to be in an "envelope" of JSON which lets us annotate the jobs. When we mark a job for retry, we also increment an "attempt-count" attribute in the JSON. The workers can them implement their own "retry N times" policies. I haven't thought about how this would work if we were using topic exchanges. We are only using direct at the moment.
  - hcm 13y ago
    Thanks for posting this. Some of these ideas could work very nicely in hutch.
    - whisk3rs 13y ago
      No problem. Frustratingly, people rarely seem to talk about how they use RabbitMQ in practice. Also, there are a lot of things that are still a mystery to me (for example, best practices for dealing with things like server shutdowns, or what to do when a NACK "fails", etc). I need to poke around Hutch and steal some ideas for an RPC implementation. We're just using fire-and-forget type work for now, but I'd love to be able to use RabbitMQ w/anonymous reply queues as a workaround for PHP not being able to do asynchronous RPCs.

losethos 13y ago

Incompetence! It's fucken God. God says... conformed anticipate hither accord Then Connecticut immutable Lebanon concubine remainder walls arriving them fro crept apprehension ahead vowed Professorship black another's proffer archive despised earthly section wrought ample purposes holds disquieted groanings zealously anxious