[rabbitmq-discuss] Fwd: Best Practice - Implementing a Consumer (subscriber)

Thu Jan 13 19:54:59 GMT 2011

Hi, Jerry...

I apologize that you didn't get a response before today.  Many of us were a bit shut down over the winter holiday period.

> The notification broker is subscribed to a queue, reads messages off of the 
> queue and does work to determine how the message content is formatted (based 
> on message type), who the message should be routed to (a person) and by what 
> means the message is routed (email, sms, tweet, etc).
> 
> Ideally, to provide some scalability, several instances of the broker would 
> be running in parallel in some sort of load balanced manner, all reading off 
> of the same queue.

This is a totally reasonable sounding scenario.  Marek Majkowski of the Rabbit team has written a nice tutorial that sketches how a core, Rabbity part of this system might be built:

http://www.rabbitmq.com/tutorial-two-python.html

For the system you describe it seems that you'd want to run multiple instances of your broker/dispatcher entity all subscribed to the same Rabbit queue into which messages are being published.  If you follow the pattern described in the tutorial you'll get a round-robin distribution of work items across your own brokers.

That will give you the ability you desire to handle greater volume on your application's end, and will also let your application continue to get work done even if some instances of your dispatcher/broker app, or the hosts they're running on, go down.

Also, note in particular the discussion of "message acknowledgment" in the tutorial above.  Following those guidelines will protect you against losing work items if your application receives a work item out of a Rabbit queue and then meets some bad fate before finishing its task.  By using the ack mechanism as described, Rabbit will maintain responsibility for keeping your message until such time as you've certified that you've successfully done the work entailed by the message.

> The broker would be running on a pool of application servers or IIS servers. 
> either or, but not both.
> 
> What is the recommend implementation of the broker? A window's service, an 
> application running under IIS, some other WCF/WAS implementation?

On that issue, Rabbit is agnostic.  I'd suggest that you implement your application side creatures in whatever form you find easiest to start, maintain, debug and monitor the health of...  Windows services that run at system boot time might be entirely reasonable for your scenario.  To a large extent it probably depends on what your ops people and developers find most comfortable and cost effective in practice.

Best regards,
Jerry