[rabbitmq-discuss] Can I get your input for a RabbitMQ cookbook?

Tue Feb 15 15:31:16 GMT 2011

On Mon, Feb 14, 2011 at 7:12 AM, Jonathan Brisbin <jbrisbin at vmware.com>wrote:

> Would anyone like to express an interest in a particular topic for a
> potential RabbitMQ cookbook project I might be working on? We're
> trying to come up with an outline and I want to make sure it's
> targeted to what people actually want to read about, so if you've got
> an idea about some topic you'd like to see covered, please let me know
> off list.
>
> Some of the topics will be:
>
> Interoperability (STOMP, Celery, etc...)
> Performance tuning
> Plugin authoring
> Configuration
> [Put your topic here]
>
>
These are a few of the topics that have come up at my company while
evaluating RabbitMQ -- some of which I've asked about on this list.  I'm not
sure if all would fit into "cookbook" type of examples.  Most are about how
to operate RabbitMQ in a production environment.

   - High Availability:  How to set up RabbitMQ such that a server failure
   will not disrupt service.
   For example, HAProxy or hardware load balancer.

   - Client connection management: Ensuring that clients gracefully handle
   connection failures.

   - Message durability: Scenarios when/why queue durability is appropriate.

   - Message replication:  When/why/how to replicate messages across brokers
   (e.g. Pacemaker example).
   And if messages are replicated how to prevent multiple consumers from
   processing the same message.

   - Clustering: As others have noted, how to scale RabbitMQ.
   How to add new servers into an active cluster and how to remove a server
   (and dealing with queues on the server that is removed).

As you might note, uptime is important to our company and those are all
about keeping RabbitMQ up and available. Oddly, at our company the message
durability/replication issue consumed the most discussion time -- that is,
making sure that a message produced is always processed[1].  (I think a
better/cleaner approach is to design with message loss in mind and when
needed, producers will re-send a message.)

   - Topic routing: This is covered in the tutorial, but it would be great
   to see some cookbook examples that demonstrate a few real-world scenarios.
   The power of topic routing was a feature that was difficult for some to
   grasp at our company.  Aggregating logs and tracking specific log types
   (e.g. error watchdog) could be one example.

   - Debugging: When things go wrong (and they always do) steps to figure
   out why RabbitMQ is not working, what happened, and how to fix.
   For example: figuring out if Mnesia DB is corrupt and need to remove.
    This is all about the 2am IT on-call person trying to solve a problem by
   following a "play book" of steps to get things running again.

   - Management of consumers: monitoring queue sizes, allocating resources
   to workers, tools available to monitor consumers across multiple machines,
   threaded consumers vs. forking model.

   - RPC in a web environment: This may be outside the scope for RabbitMQ,
   but our use cases have involved the need for RPC style of calls, but in a
   web-environment this is often meant polling by the web client (e.g. via
   AJAX) because it is not desirable to have a web process wait on a RPC
   response.  But, in a polling web environment you cannot rely on the message
   ever being consumed.  What I've done is passed the unique return-queue name
   in the AJAX request and used auto_delete and x-expires on the temporary
   queue to have it removed if the web client fails to fetch the response.
   I'm not convinced that is the best approach.

[1] Message durability was considered so important by some at our company
that we now send all messages to a single consumer that stores them in
Postgresql with Slony replication before being passed onto consumers.  I
would not recommend that approach -- and indeed with experience we've seen
messages that are that critical need to be tracked by the application's
database anyway so making the messages durable adds little benefit.

-- 
Bill Moseley
moseley at hank.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110215/72c5216d/attachment.htm>