[rabbitmq-discuss] RabbitMQ-based vcloud session clustering

[Jon Brisbin]

On Apr 7, 2010, at 4:55 PM, Alexis Richardson wrote:

>>> But perhaps the VMware product suite could be helpful too....
>> 
>> We're using the free stuff, so we don't get much help there. :) My original Nirvana was to have an environment that shrunk and grew based on user demand. When more resources were needed, the system could start additional VMs. That's still sort of my goal, but I've softened on the dynamism, which is still not well-supported in most third-party applications we use. Apache, for instance, is a beast that is anything but cloud-friendly. When my application servers start and stop, I have to update a central "master" node and a cron job keeps the ProxyMember directives up-to-date and restarts Apache if it sees membership change. RabbitMQ needs similar hand-holding because it doesn't seem well-equipped to exist within a dynamic and complex environment where it is only a citizen. I'm tackling this in my vcloud tomcat session manager by having each node keep a list of all active sessions and the node they were on last.
> 
> This is fascinating.  Are you saying, for example, that you would like
> RabbitMQ to deal with group membership and discovery in a more
> 'elastic' way?  What about things like this:
> http://cacm.acm.org/blogs/blog-cacm/83396-errors-in-database-systems-eventual-consistency-and-the-cap-theorem/fulltext

I think elasticity is absolutely essential to new software that has any desire to operate inside a "cloud" (however loosely-defined) environment. 

There's no easy answer, as that blog post pointed out. I'm trying to address issues like that myself by keeping an internal membership list that are pointers to the real data. If one were to apply this to RabbitMQ, it would look something like:

1. Broker starts and registers itself with central service registry (I'm using dynamic DNS, so as soon as RabbitMQ starts, it's accessible via internal hostname mq.cloud.npci.com).
2. Broker attaches to another random broker and issues an "add me and give me the current state" message to a fanout queue (which one depends on what "group" its in).
3. Broker receives list of currently active brokers/consumers, which now includes itself.
4. Client connects to random broker and publishes a message.
5. Broker delivers message to local consumers, if any exist, and checks the membership list to see if any other brokers list consumers for that queue; if so, it forwards the message to that broker (maybe dropping it from the list if its inaccessible), who maybe repeats the process (if membership has changed).

This is basically what I'm doing with sessions. Each node knows all the available sessions throughout the cloud and where the actual object resides. If it doesn't see a session locally, but it shows up in the list of cloud sessions, it tries to load the session from the node listed in the membership map. My idea is to have one of the replicators (who are eavesdropping) always respond, in addition to the node, so that a response is always received by the requestor, even if the node has gone down before every other node realizes it and has had a chance to update membership.

Jon Brisbin
Portal Webmaster
NPC International, Inc.