[rabbitmq-discuss] RabbitMQ-based vcloud session clustering

[Jon Brisbin]

On Apr 9, 2010, at 6:21 AM, Alexis Richardson wrote:

> Jon
> 
> Sorry to topline you, but I want to show you something similar to what
> you describe.  Let's compare notes.
> 
> http://www.oceanobservatories.org/spaces/download/attachments/7045394/AMQP-ScalableFederation_OOI-PrototypeReport-20100105.pdf
> 
> This describes work by the RabbitMQ team to enable something a bit
> like your (1-5) below.  What do you think?
> 

I'm glad to hear about 0-9-1 compatibility, for sure! We've been fighting this 0-8/0-9 thing with our C++ developers for a little bit now, trying to get something that will work for them (they're still using VS 2003). I'm going to try downloading the source and setting up an 0-9-1 server for them to play around with.

As for the cloud stuff...it sounds like you're farther down the road than I am. :) Since I'm approaching this operationally instead of academically (or even strategically ;), I don't know that I could help much with the theory. I was thinking out load and basically transliterating what I'm doing in other areas to RabbitMQ (minus the store-and-forward stuff...that was improv ;). This definitely puts some meat on the bones of the problems I'm facing in implementing applications that use these federated services.

But in the end, I'm going to have to write my own software to approximate these more long-term solutions because of the demands of an operational, production environment. I'm working on a blog post to try and lay out my thinking on this. I think I need a couple more cups of coffee to properly digest this... :)

Jon Brisbin
Portal Webmaster
NPC International, Inc.



> alexis
> 
> 
> On Thu, Apr 8, 2010 at 3:30 PM, Jon Brisbin
> <jon.brisbin at npcinternational.com> wrote:
>> 
>> On Apr 7, 2010, at 4:55 PM, Alexis Richardson wrote:
>> 
>>>>> But perhaps the VMware product suite could be helpful too....
>>>> 
>>>> We're using the free stuff, so we don't get much help there. :) My original Nirvana was to have an environment that shrunk and grew based on user demand. When more resources were needed, the system could start additional VMs. That's still sort of my goal, but I've softened on the dynamism, which is still not well-supported in most third-party applications we use. Apache, for instance, is a beast that is anything but cloud-friendly. When my application servers start and stop, I have to update a central "master" node and a cron job keeps the ProxyMember directives up-to-date and restarts Apache if it sees membership change. RabbitMQ needs similar hand-holding because it doesn't seem well-equipped to exist within a dynamic and complex environment where it is only a citizen. I'm tackling this in my vcloud tomcat session manager by having each node keep a list of all active sessions and the node they were on last.
>>> 
>>> This is fascinating.  Are you saying, for example, that you would like
>>> RabbitMQ to deal with group membership and discovery in a more
>>> 'elastic' way?  What about things like this:
>>> http://cacm.acm.org/blogs/blog-cacm/83396-errors-in-database-systems-eventual-consistency-and-the-cap-theorem/fulltext
>> 
>> I think elasticity is absolutely essential to new software that has any desire to operate inside a "cloud" (however loosely-defined) environment.
>> 
>> There's no easy answer, as that blog post pointed out. I'm trying to address issues like that myself by keeping an internal membership list that are pointers to the real data. If one were to apply this to RabbitMQ, it would look something like:
>> 
>> 1. Broker starts and registers itself with central service registry (I'm using dynamic DNS, so as soon as RabbitMQ starts, it's accessible via internal hostname mq.cloud.npci.com).
>> 2. Broker attaches to another random broker and issues an "add me and give me the current state" message to a fanout queue (which one depends on what "group" its in).
>> 3. Broker receives list of currently active brokers/consumers, which now includes itself.
>> 4. Client connects to random broker and publishes a message.
>> 5. Broker delivers message to local consumers, if any exist, and checks the membership list to see if any other brokers list consumers for that queue; if so, it forwards the message to that broker (maybe dropping it from the list if its inaccessible), who maybe repeats the process (if membership has changed).
>> 
>> This is basically what I'm doing with sessions. Each node knows all the available sessions throughout the cloud and where the actual object resides. If it doesn't see a session locally, but it shows up in the list of cloud sessions, it tries to load the session from the node listed in the membership map. My idea is to have one of the replicators (who are eavesdropping) always respond, in addition to the node, so that a response is always received by the requestor, even if the node has gone down before every other node realizes it and has had a chance to update membership.
>> 
>> Jon Brisbin
>> Portal Webmaster
>> NPC International, Inc.
>>