[rabbitmq-discuss] RabbitKernel

Tue Sep 7 20:33:56 BST 2010

Thanks for the excellent thoughts here! :) Comments below...

On Sep 7, 2010, at 2:02 PM, darrencruse wrote:

> a.  Are you familiar with the new NKP protocol?  Sounds like you should at
> least take a look if not - it's very much about cloud computing using
> NetKernel.  They've built a high speed protocol using Apache Etch for
> allowing multiple NetKernel's to work together.  
> 
> https://cwiki.apache.org/ETCH/home.html
> 
> And they're latest downloads include an example doing a pub/sub model using
> NetKernel with the NKP Protocol.

This sounds very good, indeed! I'll be investigating this more right away.

> b.  But what you said about looking at the JMS source as a reference for
> your RabbitMQ Transport was what I was going to suggest you beat me to it.
> :)

Haven't actually written any code on this yet. Too many other things demanding my attention. 

I'm reworking my cloud job scheduler using Spring AMQP right now, so it might actually be a perfect time to revisit this in earnest and see if I can make something work. If you have a need for something similar, I love to collaborate. :)

> c.  I didn't know enough to know is there definitely not a JMS Provider for
> RabbitMQ (probably a dumb question from a brief google it sounds like maybe
> there's not).  If there was I wondering if in fact you could just use
> NetKernel's JMS Transport with RabbitMQ.

IMHO it's just as well there isn't a JMS provider for RabbitMQ. ;)

I like AMQP because it's truly cross-platform. I can write producers in Python and consumers in Java. Or Ruby. Or Erlang. :) 

> d.  From your arch drawing I was surprised you were talking about Scala
> Actors - why wouldn't you just run NetKernel on both JVMs?  i.e. The
> single-within-a JVM scala actor model seems redundant with NetKernel's
> dymamic request resolution between endpoints within a single NetKernel, and
> distributed actors likewise seem redundant with using your RabbitMQ
> Transport and possibly using asynchronous requests.  It seems to me the real
> beauty whether using NetKernel or actors would be the elegance of choosing
> one approach and using that consistently throughout the architecture - it
> wasn't clear to me why you'd want to mix both.  But I'm probably just not
> following what you were after - are you integrating existing systems this
> way that already use scala actors or something?

I guess that's a little confusing because I don't fully know what I want! :) 

The thing that prompted my looking into NetKernel was the frustration with having multiple copies of all the connections, channels, consumers, etc... in a Tomcat session manager I wrote for private clouds. Since I run multiple instances of Tomcat/tcServer per virtual machine (VMware virtual machine in this case), it seemed silly to me to be running, on a single "box", several instances of my session manager (one for each webapp) times the number of tcServer instances I'm running (usually 3) when all I need on that Linux box is a single manager for messages and a single cache that can service multiple tcServer instances without all the resource usage involved in duplicating everything. The better use I can make of those small VMs, the more traffic I can funnel through them. I need a way to communicate from a client in a session manager running in a Tomcat VM to the AMQP manager that's doing the message sending and session object mangling. Scala actors seemed to make sense here because they're friendly to the kind of cloud-scale distributed computing I'm trying to do but are still Java compatible. I wasn't sure I could get the same kind of ease of component communication by embedding NetKernel inside each Tomcat (again, I was concerned about unnecessary duplication of resources, whether it's NetKernel or RabbitMQ client stuff) considering a session manager is instantiated per webapp.

> e.  Likewise I noticed you highlighted the use of cron on the one box, but
> NetKernel has nice cron like capabilites based on quartz.  So again it seems
> like there'd be more synergy by using NetKernel for those needs too(?)

I'd prefer quartz, actually. I'd gladly stay within the context of NetKernel if I could. 

In general I use the term "cron" to refer to a scheduled, recurring job of some kind rather than the UNIX "cron" specifically. :)

> Just some thoughts fwiw (again I'm not a RabbitMQ expert and I didn't look
> closely at your thread so I may be missing some things in what you're
> after),

I really appreciate your thoughts! :) You've inspired me to get back in there and check out the NKP stuff and see if that will be a good fit. I've kind of hit a brick wall on another project I'm working on, so I think I'll get back into the job scheduler code today.

Thanks!

J. Brisbin
http://jbrisbin.com/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20100907/763c9705/attachment-0001.htm>