[rabbitmq-discuss] RabbitKernel

[Jon Brisbin]

I would basically be implementing an AMQP version of the JMS stuff:

http://docs.netkernel.org/book/view/book:mod:reference/doc:tpt:jms:accessor:service

...but with consumers added, instead of the one-way street the JMS sender is.

Jon Brisbin
Portal Webmaster
NPC International, Inc.



On Jul 30, 2010, at 3:09 PM, Jon Brisbin wrote:

> I've attached a PDF of a 5-minute diagram that shows some use cases for me.
> 
> On the right:
> 
> Top is a tcServer instance that needs to load an object from the distributed cache.
> 
> Middle is a cron job that needs access to modules running in NetKernel on that machine for maintenance functions.
> 
> Bottom is the AS/400 that needs to update the PostgreSQL server at a store as the result of a database trigger.
> 
> Flow for tcServer code requesting object from distributed cache:
> 
> 1. Use Scala/Akka (or some IPC mechanism) to block on a remote Actor which checks local cache module but doesn't find the object. It shells out to the AMQP module, which is solely responsible for sending and receiving messages. 
> 
> 2. Sends a message to the cloud, asking for the object.
> 
> 3. Remote AMQP module gets a cache request message and dispatches it to the cache module, which returns an object. The remote AMQP module then replies to the requestor, handing back the Protobuf'd object.
> 
> 4. Requesting AMQP module maybe stores object in local cache and hands object back to Akka module, which replies to the tcServer code that originally requested the object.
> 
> Flow for AS/400 wanting to update a store:
> 
> 1. AS/400 trigger fires, dumping a message into RabbitMQ.
> 
> 2. AMQP module sees it and delegates to Store Updater Module.
> 
> 3. Store Updater Module sees this is a fire-and-forget update, so returns immediately and proceeds to update a store database in Birmingham, Alabama (or Portland, Oregon for that matter) asynchronously.
> 
> Jon Brisbin
> Portal Webmaster
> NPC International, Inc.
> 
> <NetKernelDiagram.pdf>
> On Jul 30, 2010, at 1:34 PM, Alexis Richardson wrote:
> 
>> Jon
>> 
>> Interesting indeed!  At the risk of being both laconic and glib... do
>> you have a diagram showing what you propose to achieve?  I think that
>> would bring this to life...
>> 
>> alexis
>> 
>> On Fri, Jul 30, 2010 at 5:01 PM, Jon Brisbin
>> <jon.brisbin at npcinternational.com> wrote:
>>> To expand some (and for the benefit of the list), I guess what I'm thinking is related to a problem I see on the horizon as I build more utilities that have at their foundation AMQP messaging. Each component needs to talk to the other components running on different VMs within the cloud on different exchanges/queues and each has their own system of subscribing consumers and producing messages. This means: if I have three instances of tcServer running on a VM and each webapp is using the cloud session manager and and the server is using the JMX invoker, I've got four different ConnectionFactories, four sets of subscribers, four sets of producers (times the number of workers I have configured; I'll likely have 40 or more of these per VM). This doesn't seem terribly scalable to me, so I'm hunting for a way to use messaging on a VM for communicating to various components (that are likely unrelated) in a more reasonable way.
>>> 
>>> OSGi has always appealed to me because of module versioning, live reloading, etc... and the ROC approach that NetKernel takes to abstracting OSGi seems to offer (in my initial estimation at least) the ability to de-duplicate the various ways I'm managing connections, channels, subscribers, and producers (which is to say: each time I write a module, I try and improve it; the side-effect being each module does these things a little differently).
>>> 
>>> I'm starting with a proof-of-concept NetKernel module that offers endpoints for publishing messages. That seems the easiest to start with. But after that, I think creating a system of consumers and resource "callbacks" would be fairly lightweight and, more than anything, better than what I've got now because it wouldn't be copied all over the place in different ways.
>>> 
>>> There seems to be a "best practice" way of managing the actual sending and receiving of messages irregardless of the application into which this functionality is injected. Once I can encapsulate that into a module, I have to figure out how IPC will happen! :/ I'm leaning toward Scala/Akka remote actors but there are more ways to go about this, I'm sure. I could also use local Redis servers that would use a variation on OSGi's "whiteboard" pattern where code inside the RabbitMQ NetKernel module would write data to a local Redis server where it could be picked up by code running in other JVMs on the same machine. I just haven't thought enough about it yet to decide which is best (my luck will be that it's actually a combination of various things).
>>> 
>>> Jon Brisbin
>>> Portal Webmaster
>>> NPC International, Inc.
>>> 
>>> 
>>> 
>>> On Jul 29, 2010, at 4:34 PM, David Wragg wrote:
>>> 
>>>> Hi Jon,
>>>> 
>>>> Jon Brisbin <jon.brisbin at npcinternational.com> writes:
>>>>> Has anyone worked with RabbitMQ inside NetKernel?
>>>>> 
>>>>> http://www.1060research.com/netkernel/
>>>>> 
>>>>> I know enough about OSGI to make myself dangerous. I've worked some
>>>>> with Apache's ESB, so when I came across NetKernel, I thought I'd at
>>>>> least explore it a little to see if it would make our cloud
>>>>> infrastructure easier to manage. I'd love to have some AMQP
>>>>> integration in NetKernel to play with, so I wondered if anyone has
>>>>> already worked on something or if there are plans to do so?
>>>> 
>>>> I don't think we've heard of RabbitMQ and NetKernel being used together.
>>>> 
>>>> I wonder how you would combine the resource oriented computing paradigm
>>>> of NetKernel with a messaging approach?  Or were you thinking of using
>>>> AMQP as a transport for Netkernel?
>>>> 
>>>> Dvid
>>>> 
>>>> --
>>>> David Wragg
>>>> Staff Engineer, RabbitMQ
>>>> SpringSource, a division of VMware
>>> 
>>> _______________________________________________
>>> rabbitmq-discuss mailing list
>>> rabbitmq-discuss at lists.rabbitmq.com
>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>> 
> 
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss