[rabbitmq-discuss] RabbitKernel

Fri Jul 30 22:01:04 BST 2010

On Jul 30, 2010, at 3:34 PM, Alexis Richardson wrote:

> Surely that would only make sense at *very* high volume?
> 

We're certainly not doing as much traffic as a lot of people, so I'm sure we've got CPU/RAM to burn. 

This duplication of resources concerns me because there's such a variation in the way each component works (which is mostly an artefact of me constantly tweaking things). I'm already hovering in the 60%-70% memory usage range under normal circumstances (VMs with 3GB RAM, 3 instances of tcServer, and sundry other apps). This is giving us great performance. Page responses are consistent and usually in the tens of milliseconds (unless it's talking to the AS/400...then it's 10 seconds-ish!). But we haven't even fully utilized all the messaging we're going to be doing in the next couple months. I'm not sure what impact that's going to have.

I'm thinking every VM will have something like this running on it, whether it's doing app serving or as a slave compute node for map/reduce jobs (which I'm trying to architect for, even though the other developers can't even spell map/reduce! ;). I've got to find a way to start these services/listeners/consumers anyway, whether as a module in NetKernel, or as a Java app using commons-cli and invoking it from a bash script. It seems to me that using an OSGi approach will yield better results with fewer headaches. I might be wrong.

You raise a good point, though. At what point does it become a theoretical exercise, done simply to see if it's possible? I just follow my instincts. My instinct says I can make this better than what it is today. I guess you could say I simply can't help myself. :)

Jon Brisbin
Portal Webmaster
NPC International, Inc.

> 
> On Fri, Jul 30, 2010 at 9:31 PM, Jon Brisbin
> <jon.brisbin at npcinternational.com> wrote:
>> I can see several advantages:
>> 
>> In addition to the management and resource efficiency of consolidating to a central (to that VM anyway) postoffice, it seems more scalable, can support higher throughput because of the dedicated dispatching (which is a pattern I use anyway inside the code where I do messaging), vastly reduces resource usage in those other components where all of this functionality is duplicated at current, and seems it will make testing easier because I can dispense with actual message sending/receiving in the test versions of modules without affecting application code.
>> 
>> It should also be easier to manage services I would otherwise have to start with a bash script like the cache, the store updater, the distributed ClassLoader, and suchlike...
>> 
>> Jon Brisbin
>> Portal Webmaster
>> NPC International, Inc.
>> 
>> 
>> 
>> On Jul 30, 2010, at 3:16 PM, Alexis Richardson wrote:
>> 
>>> Jon
>>> 
>>> Thank-you!  Now can you elaborate on what the middle tier 'gets you'
>>> please?  Is it just a convenient hosting layer or does it control
>>> something that can't be in the other two tiers?
>>> 
>>> alexis
>>> 
>>> 
>>> 
>>> On Fri, Jul 30, 2010 at 9:09 PM, Jon Brisbin
>>> <jon.brisbin at npcinternational.com> wrote:
>>>> I've attached a PDF of a 5-minute diagram that shows some use cases for me.
>>>> 
>>>> On the right:
>>>> 
>>>> Top is a tcServer instance that needs to load an object from the distributed cache.
>>>> 
>>>> Middle is a cron job that needs access to modules running in NetKernel on that machine for maintenance functions.
>>>> 
>>>> Bottom is the AS/400 that needs to update the PostgreSQL server at a store as the result of a database trigger.
>>>> 
>>>> Flow for tcServer code requesting object from distributed cache:
>>>> 
>>>> 1. Use Scala/Akka (or some IPC mechanism) to block on a remote Actor which checks local cache module but doesn't find the object. It shells out to the AMQP module, which is solely responsible for sending and receiving messages.
>>>> 
>>>> 2. Sends a message to the cloud, asking for the object.
>>>> 
>>>> 3. Remote AMQP module gets a cache request message and dispatches it to the cache module, which returns an object. The remote AMQP module then replies to the requestor, handing back the Protobuf'd object.
>>>> 
>>>> 4. Requesting AMQP module maybe stores object in local cache and hands object back to Akka module, which replies to the tcServer code that originally requested the object.
>>>> 
>>>> Flow for AS/400 wanting to update a store:
>>>> 
>>>> 1. AS/400 trigger fires, dumping a message into RabbitMQ.
>>>> 
>>>> 2. AMQP module sees it and delegates to Store Updater Module.
>>>> 
>>>> 3. Store Updater Module sees this is a fire-and-forget update, so returns immediately and proceeds to update a store database in Birmingham, Alabama (or Portland, Oregon for that matter) asynchronously.
>>>> 
>>>> Jon Brisbin
>>>> Portal Webmaster
>>>> NPC International, Inc.
>>>> 
>>>> 
>>>> 
>>>> On Jul 30, 2010, at 1:34 PM, Alexis Richardson wrote:
>>>> 
>>>>> Jon
>>>>> 
>>>>> Interesting indeed!  At the risk of being both laconic and glib... do
>>>>> you have a diagram showing what you propose to achieve?  I think that
>>>>> would bring this to life...
>>>>> 
>>>>> alexis
>>>>> 
>>>>> On Fri, Jul 30, 2010 at 5:01 PM, Jon Brisbin
>>>>> <jon.brisbin at npcinternational.com> wrote:
>>>>>> To expand some (and for the benefit of the list), I guess what I'm thinking is related to a problem I see on the horizon as I build more utilities that have at their foundation AMQP messaging. Each component needs to talk to the other components running on different VMs within the cloud on different exchanges/queues and each has their own system of subscribing consumers and producing messages. This means: if I have three instances of tcServer running on a VM and each webapp is using the cloud session manager and and the server is using the JMX invoker, I've got four different ConnectionFactories, four sets of subscribers, four sets of producers (times the number of workers I have configured; I'll likely have 40 or more of these per VM). This doesn't seem terribly scalable to me, so I'm hunting for a way to use messaging on a VM for communicating to various components (that are likely unrelated) in a more reasonable way.
>>>>>> 
>>>>>> OSGi has always appealed to me because of module versioning, live reloading, etc... and the ROC approach that NetKernel takes to abstracting OSGi seems to offer (in my initial estimation at least) the ability to de-duplicate the various ways I'm managing connections, channels, subscribers, and producers (which is to say: each time I write a module, I try and improve it; the side-effect being each module does these things a little differently).
>>>>>> 
>>>>>> I'm starting with a proof-of-concept NetKernel module that offers endpoints for publishing messages. That seems the easiest to start with. But after that, I think creating a system of consumers and resource "callbacks" would be fairly lightweight and, more than anything, better than what I've got now because it wouldn't be copied all over the place in different ways.
>>>>>> 
>>>>>> There seems to be a "best practice" way of managing the actual sending and receiving of messages irregardless of the application into which this functionality is injected. Once I can encapsulate that into a module, I have to figure out how IPC will happen! :/ I'm leaning toward Scala/Akka remote actors but there are more ways to go about this, I'm sure. I could also use local Redis servers that would use a variation on OSGi's "whiteboard" pattern where code inside the RabbitMQ NetKernel module would write data to a local Redis server where it could be picked up by code running in other JVMs on the same machine. I just haven't thought enough about it yet to decide which is best (my luck will be that it's actually a combination of various things).
>>>>>> 
>>>>>> Jon Brisbin
>>>>>> Portal Webmaster
>>>>>> NPC International, Inc.
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Jul 29, 2010, at 4:34 PM, David Wragg wrote:
>>>>>> 
>>>>>>> Hi Jon,
>>>>>>> 
>>>>>>> Jon Brisbin <jon.brisbin at npcinternational.com> writes:
>>>>>>>> Has anyone worked with RabbitMQ inside NetKernel?
>>>>>>>> 
>>>>>>>> http://www.1060research.com/netkernel/
>>>>>>>> 
>>>>>>>> I know enough about OSGI to make myself dangerous. I've worked some
>>>>>>>> with Apache's ESB, so when I came across NetKernel, I thought I'd at
>>>>>>>> least explore it a little to see if it would make our cloud
>>>>>>>> infrastructure easier to manage. I'd love to have some AMQP
>>>>>>>> integration in NetKernel to play with, so I wondered if anyone has
>>>>>>>> already worked on something or if there are plans to do so?
>>>>>>> 
>>>>>>> I don't think we've heard of RabbitMQ and NetKernel being used together.
>>>>>>> 
>>>>>>> I wonder how you would combine the resource oriented computing paradigm
>>>>>>> of NetKernel with a messaging approach?  Or were you thinking of using
>>>>>>> AMQP as a transport for Netkernel?
>>>>>>> 
>>>>>>> Dvid
>>>>>>> 
>>>>>>> --
>>>>>>> David Wragg
>>>>>>> Staff Engineer, RabbitMQ
>>>>>>> SpringSource, a division of VMware
>>>>>> 
>>>>>> _______________________________________________
>>>>>> rabbitmq-discuss mailing list
>>>>>> rabbitmq-discuss at lists.rabbitmq.com
>>>>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>>>>> 
>>>> 
>>>> 
>>>> 
>> 
>> 