[rabbitmq-discuss] Using Disruptor in web app

[Simon MacMullen]

This doesn't appear to be a very RabbitMQ-related question. Did you want 
to ask here instead? http://groups.google.com/group/lmax-disruptor

Cheers, Simon

On 10/06/12 22:57, oscar wrote:
> Hi all,
>
> Firstly this is awesome stuff and I want to use it in my next project,
> which we be run in a web server.  I'm a little worried about trying to
> architect my project for performance based around using the disruptor
> and getting worse performance than normal due to not understanding it
> well enough.  I would really appreciate some input.
>
> The project will be hosted so I won't be able to simplify by having
> all data in memory (I think).  It will be a fairy traditional (boring)
> project, order entry processing (and other stuff).  So some screens to
> view orders, products query for them edit/maintain CRUD type stuff.
> So latency isn't important in itself but only as a means to support
> the maximum number of users for a smallish server with between 1.7 -17
> Gb of memory.
>
> 1) Is the disruptor for me?  (I hope yes)
> 2) When designing the ringbuffer events, how concerned should I be
> that they don't change in terms of memory location as they are
> mutated.  For instance if I use strings in the event than these will
> be re-allocated on each mutation.  Should I use say StringBuilders or
> even something like protobuf messages and have the event type as a
> byte array and marshall each event into that byte array  (handling for
> overflow) I see this as trading memory striding against cpu overhead
> and short lived object creation (new strings etc) in the marshalling/
> demarshalling process?
> 3) Since each event will be marshalled from an http request I'm not
> sure how large to make the event.  Should it be large so that the ring
> buffer has fewer slots but the chance of an oversize object (won't fit
> in preallocated event slot) is very small?
> 4) Should I have just one input ringbuffer with an event definition
> that can handle any event (http request) or multiple input ring
> buffers?  I think that LMAX had just one.
> 5) What should the overall process flow be?  an http request will come
> in and my web container thread will quickly marshall it into an event
> and put in on the ring buffer.  Consumers will read it it and say
> fetch a customer from the DB and update the event.  Finally my
> business processor (final single threaded beast of a consumer) will
> produce an http response.  What's the best/standard way of
> communicating that back the original thread that orginated the
> request?  via a Latch that's put onto the event?
>
> 6) Lastly is there a standard pattern / best practice architecture to
> be adopted for a CRUD like web app running in a webserver using the
> Disruptor.  I feel like what I'm doing is so standard that the ought
> to be a right way to do it.  I'm thinking right now that I would only
> need one RingBuffer being fed by the webserver request threads and
> then various consumers decorating that,  or do I need another one on
> the output side that the business processor is the producer to?
>
> Hope I'm not being dense, really appreciate any advice you can give on
> any of the points I've raised.
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


-- 
Simon MacMullen
RabbitMQ, VMware