[rabbitmq-discuss] My App Isn't Scaling

[Daniel N]

Hi Guys,

I tried to bring this up in IRC but I couldn't get a bite.

I've created an application in an attempt to learn rabbitmq via the ruby gem
amqp.   My basic setup is that I have a very simple web app out front that
recieves requests and creates the initial message in the system.  The flow
goes like this:

Web App
Direct Exchange + Queue -> do some work (forward 1 message) ->
Direct Exchange + Queue -> do stuff  (forward 2 messages for each one)->
Direct Exchange + Queue -> do stuff (forward 2 messages for each one) ->
Topic Exchange + routed to a queue (forward 0 messages for each one) ->
finish

Firstly, this is all being done on my laptop and I'm hammering it with ab
and 3 instances of the web app.

So firstly this is really slow.  I was expecting that with 3 instances of
the web app out front would increase it's concurrency a bit.  What I found
though was unexpected by me.

I hit the system with 3 ab's and sent in 1000 requests each with a
concurrecy of 40.  This went through a rack app so the concurrency may not
actually matter all that much.

Throughout the run, the output from each worker (single worker on a queue)
was sporadic.  It would hammer for a bit, then pause then go again. Once the
ab's had finished making requests to the web apps, and they'd finished
creating messages on the rabbitmq network, the downstream workers didn't
stop.  Instead they continued, basically in order.   The upstream one would
empty, with the downstream ones paused and only consuming messages
sporadically.  Then once that queue was effectively finished the next
consumer would become active and empty etc.

It took a significant ammount of time to empty the queues and I was only
getting 18rps from each web app!  This is one of the most unscientific
benchmarks possible and this was only observed by watching the logs of the
consumers.  Each hop reads from a couchdb for some information.

This is about all I can think to list at the moment.  I'd really appreciate
any insights into why I'm seeing this build up of messages.  I would think
this should barely stress the rabbitmq network at all.  Is it possible to
attach many queues to a single Direct Exchange?

All my consumers are setup as subscribers to a queue, and all direct
exchanges have only one queue.

Cheers
Daniel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20090218/7a5ab8b6/attachment.htm