[rabbitmq-discuss] latency in starting/stopping a rabbitmq node

[Alexandru Scvorţov]

Hi Praveen,

> However I realized when i wanted to shutdown the broker before starting a
> new test, the stop command (rabbitmqctl stop) took a long time
> to complete.

We are aware of this problem.  The fix is currently going through QA and
will probably be in the next release, which should be around fairly
soon.

> Query 1)
> I am curious as to what causes the latency to stop the broker when issued a
> rabbitmqctl stop command. It seems to be something to do with the number of
> queues created as the stop time increase proportionally as the number of
> queues increases.

Internally, when we terminate a queue, we do a few file operations.  This
is usually not a problem, but when you close a connection with 100 000s
of queues, the same order of file operations get scheduled.  Erlang's
IO system then does some expensive operations of this long queue and
it ends up processing the operations in quadratic time.  The fix going
through QA brings this down to linear time; for instance, I can delete
40k queues in 20s (compared to 211s on the latest release).

> Query 2)
> In the case of durable queues, I measured the time taken to restart the
> broker after stopping it (a clean and unclean stop).
> I found that even after a clean/unclean stop the time to restart the broker
> was just about 20 seconds on an average.
> However, in the case where i created 50000 durable queues and did an unclean
> stop(just aborted the broker) and tried to restart the broker it didn't
> start for over to 6 minutes (when I gave up)...
> It was hung in the step of "starting exchange,queue and binding recovery.."
> It will be great if someone could explain why this could be caused.

I can't reproduce this.  Declaring 100 000 durable queues, killing the broker
and re-starting it seems to work fine.  It takes about 1 min on my
machine.

Is it really hung?  Is it using the CPU or disk at all at this time?  Is
there anything in the logs (both the rabbit and SASL logs)?

> It will be great if someone could answer the above queries or provide me
> with some pointers about the same.

There's not much you can do at the moment except avoiding terminating a
large number of queues at the same time.

Hope this clears things up.

Cheers,
Alex

On Thu, Sep 08, 2011 at 07:38:14PM -0700, Praveen M wrote:
> Hi,
> 
> I'm a rabbitmq newbie and am trying to run some experiments to figure out if
> rabbitmq would serve my use case.
> 
> I would like to create queues in the order of 100,000s. (one for each of my
> customers).
> 
> I ran various tests,
> 
> I'm using the latest 2.6.0 server and 2.6.0 client, and the following tests
> in durable queues mode and in non-durable queues mode.
> 
> Tests,
> 1) to create 1000 queues , produce, consume
> 2) to create 10000 queues , produce, consume
> 3) to create 50000 queues, produce and consume.
> 
> It works like a charm and the memory usage even with 50,000 queues seem very
> reasonable. (the order of 1-1.7G)
> 
> However I realized when i wanted to shutdown the broker before starting a
> new test, the stop command (rabbitmqctl stop) took a long time
> to complete.
> 
> I made a small chart of how long the stop command on the broker takes to
> execute after the test creates 'N' queues listed below.
> Also, in the case of durable queues, i found some weird numbers for the time
> taken to restart the queues after a clean/unclean(aborting broker) stop
> 
> *NON_DURABLE_QUEUES TEST*
> *No of Queues Stop Time*
> 1000 10.7 seconds
> 10000 2 minutes
> 50000 11 minutes
> 
> *
> *
> *DURABLE_QUEUES TEST
> *No of Queues Start Time Stop Time*
> 1000 2 seconds 10 seconds
> 10000 24 seconds 2 minutes
> 10000 after crash it recovers in 20 seconds (on improper shutdown).
> 50000 even at 6 minutes the queues doesn't start on a improper shutdown
> 
> 
> Query 1)
> I am curious as to what causes the latency to stop the broker when issued a
> rabbitmqctl stop command. It seems to be something to do with the number of
> queues created as the stop time increase proportionally as the number of
> queues increases.
> 
> Query 2)
> In the case of durable queues, I measured the time taken to restart the
> broker after stopping it (a clean and unclean stop).
> I found that even after a clean/unclean stop the time to restart the broker
> was just about 20 seconds on an average.
> However, in the case where i created 50000 durable queues and did an unclean
> stop(just aborted the broker) and tried to restart the broker it didn't
> start for over to 6 minutes (when I gave up)...
> It was hung in the step of "starting exchange,queue and binding recovery.."
> It will be great if someone could explain why this could be caused.
> 
> It will be great if someone could answer the above queries or provide me
> with some pointers about the same.
> 
> Thank you for your help,
> -- 
> -Praveen

> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss