[rabbitmq-discuss] Slow shutdown issue

[Matthew Sackman]

Hi Ray,

On Thu, Jul 21, 2011 at 03:18:11PM -0400, Raymond Murthi wrote:
> I am pretty new on RabbitMQ and I am trying to do a stress test on
> RabbitMQ (v. 2.5.1) using the following properties: 1kb persistent
> message, nondurable queue, and without any ready consumer (that is,
> just writing to the queue). However, every time I overload the RAM
> (13.7GB high watermark) and force RabbitMQ to write to disk, I always
> experience a very long wait time whenever I want to shutdown the
> server/stopping the app/purging the queue. The wait time could be over
> an hour depending on the number of messages I sent (I sent over 5
> millions messages). I  also notice that RabbitMQ tries to dump the
> messages from the memory to the disk (under /msg_store_transient) -
> since nothing is consumed.

Regardless of whether a persistent msg is written to a transient or
durable queue, it will eventually get written to disk. If you don't want
msgs to be written to disk by default then don't publish persistent
msgs. That way, they'll only get written to disk when you run low on
memory.

The test you're running results in many GB of msgs in RAM pending being
written to disk. On shutdown, Rabbit is trying to write those msgs to
disk, and thus your disk is the bottleneck.

If you declared the queue exclusive then we have some logic in there
that should make the shutdown of the queue and broker much faster. I'll
checker later today but I've a feeling that the same logic doesn't apply
to transient queues, though I can't think of a good reason why not.

Repeat your test but declare the queue as exclusive. You should find
that the shutdown of the broker is then much much faster.

> I read the documentation and it looks like it is (but isn't it too
> long?).

Writing 14GB of data to disk is likely to take a while.

> How to kill the process immediately?

kill -9, pull the power chord etc. But you'll lose messages. That said,
your test probably isnt concerned with msgs surviving restart.

> And why are messages
> stored under /msg_store_transient instead of /msg_store_persistent?

Because they're msgs going to a transient queue.

> What  is the difference between the two directories?

transient is for msgs sent to transient queues. This directory is blown
away on startup because no transient queues survive broker restart.

persistent is fro msgs sent to durable queues.

Matthew