[rabbitmq-discuss] RabbitMQ 2.3.1 high CPU usage & disks operations

[Raphael Simon]

Hello Matthew,

We set the number of available file descriptors to around 32,000 and I can
see that RabbitMQ is using around 14,000 (with the rabbitadmin management
agent) so there should be a comfortable margin. The number of queues deleted
was around 2,000 (we have a queue gc daemon that will periodically delete
queues with no consumers as we can't guarantee that clients will always
leave cleanly). Wrt to messages expiring: the message throughput is fairly
uniform and the number of messages expiring at any given time should remain
fairly constant. I guess bursts are possible though, I would estimate the
max number of messages expiring all at once to around 40k - do you see an
issue with this? etop did show the timer process often as the busiest.

Thanks for the help!

--
Raphael.

On Thu, Apr 21, 2011 at 3:42 AM, Matthew Sackman <matthew at rabbitmq.com>wrote:

> Hi Raphael,
>
> Thanks for the detailed report. This is an interesting one, and a few
> ideas spring to mind.
>
> You seem to be using the x-message-ttl feature now, with message expiry
> around 1 day. Do these events happen to correspond with the bulk expiry
> of a lot of messages? Certainly 60,000 odd msgs in the broker shouldn't
> cause any problems at all, but just want to check that one. Also, when
> you say "many queues get deleted", how many are we talking about? The
> only thing we can think of is if you're deleting more queues than you
> have file descriptors available for the process then you may run into
> problems: Rabbit won't crash, but it might thrash fds between different
> queues which can overall result in vastly more disk activity than
> necessary.
>
> The normal Linux default fd limit (ulimit -n) is 1024, though on OS X,
> it's down at 256 by default IIRC. If you're deleting more than about
> 900 queues at any one go, then I'd suggest that you try increasing that
> substantially: having about double the number of fds available than
> queues should eliminate any thrashing - if indeed this is the problem.
> If this is the problem then by strace, you should be able to see an
> awful lot of open and close calls to various files, whereas if this
> problem is solved by higher fd ulimit then you should see files stay
> open for much longer.
>
> Please let us know how you get on.
>
> Matthew
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110421/a5e4b484/attachment-0001.htm>