Hello Matthew,<div><br></div><div>We set the number of available file descriptors to around 32,000 and I can see that RabbitMQ is using around 14,000 (with the rabbitadmin management agent) so there should be a comfortable margin. The number of queues deleted was around 2,000 (we have a queue gc daemon that will periodically delete queues with no consumers as we can't guarantee that clients will always leave cleanly). Wrt to messages expiring: the message throughput is fairly uniform and the number of messages expiring at any given time should remain fairly constant. I guess bursts are possible though, I would estimate the max number of messages expiring all at once to around 40k - do you see an issue with this? etop did show the timer process often as the busiest.</div>
<div><br></div><div>Thanks for the help!</div><div><br></div><div>--</div><div>Raphael.<br><br><div class="gmail_quote">On Thu, Apr 21, 2011 at 3:42 AM, Matthew Sackman <span dir="ltr"><<a href="mailto:matthew@rabbitmq.com">matthew@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Hi Raphael,<br>
<br>
Thanks for the detailed report. This is an interesting one, and a few<br>
ideas spring to mind.<br>
<br>
You seem to be using the x-message-ttl feature now, with message expiry<br>
around 1 day. Do these events happen to correspond with the bulk expiry<br>
of a lot of messages? Certainly 60,000 odd msgs in the broker shouldn't<br>
cause any problems at all, but just want to check that one. Also, when<br>
you say "many queues get deleted", how many are we talking about? The<br>
only thing we can think of is if you're deleting more queues than you<br>
have file descriptors available for the process then you may run into<br>
problems: Rabbit won't crash, but it might thrash fds between different<br>
queues which can overall result in vastly more disk activity than<br>
necessary.<br>
<br>
The normal Linux default fd limit (ulimit -n) is 1024, though on OS X,<br>
it's down at 256 by default IIRC. If you're deleting more than about<br>
900 queues at any one go, then I'd suggest that you try increasing that<br>
substantially: having about double the number of fds available than<br>
queues should eliminate any thrashing - if indeed this is the problem.<br>
If this is the problem then by strace, you should be able to see an<br>
awful lot of open and close calls to various files, whereas if this<br>
problem is solved by higher fd ulimit then you should see files stay<br>
open for much longer.<br>
<br>
Please let us know how you get on.<br>
<br>
Matthew<br>
_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</blockquote></div><br></div>