<div dir="ltr">Following up on this to close the loop on what happened, and hopefully provide some useful information if anybody else run into a similar problem.<div><br></div><div>Long story short, I was able to find/fix the problem without needing to upgrade from RabbitMQ 3.0.2 and Erlang R15B01. (Although I hope to upgrade soon.)</div>
<div><br></div><div>Our server had ~800 connections with associated subscriptions from clients. Using top, I found a disproportionate amount of time spent in system code on core 0. Using strace, I found that poll() was being called, specifying ~800 file descriptors (i.e., all our connections).</div>
<div><br></div><div>Turns out that Erlang by default doesn't use the much more efficient epoll() mechanism. However, RabbitMQ's startup of Erlang includes the "+K true" option, which tells Erlang to use epoll.</div>
<div><br></div><div>For reasons I won't go into, I was setting the RABBITMQ_SERVER_ERL_ARGS environment variable, not realizing that what I put there *replaced* what RabbitMQ passes to Erlang. I mistakenly believed it got *added* to the Erlang start parameter.</div>
<div><br></div><div>Once I changed the RABBITMQ_SERVER_ERL_ARGS variable to include the RabbitMQ defaults as well, things started working much better.</div><div><br></div><div>For what it's worth, it might be useful for RabbitMQ to log the Erlang related stuff and highlight if any thing's been changed from the defaults.</div>
<div><br></div><div>Thanks Matthias, Michael and Zhibo for your help!</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Oct 23, 2013 at 12:08 AM, Matthias Radestock <span dir="ltr"><<a href="mailto:matthias@rabbitmq.com" target="_blank">matthias@rabbitmq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Matt,<div class="im"><br>
<br>
On 22/10/13 23:44, Matt Pietrek wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
The load on our box (it's in production) has dropped a little bit, but<br>
core 0 is still running nearly flat out.<br>
</blockquote>
<br></div>
If you are seeing *less* than 100% utilisation of one core then your system is *not* CPU bound.<br>
<br>
Check that RabbitMQ is actually a bottlneck here, rather than, say, your producers or consumers.<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Are there other useful "eval" commands that I could use to ferret out<br>
what's happening? This is a box in production and we have an upcoming<br>
release that's only going to add more load on this box.<br>
</blockquote>
<br></div>
We could come up with some eval code to identify the top running processes. Or perhaps find a way to get etop to run (Erlang's equivalent of 'top'). But...<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Also, we are running this in a cluster with another box, with all queues<br>
in HA mode.<br>
</blockquote>
<br></div>
There have been tons of improvements *and bug fixes* to mirrored queues. Some relating to performance. So I strongly recommend you upgrade RabbitMQ to the lastest version. And Erlang too, due to the SMP scheduler improvements I mentioned in my earlier email.<br>
<br>
If you still see performance anomalies after the upgrade then we can investigate further.<br>
<br>
<br>
Regards,<div class="HOEnZb"><div class="h5"><br>
<br>
Matthias.<br>
______________________________<u></u>_________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.<u></u>rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/<u></u>cgi-bin/mailman/listinfo/<u></u>rabbitmq-discuss</a><br>
</div></div></blockquote></div><br></div>