I almost feel as though rabbitmq can't push items to my consumer fast enough, I had a backed up queue with 20 consumers and spawned another watching the items get processed and it just seemed so slow, saw numerous pauses between item processing and I don't believe it is the consumer's fault.<div>
<br></div><div>I am using the Python library for the consumer part.</div><div><br></div><div>Suhail<br><br><div class="gmail_quote">On Mon, Sep 7, 2009 at 8:23 AM, Chuck Remes <span dir="ltr"><<a href="mailto:cremes.devlist@mac.com">cremes.devlist@mac.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div class="im"><br>
On Sep 7, 2009, at 1:07 AM, aisha fenton wrote:<br>
<br>
> Hi,<br>
> I'm sure I'm doing something wrong since I can't find reference to<br>
> this anywhere else. What I'm seeing is that the performance of<br>
> draining a queue gets slower as the queue size increases.<br>
><br>
> I'm aware of the issue in RabbitMQ 1.6 that means that when it runs<br>
> out of physical memory that it's performance degrades because it<br>
> starts swapping out. But I'm not anywhere close to running out of<br>
> memory yet, and the degradation starts almost immediately and<br>
> increases linearly as the queue depth grows.<br>
><br>
> I am publishing 500mps to an exchange. Each message is about 1KB. I<br>
> have a single fanout queue bound to the exchange. A single consumer is<br>
> popping messages off the queue.<br>
><br>
> When the queue is less than 20,000 messages I can pop 300mps off the<br>
> queue. When the queue is 200,000 messages the performance drop to<br>
> 40-80mps.<br>
><br>
> I'm running RabbitMQ 1.6.0. And nether rabbitmq, the consumer, or the<br>
> publisher are using more than 40% CPU.<br>
><br>
> I assume I shouldn't be seeing this? Any help much appreciated.<br>
<br>
</div>You didn't include any code, but I'm going to take a stab in the dark<br>
anyway. If you are using the ruby amqp gem you might be doing<br>
something like this:<br>
<br>
def next_message<br>
� exchange = MQ.fanout 'foo'<br>
� queue = MQ.queue 'bar'<br>
<br>
� queue.bind exchange<br>
<br>
� newest_message = queue.pop<br>
end<br>
<br>
In the code above the call to MQ.<whatever> is opening a new channel<br>
to rabbitmq each time the method is called. You are essentially<br>
leaking channels all over the place every time you try to pop a new<br>
message.<br>
<br>
If you aren't using the ruby amqp stuff, I still recommend checking<br>
your code for whatever library you chose. You only need to allocate a<br>
few channels and reuse them.<br>
<font color="#888888"><br>
cr<br>
</font><div><div></div><div class="h5"><br>
<br>
_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><a href="http://mixpanel.com">http://mixpanel.com</a><br>Blog: <a href="http://blog.mixpanel.com">http://blog.mixpanel.com</a><br>
</div>