<span class="Apple-style-span" style="font-family: arial, sans-serif; font-size: 13.2px; border-collapse: collapse; ">Thanks for the pointer looking at the queues was very helpful figured out I was never cleaning out the server queues and I kept re-creating a new one each time it was restarted and they were all bound to the same topic. So the sending of 100k messages from the client resulted in coping those 100k messages into tens of queues making millions of copies of messages.<div>
<br></div><div>I changed the server queue to be non-durable and it looked like I can do many passes with no crash now.</div><div><br></div><div>Thanks!</div><div><br></div><font color="#888888"><div>Jared</div></font></span><br>
<div class="gmail_quote">On Mon, Aug 2, 2010 at 11:36 AM, David Wragg <span dir="ltr"><<a href="mailto:david@rabbitmq.com">david@rabbitmq.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Hi Jared,<br>
<div class="im"><br>
Jared Smith <<a href="mailto:jaredtsmith@gmail.com">jaredtsmith@gmail.com</a>> writes:<br>
> Everything works good for the first pass where I send 100k<br>
> request/response pairs and I'm getting on the order of 4-5k rpc calls<br>
> per second. To make amqplib have blocking semantics I registered a<br>
> callback with basic_consume and have that deposit messages that are<br>
> read into a blocking queue which the client thread reads from this; it<br>
> seems to work ok as far as I can tell.<br>
><br>
> Problem I'm seeing is that on the second run of the program I get the<br>
> following errors:<br>
><br>
> ==============client error=================<br>
</div>[...]<br>
<div class="im">> u'PRECONDITION_FAILED - timeout waiting for channel.flow_ok{active=false}',<br>
> (0, 0), '')<br>
> =======================================<br>
><br>
> ============== matching server error ================<br>
</div>[...]<br>
<div class="im">> amqplib.client_0_8.exceptions.AMQPConnectionException: (503,<br>
> u'COMMAND_INVALID - basic.publish received after<br>
> channel.flow_ok{active=false}', (60, 40), 'Channel.basic_publish')<br>
> ==============================================<br>
<br>
</div>It's hard to be sure from the fragments of code you posted, but I wonder<br>
if the issue is that you aren't acknowledging the messages? To do this,<br>
you either need to call basic_consume with no_ack=True (to tell rabbit<br>
not to expect acknowledgements), or call basic_ack in your consuming<br>
code to explicitly ack the messages.<br>
<br>
If you don't acknowledge the messages, then they will stay in the queue,<br>
and take up memory, eventually resulting in the channel.flow methods<br>
indicated above.<br>
<br>
You could confirm that this is the issue by doing a<br>
<br>
$ sudo rabbitmqctl list_queues<br>
<br>
This will list the queues, along with the count of messages in each<br>
queue. If you do this after your first run, and see that a lot of<br>
messages are still present, then it is very likely to be the lack of<br>
acks causing the problem.<br>
<br>
David<br>
<font color="#888888"><br>
--<br>
David Wragg<br>
Staff Engineer, RabbitMQ<br>
SpringSource, a division of VMware<br>
</font></blockquote></div><br>