[rabbitmq-discuss] hanging on message sending with py-amqplib

[Matthias Radestock]

Tsuraan,

tsuraan wrote:
> Don't feel like doing the confusing name-mangling stuff, so sending
> these directly to you
> [...]

The output of list_connections show many connections in the 'blocked' 
state, which means the broker is refusing to accept any more data.

That should only happen if the memory alarm is currently active. What 
does the log file say about that? Perhaps you could post it.

The output of list_queues shows a number of queues with messages waiting 
for acknowledgement. One queue in particular has a high count. What's 
the average message size of messages in that queue?

The messages waiting for acks also show up in list_channels.

Tracing the from the queue through the consumers, channels and 
connections to find the connection which is consuming from the high-ack 
queue, we discover that connection to be blocked.


So I think what is happening here is that you have a client that is both 
a consumer and producer. At some point the memory alarm is raised. 
Because the client has been identified as a producer the server blocks 
the inbound socket data stream, thus pausing the client in its tracks. 
But messages still get delivered to the client. And they all pile up in 
memory, in the server's channel process. That is preventing the server 
from ever recovering, clearing the memory alarm and unblocking the 
connections.


I reckon one way to prevent this situation is to set a 
qos.prefetch_count, thus bounding the number of messages which are 
waiting for acknowledgement. Or, if you can consume in no-ack mode that 
would be even better.


Regards,

Matthias.