[rabbitmq-discuss] RabbitMQ blocking issue

Thu Feb 21 15:21:58 GMT 2013

(re-sending since the m/l had dropped off the 'cc')

Peter,

On 21/02/13 14:55, Konar Peter (Nokia-LC/Chicago) wrote:
> from the log:
>
> =ERROR REPORT==== 21-Feb-2013::14:01:02 ===
> closing AMQP connection <0.23035.17> (10.196.42.128:59179 -> 10.196.42.21:5672):
> {heartbeat_timeout,running}
>
> =INFO REPORT==== 21-Feb-2013::14:01:02 ===
> accepting AMQP connection <0.23611.34> (10.196.42.128:41413 -> 10.196.42.21:5672)
>
> =INFO REPORT==== 21-Feb-2013::14:06:28 ===
> Memory limit set to 1MB of 15949MB total.
>
> =INFO REPORT==== 21-Feb-2013::14:06:28 ===
> vm_memory_high_watermark set. Memory used:141211416 allowed:1672474
>
> =WARNING REPORT==== 21-Feb-2013::14:06:28 ===
> memory resource limit alarm set on node rabbit at ajtrafp063
>
> =INFO REPORT==== 21-Feb-2013::14:06:38 ===
> Memory limit set to 6379MB of 15949MB total.
>
> =INFO REPORT==== 21-Feb-2013::14:06:38 ===
> vm_memory_high_watermark clear. Memory used:129010184 allowed:6689898496
>
> =WARNING REPORT==== 21-Feb-2013::14:06:38 ===
> memory resource limit alarm cleared on node rabbit at ajtrafp063

Looks like you got there too late - the connection timed out before the 
'rabbitmqctl set_vm_memory_high_watermark ...' commands.

Hopefully you'll catch it early enough next time.

Also, before you run the pair of 'rabbitmqctl 
set_vm_memory_high_watermark ...' commands, do the following:

1) determine the pid of the connection that is blocked by running
   rabbitmqctl -q list_connections pid peer_port | grep <port>
where <port> is the peer port shown in the netstat output, e.g. 59179 in 
the case we've just looked at.

2) plug the pid, which looks something like <rabbit at host.1.289.0>, into
   rabbitmqctl eval '{backtrace, B} = 
process_info(rabbit_misc:string_to_pid("<rabbit at host.1.289.0>"), 
backtrace), io:format("~s", [binary_to_list(B)]).'
(all on one line) and post the output. NB: If you are in a cluster, you 
need to run this on the rabbit server identified by the host portion of 
the pid.

Regards,

Matthias.