[rabbitmq-discuss] Blocked connections on rabbitmq 2.2.0

[Marek Majkowski]

On Fri, Feb 4, 2011 at 12:42, Ivan Sanchez <s4nchez at gmail.com> wrote:
>  Thank you for your answer. I've got a few of the following alerts:
>
>  =INFO REPORT==== 2-Feb-2011::21:05:38 ===
> vm_memory_high_watermark set. Memory used:6611768328 allowed:
> 3351445504
>
> =INFO REPORT==== 2-Feb-2011::21:05:38 ===
>    alarm_handler: {set,{vm_memory_high_watermark,[]}}

That line means that Rabbit is under memory pressure and
won't accept any new messages until some memory
will be freed.

>  Apart from the cluster setup, the only other setting I have in my
> config is ERL_MAX_ETS_TABLES=10000.

That shouldn't make a difference.

>  The part that's confusing to me is that only one node had this issue
> (even after resetting the whole cluster), and I'm not using persistent
> messages. So now I'm not sure what else could be causing the memory
> problem.
>
>  Is there anything else I'm missing here?

Maybe the biggest queue landed on that particular node?
Maybe this node handles all the queues?



> On Feb 4, 11:53 am, Marek Majkowski <maje... at gmail.com> wrote:
>> On Thu, Feb 3, 2011 at 20:36, Ivan Sanchez <s4nc... at gmail.com> wrote:
>> >  Yesterday I've noticed some very strange behaviour in one of our
>> > rabbitmq cluster nodes. Its queues became unresponsive and running
>> > "rabbitmqctl list_connections" was returning that all the connections
>> > were either "blocking" or "blocked". The documentation doesn't mention
>> > these states. Does anyone know what they mean?
>>
>> >  To give a bit of context: we noticed this problem when good portion
>> > of our clients stopped receiving messages. Looking at all the servers
>> > we found one that was using too much CPU and also swapping to disk.
>> > This node was the only presenting this behaviour, but it seemed like
>> > this problem compromised the whole cluster. We haven't touched these
>> > servers for ages (they are dedicated to rabbitmq) and the system load
>> > was completely under normal levels. We use simple DNS round-robin for
>> > clients to connect to the cluster and none of our messages are
>> > persistent, so seeing swapping really scared me.
>>
>> >  After restarting the whole cluster a few times the problem
>> > persisted, always on the same server. We even tried force_reset in all
>> > nodes, but that also didn't help. Things just went back to normal
>> > after we removed the problematic node from the cluster. Now my task if
>> > figure out what can be the problem.
>>
>> >  Did anyone have experience with this kind of behaviour? I'm even
>> > considering hardware problem, but so far didn't find anything
>> > indicating that was the case.
>>
>> Hi,
>>
>> When RabbitMQ is using more memory than it should,
>> to avoid crashing, it stops accepting new messages.
>>
>> In AMQP publishing messages is asynchronous, and
>> it's illegal to 'reject' a published message.
>>
>> Instead, when RabbitMQ is under a memory pressure
>> it stops receiving data from tcp connections that try to
>> do 'basic_publish'.
>>
>> That's what the 'blocked' connection state means.
>>
>> It's perfectly legal to open second connection and
>> consume messages using it as long as it doesn't do 'basic_publish'.
>>
>> Please check for memory alerts in your RabbitMQ logs,
>> check if memory watermark is being set properly
>> and if you have enough memory for your Rabbit.
>>
>> Cheers,
>>   Marek
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>