[rabbitmq-discuss] Blocked connections on rabbitmq 2.2.0

[Ivan Sanchez]

  Unfortunately when I tried to run 'rabbitmqctl list_queues memory'
during the problem it would never bring a result.

  But you're right, it's possible queues got big in that node (clients
too slow, perhaps). If that's the case, is there a way to prevent the
server from running out of memory? In my context I'd rather drop
unacknowledged messages than making the queue grow forever.

--
Ivan

On Feb 4, 1:01 pm, Marek Majkowski <maje... at gmail.com> wrote:
> On Fri, Feb 4, 2011 at 12:42, Ivan Sanchez <s4nc... at gmail.com> wrote:
> >  Thank you for your answer. I've got a few of the following alerts:
>
> >  =INFO REPORT==== 2-Feb-2011::21:05:38 ===
> > vm_memory_high_watermark set. Memory used:6611768328 allowed:
> > 3351445504
>
> > =INFO REPORT==== 2-Feb-2011::21:05:38 ===
> >    alarm_handler: {set,{vm_memory_high_watermark,[]}}
>
> That line means that Rabbit is under memory pressure and
> won't accept any new messages until some memory
> will be freed.
>
> >  Apart from the cluster setup, the only other setting I have in my
> > config is ERL_MAX_ETS_TABLES=10000.
>
> That shouldn't make a difference.
>
> >  The part that's confusing to me is that only one node had this issue
> > (even after resetting the whole cluster), and I'm not using persistent
> > messages. So now I'm not sure what else could be causing the memory
> > problem.
>
> >  Is there anything else I'm missing here?
>
> Maybe the biggest queue landed on that particular node?
> Maybe this node handles all the queues?
>
>
>
>
>
>
>
>
>
> > On Feb 4, 11:53 am, Marek Majkowski <maje... at gmail.com> wrote:
> >> On Thu, Feb 3, 2011 at 20:36, Ivan Sanchez <s4nc... at gmail.com> wrote:
> >> >  Yesterday I've noticed some very strange behaviour in one of our
> >> > rabbitmq cluster nodes. Its queues became unresponsive and running
> >> > "rabbitmqctl list_connections" was returning that all the connections
> >> > were either "blocking" or "blocked". The documentation doesn't mention
> >> > these states. Does anyone know what they mean?
>
> >> >  To give a bit of context: we noticed this problem when good portion
> >> > of our clients stopped receiving messages. Looking at all the servers
> >> > we found one that was using too much CPU and also swapping to disk.
> >> > This node was the only presenting this behaviour, but it seemed like
> >> > this problem compromised the whole cluster. We haven't touched these
> >> > servers for ages (they are dedicated to rabbitmq) and the system load
> >> > was completely under normal levels. We use simple DNS round-robin for
> >> > clients to connect to the cluster and none of our messages are
> >> > persistent, so seeing swapping really scared me.
>
> >> >  After restarting the whole cluster a few times the problem
> >> > persisted, always on the same server. We even tried force_reset in all
> >> > nodes, but that also didn't help. Things just went back to normal
> >> > after we removed the problematic node from the cluster. Now my task if
> >> > figure out what can be the problem.
>
> >> >  Did anyone have experience with this kind of behaviour? I'm even
> >> > considering hardware problem, but so far didn't find anything
> >> > indicating that was the case.
>
> >> Hi,
>
> >> When RabbitMQ is using more memory than it should,
> >> to avoid crashing, it stops accepting new messages.
>
> >> In AMQP publishing messages is asynchronous, and
> >> it's illegal to 'reject' a published message.
>
> >> Instead, when RabbitMQ is under a memory pressure
> >> it stops receiving data from tcp connections that try to
> >> do 'basic_publish'.
>
> >> That's what the 'blocked' connection state means.
>
> >> It's perfectly legal to open second connection and
> >> consume messages using it as long as it doesn't do 'basic_publish'.
>
> >> Please check for memory alerts in your RabbitMQ logs,
> >> check if memory watermark is being set properly
> >> and if you have enough memory for your Rabbit.
>
> >> Cheers,
> >>   Marek
> >> _______________________________________________
> >> rabbitmq-discuss mailing list
> >> rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> > _______________________________________________
> > rabbitmq-discuss mailing list
> > rabbitmq-disc... at lists.rabbitmq.com
> >https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss