[rabbitmq-discuss] Monitoring for busy_dist_port

Wed May 29 11:48:14 BST 2013

Hi Iain.

I read that blog post some time ago and am still not very clear on what 
is *really* going on, but I'm certainly sure that from our perspective, 
busy_dist_port is nothing to be scared of, which is why we don't log it.

So I would not worry.

Cheers, Simon

On 24/05/13 15:35, Iain Hull wrote:
> Hi Simon, Tim,
>
> Thanks for your replies and effort analysing my query.
>
> On Riak we have experienced stability issues when large data is
> being transferred.  I am not too sure the exact root cause, but we see
> busy_dist_port messages then one of the Riak nodes crashes and has to be
> restarted by Chef.  Increasing "+zdbbl 18432" resolved this issue.
>
> This contains more information about Riak and this setting
> http://boundary.com/blog/2012/09/26/incuriosity-killed-the-infrastructur/
>
> We have not seen any stability issues with our use of RabbitMQ to date.
> After our Riak experience I was concerned if our RabbitMQ usage pattern
> changes over time (for example message sizes increase), we could
> experience similar stability issues too.  If there is a chance of this I
> would like to be warned before its a problem.
>
> Iain.
>
>
> On 21 May 2013 15:38, Simon MacMullen <simon at rabbitmq.com
> <mailto:simon at rabbitmq.com>> wrote:
>
>     To add to what Tim said: I've run some tests with busy_dist_port
>     monitoring patched into RabbitMQ and while I can certainly see
>     busy_dist_port messages, I haven't been able to demonstrate any
>     improvement in throughput by tweaking zdbbl until they went away -
>     so I don't see the point in exposing them.
>
>     Out of interest, did you see a performance improvement in Riak after
>     modifying zdbbl, or were you just trying to make the warnings go
>     away? :-)
>
>     Cheers, Simon
>
>
>     On 20/05/13 12:15, Iain Hull wrote:
>
>         We are in the process of deploying a Riak cluster, and are
>         wondering if
>         some of the lessons we learned monitoring Erlang in Riak should be
>         applied to RabbitMQ.
>
>         During testing we noticed a lot of busy_dist_port in the logs,
>         the root
>         was cause was a the Erlang kernel parameter zdbbl was set too
>         low.  We
>         have adjusted the parameter and are now monitoring the logs for
>         busy_dist_port errors.
>
>         I have searched the RabbitMQ mailing list for cases of
>         busy_dist_port or
>         zdbbl, but cannot find any.  I am wondering if RabbitMQ can
>         suffer from
>         problems with busy_dist_port or if zdbbl should be set to any
>         particular
>         value? Or I should I monitor the RabbitMQ logs for busy_dist_port?
>
>         Thanks,
>         Iain.
>
>         --
>         Iain Hull
>         Senior Software Engineer
>         iain.hull.workday (Skype)
>         iain.hull at workday.com <mailto:iain.hull at workday.com>
>         <mailto:iain.hull at workday.com <mailto:iain.hull at workday.com>>
>
>         <http://www.workday.com/__signature-uk
>         <http://www.workday.com/signature-uk>>
>
>
>
>         _________________________________________________
>         rabbitmq-discuss mailing list
>         rabbitmq-discuss at lists.__rabbitmq.com
>         <mailto:rabbitmq-discuss at lists.rabbitmq.com>
>         https://lists.rabbitmq.com/__cgi-bin/mailman/listinfo/__rabbitmq-discuss
>         <https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss>
>
>
>
>     --
>     Simon MacMullen
>     RabbitMQ, Pivotal
>
>
>
>
> --
> Iain Hull
> Senior Software Engineer
> iain.hull.workday (Skype)
> iain.hull at workday.com <mailto:iain.hull at workday.com>
>
> <http://www.workday.com/signature-uk>

-- 
Simon MacMullen
RabbitMQ, Pivotal