[rabbitmq-discuss] Monitoring for busy_dist_port

[Iain Hull]

Hi Simon, Tim,

Thanks for your replies and effort analysing my query.

On Riak we have experienced stability issues when large data is
being transferred.  I am not too sure the exact root cause, but we see
busy_dist_port messages then one of the Riak nodes crashes and has to be
restarted by Chef.  Increasing "+zdbbl 18432" resolved this issue.

This contains more information about Riak and this setting
http://boundary.com/blog/2012/09/26/incuriosity-killed-the-infrastructur/

We have not seen any stability issues with our use of RabbitMQ to date.
After our Riak experience I was concerned if our RabbitMQ usage pattern
changes over time (for example message sizes increase), we could experience
similar stability issues too.  If there is a chance of this I would like to
be warned before its a problem.

Iain.


On 21 May 2013 15:38, Simon MacMullen <simon at rabbitmq.com> wrote:

> To add to what Tim said: I've run some tests with busy_dist_port
> monitoring patched into RabbitMQ and while I can certainly see
> busy_dist_port messages, I haven't been able to demonstrate any improvement
> in throughput by tweaking zdbbl until they went away - so I don't see the
> point in exposing them.
>
> Out of interest, did you see a performance improvement in Riak after
> modifying zdbbl, or were you just trying to make the warnings go away? :-)
>
> Cheers, Simon
>
>
> On 20/05/13 12:15, Iain Hull wrote:
>
>> We are in the process of deploying a Riak cluster, and are wondering if
>> some of the lessons we learned monitoring Erlang in Riak should be
>> applied to RabbitMQ.
>>
>> During testing we noticed a lot of busy_dist_port in the logs, the root
>> was cause was a the Erlang kernel parameter zdbbl was set too low.  We
>> have adjusted the parameter and are now monitoring the logs for
>> busy_dist_port errors.
>>
>> I have searched the RabbitMQ mailing list for cases of busy_dist_port or
>> zdbbl, but cannot find any.  I am wondering if RabbitMQ can suffer from
>> problems with busy_dist_port or if zdbbl should be set to any particular
>> value? Or I should I monitor the RabbitMQ logs for busy_dist_port?
>>
>> Thanks,
>> Iain.
>>
>> --
>> Iain Hull
>> Senior Software Engineer
>> iain.hull.workday (Skype)
>> iain.hull at workday.com <mailto:iain.hull at workday.com>
>>
>> <http://www.workday.com/**signature-uk<http://www.workday.com/signature-uk>
>> >
>>
>>
>>
>> ______________________________**_________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.**rabbitmq.com<rabbitmq-discuss at lists.rabbitmq.com>
>> https://lists.rabbitmq.com/**cgi-bin/mailman/listinfo/**rabbitmq-discuss<https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss>
>>
>>
>
> --
> Simon MacMullen
> RabbitMQ, Pivotal
>



-- 
Iain Hull
Senior Software Engineer
iain.hull.workday (Skype)
iain.hull at workday.com

<http://www.workday.com/signature-uk>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130524/e536b9e2/attachment.htm>