[rabbitmq-discuss] RabbitMQ 'hang' when trying to access HA queues

[Andrew Rendell]

Hello,

We have been using RabbitMQ in a high volume application for about two years. We use queues which are durable and mirrored to both nodes in the cluster. We have done a great many load tests and failure tests successfully.

Last week, during a very important test, under very low load conditions, suddenly, two queues 'hung'. Attempts to view the queues via rabbitmqctl resulted in the rabbitmqctl command hanging. Our Java clients called queue declare to check the length of the affected queues but these calls hung (until timed out by application code). We know that publishers were attempting to send messages to the exchange which routed to the affected queues and we had consumers which appeared to be attached to the queues in question but no messages flowed through the system. Eventually the support team started bouncing RabbitMQ brokers, producers and consumers and the messages which had failed to materialise during the 'hang' suddenly appeared in our application. The rabbitmqctl list queues command and queue declare also started working.

During this extended period there were other queues in the broker which were working without issue. Cluster status reported that both nodes of the RabbitMQ cluster were running. We could confirm that memory usage was not higher than the 40% watermark during the outage.

Since memory usage looked normal and producers and consumers not associated with the two failing HA queues we discounted the rudimentary flow control in place at 2.7.1.

Our RabbitMQ broker is version 2.7.1 and is running on Solaris.

We are painfully aware that version 2.7.1 is woefully out of date and that there has been an enormous number of fixes and enhancements applied to HA queues since.

What we are hoping is that somebody could give us any indication of what could cause this issue. In an ideal world, somebody would report that this is a known issue with 2.7.1 and that the issue is resolved at a later release.

Any help much appreciated.

Andrew.






*****************************************************
This email is issued by a VocaLink group company. It is confidential and intended for the exclusive use of the addressee only. You should not disclose its contents to any other person. If you are not the addressee (or responsible for delivery of the message to the addressee), please notify the originator immediately by return message and destroy the original message. The contents of this email will have no contractual effect unless it is otherwise agreed between a specific VocaLink group company and the recipient.

The VocaLink group companies include, among others: VocaLink Limited (Company No 06119048, VAT No. 907 9619 87) which is registered in England and Wales at registered office Drake House, Homestead Road, Rickmansworth, WD3 1FX. United Kingdom, Voca Limited (Company no 1023742, VAT No. 907 9619 87) which is registered in England and Wales at registered office Drake House, Three Rivers Court, Homestead Road, Rickmansworth, Hertfordshire. WD3 1FX. United Kingdom, LINK Interchange Network Limited (Company No 3565766, VAT No. 907 9619 87) which is registered in England and Wales at registered office Drake House, Homestead Road, Rickmansworth, WD3 1FX and VocaLink Holdings Limited (Company No 06119036, VAT No. 907 9619 87) which is registered in England and Wales at registered office Drake House, Homestead Road, Rickmansworth, WD3 1FX. United Kingdom.

The views and opinions expressed in this email may not reflect those of any member of the VocaLink group. This message and any attachments have been scanned for viruses prior to leaving the VocaLink group network; however, VocaLink does not guarantee the security of this message and will not be responsible for any damages arising as a result of any virus being passed on or arising from any alteration of this message by a third party. The VocaLink group may monitor emails sent to and from the VocaLink group network.

*************************************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130911/45e786eb/attachment.htm>