[rabbitmq-discuss] Weird behaviour with mirrored queues

Mon Sep 2 09:21:52 BST 2013

Hi,

Tim. cheers for the clarifications.  I'm aware only individual queues are
mirrored but am using the master slave as an analogy (I'm trying to set up
the policies so it looks like 2 groups of 2 within the cluster using the
nodes policy).

All queues are pre-existing and created on mq1 - they all have publishers
and consumers connected.  We start with no policies.  When we create the
appx policy the appx queues successfully move from mq1 to be mirrored across
mq3 and mq4 and change the master to the first node in the policy (as
expected).  If we set the app_y policy without the catchall policy the same
thing happens and everything is hunky-dory.  However if we have the catch
all policy set before we set the app_y policy (effectively meaning the app_y
queues are mirrored across mq1 and mq2) and we then set the app_y policy the
queues disappear from list_queues and we are unable to consume from the
queue (note we don't have the consumer cancel notification property set in
our consumers so we have to manually restart consumers).

They app_y policy takes the exact form of the appx policy but with x
replaced by y.

Unfortunately we don't have the management plugin enabled.  I can try
setting up a similar situation in our dev environment (unfortunately this
has been happening in prod) to see if they will show up in the management
plugin.

The reason we want to move queues is for load balancing.  We want to enable
mirroring for redundancy (losing messages is bad news for us) but with the
overhead we're hitting max CPU and we'd like to scale out rather than up, so
we're balancing queues (we have almost 300) across 2 sets of 2 replicated
nodes (that seems to be the most deterministic way for us to scale).

There's no messages in the sasl log or in the normal log.  The server just
hangs indefinitely when trying to shutdown the node. I suspected the queues
still existed as well.  We have a python script for deleting queues and when
I tried that to delete one of the hidden queues it hung as well, but didn't
error (which is what happens when the queue doesn't exist).

An example of a queue name that matches is
OdometerService_appy_boundto_cooked_tx_y

>Just to clarify then, any queue (name) that matches the `appx_' regex will
be mirrored over the nodes on mq3 and mq4, whereas everything else will be
mirrored across those on hosts mq1 and mq2. 

This is correct.  The problem only presents when we use a policy with a
higher priority to change a queue that is already mirrored by the catchall
policy.  If we just change the nodes in an existing policy it is fine. 
Changing masters by restarting nodes also works fine.

--
View this message in context: http://rabbitmq.1065348.n5.nabble.com/Weird-behaviour-with-mirrored-queues-tp29254p29262.html
Sent from the RabbitMQ mailing list archive at Nabble.com.