[rabbitmq-discuss] Question regarding setting up a shovel within a cluster

[ron.jaeger at thomsonreuters.com]

I am looking for a little help around the topic of setting up a shovel in a rabbitmq clustered environment.  We have an application that will need to send messages across two different datacenters.   Each datacenter will have a RabbitMQ cluster.   I have set up a shovel in our lab to verify that this will meet the applications requirements.  It is fairly straightforward to set up the shovel between standalone RabbitMQ nodes.  However,  I have ran into issues implementing  the shovel between RabbitMQ clusters, with the error occurring when trying to failover the shovel from one node to another within the cluster.   The problem is mostly around the queues the shovel is using.  The configuration I am testing with is as follows:

[
    {rabbitmq_shovel,
       [{shovels,
         [{test_shovel,
            [{sources, [{brokers,
                          ["amqp://guest:guest@ron-rabbitmq:5672",
                           "amqp://guest:guest@ron-rabbitmq1:5672"] },
                         {declarations,
                            [{'queue.declare',
                                    [{queue, <<"shovel_out_queue">>},
                                      durable]},
                            {'exchange.declare',
                                    [{exchange, <<"shovel_exchange">>},
                                     {type, <<"direct">>},
                                     durable]},
                            {'queue.bind',
                              [{exchange, <<"shovel_exchange">>},
                               {queue, <<"shovel_out_queue">>},
                               {routing_key, <<"shovel_rk">>}]}
                            ]}]},
            {destinations, [{broker, "amqp://guest:guest@ron-rabbitmq2:5672"},
                         {declarations,
                            [{'queue.declare',
                                    [{queue, <<"shovel_in_queue">>},
                                      durable]},
                            {'exchange.declare',
                                    [{exchange, <<"shovel_exchange">>},
                                     {type, <<"direct">>},
                                      durable]},
                            {'queue.bind',
                                    [{exchange, <<"shovel_exchange">>},
                                     {queue, <<"shovel_in_queue">>},
                                     {routing_key, <<"shovel_rk">>}]}
                            ]}]},
            {queue, <<"shovel_out_queue">>},
            {ack_mode, no_ack},
            {publish_properties, [{delivery_mode, 2}]},
            {publish_fields, [{exchange, <<"shovel_exchange">>},
                            {routing_key, <<"shovel_rk">>}]},
            {reconnect_delay, 5}
            ]}
           ]
         }]
      }
].

The error I am enountering is:
** Reason for termination ==
** {{shutdown,
        {server_initiated_close,404,
            <<"NOT_FOUND - home node 'RABBIT1 at ron-rabbitmq' of durable queue 'shovel_out_queue' in vhost '/' is down or inaccessible">>}},

It is evident that it can't access the queue because it was on the other cluster member that is  down.   I have considered having the queues be transient, but there are additional risks associated with that, such as loosing messages during a node failure.  I have also thought about mirroring the queues, but additional overhead may be a concern due to the high volume of traffic.  Any thoughts or suggestions would be appreciated.

Ron Jaeger
Thomson Reuters
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20131003/ef7b77e5/attachment.htm>