[rabbitmq-discuss] Inconsistent Shovel Delivery

[rparker]

We have a simple shovel that takes messages published to a queue on a
rabbitmq cluster by a custom Windows service (using Mass Transit ) and then
shovels these messages across a WAN to a fanout exchange on a remote
rabbitmq cluster that routes to a queue of the same name on that remote
cluster.  Here's the basic config:

  {rabbitmq_shovel,
             [ {shovels, [ {my_shovel,
                    [ {sources,
                        [ {broker, "amqp://" }
                        ]}
                    , {destinations,
                        [ {broker, "amqp://guest:guest@server2:5672"}
                        ]}
                    , {queue, <<"My_Queue">>}
                    , {prefetch_count, 100}
                    , {ack_mode, on_confirm}
                    , {publish_properties, [ {delivery_mode, 2} ]}
                    , {publish_fields, [ {exchange, <<"My_Exchange">>}]}
                    , {reconnect_delay, 5}
                    ]}
                ]}
    ]}

So, what's happening is that only every other message makes it to the
ultimate queue on the remote cluster.  We originally were addressing the
remote cluster via a load balancer and we thought this was an easy culprit,
but after changing the shovel config to go directly to the master node of
the remote cluster bypassing the LB, the problem still persisted.  We tried
decoupling both clusters and shoveling between two standalone nodes, and the
problem still persisted.  We tried unconfiguring shovel and published
directly to the exchanges/queues on both the local and remote clusters and
the correct number of messages were published.  Its only when shoveling is
used that we are getting an inconsistent number of messages, and when
examining the messages it shows that exactly every other message is making
it to the remote cluster.  No errors are evident in the log file and when I
show a status of the shovel, it shows it as running:

[{my_shovel,
     {running,
         {source,
             {amqp_params_direct,<<"guest">>,none,<<"/">>,
                 rabbit at server1,none,[]}},
         {destination,
             {amqp_params_network,<<"guest">>,<<"guest">>,<<"/">>,
                 "server2",5672,0,0,5,infinity,none,
                 [#Fun<amqp_uri.7.121371571>,#Fun<amqp_uri.7.121371571>],
                 [],[]}}},
     {{2014,3,12},{15,45,57}}}]
...done.

What on earth would cause this behavior other than some sort of faulty
firewall?  Why would only shovel messages be affected and not directly
published messages?  Is there a way to turn up the verbosity of the
shoveling logging?

RabbitMQ version is 3.2.4 with Erlang R16B03-1

Thanks in advance



--
View this message in context: http://rabbitmq.1065348.n5.nabble.com/Inconsistent-Shovel-Delivery-tp34070.html
Sent from the RabbitMQ mailing list archive at Nabble.com.