[rabbitmq-discuss] Shovel messages get lost or stay unacknowledged.

[Jelle Smet]

Hi all,

Sorry for double posting, but I wanted to start a new thread on this topic 
as the previous one<https://groups.google.com/forum/#!topic/rabbitmq-discuss/7grvcHWvp3s/discussion>had multiple angles covering multiple cases.

I'm having issues with the Shovel plugin, which currently makes my work 
with RabbitMQ difficult.
My goal is to get the content of a queue to a queue of a remote broker.

Here's a recap:

   - All RabbitMQ instances run the latest stable version 2.7.0 (the 
   problem also occurred with 2.6.1)
   - Messages which should be shoveled to a remote broker stay 
   unacknowledged in the queue which Shovel is using as a source.
   - Restarting RabbitMQ on that moment isn't possible.  Killing is 
   required, often having to issue "epmd -kill"
   - When I restart RabbitMQ which has the Shovel plugin, all messages get 
   delivered correctly.

By default I use ack_mode with value "on_confirm".  When I change this into 
"no_ack" and I run into the above situation all messages get lost.

I must say occasional network interruptions, or occasional packetloss is a 
fact.  But not to that extent it should be a problem. Besides that, it 
doesn't explain that restarting Rabbit, makes all messages arrive correctly 
all of a sudden.

Here's my complete Shovel configuration (I replaced hostnames & queue 
names):

[{rabbitmq_shovel,
  [{shovels,
    [
     {scan_orders_amsterdam,
      [{sources,      [{broker,"amqp://localhost"}]},
       {destinations, [{broker, "amqp://server01"}]},
       {queue, <<"messages_server01">>},
       {ack_mode, on_confirm},
       {publish_properties, [{delivery_mode, 2}]},
       {publish_fields, [{exchange, <<"">>},
                         {routing_key, <<"assignments">>}]},
       {reconnect_delay, 5}
      ]},
     {scan_orders_gent,
      [{sources,      [{broker,"amqp://localhost"}]},
       {destinations, [{broker, "amqp://server02"}]},
       {queue, <<"messages_server02">>},
       {ack_mode, on_confirm},
       {publish_properties, [{delivery_mode, 2}]},
       {publish_fields, [{exchange, <<"">>},
                         {routing_key, <<"assignments">>}]},
       {reconnect_delay, 5}
      ]},
     {scan_orders_lebanon,
      [{sources,      [{broker,"amqp://localhost"}]},
       {destinations, [{broker, "amqp://server03"}]},
       {queue, <<"messages_server03">>},
       {ack_mode, on_confirm},
       {publish_properties, [{delivery_mode, 2}]},
       {publish_fields, [{exchange, <<"">>},
                         {routing_key, <<"assignments">>}]},
       {reconnect_delay, 5}
      ]},
     {scan_orders_pune,
      [{sources,      [{broker,"amqp://localhost"}]},
       {destinations, [{broker, "amqp://server04"}]},
       {queue, <<"messages_server04">>},
       {ack_mode, on_confirm},
       {publish_properties, [{delivery_mode, 2}]},
       {publish_fields, [{exchange, <<"">>},
                         {routing_key, <<"assignments">>}]},
       {reconnect_delay, 5}
      ]},
     {scan_orders_lodz,
      [{sources,      [{broker,"amqp://localhost"}]},
       {destinations, [{broker, "amqp://server05"}]},
       {queue, <<"messages_server05">>},
       {ack_mode, on_confirm},
       {publish_properties, [{delivery_mode, 2}]},
       {publish_fields, [{exchange, <<"">>},
                         {routing_key, <<"assignments">>}]},
       {reconnect_delay, 5}
      ]}
     ]
   }]
}].

The problem I would like to get solved is that my messages just arrive on 
their destination, without having to restart Rabbit frequently.
Any pointers, work-arounds, fixes, tips are welcome.

Cheers,

Jelle
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20111118/376fc0dd/attachment.htm>