[rabbitmq-discuss] RabbitMQ Shovel Connections

Mon Sep 26 12:54:21 BST 2011

Hi Mihir,

On Thu, Sep 22, 2011 at 09:25:26AM +1000, Mihir Mone wrote:
> I have currently setup the shovelling of messages from one of our client
> servers to our end via SSL. Everything works beautifully when you first
> start the servers. However, over time (2-3 days) messages just seem to
> get lost in transit. I have setup shovel to do transactional forwards so
> as to ascertain that the messages are indeed getting delivered. When I
> look at the client side server it does not show any piled up messages
> (making me believe that the messages are indeed getting forwarded
> properly). However, at our end there are no messages getting delivered
> and the queues are empty. When I look at the connections list (using the
> management plugin) I can see that the connection is still active. The
> only solution I have found so far is to restart both the servers so that
> the shovel connections get recycled.

That's mighty odd. I'd suggest you check the routing topology and such -
there's no other process deleting bindings or such that the shovel is
rely on is there? I.e. check that by the time you reach this problematic
state, that the shovel publishing messages on the downstream side isn't
just throwing those messages into the ether - sufficient
bindings/exchanges/queues exist such that those msgs really will reach
the intended queue.

> Restarting the servers can be a hassle sometimes since we have multiple
> client servers connecting to the same servers at our end. It is only
> recently that I have started using the shovel plugin. Earlier we had
> just written a shovelling application ourselves which used to perform
> the consume-publish operation but wasn't as flexible or featureful as
> the shovel plugin.

Ok. Restarting should definitely not be necessary - something is going
wrong here. As well as checking that everything is remaining set up to
correctly route messages on the down-stream side, please also have a
look through the logs on the downstream to see whether any errors are
occurring. If the shovel itself is actually running as a plugin on the
upstream side then check those logs too as the shovel will report
problems to its local node.

> So, what I was thinking was to put may be an option in the shovel
> configuration (something like "recycle_connection") so that we can force
> a connection recycle every few days (ideally I would like to do it every
> 24-36 hrs). The whole point of restarting the servers is so that the
> connections get recycled. So if there was a way in the plugin itself to
> do this, the whole server won't need a restart.

There isn't such an option, but this shouldn't be necessary either.
Something else is going wrong which should be found and fixed.

Matthew