[rabbitmq-discuss] Shovel bug? Consumer hangs after network failure.

[Matthew Sackman]

Hi,

On Thu, Jan 20, 2011 at 09:03:13AM +0100, Øyvind Tjervaag wrote:
> I needed to move messages in both directions between the rabbit
> servers. I first installed the shovel plugin on rabbit2 only, and
> configured that to move messages in both directions. If I consume from
> a queue on Rabbit1 with the shovel on Rabbit2 and disconnect the
> network between them, the consumer doesn't seem to be able to
> reconnect to rabbit1 when the network comes back up, generating errors
> like this:
> 
> The shovel that moves messages from Rabbit2 to Rabbit1 however detects
> that the network was down, and reconnects properly. I've done a
> temporary workaround by always having my shovel plugin on the same
> server where I consume messages.

Yes, this is largely to be expected. Consuming is an asynchronous
activity and the Rabbit1 -> Rabbit2 shovel is just consuming from queues
on Rabbit1. Until the socket is closed by the kernel, it won't
necessarily notice that Rabbit1 has gone down. TCP timeouts can
sometimes be quite long. So, I would recommend that you turn on
heartbeats on that shovel. Unfortunately, looking through the source, I
see there is no way to turn on heartbeats. Sigh. I'll file a bug and get
that added.

Writing to a socket rather than reading from it is much more likely to
prompt an error quickly. Thus if you make your Rabbit1 -> Rabbit2 shovel
a shovel running in Rabbit1 (i.e. Rabbit1 is pushing to Rabbit2 rather
than Rabbit2 pulling from Rabbit1 as it is currently) then you should
find that's more reliable and copes without the addition of heartbeats.

Best wishes,

Matthew