[rabbitmq-discuss] Shovel stuck?

[Benjamin Ballard]

Hello,

I have been seeing similar issues to Alex. I also have a non reliable network that is used to shovel data between two rabbit nodes.  I also see my shovels go down in a terminated state and then never come up again.  My config shovel file has declarations in it to create the exchanges on the remote node.  However sometimes I get the error shown by Alex, and sometimes the rabbit shovel terminates when declaring the exchange (even though it was just connected to that exchange and the connection was working).  It also seems to happen much more frequently when the heartbeat time is set smaller, in the environment that the nodes are working in a 1 second heart beat causes a problem in under 30 min, but moving the heart beat to 5 seconds causes the issue to happen about once a day (Probably environment (non reliable network) specific).  Based on this it would seem to me that that there might be network problems during the starting phase of the shovel (after a disconnect due to heart beat), which cause declarations to fail and the shovel to terminate since having a shovel continuously reconnect with a failing declarations would just cause an infinite loop.  The data (config, shovel status and sasl logs) for these is in https://gist.github.com/3360344 . Since i can detect this failure is there anyway to restart the shovel without restarting rabbit?  Or is there a config option that i am missing to have rabbit shovels to attempt to reconnect even if there is a failure during the starting phase of a rabbit shovel. Could i just detect the shovel failure and then create a new shovel when the first crashes out without restarting all of rabbit?

Thank you,

Ben Ballard
________________________________________
From: rabbitmq-discuss-bounces at lists.rabbitmq.com [rabbitmq-discu 

ss-bounces at lists.rabbitmq.com] on behalf of Emile Joubert [emile at rabbitmq.com]
Sent: Wednesday, August 15, 2012 6:26 AM
To: Discussions about RabbitMQ
Subject: Re: [rabbitmq-discuss] Shovel stuck?

Hi,

On 14/08/12 21:29, Alex Zepeda wrote:
> 1.) The logging is exceedingly verbose.  I've set
> {log_levels, [{connection, none}]}, but shovel still dumps a bunch of info in
> the SASL log.  Are there any tunables to turn this down, or would that require
> recompiling shovel?

There are no other tunables, but it should be possible to rotate and
archive old logs regularly. The only other option is to turn logging off
completely in Erlang, something we don't recommend.

> 2.) Occasionally the shovel gets wedged.  Initially it was from running into
> the free disk space limit, but occasionally I'll run into a situation where
> the shovel just terminates for no good reason (stop_app; start_app seems to
> resolve the issue).  Here's the error I'm seeing:

Are there any log entries in the regular or the sasl log that might help
to explain this? How long after starting a shovel does this happen? Can
you reliably reproduce the problem?


-Emile

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss at lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss