[rabbitmq-discuss] Proposed change to shovel

Thu Apr 18 16:14:19 BST 2013

Hi Fred,

So this looks reasonable enough to me, although it's not really 'failover' as such. Perhaps 'robust endpoint selection' is a better description. 

On 18 Apr 2013, at 16:04, Fred Dushin wrote:

> Hi,
> 
> I would like to consider a change to the shovel plugin, to allow the shovel to failover more predictably against the list of broker endpoints in the rabbitmq_shovel configuration.  Essentially, the idea is to try each endpoint in order of the declaration, and to use the first endpoint to which a connection can be made, instead of selecting an endpoint at random.
> 
> This does slightly violate the spirit of Erlang's "let it fail" philosophy, and I am not sure if the random endpoint selection in the current code is really more designed for clustered environments.  Perhaps it would make more sense to make the iterative failover strategy an "opt-in" feature, if not simply for the sake of backwards-compatibility.

I'm not sure why random endpoint selection was previously selected. I'll do some hg archeology and try to figure that out at some point...

BTW have you see this: http://hg.rabbitmq.com/rabbitmq-shovel/rev/81231b905a3f ?

That allows for proper 'failover' when shovels are used in a cluster, but also means that a named shovel can only run on one node at once. With that in place, a shovel worker can fail over (i.e., migrate) to another node if it's node crashes. The two things are clearly orthogonal, but it's worth being aware of.

> 
> I am providing a patch, but not suggesting that this actual change go into production code, as I really did the change for testing purposes.  I'm really more interested in whether others would find the patch useful, and whether there would be a way for us to fold a feature like this into the shovel.  If there is interest, I would be happy to contribute something more production-worthy, for review.
> 

If there is sufficient interest, then we could certainly consider a patch. Let's see how this plays out for now... 

Cheers,
Tim