[rabbitmq-discuss] Proposed change to shovel

Fri Apr 19 14:20:59 BST 2013

On Apr 18, 2013, at 11:23 AM, Simon MacMullen <simon at rabbitmq.com> wrote:

> On 18/04/13 16:04, Fred Dushin wrote:
>> I would like to consider a change to the shovel plugin, to allow the
>> shovel to failover more predictably against the list of broker endpoints
>> in the rabbitmq_shovel configuration.  Essentially, the idea is to try
>> each endpoint in order of the declaration, and to use the first endpoint
>> to which a connection can be made, instead of selecting an endpoint at
>> random.
> 
> Why? How is this better than the random selection?

In my case, I am pushing messages across WANs, potentially to different data centers, where some are treated as backups in the case where primaries go down.  So having a list of destinations provides some resiliency in the case of network or hardware failure.  I think in cases like this, operators need some predictability in where the messages are being sent.

> 
>> This does slightly violate the spirit of Erlang's "let it fail"
>> philosophy, and I am not sure if the random endpoint selection in the
>> current code is really more designed for clustered environments.
> 
> Yeah, that's the intention.
> 
>>  Perhaps it would make more sense to make the iterative failover
>> strategy an "opt-in" feature, if not simply for the sake of
>> backwards-compatibility.
> 
> We could do that. But I'd like to see a reason for this feature to exist.

I agree that is adds complexity to a fairly tight bit of code, hence increases test complexity, imposes an maintenance burden, etc.  So I can certainly understand the reluctance to fold it in.  I am mostly curious if other users have the same set of requirements, or whether instead the source and destination lists are really more used in clustering scenarios.  For all I know, I am abusing the use-case for which the multi-endpoint feature was designed.

-Fred

> 
> Cheers, Simon
> 
> -- 
> Simon MacMullen
> RabbitMQ, VMware