[rabbitmq-discuss] Shovel questions

Mon Jan 23 21:44:42 GMT 2012

On Sun, Jan 22, 2012 at 4:46 AM, Simone Busoli <simone.busoli at gmail.com>wrote:

> On Sat, Jan 21, 2012 at 23:58, Elias Levy <fearsome.lucidity at gmail.com>wrote:
>
>> We are thinking of exposing a portion of our message bus to customers.
>>  To do we we will be creating a new cluster that can be exposed and will be
>> linking it to our existing cluster.
>>
>> I've looked a the Federation plug-in, but it has a feature disqualifies
>> it for our purposes.   We'd like to keep communications from our existing
>> cluster to the new exposed cluster unidirectional.  That means now allowing
>> incoming connections into the existing cluster, in case there is ever a
>> RabbitMQ vulnerability (or say, an Erlang or OpenSSL vulnerability) that
>> would allow the external cluster to be compromised.  Since both cluster are
>> running RabbitMQ, we don't want an attacker to be able to jump from the
>> external cluster to the internal one.  From what I've gathered, the AMQP
>> client that the Federation plug-in uses executed within the downstream
>> broker, and thus it has to be able to make an AMQP connection to the
>> upstream broker.
>>
>> Is that correct?  Any plans to allow for the opposite behavior (upstream
>> connects to the downstream broker)?
>>
>
> I don't know the internals of the federation plugin well enough to be able
> to answer this question. I know that it uses exchange-exchange bindings and
> I'm not sure if it's even feasible to have it work in the way you describe.
>

I think you are wrong.  The Federation plug-in does not use
exchange-to-exchange bindings.  Such bindings are not supported across
virtual hosts, never mind across clusters.  The Federation plug-in merely
provides a nice illusion of such binding by using an internal AMQP client
as I described.

Any other takers?

That leaves us with the Shovel plug-in, which can be configured to run on
>> either the upstream or downstream broker.
>>
>> One problem with the Shovel plug-in is that it appears you can only
>> manage the shovels within the configuration file.  Is there no way to
>> manage shovels online, either through AMQP or a management plug-in?
>>
>
> No, there is no way to do that except in the configuration file.
>

That's too bad.

> If neither of those are possible, can RabbitMQ reload its configuration
>> without restarting?
>>
>
> No, you cannot do that. Nonetheless be aware that shovel is not hard to
> set up manually as an external component from the broker, although with
> some gotchas. We've been doing that for some time before the federation
> plugin was published.
>

Interesting.  Do you have any documentation on how one may go about doing
so?  Or are you merely standing up a new RabbitMQ instance with the shovel
configured, and not using the broker in those new instances?

One disadvantage of running the shovel appart from the downstream and
upstream brokers is that now the messages must cross the network twice,
once from the upstream to the shovel, and then from the shovel to the
downstream.  With the shovel within the upstream or downstream broker only
one network transfer is needed.

> Finally, its unclear whether you should only define a shovel in a single
>> node in a cluster or in multiple ones?  If you define it in multiple ones,
>> does it create multiple shovel clients competing for messages?  If not,
>> will a shovel fail over if the node with the existing client fails?
>>
>
> Shovel is a low level mechanis, so whatever you configure it, it will do
> what you told it to, i.e. pick messages from one queue and publish them to
> some exchange. How you set it up is then up to you according to what you
> want to accomplish.
>

I understand that.  My question is whether the shovel is cluster aware,
like the federation plug-in.

The federation plug-in, when configured on all nodes, will only make a
single connection to the upstream broker, and if the nodes with the
connection dies, another node will create a new one.

My understanding is that the shovel is not cluster aware.  If you configure
the shovel on each node in your cluster, you'll end up with as many shovels
and connections as there are nodes.  That's not particularly bad, as it
will give you load balancing across nodes, but I want to confirm my
understanding.

Elias
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120123/212f988e/attachment.htm>