[rabbitmq-discuss] Shovel configuration for a clustered broker with mirrored high-availability queues

Sat Jun 30 14:12:49 BST 2012

On Sat, Jun 30, 2012 at 12:42 AM, Eric <ejarendt at gmail.com> wrote:

> I thought I sent along another message, but apparently it didn't go
> through.  Viewing this group through the new Google Groups UI is
> confusing.
>
>
I find it's better to avoid the Google Group, and just use the mailing list
directly.

> I misunderstood the federation configuration; I see now that you
> declare a backing type when you declare the federated exchange, which
> makes sense.  I thought the backing type was based on the upstream
> exchange.
>
> Going back to shovel for a second... if I have two shovels running in
> a cluster, one on each broker, and they connect to their host broker
> and consume from the same mirrored queue (the queue is 'shared' across
> the cluster) will they simply behave like two consumers on the same
> queue, and basically receive messages round-robin?  Because that would
> work fine for my scenario.  They'll share the workload unless one of
> the brokers in the cluster dies, after which one shovel would be doing
> all the work.
>
>
Correct, the shovels will behave as you outlined when consuming.

> On Jun 29, 9:00 am, Eric <ejare... at gmail.com> wrote:
> > Thanks for the reply.
> >
> > The odd thing about the federation plugin is that the downstream broker,
> > which is the 'master' conceptually in my case, has to declare its
> exchange
> > type as 'federation', which means the actual type is based on the
> upstream
> > 'slave' exchange.  That feels strange to me, because I don't want the
> > master to really care about the upstream exchange - it's sort of
> optional.
> >  If the upstream broker is alive that's great, and I want it to forward
> > messages along to the master.  If it's not, or isn't present when the
> > master starts, I don't want that to be a problem.
> >
> > Maybe I'm not understanding federation correctly?  It just struck me as
> odd
> > that the master has to go declaring all of the upstream sources it
> expects,
> > and the other way (with shovel) seemed more natural.
> >
> > I understand that the sources and destination fields accept a list for
> the
> > purpose of failover.  I could configure a single shovel that consumes
> from
> > either broker in the 'slave' cluster and publishes to either broker on
> the
> > master cluster, but I'm worried about the shovel's host broker failing
> and
> > taking the shovel down.
> >
> >
> >
> >
> >
> >
> >
> > On Friday, June 29, 2012 12:31:41 AM UTC-7, Brendan Hay wrote:
> >
> > > Hi Eric,
> >
> > > FYI: The 'sources' and 'destinations' configuration fields both accept
> a
> > > list. The shovel plugin doesn't actually pull/push to/from all of these
> > > simultaneously, it uses them as a form of simple failover - if a
> connection
> > > fails, it uses the next one in the list.
> >
> > > It sounds like for your scenario (when clustering in general), it
> would be
> > > easier to use the federation plugin (
> > >http://www.rabbitmq.com/federation.html). When you declare a federated
> > > exchange on the downstream/master cluster, the plugin auto-magically
> > > declares queues (mirrored if configured) on the upstream/AWS cluster
> .. you
> > > would then bind a mirrored queue to the federated exchange on the
> > > downstream/master cluster to cause messages to be routed across the
> link to
> > > that queue.
> >
> > > The plugin will then stay connected even if one of your nodes on either
> > > cluster go down .. if the whole downstream/master cluster goes
> away/down,
> > > then messages will pile up in the upstream/AWS queues, and be
> transfered
> > > once the link is re-established.
> >
> > > Does that solve your use-case?
> >
> > > Regards,
> > > Brendan
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120630/d069edf5/attachment.htm>