[rabbitmq-discuss] Architecture Question: 2 Brokers, or Not 2 Brokers ?

[Simone Busoli]

Hi dns, I'll try to bring my own experience on the subject. When you
federate exchanges between brokers you are in the loose sense having the
downstream exchange acting as a consumer for the upstream exchange. I may
have misunderstood you but I think that having 8000 brokers rather 8000
remote consumers on the central broker doesn't solve the consistency
problem you are describing.

Let's make an example with federated brokers on the happy path:

   1. One message is published on the upstream exchange and reaches the
   downstream exchange, and in turn the queue and then the consumer on the
   downstream broker
   2. The downstream broker confirms the message to the upstream broker so
   the upstream broker can forget about it
   3. The connection between the brokers goes down
   4. The consumer on the downstream broker processes the message,
   acknowledges it and everyone's happy

Here's the sad path with federated brokers:

   1. same as above
   2. The connection between the brokers is lost before the downstream can
   confirm the message to the upstream (unlikely, but can definitely happen)
   3. The consumer on the downstream broker processes the message and
   acknowledges it to its local broker
   4. The connection between the two brokers is restored, and thus the
   upstream will resend the unconfirmed messages once again
   5. The consumer on the downstream will receive the same message it has
   already consumed, therefore it will need a way to deduplicate it

Now the safe path with just the central broker:

   1. One message is published on the central broker and reaches a remote
   consumers
   2. The consumer processes the message but the connection to the central
   broker will go down before it can ack the message
   3. Once the connection is restored the consumer will receive the unacked
   message again, so also in this case it will need to deduplicate it

Some observations:

   - in any case your consumer will need to be somewhat aware of the fact
   that it is quite distant from the original broker and cope with what this
   implies in some way (message duplication, for one)
   - with the central broker the remote consumers will have to handle
   connection failures themselves, something which is less likely to happen
   with a local broker, although not impossible of course. This is handled by
   the federation plugin in case of federated brokers instead
   - Having 8000 brokers rather than 8000 consumers is going to require
   quite some maintenance, and I'm not sure how well the federation plugin
   RabbitMQ can handle that, I am under the impression that it was not
   designed for this many federated exchanges, but I may be wrong
   - You mention synchronicity between the central location and the remote
   consumers. Consider that even if you federate brokers and you see all
   messages delivered and acked on the federation queues, it doesn't tell you
   anything about whether the remote consumers have processed the messages or
   not


On Mon, Jan 23, 2012 at 19:54, dnsplus <developer at dnsplus.net> wrote:

>
>
>
> Jerry Kuch wrote:
> >
> > Hi, dnsplus:
> >
> > Could you perhaps clarify a bit what you mean by "remote?"  Are you
> > talking about systems that are geographically separated from your
> > "central broker" for some reason external to your application?  Or
> > are you just talking about having 8,000 consumers?  The latter
> > shouldn't be a daunting number at all for even fairly modest Rabbit
> > deployments.
> >
> > Best regards,
> > Jerry
> >
> >
>
> Its a little complicated, but I'll give it my best shot:
>
> Jerry, for you ... Definition of a Site:  A retail store location, from 1
> to
> ~1800 miles away from the data center where the Central Broker runs).
>
> I am working with a large environment which has no guarantee of a stable
> network, or guaranteed bandwidth to the remote sites. The typical CIR is
> 384Kbps, with a 1.544Mbps MAX (yep, fractional T1s). To mitigate the
> possibility that network connectivity to a site, or region, can be
> unavailable for an undetermined period of time, at any time, and for any
> reason ... we think guaranteed message delivery with RabbitMQ is the
> foundation we need for this 'application' to work.
>
> The message payload for this application will be well formatted XML which
> contains instructions that will be parsed, interpreted, and executed via a
> consumer at each client site, implemented in PERL.
>
> So ... should I be using a Central Broker, which forwards a message to a
> queue at a remote broker, which is consumed by the client process?
>
> Or, do I leverage the Central Broker with a Queue for each remote site, and
> configure the consumers to talk to the Central Broker for message
> collection
> and processing.
>
> From an accounting perspective, I am thinking the Central Broker is going
> to
> be easier to query for unconsumed messages, or query a common error queue
> which is written to by the remote client, when things go amiss.
>
> But, should there be a network interruption between the time a client
> collects the message from Central, and starts the execution of instructions
> it contains, the Central Que would fail to get an ACK, or possible error
> message from the client. This could result in a lack of synchronicity
> between reality as understood by the remote client consumer, and the
> accounting and reporting mechanisms at Central.
>
> Hopefully I have conveyed more than my own confusion about this project.
>
> Thanks !
>
>
>
> --
> View this message in context:
> http://old.nabble.com/Architecture-Question%3A-2-Brokers%2C-or-Not-2-Brokers---tp33189279p33190235.html
> Sent from the RabbitMQ mailing list archive at Nabble.com.
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120123/6fbff298/attachment.htm>