[rabbitmq-discuss] RabbitMQ clustering

Mon Nov 29 10:30:47 GMT 2010

Hi Marek,

For POC it worked, but we know that since ours is a "cloud" based
application, it will scale. So we have to chose a broker carefully. Do we
have a workaround for this ?

Regards,
Kshitiz Garg
HP R&D Division

 On Mon, Nov 29, 2010 at 3:49 PM, Marek Majkowski <majek04 at gmail.com> wrote:

> Hi,
>
> You have a valid point, just a quick question...
>
> On Mon, Nov 29, 2010 at 07:18, Kshitiz Garg <stephanion2002 at gmail.com>
> wrote:
> > Hi Gustavo,
> >
> > Sorry for replying late,
> >
> > We don't have any figure yet, but on our pub-sub system, millions of of
> > publishers will be publishing messages.
> >
> > Our system is as follows:
> >
> > 1. A publisher publishes message on our rest APIs.
> > 2. From there, we first store those messages in DB.
> > 3. After storing, we publish the message on RabbitMQ setup using a load
> > balancer.
> > 4. Behind the load balancer, we have a cluster of 4 rabbitMQ nodes.
> > 5. At the consuming end,  we have perl and java consumers fetching
> messages.
> > 7. These internal consumers send messages to the subscribers as per the
> > subscription protocol.
> >
> > Now, if we add 4 RabbitMQ nodes in a cluster, all the credentials,
> exchanges
> > and bindings get copied to each mnesia db. However, since the queues are
> > specific to nodes, the message directed to a particular queue will be
> lost
> > (if the node on which that queue was created is down). This seems to be a
> > weak link for a scalable and High availability solution we need. Will the
> > message be persisted somewhere and delivered to that queue after that
> > particular node comes up?
> >
> > If the message is gonna lost, can we have a workaround that we create
> copies
> > of the queues explicitly on each node. Now since they are copies, the
> > exchange will send the message to all the nodes and even if a node is
> down,
> > message will be delivered to all the copies of that queue.
> >
> > We know that Clustering!=HA, but we do need Clustering with HA. Here
> since
> > even after clustering of 4 rabbitMQ nodes, the active-passive setup for
> each
> > of these 4 nodes becomes the bottleneck because of the complexity of the
> > non-replication of "queues". Is there any workaround or any other setup u
> > can suggest ?
>
> ... have you actually tried a setup with only a single RabbitMQ node, and
> did
> it really became a bottleneck?
>
> Marek
>
>
> > Regards,
> > Kshitiz Garg
> >
> > On Sun, Nov 28, 2010 at 10:01 PM, Gustavo Aquino <
> aquino.gustavo at gmail.com>
> > wrote:
> >>
> >> Hi Garg,
> >> Could you give to us more details about what you need? eg.  it's
> messages
> >> throughput is per second? and etc.
> >> Regards.
> >>
> >>
> >>
> >> On Wed, Nov 24, 2010 at 10:35 AM, Kshitiz Garg <
> stephanion2002 at gmail.com>
> >> wrote:
> >>>
> >>> Hi,
> >>>
> >>> Can someone present a pictorial representation of 4-5 Rabbit MQ nodes
> >>> with Clustering and HA (it can also go on the RabbitMQ site). I wanted
> to
> >>> know about a RabbitMQ setup which is able to handle millions of
> messages.
> >>>
> >>> I hope there would be many more users with such a requirement.
> >>>
> >>> Thanks
> >>> Kshitiz Garg
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> rabbitmq-discuss mailing list
> >>> rabbitmq-discuss at lists.rabbitmq.com
> >>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> >>>
> >>
> >
> >
> > _______________________________________________
> > rabbitmq-discuss mailing list
> > rabbitmq-discuss at lists.rabbitmq.com
> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20101129/e6c822bc/attachment-0001.htm>