[rabbitmq-discuss] RabbitMQ clustering

[Kshitiz Garg]

Hi Gustavo/Marek,

Please help us in designing the setup.

Regards,
Kshitiz Garg

On Mon, Nov 29, 2010 at 4:00 PM, Kshitiz Garg <stephanion2002 at gmail.com>wrote:

> Hi Marek,
>
> For POC it worked, but we know that since ours is a "cloud" based
> application, it will scale. So we have to chose a broker carefully. Do we
> have a workaround for this ?
>
> Regards,
> Kshitiz Garg
> HP R&D Division
>
>
>  On Mon, Nov 29, 2010 at 3:49 PM, Marek Majkowski <majek04 at gmail.com>wrote:
>
>> Hi,
>>
>> You have a valid point, just a quick question...
>>
>> On Mon, Nov 29, 2010 at 07:18, Kshitiz Garg <stephanion2002 at gmail.com>
>> wrote:
>> > Hi Gustavo,
>> >
>> > Sorry for replying late,
>> >
>> > We don't have any figure yet, but on our pub-sub system, millions of of
>> > publishers will be publishing messages.
>> >
>> > Our system is as follows:
>> >
>> > 1. A publisher publishes message on our rest APIs.
>> > 2. From there, we first store those messages in DB.
>> > 3. After storing, we publish the message on RabbitMQ setup using a load
>> > balancer.
>> > 4. Behind the load balancer, we have a cluster of 4 rabbitMQ nodes.
>> > 5. At the consuming end,  we have perl and java consumers fetching
>> messages.
>> > 7. These internal consumers send messages to the subscribers as per the
>> > subscription protocol.
>> >
>> > Now, if we add 4 RabbitMQ nodes in a cluster, all the credentials,
>> exchanges
>> > and bindings get copied to each mnesia db. However, since the queues are
>> > specific to nodes, the message directed to a particular queue will be
>> lost
>> > (if the node on which that queue was created is down). This seems to be
>> a
>> > weak link for a scalable and High availability solution we need. Will
>> the
>> > message be persisted somewhere and delivered to that queue after that
>> > particular node comes up?
>> >
>> > If the message is gonna lost, can we have a workaround that we create
>> copies
>> > of the queues explicitly on each node. Now since they are copies, the
>> > exchange will send the message to all the nodes and even if a node is
>> down,
>> > message will be delivered to all the copies of that queue.
>> >
>> > We know that Clustering!=HA, but we do need Clustering with HA. Here
>> since
>> > even after clustering of 4 rabbitMQ nodes, the active-passive setup for
>> each
>> > of these 4 nodes becomes the bottleneck because of the complexity of the
>> > non-replication of "queues". Is there any workaround or any other setup
>> u
>> > can suggest ?
>>
>> ... have you actually tried a setup with only a single RabbitMQ node, and
>> did
>> it really became a bottleneck?
>>
>> Marek
>>
>>
>> > Regards,
>> > Kshitiz Garg
>> >
>> > On Sun, Nov 28, 2010 at 10:01 PM, Gustavo Aquino <
>> aquino.gustavo at gmail.com>
>> > wrote:
>> >>
>> >> Hi Garg,
>> >> Could you give to us more details about what you need? eg.  it's
>> messages
>> >> throughput is per second? and etc.
>> >> Regards.
>> >>
>> >>
>> >>
>> >> On Wed, Nov 24, 2010 at 10:35 AM, Kshitiz Garg <
>> stephanion2002 at gmail.com>
>> >> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> Can someone present a pictorial representation of 4-5 Rabbit MQ nodes
>> >>> with Clustering and HA (it can also go on the RabbitMQ site). I wanted
>> to
>> >>> know about a RabbitMQ setup which is able to handle millions of
>> messages.
>> >>>
>> >>> I hope there would be many more users with such a requirement.
>> >>>
>> >>> Thanks
>> >>> Kshitiz Garg
>> >>>
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> rabbitmq-discuss mailing list
>> >>> rabbitmq-discuss at lists.rabbitmq.com
>> >>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>> >>>
>> >>
>> >
>> >
>> > _______________________________________________
>> > rabbitmq-discuss mailing list
>> > rabbitmq-discuss at lists.rabbitmq.com
>> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>> >
>> >
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20101130/57edda52/attachment.htm>