[rabbitmq-discuss] RabbitMQ clustering

[Kshitiz Garg]

Hi Gustavo,

Sorry for replying late,

We don't have any figure yet, but on our pub-sub system, millions of of
publishers will be publishing messages.

Our system is as follows:

1. A publisher publishes message on our rest APIs.
2. From there, we first store those messages in DB.
3. After storing, we publish the message on RabbitMQ setup using a load
balancer.
4. Behind the load balancer, we have a cluster of 4 rabbitMQ nodes.
5. At the consuming end,  we have perl and java consumers fetching messages.
7. These internal consumers send messages to the subscribers as per the
subscription protocol.

Now, if we add 4 RabbitMQ nodes in a cluster, all the credentials, exchanges
and bindings get copied to each mnesia db. However, since the queues are
specific to nodes, the message directed to a particular queue will be lost
(if the node on which that queue was created is down). This seems to be a
weak link for a scalable and High availability solution we need. Will the
message be persisted somewhere and delivered to that queue after that
particular node comes up?

If the message is gonna lost, can we have a workaround that we create copies
of the queues explicitly on each node. Now since they are copies, the
exchange will send the message to all the nodes and even if a node is down,
message will be delivered to all the copies of that queue.

We know that Clustering!=HA, but we do need Clustering with HA. Here since
even after clustering of 4 rabbitMQ nodes, the active-passive setup for each
of these 4 nodes becomes the bottleneck because of the complexity of the
non-replication of "queues". Is there any workaround or any other setup u
can suggest ?

Regards,
Kshitiz Garg

On Sun, Nov 28, 2010 at 10:01 PM, Gustavo Aquino
<aquino.gustavo at gmail.com>wrote:

> Hi Garg,
>
> Could you give to us more details about what you need? eg.  it's messages
> throughput is per second? and etc.
>
> Regards.
>
>
>
>
> On Wed, Nov 24, 2010 at 10:35 AM, Kshitiz Garg <stephanion2002 at gmail.com>wrote:
>
>> Hi,
>>
>> Can someone present a pictorial representation of 4-5 Rabbit MQ nodes with
>> Clustering and HA (it can also go on the RabbitMQ site). I wanted to know
>> about a RabbitMQ setup which is able to handle millions of messages.
>>
>> I hope there would be many more users with such a requirement.
>>
>> Thanks
>> Kshitiz Garg
>>
>>
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20101129/e27bdf0e/attachment.htm>