[rabbitmq-discuss] RabbitMQ clustering

[Marek Majkowski]

Hi,

You have a valid point, just a quick question...

On Mon, Nov 29, 2010 at 07:18, Kshitiz Garg <stephanion2002 at gmail.com> wrote:
> Hi Gustavo,
>
> Sorry for replying late,
>
> We don't have any figure yet, but on our pub-sub system, millions of of
> publishers will be publishing messages.
>
> Our system is as follows:
>
> 1. A publisher publishes message on our rest APIs.
> 2. From there, we first store those messages in DB.
> 3. After storing, we publish the message on RabbitMQ setup using a load
> balancer.
> 4. Behind the load balancer, we have a cluster of 4 rabbitMQ nodes.
> 5. At the consuming end,  we have perl and java consumers fetching messages.
> 7. These internal consumers send messages to the subscribers as per the
> subscription protocol.
>
> Now, if we add 4 RabbitMQ nodes in a cluster, all the credentials, exchanges
> and bindings get copied to each mnesia db. However, since the queues are
> specific to nodes, the message directed to a particular queue will be lost
> (if the node on which that queue was created is down). This seems to be a
> weak link for a scalable and High availability solution we need. Will the
> message be persisted somewhere and delivered to that queue after that
> particular node comes up?
>
> If the message is gonna lost, can we have a workaround that we create copies
> of the queues explicitly on each node. Now since they are copies, the
> exchange will send the message to all the nodes and even if a node is down,
> message will be delivered to all the copies of that queue.
>
> We know that Clustering!=HA, but we do need Clustering with HA. Here since
> even after clustering of 4 rabbitMQ nodes, the active-passive setup for each
> of these 4 nodes becomes the bottleneck because of the complexity of the
> non-replication of "queues". Is there any workaround or any other setup u
> can suggest ?

... have you actually tried a setup with only a single RabbitMQ node, and did
it really became a bottleneck?

Marek


> Regards,
> Kshitiz Garg
>
> On Sun, Nov 28, 2010 at 10:01 PM, Gustavo Aquino <aquino.gustavo at gmail.com>
> wrote:
>>
>> Hi Garg,
>> Could you give to us more details about what you need? eg.  it's messages
>> throughput is per second? and etc.
>> Regards.
>>
>>
>>
>> On Wed, Nov 24, 2010 at 10:35 AM, Kshitiz Garg <stephanion2002 at gmail.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> Can someone present a pictorial representation of 4-5 Rabbit MQ nodes
>>> with Clustering and HA (it can also go on the RabbitMQ site). I wanted to
>>> know about a RabbitMQ setup which is able to handle millions of messages.
>>>
>>> I hope there would be many more users with such a requirement.
>>>
>>> Thanks
>>> Kshitiz Garg
>>>
>>>
>>>
>>> _______________________________________________
>>> rabbitmq-discuss mailing list
>>> rabbitmq-discuss at lists.rabbitmq.com
>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>>
>>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>