[rabbitmq-discuss] RabbitMQ clustering

[Gustavo Aquino]

Hi Kshitiz,

Unfortunately until now in my projects using Rabbit we don't found a way to
use Rabbit in HA/HP mode, just HA active/passive and I think that you are
trying to found one HA/HP that represents all nodes active.

Just to understand you are talking about millions of messages, do you really
need 4 nodes to process it?, We process 1kk messages per second with a
single node, messages with ~260bytes.

Talk about your case to us.

Regards.



On Mon, Nov 29, 2010 at 5:18 AM, Kshitiz Garg <stephanion2002 at gmail.com>wrote:

> Hi Gustavo,
>
> Sorry for replying late,
>
> We don't have any figure yet, but on our pub-sub system, millions of of
> publishers will be publishing messages.
>
> Our system is as follows:
>
> 1. A publisher publishes message on our rest APIs.
> 2. From there, we first store those messages in DB.
> 3. After storing, we publish the message on RabbitMQ setup using a load
> balancer.
> 4. Behind the load balancer, we have a cluster of 4 rabbitMQ nodes.
> 5. At the consuming end,  we have perl and java consumers fetching
> messages.
> 7. These internal consumers send messages to the subscribers as per the
> subscription protocol.
>
> Now, if we add 4 RabbitMQ nodes in a cluster, all the credentials,
> exchanges and bindings get copied to each mnesia db. However, since the
> queues are specific to nodes, the message directed to a particular queue
> will be lost (if the node on which that queue was created is down). This
> seems to be a weak link for a scalable and High availability solution we
> need. Will the message be persisted somewhere and delivered to that queue
> after that particular node comes up?
>
> If the message is gonna lost, can we have a workaround that we create
> copies of the queues explicitly on each node. Now since they are copies, the
> exchange will send the message to all the nodes and even if a node is down,
> message will be delivered to all the copies of that queue.
>
> We know that Clustering!=HA, but we do need Clustering with HA. Here since
> even after clustering of 4 rabbitMQ nodes, the active-passive setup for each
> of these 4 nodes becomes the bottleneck because of the complexity of the
> non-replication of "queues". Is there any workaround or any other setup u
> can suggest ?
>
> Regards,
> Kshitiz Garg
>
>
> On Sun, Nov 28, 2010 at 10:01 PM, Gustavo Aquino <aquino.gustavo at gmail.com
> > wrote:
>
>> Hi Garg,
>>
>> Could you give to us more details about what you need? eg.  it's messages
>> throughput is per second? and etc.
>>
>> Regards.
>>
>>
>>
>>
>> On Wed, Nov 24, 2010 at 10:35 AM, Kshitiz Garg <stephanion2002 at gmail.com>wrote:
>>
>>> Hi,
>>>
>>> Can someone present a pictorial representation of 4-5 Rabbit MQ nodes
>>> with Clustering and HA (it can also go on the RabbitMQ site). I wanted to
>>> know about a RabbitMQ setup which is able to handle millions of messages.
>>>
>>> I hope there would be many more users with such a requirement.
>>>
>>> Thanks
>>> Kshitiz Garg
>>>
>>>
>>>
>>> _______________________________________________
>>> rabbitmq-discuss mailing list
>>> rabbitmq-discuss at lists.rabbitmq.com
>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20101130/bc87ccef/attachment.htm>