Hi Kshitiz,<div><br></div><div>Unfortunately until now in my projects using Rabbit we don't found a way to use Rabbit in HA/HP mode, just HA active/passive and I think that you are trying to found one HA/HP that represents all nodes active.</div>
<div><br></div><div>Just to understand you are talking about millions of messages, do you really need 4 nodes to process it?, We process 1kk messages per second with a single node, messages with ~260bytes.</div><div><br></div>
<div>Talk about your case to us.</div><div><br></div><div>Regards.</div><div><br></div><div><br><br><div class="gmail_quote">On Mon, Nov 29, 2010 at 5:18 AM, Kshitiz Garg <span dir="ltr"><<a href="mailto:stephanion2002@gmail.com">stephanion2002@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Hi Gustavo,<br><br>Sorry for replying late,<br><br>We don't have any figure yet, but on our pub-sub system, millions of of publishers will be publishing messages.<br>
<br>Our system is as follows:<br><br>1. A publisher publishes message on our rest APIs. <br>
2. From there, we first store those messages in DB.<br>3. After storing, we publish the message on RabbitMQ setup using a load balancer.<br>4. Behind the load balancer, we have a cluster of 4 rabbitMQ nodes.<br>5. At the consuming end, we have perl and java consumers fetching messages.<br>
7. These internal consumers send messages to the subscribers as per the subscription protocol.<br><br>Now, if we add 4 RabbitMQ nodes in a cluster, all the credentials, exchanges
and bindings get copied to each mnesia db. However, since the queues are
specific to nodes, the message directed to a particular queue will be
lost (if the node on which that queue was created is down). This seems to be a weak link for a
scalable and High availability solution we need. Will the message be persisted
somewhere and delivered to that queue after that particular node
comes up?<br>
<br>
If the message is gonna lost, can we have a workaround that we create
copies of the queues explicitly on each node. Now since they are copies,
the exchange will send the message to all the nodes and even if a node
is down, message will be delivered to all the copies of that queue.<br>
<br>We know that Clustering!=HA, but we do need Clustering with HA. Here since even after clustering of 4 rabbitMQ nodes, the active-passive setup for each of these 4 nodes becomes the bottleneck because of the complexity of the non-replication of "queues". Is there any workaround or any other setup u can suggest ?<br>
<br>Regards,<br><font color="#888888">Kshitiz Garg</font><div><div></div><div class="h5"><br><br><div class="gmail_quote">On Sun, Nov 28, 2010 at 10:01 PM, Gustavo Aquino <span dir="ltr"><<a href="mailto:aquino.gustavo@gmail.com" target="_blank">aquino.gustavo@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">Hi Garg,<div><br></div><div>Could you give to us more details about what you need? eg. it's messages throughput is per second? and etc.</div>
<div><br></div><div>Regards.</div><div><br></div><div><br></div><div><br><br>
<div class="gmail_quote"><div><div></div><div>On Wed, Nov 24, 2010 at 10:35 AM, Kshitiz Garg <span dir="ltr"><<a href="mailto:stephanion2002@gmail.com" target="_blank">stephanion2002@gmail.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex"><div><div></div><div>
Hi,<br><br>Can someone present a pictorial representation of 4-5 Rabbit MQ nodes with Clustering and HA (it can also go on the RabbitMQ site). I wanted to know about a RabbitMQ setup which is able to handle millions of messages.<br>
<br>I hope there would be many more users with such a requirement.<br><br>Thanks <br><font color="#888888">Kshitiz Garg<br><br><br>
</font><br></div></div><div>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br></div></blockquote></div><br></div>
</blockquote></div><br>
</div></div></blockquote></div><br></div>