<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Gustavo,</div><div><br></div><div>1) We haven't benchmarked the overhead of using LVS</div><div><br></div><div>2) As you can find out from our numbers, not that much, and it's hard to tell, since during peak time we have 20.000 concurrent users, but maybe in the morning the are only 5000. And at the end it depends on what they do on the site. Soon we plan to add other 300.000 messages a day.</div><div><br></div><div>3) The queues are persistent. </div><div><br></div><div>This means that if one node is out of the LVS, anyway we can reach that node by it's exact IP. In our setup that node won't receive more messages –because the publishers connect to the LVS IP–, but we are able to consume the old ones.</div><div><br></div><div>4) We don't share storage. 1 message will go to one queue, there's no replication. The sysadmin is working on solving that. I prefer to try something "native" i.e.: provided or implemented inside RabbitMQ. He's researching some Linux technologies for filesystems.</div><div><br></div><div>Regards,</div><div><br></div><div>Alvaro</div><div><br></div><div>P. S. My second name is Gustavo, so this feels like replying to myself :)</div><br><div><div>On Mar 18, 2010, at 11:59 PM, Gustavo Aquino wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">Alvaro,<div><br></div><div>Thank you so much to report your experience.</div><div><br></div><div>I will take a look about LVM now.. so I would like to do some questions that I'm in doubt about your solution. Do you know what is the overhead using LVM ? How many messages per second do you have ? Do you use a persistent queue ? If yes how did you to continue consume queue when one node is down ?</div>
<div><br></div><div>You are using a heartbeat, so how you share inter nodes the same disk ? NFS ? Storage ?</div><div><br></div><div>Thank you so much for your report it helps a lot.</div><div><br></div><div>Best wishes. <br>
<br><div class="gmail_quote">On Thu, Mar 18, 2010 at 12:17 PM, Alvaro Videla <span dir="ltr"><<a href="mailto:videlalvaro@gmail.com">videlalvaro@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Sorry, It's me again, forgot to add something :)<br><br>That LVS setup was live tested when we got the dreaded out of memory problem, when the Erlang VM just crash shutting down the RabbitMQ node. One of the brokers went down, but we continued working from the other.<br>
<br>After that we decided to finally upgrade our RabbitMQ server to the latest version<div><div></div><div class="h5"><br><br><div class="gmail_quote">On Thu, Mar 18, 2010 at 11:05 PM, Alvaro Videla <span dir="ltr"><<a href="mailto:videlalvaro@gmail.com" target="_blank">videlalvaro@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex"><div style="word-wrap:break-word"><div><font face="arial, helvetica, sans-serif" size="4"><span style="font-size:14px;white-space:pre-wrap">Hi,</span></font></div>
<div><font face="arial, helvetica, sans-serif" size="4"><span style="font-size:14px;white-space:pre-wrap"><br></span></font></div><div><font face="arial, helvetica, sans-serif" size="4"><span style="font-size:14px;white-space:pre-wrap">After some feedback by @etrepum at Twitter, I feel that what I commented needs some more details, so I did some research about LVS, since that set up was done by one of our sysadmins.</span></font></div>
<div><font face="arial, helvetica, sans-serif" size="4"><span style="font-size:14px;white-space:pre-wrap"><br></span></font></div><div><font face="arial, helvetica, sans-serif" size="4"><span style="font-size:14px;white-space:pre-wrap">You can check about a LVS setup here: </span></font></div>
<div><font face="arial, helvetica, sans-serif" size="4"><span style="font-size:14px;white-space:pre-wrap"><br></span></font></div><div><font face="arial, helvetica, sans-serif" size="4"><span style="font-size:14px;white-space:pre-wrap"><a href="http://www.ibiblio.org/oswg/oswg-nightly/oswg/en_US.ISO_8859-1/articles/cluster-howto/cluster-howto/index.html" target="_blank">http://www.ibiblio.org/oswg/oswg-nightly/oswg/en_US.ISO_8859-1/articles/cluster-howto/cluster-howto/index.html</a></span></font></div>
<div><font face="arial, helvetica, sans-serif" size="4"><span style="font-size:14px;white-space:pre-wrap"><br></span></font></div><div><font face="arial, helvetica, sans-serif" size="4"><span style="font-size:14px;white-space:pre-wrap">From that page:</span></font></div>
<div><font face="arial, helvetica, sans-serif" size="4"><span style="font-size:14px;white-space:pre-wrap"><br></span></font></div><div><font face="arial, helvetica, sans-serif" size="4"><span style="font-size:14px;white-space:pre-wrap">"<span style="font-family:arial;font-size:medium;white-space:normal">the role of the <i>active router</i> is to redirect service requests from the virtual server address to the real servers." [...] </span></span></font></div>
<div><font face="arial, helvetica, sans-serif"><br></font></div><div><font face="arial, helvetica, sans-serif" size="4"><span style="font-size:14px;white-space:pre-wrap"><span style="font-family:arial;font-size:medium;white-space:normal">"<span style="font-family:arial">The active router dynamically monitors the health of the real servers, and the workload on each." [...] </span></span></span></font></div>
<div><font face="arial, helvetica, sans-serif"><br></font></div><div><font face="arial, helvetica, sans-serif" size="4"><span style="font-size:14px;white-space:pre-wrap"><span style="font-family:arial;font-size:medium;white-space:normal"><span style="font-family:arial">"<span style="font-family:arial">If a real server becomes disabled, the active router stops sending jobs to the server until it returns to normal operation."</span></span></span></span></font></div>
<div><font face="arial, helvetica, sans-serif"><br></font></div><div><font face="arial, helvetica, sans-serif"><span style="font-size:14px;white-space:pre-wrap">I hope this mails clarifies things up,</span></font></div>
<div><font face="arial, helvetica, sans-serif" size="4"><span style="font-size:14px;white-space:pre-wrap"><br></span></font></div><font color="#888888"><div><font face="arial, helvetica, sans-serif" size="4"><span style="font-size:14px;white-space:pre-wrap">Alvaro</span></font></div>
<div><br></div><br></font><div><div><div>On Mar 18, 2010, at 7:57 PM, Gustavo Aquino wrote:</div><br></div><blockquote type="cite"><div><div></div><div>Hi,<div><br></div><div>I have done this question before for many peoples, without success, because I don't found (Documentation, discussion lists and etc) any way to do High Availability with RabbitMQ without a lot of workaround, so exist a way to do HA with RabbitMQ without implementing a lot of stuffs by client side, like recreating queues when node down, recreating configurations, recreating client connections and etc ?</div>
<div><br></div><div>What's recommendation from RabbitMQ to do HA ?</div><div><br></div><div>Someone here have done some HA implementation to RabbitMQ ?</div><div><br></div><div>Regards.</div><div><br></div><div>Gustavo</div>
<div><br></div></div></div><div>
_______________________________________________<br>rabbitmq-discuss mailing list<br><a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br><a href="http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</div></blockquote></div><br></div></blockquote></div><br>
</div></div><br>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br></body></html>