Hey Kalpesh<br><br>Thank u for ur response<br><br><div class="gmail_quote">On Mon, Nov 9, 2009 at 7:58 AM, Kalpesh <span dir="ltr"><<a href="mailto:kpatel@egnyte.com">kpatel@egnyte.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div link="blue" vlink="purple" lang="EN-US">
<div>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">Hello Arun,</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">�</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">The current production in egnyte uses Rabbit in a low volume
scenario like 10/15 per min. In future versions we are planning to use Rabbit
as the backend messaging system for handling most of our background jobs. The
volume can comes in bursts and if background job consumers are down it can accumulate
in worse case to 200-500K per day (This is worse case, in normal case the queue
should be exhausted immediately).� </span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">�</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">I tried a simple test of pushing 200K messages each 1KB then
killed the rabbit and started Rabbit again and Rabbit was able to recover all
messages. Then in another test I tried pushing 500K messages 1 KB each and
Rabbit died, I wasn't able to recover the mnesia database (I had to purge the
db and start over with fresh db and lost all messags). �The only solution
I found was to have 2 or more rabbits and use round robin to increase the no of
messages Rabbit can hold. In addition we can't afford to lose messages and I
will have to use the control-flow mechanism to throttle the consumers to log to
some db or filesystem in case both Rabbits are full (but then the solution is
becoming complex if I have to introduce a db in the mix as the consumers will
have to keep checkpoints and other recovery mechanism).</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">�</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">The ideal solution is that Rabbit should keep only those many
messages in memory that it can hold and rest should be persisted in the disk
that way we are limited by disk and not physical memory of the machine. I saw
that Matthew and team are already working on it that's why I am interested in
testing it out if its available as beta.</span></p></div></div></blockquote><div>Guess this is a fundamental problem with mnesia (storing data both in ram and fs)..<br>Looking forward to the new persister<br><br>-Arun <br>
</div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div link="blue" vlink="purple" lang="EN-US"><div>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">�</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">Thanks</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">-Kalpesh</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">�</span></p>
<div style="border-style: solid none none; border-color: rgb(181, 196, 223) -moz-use-text-color -moz-use-text-color; border-width: 1pt medium medium; padding: 3pt 0in 0in;">
<p class="MsoNormal"><b><span style="font-size: 10pt;">From:</span></b><span style="font-size: 10pt;"> Arun Suresh
[mailto:<a href="mailto:arun.suresh@gmail.com" target="_blank">arun.suresh@gmail.com</a>] <br>
<b>Sent:</b> Sunday, November 08, 2009 7:56 PM<br>
<b>To:</b> Kalpesh<br>
<b>Cc:</b> <a href="mailto:matthew@lshift.net" target="_blank">matthew@lshift.net</a>; <a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<b>Subject:</b> Re: [rabbitmq-discuss] RabbitMQ new Persister</span></p>
</div><div><div></div><div class="h5">
<p class="MsoNormal">�</p>
<p class="MsoNormal" style="margin-bottom: 12pt;">Hello Kalpesh<br>
<br>
We are using RabbitMQ as well in our team... Most of our messaging needs are
not high volume.. but we do have a set of messages where we expect something
like 350/min at peek load to be distributed to consumers..<br>
<br>
Just curious to know what is the load at which the current implementation of
the RabbitMQ persistor bails out on you..<br>
<br>
-Arun Suresh<br>
Yahoo! India</p>
<div>
<p class="MsoNormal">On Mon, Nov 9, 2009 at 7:11 AM, Kalpesh <<a href="mailto:kpatel@egnyte.com" target="_blank">kpatel@egnyte.com</a>> wrote:</p>
<div>
<div>
<p class="MsoNormal">Hello
Matthew,</p>
<p class="MsoNormal">�</p>
<p class="MsoNormal">We
are looking forward for the new persister that RabbitMQ team is working on. We
are currently using Rabbit in a low volume scenario, now we are planning to use
it for a high volume scenario. We can't afford to lose any messages,
�currently Rabbit runs out of memory if we overfed it and then it crashes.
The only way to not lose message is to store it in some db or filesystem and
than push it to Rabbit just for distribution to consumers.</p>
<p class="MsoNormal">�</p>
<p class="MsoNormal">Is
there any way we can get access to the new persister code? I would be happy to
do some tests on it.</p>
<p class="MsoNormal">�</p>
<p class="MsoNormal">Thanks</p>
<p class="MsoNormal">-Kalpesh</p>
<p class="MsoNormal"><span style="color: rgb(126, 126, 126);">Kalpesh Patel<br>
</span><b><span style="color: rgb(230, 102, 27);">Egnyte </span></b><span style="color: rgb(230, 102, 27);">- </span><span style="font-size: 9pt; color: rgb(230, 102, 27);">your <b>On Demand
File Server<br>
</b></span><span style="font-size: 8pt; color: rgb(126, 126, 126);">Big infrastructure;
Small price tag<br>
</span><span style="font-size: 9pt; color: rgb(126, 126, 126);">W 650.265.4054 | C
650.740.3062<br>
sign up at </span><span style="font-size: 9pt; color: rgb(0, 0, 254);"><a href="http://www.egnyte.com" target="_blank">http://www.egnyte.com</a></span></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom: 12pt;"><br>
_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com" target="_blank">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a></p>
</div>
<p class="MsoNormal">�</p>
</div></div></div>
</div>
</blockquote></div><br>