<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Hello Arun,<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>The current production in egnyte uses Rabbit in a low volume
scenario like 10/15 per min. In future versions we are planning to use Rabbit
as the backend messaging system for handling most of our background jobs. The
volume can comes in bursts and if background job consumers are down it can accumulate
in worse case to 200-500K per day (This is worse case, in normal case the queue
should be exhausted immediately). <o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>I tried a simple test of pushing 200K messages each 1KB then
killed the rabbit and started Rabbit again and Rabbit was able to recover all
messages. Then in another test I tried pushing 500K messages 1 KB each and
Rabbit died, I wasn't able to recover the mnesia database (I had to purge the
db and start over with fresh db and lost all messags). The only solution
I found was to have 2 or more rabbits and use round robin to increase the no of
messages Rabbit can hold. In addition we can't afford to lose messages and I
will have to use the control-flow mechanism to throttle the consumers to log to
some db or filesystem in case both Rabbits are full (but then the solution is
becoming complex if I have to introduce a db in the mix as the consumers will
have to keep checkpoints and other recovery mechanism).<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>The ideal solution is that Rabbit should keep only those many
messages in memory that it can hold and rest should be persisted in the disk
that way we are limited by disk and not physical memory of the machine. I saw
that Matthew and team are already working on it that's why I am interested in
testing it out if its available as beta.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Thanks<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>-Kalpesh<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'>
<p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span
style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Arun Suresh
[mailto:arun.suresh@gmail.com] <br>
<b>Sent:</b> Sunday, November 08, 2009 7:56 PM<br>
<b>To:</b> Kalpesh<br>
<b>Cc:</b> matthew@lshift.net; rabbitmq-discuss@lists.rabbitmq.com<br>
<b>Subject:</b> Re: [rabbitmq-discuss] RabbitMQ new Persister<o:p></o:p></span></p>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal style='margin-bottom:12.0pt'>Hello Kalpesh<br>
<br>
We are using RabbitMQ as well in our team... Most of our messaging needs are
not high volume.. but we do have a set of messages where we expect something
like 350/min at peek load to be distributed to consumers..<br>
<br>
Just curious to know what is the load at which the current implementation of
the RabbitMQ persistor bails out on you..<br>
<br>
-Arun Suresh<br>
Yahoo! India<o:p></o:p></p>
<div>
<p class=MsoNormal>On Mon, Nov 9, 2009 at 7:11 AM, Kalpesh <<a
href="mailto:kpatel@egnyte.com">kpatel@egnyte.com</a>> wrote:<o:p></o:p></p>
<div>
<div>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Hello
Matthew,<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>We
are looking forward for the new persister that RabbitMQ team is working on. We
are currently using Rabbit in a low volume scenario, now we are planning to use
it for a high volume scenario. We can't afford to lose any messages,
currently Rabbit runs out of memory if we overfed it and then it crashes.
The only way to not lose message is to store it in some db or filesystem and
than push it to Rabbit just for distribution to consumers.<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Is
there any way we can get access to the new persister code? I would be happy to
do some tests on it.<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Thanks<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>-Kalpesh<o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span
style='color:#7E7E7E'>Kalpesh Patel<br>
</span><b><span style='color:#E6661B'>Egnyte </span></b><span style='color:
#E6661B'>- </span><span style='font-size:9.0pt;color:#E6661B'>your <b>On Demand
File Server<br>
</b></span><span style='font-size:8.0pt;color:#7E7E7E'>Big infrastructure;
Small price tag<br>
</span><span style='font-size:9.0pt;color:#7E7E7E'>W 650.265.4054 | C
650.740.3062<br>
sign up at </span><span style='font-size:9.0pt;color:#0000FE'><a
href="http://www.egnyte.com" target="_blank">http://www.egnyte.com</a></span><o:p></o:p></p>
</div>
</div>
<p class=MsoNormal style='margin-bottom:12.0pt'><br>
_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss"
target="_blank">http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><o:p></o:p></p>
</div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
</body>
</html>