[rabbitmq-discuss] RabbitMQ new Persister

[Kalpesh]

Hello Arun,

 

The current production in egnyte uses Rabbit in a low volume scenario like
10/15 per min. In future versions we are planning to use Rabbit as the
backend messaging system for handling most of our background jobs. The
volume can comes in bursts and if background job consumers are down it can
accumulate in worse case to 200-500K per day (This is worse case, in normal
case the queue should be exhausted immediately).  

 

I tried a simple test of pushing 200K messages each 1KB then killed the
rabbit and started Rabbit again and Rabbit was able to recover all messages.
Then in another test I tried pushing 500K messages 1 KB each and Rabbit
died, I wasn't able to recover the mnesia database (I had to purge the db
and start over with fresh db and lost all messags).  The only solution I
found was to have 2 or more rabbits and use round robin to increase the no
of messages Rabbit can hold. In addition we can't afford to lose messages
and I will have to use the control-flow mechanism to throttle the consumers
to log to some db or filesystem in case both Rabbits are full (but then the
solution is becoming complex if I have to introduce a db in the mix as the
consumers will have to keep checkpoints and other recovery mechanism).

 

The ideal solution is that Rabbit should keep only those many messages in
memory that it can hold and rest should be persisted in the disk that way we
are limited by disk and not physical memory of the machine. I saw that
Matthew and team are already working on it that's why I am interested in
testing it out if its available as beta.

 

Thanks

-Kalpesh

 

From: Arun Suresh [mailto:arun.suresh at gmail.com] 
Sent: Sunday, November 08, 2009 7:56 PM
To: Kalpesh
Cc: matthew at lshift.net; rabbitmq-discuss at lists.rabbitmq.com
Subject: Re: [rabbitmq-discuss] RabbitMQ new Persister

 

Hello Kalpesh

We are using RabbitMQ as well in our team... Most of our messaging needs are
not high volume.. but we do have a set of messages where we expect something
like 350/min at peek load to be distributed to consumers..

Just curious to know what is the load at which the current implementation of
the RabbitMQ persistor bails out on you..

-Arun Suresh
Yahoo! India

On Mon, Nov 9, 2009 at 7:11 AM, Kalpesh <kpatel at egnyte.com> wrote:

Hello Matthew,

 

We are looking forward for the new persister that RabbitMQ team is working
on. We are currently using Rabbit in a low volume scenario, now we are
planning to use it for a high volume scenario. We can't afford to lose any
messages,  currently Rabbit runs out of memory if we overfed it and then it
crashes. The only way to not lose message is to store it in some db or
filesystem and than push it to Rabbit just for distribution to consumers.

 

Is there any way we can get access to the new persister code? I would be
happy to do some tests on it.

 

Thanks

-Kalpesh

Kalpesh Patel
Egnyte - your On Demand File Server
Big infrastructure; Small price tag
W 650.265.4054 | C 650.740.3062
sign up at http://www.egnyte.com


_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss at lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20091108/f9cc9196/attachment.htm