[rabbitmq-discuss] RabbitMQ new Persister

Mon Nov 9 04:08:11 GMT 2009

Hey Kalpesh

Thank u for ur response

On Mon, Nov 9, 2009 at 7:58 AM, Kalpesh <kpatel at egnyte.com> wrote:

>  Hello Arun,
>
>
>
> The current production in egnyte uses Rabbit in a low volume scenario like
> 10/15 per min. In future versions we are planning to use Rabbit as the
> backend messaging system for handling most of our background jobs. The
> volume can comes in bursts and if background job consumers are down it can
> accumulate in worse case to 200-500K per day (This is worse case, in normal
> case the queue should be exhausted immediately).
>
>
>
> I tried a simple test of pushing 200K messages each 1KB then killed the
> rabbit and started Rabbit again and Rabbit was able to recover all messages.
> Then in another test I tried pushing 500K messages 1 KB each and Rabbit
> died, I wasn't able to recover the mnesia database (I had to purge the db
> and start over with fresh db and lost all messags).  The only solution I
> found was to have 2 or more rabbits and use round robin to increase the no
> of messages Rabbit can hold. In addition we can't afford to lose messages
> and I will have to use the control-flow mechanism to throttle the consumers
> to log to some db or filesystem in case both Rabbits are full (but then the
> solution is becoming complex if I have to introduce a db in the mix as the
> consumers will have to keep checkpoints and other recovery mechanism).
>
>
>
> The ideal solution is that Rabbit should keep only those many messages in
> memory that it can hold and rest should be persisted in the disk that way we
> are limited by disk and not physical memory of the machine. I saw that
> Matthew and team are already working on it that's why I am interested in
> testing it out if its available as beta.
>
Guess this is a fundamental problem with mnesia (storing data both in ram
and fs)..
Looking forward to the new persister

-Arun

>
>
> Thanks
>
> -Kalpesh
>
>
>
> *From:* Arun Suresh [mailto:arun.suresh at gmail.com]
> *Sent:* Sunday, November 08, 2009 7:56 PM
> *To:* Kalpesh
> *Cc:* matthew at lshift.net; rabbitmq-discuss at lists.rabbitmq.com
> *Subject:* Re: [rabbitmq-discuss] RabbitMQ new Persister
>
>
>
> Hello Kalpesh
>
> We are using RabbitMQ as well in our team... Most of our messaging needs
> are not high volume.. but we do have a set of messages where we expect
> something like 350/min at peek load to be distributed to consumers..
>
> Just curious to know what is the load at which the current implementation
> of the RabbitMQ persistor bails out on you..
>
> -Arun Suresh
> Yahoo! India
>
> On Mon, Nov 9, 2009 at 7:11 AM, Kalpesh <kpatel at egnyte.com> wrote:
>
> Hello Matthew,
>
>
>
> We are looking forward for the new persister that RabbitMQ team is working
> on. We are currently using Rabbit in a low volume scenario, now we are
> planning to use it for a high volume scenario. We can't afford to lose any
> messages,  currently Rabbit runs out of memory if we overfed it and then it
> crashes. The only way to not lose message is to store it in some db or
> filesystem and than push it to Rabbit just for distribution to consumers.
>
>
>
> Is there any way we can get access to the new persister code? I would be
> happy to do some tests on it.
>
>
>
> Thanks
>
> -Kalpesh
>
> Kalpesh Patel
> *Egnyte *- your *On Demand File Server
> *Big infrastructure; Small price tag
> W 650.265.4054 | C 650.740.3062
> sign up at http://www.egnyte.com
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20091109/0f6a534e/attachment.htm 