[rabbitmq-discuss] RabbitMQ memory management

[Eran Sandler]

The reason we are using a message queue is to be able to send various 
messages to perform actions in an async manner.


I prefer not to lose these messages, hence the durable queue, but I can 
recover from not getting them.


The behavior I would expect, such as flowing back to the disk is to 
flush them to the disk and only keep a small portion in-memory, when 
"the cows come home" (or more exactly, the cowboy comes back and pushes 
the cows to their rightful place) it will start reading from the disk 
chunk by chunk until it reaches a point where all the backlog finished 
and from there it will go back to work as it was before.


Having that combined with a key that says when we should start throwing 
things directly to the disk without keeping them in memory (or without 
keeping them in memory too much) as well as a max queue length (depth, 
depends on where you look at it) would be really great.


Having the ability to write a queue plugin would be really great. There 
is always that 20% that requires a certain behavior that is not off the 
shelf :-)


Eran 


Ben Hood wrote:

> Eran,
>
> On Thu, Sep 11, 2008 at 12:14 PM, Ben Hood <0x6e6562 at gmail.com> wrote:
>   
>> You've got 2 options:
>>     
>
> You might also have a third option: setting the auto-delete flag. By
> setting this flag, when the consumer goes away, the queue will be
> auto-deleted and hence the messages will be dropped.
>
> ATM, I assume you have a durable queue that you are sending persistent
> messages to. Does this necessarily fit in with the nature of your
> application from a design perspective? Do you *really* need persistent
> messages for stuff that you are apparently not interested in
> consuming?
>
> Off on a tangent, this *may* be a use case for pluggable queues. If
> you implemented a queue backend that overflowed to disk, then you
> could happily queue up messages until the cows come home. Or at least
> until your SAN fills up. Again, this could be a future feature.
>
> HTH,
>
> Ben
>   

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20080911/2042abda/attachment.htm