[rabbitmq-discuss] rabbitmq-discuss Digest, Vol 18, Issue 34

[Alexis Richardson]

Ilya

I recommend persisting with this.  (sorry, I could not resist that)

Can you say a bit more on the list about the case in which you need to
buffer a day's worth of messages please?

* I am assuming flow control does not solve your problem - I think you
have made that clear, but can I just check?
* How often do you expect the 250Gb case to happen - do you consider
it exceptional or normal?
* What about 2Gb, 8Gb, ...?
* Do you have ordering constraints, and/or must all the messages be on one queue
* In the case where you need to page to disk due to a lack of machine
memory, and because your queues are not being drained by consumers,
does it matter what the latency of message delivery is once the
consumers come back?

The point being that there may be some workaround that you can try,
depending on the answer to the above.  You may be able to see where I
am going by the last question... ;-)

alexis


On Sun, Nov 16, 2008 at 3:26 PM, Ilya Grigorik <ilya at aiderss.com> wrote:
> Ben, I may be running out of patience on the recovery of the queue (after
> the crash), I'll try overloading it once more later tonight and report on my
> findings.
>
>> https://dev.rabbitmq.com/wiki/DiskOverflow
>>
>> Comment and add suggestions if any.
>
> Lots of great ideas in there, a few things that stood out to me:
>
> 1. I would love to have control of overflow conditions at at queue level.
> Having said that it's a double edged sword. Once you're that granular it
> also becomes painful do resource planning (woops, forgot a queue) and server
> has gone to hell - though I think it's a fair tradeoff.
>
> 2. I really like the idea of being able to fix the maximum size on a queue,
> as that would allow me to plan for the amount of available memory + disk and
> guarantee some QoS thresholds. Drop, Trip, Warn would be perfect for a hard
> limit, and I imagine TTL could be an interesting solution to anyone working
> with time-sensitive data.
>
>> Filling up a SAN though can be considered an exceptional and absolutely
>> unavoidable last barrier, after all if you manage to fill up to
>> 500GB of space of an average consumer disk probably you forgot that the
>> server even exists.
>
> Not entirely true, based on my calculations we (AideRSS) would need ~250GB
> of space to buffer a day's worth of messages, and that is something we're
> aiming for. (Though we may be an edge case)
>
> ig
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>