[rabbitmq-discuss] rabbitmq-discuss Digest, Vol 18, Issue 34

Sun Nov 16 23:59:03 GMT 2008

Ben: running latest version as of this past Friday (much better than 1.4.0,
which I tried at first)

I recommend persisting with this.  (sorry, I could not resist that)
>

;-)

> * I am assuming flow control does not solve your problem - I think you
> have made that clear, but can I just check?

Yep, in fact, I would much rather not throttle the producers - the queue
becomes a leaky abstraction. I like to think of Rabbit (or any other
router/queue in between) as a buffer, hence it should accept and fan out the
messages without exposing the lag on the consumer side.

I may be conflating two different requirements, but here is an interesting
scenario to think about: our entire infrastructure is deployed on EC2, and
time and time again we've found SQS (simple queue service) a life savior for
big data migrations or general maintenance. We're generating a lot of
real-time data, and SQS allows us to reroute all of it into a temporary
queue, and let it accumulate (we've pushed hundreds of GB's of data into it)
while we do our work, and once we're done, we just flush the queue. No
additional infrastructure required. (same thing for aggregate jobs, or even
one-off trial projects)

Arguably, we could use a different piece of infrastructure to cover this
case (another rabbit consumer which stores data into a database), but if
Rabbit could handle this, life is much easier.

(Wild idea: provide an SQS adapter to reroute packets into at a certain
threshold.. Perhaps not so wild if we get a pluggable interface)

* How often do you expect the 250Gb case to happen - do you consider
> it exceptional or normal?

250GB is an exception, but smaller amounts (1-5Gb) are fairly frequent
(which is theoretically possible even with the current pure memory
deployment).

> * Do you have ordering constraints, and/or must all the messages be on one
> queue

Depends on the data. Most of it is order agnostic, but there is some data
which only makes sense when delivered in order.

> * In the case where you need to page to disk due to a lack of machine
> memory, and because your queues are not being drained by consumers,
> does it matter what the latency of message delivery is once the
> consumers come back?
>

No - within reason, of course. ;-)

The point being that there may be some workaround that you can try,
> depending on the answer to the above.  You may be able to see where I
> am going by the last question... ;-)
>

Absolutely. I'm not saying Rabbit should do all of the things I've described
either. Of course the more the better, but even more importantly I'd like to
see some sane overflow conditions: ttl, max queue size, etc. That I can plan
for, crashing.. won't work. ;-)

ig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20081116/044fe282/attachment.htm 