Ben: running latest version as of this past Friday (much better than 1.4.0, which I tried at first)<br><br><div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
I recommend persisting with this. (sorry, I could not resist that)<br>
</blockquote><div><br>;-) <br></div><div> </div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">* I am assuming flow control does not solve your problem - I think you<br>
have made that clear, but can I just check?</blockquote><div> <br>Yep, in fact, I would much rather not throttle the producers - the queue becomes a leaky abstraction. I like to think of Rabbit (or any other router/queue in between) as a buffer, hence it should accept and fan out the messages without exposing the lag on the consumer side. <br>
<br>I may be conflating two different requirements, but here is an interesting scenario to think about: our entire infrastructure is deployed on EC2, and time and time again we've found SQS (simple queue service) a life savior for big data migrations or general maintenance. We're generating a lot of real-time data, and SQS allows us to reroute all of it into a temporary queue, and let it accumulate (we've pushed hundreds of GB's of data into it) while we do our work, and once we're done, we just flush the queue. No additional infrastructure required. (same thing for aggregate jobs, or even one-off trial projects)<br>
<br>Arguably, we could use a different piece of infrastructure to cover this case (another rabbit consumer which stores data into a database), but if Rabbit could handle this, life is much easier.<br><br>(Wild idea: provide an SQS adapter to reroute packets into at a certain threshold.. Perhaps not so wild if we get a pluggable interface) <br>
<br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">* How often do you expect the 250Gb case to happen - do you consider<br>
it exceptional or normal?</blockquote><div><br>250GB is an exception, but smaller amounts (1-5Gb) are fairly frequent (which is theoretically possible even with the current pure memory deployment). <br> </div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
* Do you have ordering constraints, and/or must all the messages be on one queue</blockquote><div><br>Depends on the data. Most of it is order agnostic, but there is some data which only makes sense when delivered in order.<br>
</div><div> <br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
* In the case where you need to page to disk due to a lack of machine<br>
memory, and because your queues are not being drained by consumers,<br>
does it matter what the latency of message delivery is once the<br>
consumers come back?<br>
</blockquote><div> <br>No - within reason, of course. ;-)<br><br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
The point being that there may be some workaround that you can try,<br>
depending on the answer to the above. You may be able to see where I<br>
am going by the last question... ;-)<br>
<font color="#888888"></font></blockquote><div><br>Absolutely. I'm not saying Rabbit should do all of the things I've described either. Of course the more the better, but even more importantly I'd like to see some sane overflow conditions: ttl, max queue size, etc. That I can plan for, crashing.. won't work. ;-) </div>
<div><br>ig<br><br></div></div>