would a slower broker (<1000msg/sec), that provides infinite message storage fit the bill?<br><br>If there were an option in rabbit to store all messages on disk,<br>and forget them there, then reload them when ready to send, then<br>
the retention and queuing limits go away (if queues are also on disk.) basically if no-one is consuming from a queue for a certain amount of time... say minutes, those queues should flush to disk, and free up memory for active queues.<br>
<br>I'm not suggesting rabbit modify their high speed way of doing things, but there might be another mode... the trivially primitive way to do it:<br>ie.<br>exchange stores into directory tree:<br><br>db/2008/11/08/23/59/msg324 msg325...<br>
<br>the routing algorithm makes links for the queues:<br><br>queues/myqueue/msg324 --> unix hard link, to msg324 in the db tree.<br> msg325...<br><br>etc...<br>we timed a reasonably linux server a few years ago at being able to perform 20,000 file creations per second, which was enough to support the messaging rate we needed.<br>
<br>yes, you need a single file system for the message store. We run such db's with upwards of 20,000,000 messages in an ext3, with decent performance. LRU is a wonderful gift from the paging system and buffer cache that you get for free. It manages what to keep in memory completely transparently to the application code.<br>
<br>when I can get away with a trivially primitive implementation, I take it.<br><br>(before someone says show me the code... <a href="http://metpx.sf.net">http://metpx.sf.net</a>, and no it doesn't do AMQP... but the ideas are vanishingly alike.)<br>
<br><br><br><div class="gmail_quote">On Sat, Nov 15, 2008 at 4:47 PM, rrabarg2 <span dir="ltr"><<a href="mailto:rrabarg2@yahoo.com.sg">rrabarg2@yahoo.com.sg</a>></span> wrote: <br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div bgcolor="#ffffff" text="#000000">
Ben,<br>
<br>
We have a system which includes a homogeneous set of message producers
(we call switches) and a backoffice application which consumes these
messages. We have a guaranteed delivery requirement for the messages,
we also require that the switches can continue to operate unhindered
(assuming infinite disk space) during long periods of slow or
non-consumption of messages by the backoffice. I'm not familiar with
rabbitMQ, but the implication I take from this thread is that rabbitMQ
would not meet our requirements (at least not without significant extra
bespoke work) - is that a fair statement?<br>
<br>
Regards<br>
<br>
Graham<div><div></div><br></div></div></blockquote></div><br>