[rabbitmq-discuss] Persister crashes Rabbit

[Peter Silva]

would a slower broker (<1000msg/sec), that provides infinite message storage
fit the bill?

If there were an option in rabbit to store all messages on disk,
and forget them there, then reload them when ready to send, then
the retention and queuing limits go away (if queues are also on disk.)
basically if no-one is consuming from a queue for a certain amount of
time... say minutes, those queues should flush to disk, and free up memory
for active queues.

I'm not suggesting rabbit modify their high speed way of doing things, but
there might be another mode... the trivially primitive way to do it:
ie.
exchange stores into directory tree:

db/2008/11/08/23/59/msg324 msg325...

the routing algorithm makes links for the queues:

queues/myqueue/msg324 --> unix hard link, to msg324 in the db tree.
                          msg325...

etc...
we timed a reasonably linux server a few years ago at being able to perform
20,000 file creations per second, which was enough to support the messaging
rate we needed.

yes, you need a single file system for the message store.  We run such db's
with upwards of 20,000,000 messages in an ext3, with decent performance.
LRU is a wonderful gift from the paging system and buffer cache that you get
for free.  It manages what to keep in memory completely transparently to the
application code.

when I can get away with a trivially primitive implementation, I take it.

(before someone says show me the code... http://metpx.sf.net, and no it
doesn't do AMQP... but the ideas are vanishingly alike.)



On Sat, Nov 15, 2008 at 4:47 PM, rrabarg2 <rrabarg2 at yahoo.com.sg> wrote:

>  Ben,
>
> We have a system which includes a homogeneous set of message producers (we
> call switches) and a backoffice application which consumes these messages.
> We have a guaranteed delivery requirement for the messages, we also require
> that the switches can continue to operate unhindered (assuming infinite disk
> space) during long periods of slow or non-consumption of messages by the
> backoffice.  I'm not familiar with rabbitMQ, but the implication I take from
> this thread is that rabbitMQ would not meet our requirements (at least not
> without significant extra bespoke work)  - is that a fair statement?
>
> Regards
>
> Graham
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20081115/b741c7d1/attachment.htm