[rabbitmq-discuss] tcp -> rabbitmq gives emfile + 541

[Matthew Sackman]

Hi Michael,

Just replying specifically about the new persister, I'll let Tony respond
to your other comments.

On Wed, Jul 01, 2009 at 04:31:53PM +0100, Michael Nacos wrote:
> Now, I distinctly remember Matthew speaking about a mechanism which monitors
> free memory and offloads to disk to the point of RabbitMQ working as long as
> there is still room for a single message in RAM, but I don't remember if
> this is already available or when it's going to be.

I'm afraid it's still not quite ready. Most of the core work is done and
it is pretty well tested, but I'm still finding the odd bug (which are
becoming increasingly tricky to track down!), coming up with new
optimisations, and generally still tuning the solution. This means that
it hasn't been through QA yet and so we don't recommend people use it.

Further, what's currently eating my time is trying to work out the best
policy of when to decide when a queue should be paged out, and how
resources should be allocated between queues. We have a number of ideas
in this space, some of which smell much better than others, but it is a
rather open problem and will take a little while before we settle on a
solution that we think works well.

> If I am right, attaching a fast enough consumer to the queue while running
> my tests will probably take care of the crashes, I have only experienced
> crashes when the number of undelivered messages exceeded 200000 messages.

Can you check the logs and say whether you get alarm high_water_mark
type messages in there? Also, in these cases, does free, or similar,
show pretty much no free RAM? Also, does a ball park calcualation of
message size * 200000 give you about your amount of RAM (within a factor
of 2)?

Matthew