[rabbitmq-discuss] What to do when high watermark reached?

Tue Sep 27 16:08:49 BST 2011

François,

2011/9/27 François Beausoleil <francois at teksol.info>:
> According to the management console's overview tab, I've reached my memory high watermark. I have two queues with a combined total of 2.5M messages. My workers are up and running, but in both cases, the queue consumers read messages and may publish messages to another exchange.
>
> I can restart my workers, in which case the messages are unack'd and put back in the queue, to be processed again.
>
> Is there a way around this? What can I do to process those messages? Can I tweak something in the server while it's up? I don't want to reboot it because I've lost messages before, even though my messages are published using :persistent => true (Bunny).
>
> Rereading http://www.rabbitmq.com/admin-guide.html#memsup, I notice that it says I should use multiple connections: one for producing, one for consuming. I'll have to change my apps to do that, but in the meantime?

The easiest thing to do is to beef up your machine and give Rabbit
more RAM (adding a SSD drive may additionally help :P).

When Rabbit hits the memory limit, there's really not much that can be
done. In theory the *consumers* should still be able to consume
messages, and thus free some memory over time.

But in practice, due to AMQP constraints RabbitMQ can't really
distinguish producers and consumers on per-channel basis. Instead,
Rabbit looks at the connection. If there are messages produced on a
connection - it will be blocked to avoid the need to cope with even
more messages.

Erlang VM is likely to crash if there isn't enough free memory to do a
full garbage collection cycle.

If you don't care about messages, you can try to purge messages or
remove unused queues. But I guess, you should be using different
connections for producing and consuming. Sorry.

Hope that helps,
 Marek