[rabbitmq-discuss] exessive memory usage and blocked connection, even when no messages left

Thu May 19 11:19:47 BST 2011

Theo,

On 19/05/11 11:00, Theo wrote:
> Ok, thanks for the clarification, it makes sense, and it's more or
> less what we figured. I was hoping there was another explanation,
> that would be fixable.
>
> What kind of performance should we expect when running on EC2? It
> feels like 20K messages per second shouldn't be a problem for a four
>  node cluster where each node has 8 CPUs and 7 Gb RAM.

20kHz shouldn't be a problem even for just a single 4-core machine,
assuming messages sizes are fairly small.

> to get any kind of performance and stability we have had to tune how
> we work with the cluster, making sure that queues are distributed
> over the nodes, that each node has more than one queue, that the
> connections from the producers and consumers are evenly distributed
> over the nodes, etc. even slight asymmetries show up quite quickly
> in the memory usage of a node, and as soon as that happens it's only
> minutes before the whole cluster goes bad.

You shouldn't have to do that kind of tuning to get 20kHz. As I said in
your previous email, your publishers publish at a rate far higher than
20kHz. And they do that all the time, not just for some brief peak
period. That's why rabbit eventually fills up all memory with buffered
messages and has to pause producers. If you run tests with throttled
producer, e.g. get producers to publish messages at a certain rate, then
rabbit will happily keep up with that rate forever.

> Topic exchanges seem to be completely out of the question, it only
> takes minutes before every node gets overloaded.

Again, that is due to the fact that the publishers operate at a far 
higher rate than is sustainable.

Regards,

Matthias.