[rabbitmq-discuss] 2 nodes, 4k queues

[Matthew Sackman]

Hi Gojko,

On Thu, Nov 19, 2009 at 07:27:24AM +0000, Gojko Adzic wrote:
> We had a case of a 2 nodes cluster where message delivery latency was
> unacceptably slow (~ 1-2 min) when the number of queues rose above 4k.
> All the messages were going out to a single topic exchange, average
> message size is 5KB, the number of messages was not more than 10/s and
> most of the messages matched 3-4 queues. We're using topic matching
> between exchange and queues. RabbitMQ was running at about 20-30% CPU on
> large EC2 instances, and about 9% memory, so there doesn't seem to be a
> memory or CPU bottleneck on the machines. Is this kind of performance
> expected (and if it is, how many queues per node would you suggest), or 
> have we hit a configuration problem? Any ideas on monitoring/measuring 
> what starts slowing down processing when the number
> of queues goes up.

Yeah, our topic matching is inefficient - it's O(n) where n is the
number of bindings. We do have a bug open to improve this. You didn't
mention the number of bindings that you have, but if you're up to 20k
bindings or so then I wouldn't be surprised if this is the performance
you're getting. In general, until we fix this bug (it should become
O(log₂(n)) because really it should just be walking a tree), we'd
probably recommend to use a direct or fanout exchange and then drop
inappropriate messages in the consumer. I know this isn't ideal, but it
might solve the problems you're seeing.

Wrt CPU load, the matching of routing keys against bindings is not
parallelised - thus you can burn a max of 1 CPU for each channel doing
the publish. So if you only have 1 publisher and several CPUs then you
will only really be able to make work for 1 CPU. The situation should be
better if you have many channels or connections and are publishing in
parallel, but there may be something else I've forgotten that limits
this case. What is your topology on the publishing side?

Matthew