[rabbitmq-discuss] another performance question

Mon Jul 15 17:45:15 BST 2013

We have been using rabbitmq for a number or years with great success, 
thanks.

All the following relates to rabbitmq 3.1.1 on debian.

Recently we started a new project using rabbitmq to handle large amounts of 
data, over 40kHz message rate and message sizes of a few kilobytes. The 
project depends on using a topic exchange with routing keys of the form 
AA.BB.CC.DD.EE. Consumers are able to bind a queue with a given wildcard 
pattern in order to receive a subset of the message stream published by one 
publisher. There is always one consumer that binds with the "#" routing key 
to receive all messages.

We have started testing against a broker running on a machine with 8 cores, 
about 2GHz each, with 32GB ram, and over a terabyte of free disk space. I 
am publishing with a single erlang client to a topic exchange. I am only 
able to publish at about 15kHz before I see flow control kick in on the 
management UI for the connection. This is with no queue bound to the 
exchange, so rabbitmq only needs to receive the messages and drop them. So 
at 15kHz I see flow control kicking in, but the machine is not busy, all 8 
cores show 85-95% idle. It looks like the server is basically idle. If I 
then bind a queue with the routing key "#" I see the enqueue rate hover 
around 5-7kHz. If I add a consumer to keep the queue empty I see no 
improvement. And during all this the machine still never gets busy.

My confusion lies in the fact that the flow control kicks in, but the 
machine is not busy. There is no ram or disk shortage and the machine's NIC 
can certainly handle much higher bandwidth loads than the 15kHz message 
rate I see in rabbitmq.

I have read everything I can on performance and expected to see a message 
receiving rate, that is with no queues, of over 100kHz. Once I add queues, 
etc I was expecting 20kHz or better.

I am wondering if 40kHz throughput is un-realistic at this point.

Thanks,
Mark.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130715/6f7d9d0d/attachment.htm>