[rabbitmq-discuss] RabbitMQ perfomance testing & troubles

Tue Apr 22 14:32:03 BST 2008

В Срд, 16/04/2008 в 13:48 +0100, Matthias Radestock пишет:

Hello

> Erlang in SMP mode will indeed happily use all your CPUs. It won't 
> necessarily go any faster though. In our experience - and I should 
> stress that this is for RabbitMQ and R11B-x only - it doesn't.
> 
> You don't need to know anything about Erlang to configure a local 
> cluster. Take a look at the http://www.rabbitmq.com/clustering.html - if 
> anything isn't clear in those instructions then please let us know ...

> OTOH, you should only look into clustering once the other issues have 
> been resolved.

Well, I was configured the local cluster for 8 nodes:

# su rabbitmq -c "/usr/sbin/rabbitmq-multi start_all 8"
Starting all nodes...   
Starting node 0....
RabbitMQ 1.3.0 (AMQP 8-0)
Copyright (C) 2007-2008 LShift Ltd., Cohesive Financial Technologies
LLC., and Rabbit Technologies Ltd.
Licensed under the MPL.  See http://www.rabbitmq.com/

Logging to "/var/log/rabbitmq/rabbit.log"
SASL logging to "/var/log/rabbitmq/rabbit-sasl.log"

starting database             ...done
starting core processes       ...done
starting recovery             ...done
starting persister            ...done
starting builtin applications ...done
starting TCP listeners        ...done

broker running
OK
....

and modify the my test client to connect for different nodes

    Random random = new Random();
    int port = Math.abs( random.nextInt() ) % 8;
    ConnectionFactory factory = new ConnectionFactory(params);
    this.connection = factory.newConnection(this.HOSTNAME,
AMQP.PROTOCOL.PORT + port);

So, then I was run 250 producer's and 250 consumers - 2500 mps:

$ java -jar RabbitmqFastMessagingProducerTest.jar 250 > client.log

the server side looks like:

Tasks: 140 total,   8 running, 132 sleeping,   0 stopped,   0 zombie
Cpu(s): 92.7%us,  0.7%sy,  0.0%ni,  6.1%id,  0.0%wa,  0.0%hi,  0.4%si,
0.0%st
Mem:   4139508k total,  2296220k used,  1843288k free,   324364k buffers
Swap:  4194296k total,        0k used,  4194296k free,   853184k cached

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 4499 rabbitmq  25   0  136m  91m 1612 R  100  2.3  25:26.72 beam
 5131 rabbitmq  25   0  165m  99m 1612 R  100  2.5  27:13.94 beam
 6395 rabbitmq  25   0  121m  78m 1612 R  100  1.9  28:48.16 beam
 4183 rabbitmq  25   0  322m 282m 1984 R  100  7.0  47:28.18 beam
 5447 rabbitmq  25   0  146m 105m 1612 R  100  2.6  29:58.29 beam
 6079 rabbitmq  16   0  128m  85m 1612 R   88  2.1  31:46.84 beam
 4815 rabbitmq  16   0 97216  64m 1612 R   84  1.6  32:59.81 beam
 5763 rabbitmq  15   0  120m  79m 1612 S   79  2.0  28:58.69
beam                    

and the client log is 

$ tail -f client.log | awk '/CLIENT_CONSUMER_THREAD_ID/{print "DELAY: "
$4-$8 " " $0}'

DELAY: 13842 CLIENT_CONSUMER_THREAD_ID: 202 TIME: 1208870263522 MESSAGE:
1784 TIME: 1208870249680
DELAY: 42 CLIENT_CONSUMER_THREAD_ID: 229 TIME: 1208870263522 MESSAGE:
1911 TIME: 1208870263480
DELAY: 2597 CLIENT_CONSUMER_THREAD_ID: 21 TIME: 1208870263527 MESSAGE:
1888 TIME: 1208870260930
DELAY: 21917 CLIENT_CONSUMER_THREAD_ID: 230 TIME: 1208870263527 MESSAGE:
1709 TIME: 1208870241610
DELAY: 47 CLIENT_CONSUMER_THREAD_ID: 5 TIME: 1208870263527 MESSAGE: 1913
TIME: 1208870263480
DELAY: 43778 CLIENT_CONSUMER_THREAD_ID: 241 TIME: 1208870263528 MESSAGE:
1508 TIME: 1208870219750
DELAY: 16053 CLIENT_CONSUMER_THREAD_ID: 4 TIME: 1208870263533 MESSAGE:
1766 TIME: 1208870247480
DELAY: 76736 CLIENT_CONSUMER_THREAD_ID: 79 TIME: 1208870263536 MESSAGE:
1207 TIME: 1208870186800
DELAY: 50 CLIENT_CONSUMER_THREAD_ID: 39 TIME: 1208870263540 MESSAGE:
1911 TIME: 1208870263490
DELAY: 50 CLIENT_CONSUMER_THREAD_ID: 120 TIME: 1208870263540 MESSAGE:
1911 TIME: 1208870263490
DELAY: 8 CLIENT_CONSUMER_THREAD_ID: 123 TIME: 1208870263548 MESSAGE:
1912 TIME: 1208870263540
DELAY: 14131 CLIENT_CONSUMER_THREAD_ID: 185 TIME: 1208870263561 MESSAGE:
1782 TIME: 1208870249430
DELAY: 80914 CLIENT_CONSUMER_THREAD_ID: 175 TIME: 1208870263564 MESSAGE:
1170 TIME: 1208870182650

In this configuration latency may very good - 8-50 ms or very bad -
76736-80914 ms.

> Note that this will result in the consumers receiving the messages from 
> *all* producers. The program is called "MulticastMain" for a reason :)

OK, In variant N producers => 1 consumer I see, that RabbitMQ may act as very fast "message collector":

$ sh ./runjava.sh com.rabbitmq.examples.MulticastMain -h host.domain.tld -a -r 100 -x 100 -y 1 | grep recving
...
recving rate: 9544 msg/s, min/avg/max latency: 1767/32375/113036 microseconds
recving rate: 10049 msg/s, min/avg/max latency: 1075/29551/102870 microseconds
recving rate: 9861 msg/s, min/avg/max latency: 1617/47801/138087 microseconds
recving rate: 9824 msg/s, min/avg/max latency: 1368/42204/131378 microseconds
recving rate: 10516 msg/s, min/avg/max latency: 1385/31713/99587 microseconds

And in variant 1 producers => N consumer I, see that RabbitMQ may act as very fast "message brodcaster":

$ sh ./runjava.sh com.rabbitmq.examples.MulticastMain -h host.domain.tld -a -r 100 -x 1 -y 100 | grep recving
recving rate: 0 msg/s, min/avg/max latency: 48143/48143/48143 microseconds
recving rate: 10024 msg/s, min/avg/max latency: 5343/38838/96254 microseconds
recving rate: 9867 msg/s, min/avg/max latency: 5649/44256/93694 microseconds
recving rate: 9814 msg/s, min/avg/max latency: 2857/41523/92171 microseconds
recving rate: 10218 msg/s, min/avg/max latency: 2951/41081/101019 microseconds
recving rate: 9995 msg/s, min/avg/max latency: 5634/39705/92788 microseconds
recving rate: 9933 msg/s, min/avg/max latency: 2846/42851/100338 microseconds

I am very sorry about my (may be very stupid ?) question: 

Is any way to improve perfomance and reduce latency in my N producers => M consumers scheme ?

WBR,
Alter