[rabbitmq-discuss] questions about RabbitMQ linear scalability

[Junius Wang]

I'm sorry to pick this email again because we really want to see any
comments from all of you.

The key point is that we try to prove the linear scaling of RabbitMQ high
availability queues but no lucky. What we see are that:

1.       The throughput of two node cluster is 50%-60% worse than a single
node broker.

2.       Adding more node did have improvement on throughput but we only got
25% improvement(throughput of 3 node cluster is 25% better than 2 node
cluster. 4 node cluster is 25% better than 3 node cluster too). What we
expected is a 45-degree line, that means when 2 nodes are used the
throughput is double. With 3 nodes, then triple.

 

Can anybody give any advice? What's the best practice in using RabbitMQ
mirror queues?

 

 

From: Junius Wang [mailto:wangjunbo924 at gmail.com] 
Sent: Thursday, August 08, 2013 6:50 PM
To: rabbitmq-discuss at lists.rabbitmq.com
Subject: questions about RabbitMQ linear scalability

 

Hi guys,

Our team are performing some performance tests and we try to prove that we
can get linear scalability of our services, it means when we add more
RabbitMQ nodes, the throughputs should increase as well.  We use RabbitMQ as
our backend message system. However, when we add more RabbitMQ servers to
cluster, the throughput decreases which is not expected and I would like to
ask for your help. Following are the details, any comments are appreciated.

 

We setup the our testing environment on AWS. All instances, including app
servers, RabbitMQ servers and Jmeter servers(host test tool Jmeter 2.7), are
of  type m1.xlarge(8ECUs, 4 Cores and 15Gib memory). For RabbitMQ servers,
"EBS optimized" are enabled, and the PIOPs are set to 2000 to improve the
disc read/write speed. There are two ELBs. One for App servers and the
another for RabbitMQ servers.  Here are some more environment
configurations:


Instances Type

m1.xlarge


connect pool size for App servers

40000


JVM args for App servers

-Xmx12G -Xms6G -XX:MaxPermSize=2G


App servers count

2


Jmeter servers

10


total concurrent users

4000


PIOPs for RabbitMQ Servers

2000


Consumer Prefetch Count

1000


message size

1KB


hipe_compile

FALSE

 

The designed topologic of broker:



1)      We create only one global topic exchange "events" to route all
incoming messages. It's durable.

2)      There will be an event log queue which receive all messages with
binding key "#". It's durable as well.

3)      There will be several queues for our external services and each
service queue may have several bindings to the global exchange which we
called eventypes.

Actually In our test scenarios,  we create only one service queue and only
one binding on this queue. Message publishers publish messages using the
only binding routing key.  So in fact, every message resides on two queues:
the event log queue and a test service queue. And all messages are
persistent.

 

We start with 1 single RabbitMQ node. And then run different configurations
with 2-RabbitMQ node cluster. For example ,1) change the node type 2)
enable/disable mirrored queues(also called high available queue, showed as
"ha" in the following tables). The test results shown as following:


configurations

Avg response time(ms)

Throughput(requests/sec)


1 rabbitmq node

1125

2702.904


2disc,ha

2957

1192.645


1disc+1ram,ha

3654

998.126


1disc+1ram,non-ha

1110

2799.826


 

1015

2720.422


2disc,non-ha

1289

2570.826


 

1080

2845.821


 

1118

2802.278

We can see that, while two rabbitmq servers are used, we see no throughput
improvement. This doesn't make sense to me. To my understanding,  Here comes
my questions:

1.       What does RabbitMQ cluster scalability means exactly? Does it refer
to the capacity of handing messages? The queue capacity? Or the message
number?

2.       The  mirrored queues seems to affect performance in a large scale.
What's the possible cause of this? 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130822/172471b1/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 14245 bytes
Desc: not available
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130822/172471b1/attachment.jpeg>