[rabbitmq-discuss] Question on throughput with RabbitMQ-3.1.1 single node as well as cluster

[Priyanki Vashi]

Hi there,

I am doing a performance study of RabbitMQ-3.1.1 and this is my first time 
to do such a performance study with any messaging broker :))-

1) I have thoroughly gone through rabbitMQ in action' and learnt important 
concepts. 

2) Tried single node broker to get a feel of how it is working and then set 
up a four node cluster (with two disk and two RAM type of node). Also 
configured HAproxy TCP Load balancer so that I can just provide single port 
to connect to the Cluster.

3) I am simulating producer and consumers through Python scripts ( using 
Python-pika library methods to connect to server , publish subscribe etc.)

4) My scripts are working fine but where I am stuck is no matter what I do 
my throughout is always 300 msg/sec. 

5) I have defines durable exchanges and queues

My final requirement is to run atleast 10 to 15 producer and 60 to 70 
consumer simultaneously and I want to start with linear increase in number 
of producer and consumer so that I can make conclusions about  throughout, 
fault handling, processor utilization etc. etc but I am seriously stuck now 
after trying to start in initial steps only. This group's help would be 
really appreciated. 

I have started with following different scenarios but no matter what I do 
my throughput is more or less remaining same (300 msg/sec) except for 
Scenario-1

Scenario-1
-1 producer and No consumer and no queue binded to exchange
-Producer is running in infinite loop and publishing to one fanout exchange
- publisher/confirm disabled
-Publisher rate - 6200 msg/sec ( checked through web management plugin)

Tried scenario-1 with also fanout type of exchange and it's the same 
publish rate
I know that Scenario-1 is not really useful, since there are no queues and 
ultimately messages will be dropped but as a part of debugging process I 
tried this and I see above mentioned results.

Scenario-2
-1 producer and 1 consumer 
-Producer is running in infinite loop and publishing to one direct exchange
-A consumer has it's own dedicated queue and listening to above exchange 
- publisher/confirm and consumer ack are disabled
Throughput - 300 msg/sec ( which is basically publish rate = 300 msg/sec 
and deliver rate - 300 msg/sec)

Tried Scenario-2 also with fanout type of exchange and enabling publisher 
confirm and consumer ack
Still the same throughput as 300 msg/sec

Scenario-3
-1 producer and 4 consumer 
-Producer is running in infinite loop and publishing to four direct exchange
-A consumer has it's own dedicated queue and listening to respective 
exchange 
- publisher/confirm and consumer ack are disabled
Throughput - 300 msg/sec ( which is basically publish rate = 300 msg/sec 
and deliver rate - 300 msg/sec)

Tried Scenario-3 also with fanout type of exchange and enabling publisher 
confirm and consumer ack
Still the same throughput as 300 msg/sec

Tried configuring prefetch_count parameters also to 100 but it still gives 
me same throughput of 300 msg/sec.
I am honestly going crazy with this.

After seeing this behavior, I am seriously suspecting that there is some 
serious limitation with my simulated producers and consumers.
Has anyone else has tried Python-pika client and any idea on throughput 
with this version of rabbit ? 
Did anyone have rough idea about throughout with rabbitMQ-3.1.1 ?

I can also share my python scripts if required but I would really 
appreciate some light on this situation
Also what points to take care, in order to improve throughput ?

Best Regards,
Priyanki
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130625/cfa3ab92/attachment.htm>