[rabbitmq-discuss] Question on throughput with RabbitMQ-3.1.1

[Priyanki Vashi]

Hi there,

I am doing a performance study of RabbitMQ-3.1.1 and this is my first time
to do such a performance study with any messaging broker :))-

1) I have thoroughly gone through rabbitMQ in action' and learnt important
concepts.

2) Tried single node broker to get a feel of how it is working and then set
up a four node cluster (with two disk and two RAM type of node). Also
configured HAproxy TCP Load balancer so that I can just provide single port
to connect to the Cluster.

3) I am simulating producer and consumers through Python scripts ( using
Python-pika library methods to connect to server , publish subscribe etc.)

4) My scripts are working fine but where I am stuck is no matter what I do
my throughout is always 300 msg/sec.

5) I have defines durable exchanges and queues

My final requirement is to run atleast 10 to 15 producer and 60 to 70
consumer simultaneously and I want to start with linear increase in number
of producer and consumer so that I can make conclusions about  throughout,
fault handling, processor utilization etc. etc but I am seriously stuck now
after trying to start in initial steps only. This group's help would be
really appreciated.

I have started with following different scenarios but no matter what I do
my throughput is more or less remaining same (300 msg/sec) except for
Scenario-1

Scenario-1
-1 producer and No consumer and no queue binded to exchange
-Producer is running in infinite loop and publishing to one fanout exchange
- publisher/confirm disabled
-Publisher rate - 6200 msg/sec ( checked through web management plugin)

Tried scenario-1 with also fanout type of exchange and it's the same
publish rate
I know that Scenario-1 is not really useful, since there are no queues and
ultimately messages will be dropped but as a part of debugging process I
tried this and I see above mentioned results.

Scenario-2
-1 producer and 1 consumer
-Producer is running in infinite loop and publishing to one direct exchange
-A consumer has it's own dedicated queue and listening to above exchange
- publisher/confirm and consumer ack are disabled
Throughput - 300 msg/sec ( which is basically publish rate = 300 msg/sec
and deliver rate - 300 msg/sec)

Tried Scenario-2 also with fanout type of exchange and enabling publisher
confirm and consumer ack
Still the same throughput as 300 msg/sec

Scenario-3
-1 producer and 4 consumer
-Producer is running in infinite loop and publishing to four direct exchange
-A consumer has it's own dedicated queue and listening to respective
exchange
- publisher/confirm and consumer ack are disabled
Throughput - 300 msg/sec ( which is basically publish rate = 300 msg/sec
and deliver rate - 300 msg/sec)

Tried Scenario-3 also with fanout type of exchange and enabling publisher
confirm and consumer ack
Still the same throughput as 300 msg/sec

Tried configuring prefetch_count parameters also to 100 but it still gives
me same throughput of 300 msg/sec.
I am honestly going crazy with this.

After seeing this behavior, I am seriously suspecting that there is some
serious limitation with my simulated producers and consumers.
Has anyone else has tried Python-pika client and any idea on throughput
with this version of rabbit ?
Did anyone have rough idea about throughout with rabbitMQ-3.1.1 ?

I can also share my python scripts if required but I would really
appreciate some light on this situation
Also what points to take care, in order to improve throughput ?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130624/9f4f74d9/attachment.htm>