[rabbitmq-discuss] Question on throughput with RabbitMQ-3.1.1 single node as well as cluster

[Tim Watson]

Reposting

On 25 Jun 2013, at 09:38, Tim Watson <tim at rabbitmq.com> wrote:

> What does your publishing code look like? The figures below are expected in that the consumer can keep pace with the producer - it could hardly be expected to consume faster than messages are arriving in the queue(s). So the slowness is very likely on the producing side.
> 
> Are you using persistent messages and either publisher confirms or transactions? If so, how often are you waiting on confirms/commits?
> 
> With the official clients we typically see avg rates of 50 - 60Khz with non-persistent messages. Persistence slows things down a tad, as do confirms (and more so transactions) but even with persistent messages and confirms, rates >= 5Khz are expected. It /sounds/ like you might be publishing persistent messages with confirms enabled and waiting for a confirm (ack) from the broker for each message. That  involves disk I/O on the server for each message plus network latency, effectively making publishing synchronous (and very slow by comparison).
> 
> Cheers,
> Tim
> 
> On 25 Jun 2013, at 08:44, Priyanki Vashi <vashi.priyanki at gmail.com> wrote:
> 
>> Hi there,
>> 
>> I am doing a performance study of RabbitMQ-3.1.1 and this is my first time to do such a performance study with any messaging broker :))-
>> 
>> 1) I have thoroughly gone through rabbitMQ in action' and learnt important concepts. 
>> 
>> 2) Tried single node broker to get a feel of how it is working and then set up a four node cluster (with two disk and two RAM type of node). Also configured HAproxy TCP Load balancer so that I can just provide single port to connect to the Cluster.
>> 
>> 3) I am simulating producer and consumers through Python scripts ( using Python-pika library methods to connect to server , publish subscribe etc.)
>> 
>> 4) My scripts are working fine but where I am stuck is no matter what I do my throughout is always 300 msg/sec. 
>> 
>> 5) I have defines durable exchanges and queues
>> 
>> My final requirement is to run atleast 10 to 15 producer and 60 to 70 consumer simultaneously and I want to start with linear increase in number of producer and consumer so that I can make conclusions about  throughout, fault handling, processor utilization etc. etc but I am seriously stuck now after trying to start in initial steps only. This group's help would be really appreciated. 
>> 
>> I have started with following different scenarios but no matter what I do my throughput is more or less remaining same (300 msg/sec) except for Scenario-1
>> 
>> Scenario-1
>> -1 producer and No consumer and no queue binded to exchange
>> -Producer is running in infinite loop and publishing to one fanout exchange
>> - publisher/confirm disabled
>> -Publisher rate - 6200 msg/sec ( checked through web management plugin)
>> 
>> Tried scenario-1 with also fanout type of exchange and it's the same publish rate
>> I know that Scenario-1 is not really useful, since there are no queues and ultimately messages will be dropped but as a part of debugging process I tried this and I see above mentioned results.
>> 
>> Scenario-2
>> -1 producer and 1 consumer 
>> -Producer is running in infinite loop and publishing to one direct exchange
>> -A consumer has it's own dedicated queue and listening to above exchange 
>> - publisher/confirm and consumer ack are disabled
>> Throughput - 300 msg/sec ( which is basically publish rate = 300 msg/sec and deliver rate - 300 msg/sec)
>> 
>> Tried Scenario-2 also with fanout type of exchange and enabling publisher confirm and consumer ack
>> Still the same throughput as 300 msg/sec
>> 
>> Scenario-3
>> -1 producer and 4 consumer 
>> -Producer is running in infinite loop and publishing to four direct exchange
>> -A consumer has it's own dedicated queue and listening to respective exchange 
>> - publisher/confirm and consumer ack are disabled
>> Throughput - 300 msg/sec ( which is basically publish rate = 300 msg/sec and deliver rate - 300 msg/sec)
>> 
>> Tried Scenario-3 also with fanout type of exchange and enabling publisher confirm and consumer ack
>> Still the same throughput as 300 msg/sec
>> 
>> Tried configuring prefetch_count parameters also to 100 but it still gives me same throughput of 300 msg/sec.
>> I am honestly going crazy with this.
>> 
>> After seeing this behavior, I am seriously suspecting that there is some serious limitation with my simulated producers and consumers.
>> Has anyone else has tried Python-pika client and any idea on throughput with this version of rabbit ? 
>> Did anyone have rough idea about throughout with rabbitMQ-3.1.1 ?
>> 
>> I can also share my python scripts if required but I would really appreciate some light on this situation
>> Also what points to take care, in order to improve throughput ?
>> 
>> Best Regards,
>> Priyanki
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130625/9df8fcac/attachment.htm>