[rabbitmq-discuss] Question on throughput with RabbitMQ-3.1.1 single node as well as cluster

[Tim Watson]

What does your publishing code look like? The figures below are expected in that the consumer can keep pace with the producer - it could hardly be expected to consume faster than messages are arriving in the queue(s). So the slowness is very likely on the producing side.

Are you using persistent messages and either publisher confirms or transactions? If so, how often are you waiting on confirms/commits?

With the official clients we typically see avg rates of 50 - 60Khz with non-persistent messages. Persistence slows things down a tad, as do confirms (and more so transactions) but even with persistent messages and confirms, rates >= 5Khz are expected. It /sounds/ like you might be publishing persistent messages with confirms enabled and waiting for a confirm (ack) from the broker for each message. That  involves disk I/O on the server for each message plus network latency, effectively making publishing synchronous (and very slow by comparison).

Cheers,
Tim

On 25 Jun 2013, at 08:44, Priyanki Vashi <vashi.priyanki at gmail.com> wrote:

> Hi there,
> 
> I am doing a performance study of RabbitMQ-3.1.1 and this is my first time to do such a performance study with any messaging broker :))-
> 
> 1) I have thoroughly gone through rabbitMQ in action' and learnt important concepts. 
> 
> 2) Tried single node broker to get a feel of how it is working and then set up a four node cluster (with two disk and two RAM type of node). Also configured HAproxy TCP Load balancer so that I can just provide single port to connect to the Cluster.
> 
> 3) I am simulating producer and consumers through Python scripts ( using Python-pika library methods to connect to server , publish subscribe etc.)
> 
> 4) My scripts are working fine but where I am stuck is no matter what I do my throughout is always 300 msg/sec. 
> 
> 5) I have defines durable exchanges and queues
> 
> My final requirement is to run atleast 10 to 15 producer and 60 to 70 consumer simultaneously and I want to start with linear increase in number of producer and consumer so that I can make conclusions about  throughout, fault handling, processor utilization etc. etc but I am seriously stuck now after trying to start in initial steps only. This group's help would be really appreciated. 
> 
> I have started with following different scenarios but no matter what I do my throughput is more or less remaining same (300 msg/sec) except for Scenario-1
> 
> Scenario-1
> -1 producer and No consumer and no queue binded to exchange
> -Producer is running in infinite loop and publishing to one fanout exchange
> - publisher/confirm disabled
> -Publisher rate - 6200 msg/sec ( checked through web management plugin)
> 
> Tried scenario-1 with also fanout type of exchange and it's the same publish rate
> I know that Scenario-1 is not really useful, since there are no queues and ultimately messages will be dropped but as a part of debugging process I tried this and I see above mentioned results.
> 
> Scenario-2
> -1 producer and 1 consumer 
> -Producer is running in infinite loop and publishing to one direct exchange
> -A consumer has it's own dedicated queue and listening to above exchange 
> - publisher/confirm and consumer ack are disabled
> Throughput - 300 msg/sec ( which is basically publish rate = 300 msg/sec and deliver rate - 300 msg/sec)
> 
> Tried Scenario-2 also with fanout type of exchange and enabling publisher confirm and consumer ack
> Still the same throughput as 300 msg/sec
> 
> Scenario-3
> -1 producer and 4 consumer 
> -Producer is running in infinite loop and publishing to four direct exchange
> -A consumer has it's own dedicated queue and listening to respective exchange 
> - publisher/confirm and consumer ack are disabled
> Throughput - 300 msg/sec ( which is basically publish rate = 300 msg/sec and deliver rate - 300 msg/sec)
> 
> Tried Scenario-3 also with fanout type of exchange and enabling publisher confirm and consumer ack
> Still the same throughput as 300 msg/sec
> 
> Tried configuring prefetch_count parameters also to 100 but it still gives me same throughput of 300 msg/sec.
> I am honestly going crazy with this.
> 
> After seeing this behavior, I am seriously suspecting that there is some serious limitation with my simulated producers and consumers.
> Has anyone else has tried Python-pika client and any idea on throughput with this version of rabbit ? 
> Did anyone have rough idea about throughout with rabbitMQ-3.1.1 ?
> 
> I can also share my python scripts if required but I would really appreciate some light on this situation
> Also what points to take care, in order to improve throughput ?
> 
> Best Regards,
> Priyanki
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130625/2c92c1a4/attachment.htm>