<html><head></head><body bgcolor="#FFFFFF"><div>Reposting</div><div><br>On 25 Jun 2013, at 09:38, Tim Watson <<a href="mailto:tim@rabbitmq.com">tim@rabbitmq.com</a>> wrote:<br><br></div><div></div><blockquote type="cite"><div><div>What does your publishing code look like? The figures below are expected in that the consumer can keep pace with the producer - it could hardly be expected to consume faster than messages are arriving in the queue(s). So the slowness is very likely on the producing side.</div><div><br></div><div>Are you using persistent messages and either publisher confirms or transactions? If so, how often are you waiting on confirms/commits?</div><div><br></div><div>With the official clients we typically see avg rates of 50 - 60Khz with non-persistent messages. Persistence slows things down a tad, as do confirms (and more so transactions) but even with persistent messages and confirms, rates >= 5Khz are expected. It /sounds/ like you might be publishing persistent messages with confirms enabled and waiting for a confirm (ack) from the broker for each message. That involves disk I/O on the server for each message plus network latency, effectively making publishing synchronous (and very slow by comparison).</div><div><br></div><div>Cheers,</div><div>Tim</div><div><br>On 25 Jun 2013, at 08:44, Priyanki Vashi <<a href="mailto:vashi.priyanki@gmail.com">vashi.priyanki@gmail.com</a>> wrote:<br><br></div><div></div><blockquote type="cite"><div><span style="font-family: arial, sans-serif; font-size: 12.800000190734863px;">Hi there,</span><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;"><br></div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;">I am doing a performance study of RabbitMQ-3.1.1 and this is my first time to do such a performance study with any messaging broker :))-</div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;"><br></div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;">1) I have thoroughly gone through rabbitMQ in action' and learnt important concepts. </div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;"><br></div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;">2) Tried single node broker to get a feel of how it is working and then set up a four node cluster (with two disk and two RAM type of node). Also configured HAproxy TCP Load balancer so that I can just provide single port to connect to the Cluster.</div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;"><br></div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;">3) I am simulating producer and consumers through Python scripts ( using Python-pika library methods to connect to server , publish subscribe etc.)</div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;"><br></div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;">4) My scripts are working fine but where I am stuck is no matter what I do my throughout is always 300 msg/sec. </div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;"><br></div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;"><div>5) I have defines durable exchanges and queues</div><div><br></div><div>My final requirement is to run atleast 10 to 15 producer and 60 to 70 consumer simultaneously and I want to start with linear increase in number of producer and consumer so that I can make conclusions about throughout, fault handling, processor utilization etc. etc but I am seriously stuck now after trying to start in initial steps only. This group's help would be really appreciated. </div><div><br></div></div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;">I have started with following different scenarios but no matter what I do my throughput is more or less remaining same (300 msg/sec) except for Scenario-1</div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;"><br></div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;">Scenario-1</div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;">-1 producer and No consumer and no queue binded to exchange</div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;">-Producer is running in infinite loop and publishing to one fanout exchange</div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;">- publisher/confirm disabled<br></div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;">-Publisher rate - 6200 msg/sec ( checked through web management plugin)</div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;"><br></div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;">Tried scenario-1 with also fanout type of exchange and it's the same publish rate</div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;">I know that Scenario-1 is not really useful, since there are no queues and ultimately messages will be dropped but as a part of debugging process I tried this and I see above mentioned results.</div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;"><br></div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;"><div>Scenario-2</div><div>-1 producer and 1 consumer </div><div>-Producer is running in infinite loop and publishing to one direct exchange</div><div>-A consumer has it's own dedicated queue and listening to above exchange </div><div>- publisher/confirm and consumer ack are disabled</div><div>Throughput - 300 msg/sec ( which is basically publish rate = 300 msg/sec and deliver rate - 300 msg/sec)</div><div><br></div><div>Tried Scenario-2 also with fanout type of exchange and enabling publisher confirm and consumer ack</div><div>Still the same throughput as 300 msg/sec</div></div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;"><br></div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;"><div>Scenario-3</div><div>-1 producer and 4 consumer </div><div>-Producer is running in infinite loop and publishing to four direct exchange</div><div>-A consumer has it's own dedicated queue and listening to respective exchange </div><div>- publisher/confirm and consumer ack are disabled</div><div>Throughput - 300 msg/sec ( which is basically publish rate = 300 msg/sec and deliver rate - 300 msg/sec)</div><div><br></div><div><div>Tried Scenario-3 also with fanout type of exchange and enabling publisher confirm and consumer ack</div><div>Still the same throughput as 300 msg/sec</div></div><div><br></div><div>Tried configuring prefetch_count parameters also to 100 but it still gives me same throughput of 300 msg/sec.</div><div>I am honestly going crazy with this.</div><div><br></div></div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;">After seeing this behavior, I am seriously suspecting that there is some serious limitation with my simulated producers and consumers.</div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;">Has anyone else has tried Python-pika client and any idea on throughput with this version of rabbit ? </div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;">Did anyone have rough idea about throughout with rabbitMQ-3.1.1 ?</div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;"><br></div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;">I can also share my python scripts if required but I would really appreciate some light on this situation</div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;">Also what points to take care, in order to improve throughput ?</div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;"><br></div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;">Best Regards,</div><div style="font-family: arial, sans-serif; font-size: 12.800000190734863px;">Priyanki</div></div></blockquote><blockquote type="cite"><div><span>_______________________________________________</span><br><span>rabbitmq-discuss mailing list</span><br><span><a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a></span><br><span><a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a></span><br></div></blockquote></div></blockquote></body></html>