[rabbitmq-discuss] Throughput observation with RabbitMQ-3.1.3 and Erlang R16B01 - Single Node and Cluster Node

Fri Aug 16 19:50:12 BST 2013

Priyanki Vashi:

> 1) how good or bad my pika clients efficiciency is and then how exactly my system resources are used by both client and rabbit server.

The goal really is still to see some evidence about producer being not fast enough. If you get much better results with faster publishers, esp.
if they can use multiple cores (MulticastMain can, I expect the same to be true of amqpc), that
would prove the hypothesis we have.

> So far I can only see that if I send 100 byes of payload, maximum throughput I get is around 30,000 msg/sec and maximum bandwidth usage is 160-170 mbps. Trying to understand what is happening in system with RAM, memory etc. etc.... 
> 
> As you mentioned I see comapritively better bandwidth usage with 500 bytes and 1000 bytes and they are in range of 350 Mbps and 580 Mpbs. Which I don't know is appropriate or not.
> 
> Do you know if one can saturate 8Gbps link with Rabbit ? 

Start publishing messages of 1-8 MB in size. You will see throughput drop but overall link saturation
should be higher.

> For the earlier question of Micheal on number of cores, here is some more info. 
> 
> I have 10 physical cores and they are hyperthreaded so that's how I have 20 cores. I have such two machines. So then,
> 
> Machine-1 with 20 cores running exclusively my Producer & Receiver based on pika
> Machine-2, again with 20 cores running exclusively RABBITMQ server

On this kind of machines, you should run 5-20 instances of both producer and consumer and context
switching won't be a problem.

You can always check core saturation and load average (a.k.a. how many processes expect the OS to give it a slice of CPU time) in htop. Increase numbers of producer and consumer until all 10 cores are
busy most of the time.

On the RabbitMQ side, running Erlang R16B01 and using multiple queues
(e.g. at least 1 per consumer) should help saturate more cores given that the runtime actually
needs them.
--
MK

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130816/1a0082a3/attachment.pgp>