<div dir="ltr">Hi Again,<div><br></div><div>1) Regarding Overhead point:</div><div>You mentioned AMQP is a binary protocol. Could you give some importance of this type of protocol with overhead ? Do you mean that it will have less overhead than what I am saying ? </div>
<div><br></div><div>I might have made some mistake since I am new to this world. So please guide me. </div><div><br></div><div>In my wireshark traces, I see following frames for every message I send. I have publisher confirm and consumer ack enabled with prefetch_count not set. </div>
<div><br></div><div>content header = 88 bytes ( does not show any direct info so not sure)</div><div>basic.publish frame = 85 bytes ( shows details of queue, exchange, routing key etc) </div><div>content body = 137 bytes ( in this my payload is 100 bytes of string + 37 bytes might be checksum etc.)<br>
</div><div>basic.ack = 87 bytes (shows delivery tag number) </div><div><br></div><div>so from this I interpreted as 297 bytes non-payload + 100 bytes of actual payload = 397 total bytes for every message on wire. </div><div>
<br></div><div>Is this not normal ? </div><div><br></div><div>My publisher and consumer code looks like this. </div><div><br></div><div><br></div><div>2) Regarding Interrupts and context switches point:</div><div>I agree and understand what you said about context switch but I wanted to bring this up since I felt my CPU was spending lot of time in doing context switches and that could be one of the reason that I don't get any higher throughput even if I increase number of producer and receiver. </div>
<div><br></div><div>This logic might be completely wrong but good to ask experts in this field about this context switch increase. </div><div><br></div><div>3) Regarding changing to faster clients like Java or amqpc:</div>
<div>My current concern is not that I don't have sufficinet throughout, I would like to understand given the fact later project need to use python based client and not java based client, would like to understand how system resources are used by rabbitmq. </div>
<div><br></div><div>Could you please guide me on </div><div><br></div><div>1) How to measure load on the disk, when running as high as I can run.</div><div><br></div><div>Thank you again !</div><div><br></div><div>Best Regards,</div>
<div>Priyanki. </div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Aug 15, 2013 at 12:06 PM, Michael Klishin <span dir="ltr"><<a href="mailto:mklishin@gopivotal.com" target="_blank">mklishin@gopivotal.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Priyanki Vashi:<br>
<div class="im"><br>
> 1) I have a quite high bandwidth between my client (producer & receiver) and server. Its around 9.2 Gbps. I have confirmed this by running iperf utility. But the maximum bandwidth I can reach is only 160 mbps. (I have pika client and single rabbitmq DISK type)<br>
<br>
</div>This means your client cannot publish faster.<br>
<div class="im"><br>
> 2) I then also took network traces using wireshark, and what I observed that the AMQP overhead was quite high then the actual payload. If I send 100 bytes of string then overhead is 300 bytes so for every message of 100 bytes I end up sending 400 bytes on wire.<br>
<br>
</div>That does not sound right. AMQP 0-9-1 is a binary protocol and for every published message<br>
without custom headers, you should get a few dozens of bytes + message payload.<br>
<br>
The rest is probably TCP and IP overhead. There is no way to use RabbitMQ over UDP.<br>
<div class="im"><br>
> Since I have publish confirm/ack enabled so I understand part of this but there is also content header and some overhead in content body as well.<br>
> Is this standard for most of the clients or its specific for pika-clients only ?<br>
<br>
</div>For a basic.publish with payload, you will get 3 frames:<br>
<br>
[basic.publish method frame] [content headers] [content payload]<br>
<br>
If the message has no content, it becomes<br>
<br>
[basic.publish method frame] [content headers]<br>
<br>
with every correct client.<br>
<div class="im"><br>
> 3) I then also try to observer how my system resources are used. I am new to this but after some googling I learnt a linux command 'dstat -dmnsy'. This provides me disk usage, RAM usage, interrupts and context switches, network send/recev. What I observer that as I increase no of producer/receiver, compare to DISK and RAM usage, its interrupts and context switches, which increases rapidly.<br>
><br>
> what does this really mean ?<br>
<br>
</div>When you publish rapidly, RabbitMQ stores messages (and some metadata about them) in RAM and on disk. Context switches means that OS gives various processes (your publisher, RabbitMQ, other processes running) time to execute. Flat network traffic rate suggests your publisher cannot<br>
publish any faster.<br>
<div class="im"><br>
> 4) Also, after certain point the value under network send/receive field remains constant. Not sure what does this mean also ?<br>
<br>
</div>See above.<br>
<div><div class="h5"><br>
><br>
> I am pasting here one of the observed sample from dstat.<br>
><br>
> mq10@mqserver10:~$ sudo rabbitmqadmin list queues vhost name node messages message_stats.publish_details.rate<br>
> +-------+------+-----------------+----------+------------------------------------+<br>
> | vhost | name | node | messages | message_stats.publish_details.rate |<br>
> +-------+------+-----------------+----------+------------------------------------+<br>
> | / | 1 | Moon@mqserver10 | 5 | 3380.4 |<br>
> | / | 2 | Moon@mqserver10 | 3 | 3649.0 |<br>
> | / | 3 | Moon@mqserver10 | 6 | 3402.4 |<br>
> | / | 4 | Moon@mqserver10 | 2 | 3450.8 |<br>
> | / | 5 | Moon@mqserver10 | 3 | 3444.8 |<br>
> +-------+------+-----------------+----------+------------------------------------+<br>
> mq10@mqserver10:~$ dstat -dmnsy<br>
> -dsk/total- ------memory-usage----- -net/total- ----swap--- ---system--<br>
> read writ| used buff cach free| recv send| used free| int csw<br>
> 100k 8974B| 188M 25.7M 115M 9674M| 0 0 | 0 0 | 15k 9759<br>
> 0 16k| 189M 25.7M 115M 9673M|6363k 4615k| 0 0 | 22k 6590<br>
> 0 0 | 189M 25.7M 115M 9673M|6666k 4847k| 0 0 | 22k 7846<br>
> 0 0 | 189M 25.7M 115M 9673M|6293k 4500k| 0 0 | 21k 7518<br>
> 0 0 | 189M 25.7M 115M 9674M|7049k 5121k| 0 0 | 22k 5936<br>
> 0 0 | 188M 25.7M 115M 9674M|6947k 5081k| 0 0 | 23k 6309<br>
> 0 0 | 188M 25.7M 115M 9674M|6387k 4741k| 0 0 | 22k 7020<br>
> 0 0 | 188M 25.7M 115M 9674M|6482k 4739k| 0 0 | 22k 7094<br>
> 0 0 | 189M 25.7M 115M 9673M|5642k 3976k| 0 0 | 18k 7586<br>
> 0 0 | 189M 25.7M 115M 9673M|6246k 4487k| 0 0 | 21k 7497<br>
> 0 0 | 189M 25.7M 115M 9673M|6631k 4827k| 0 0 | 21k 5924<br>
> 0 0 | 189M 25.7M 115M 9673M|6615k 4722k| 0 0 | 21k 5448<br>
> 0 0 | 190M 25.7M 115M 9672M|5677k 4051k| 0 0 | 20k 5757<br>
> 0 0 | 190M 25.7M 115M 9672M|6007k 4284k| 0 0 | 20k 6581<br>
> 0 0 | 190M 25.7M 115M 9672M|6983k 5052k| 0 0 | 23k 5704<br>
> 0 0 | 190M 25.7M 115M 9672M|6318k 4539k| 0 0 | 22k 6208<br>
> 0 0 | 190M 25.7M 115M 9672M|6769k 4862k| 0 0 | 23k 5952<br>
> 0 0 | 189M 25.7M 115M 9673M|6334k 4586k| 0 0 | 22k 6518<br>
> 0 0 | 191M 25.7M 115M 9671M|7019k 5089k| 0 0 | 23k 5937 ^C<br>
<br>
</div></div>This suggests your publisher publishes at a roughly constant rate.<br>
Which, given spare network capacity (per your own words) suggests<br>
that the publisher cannot go any faster.<br>
<div class="im"><br>
> Also, what is the good profiler to run for rabbitmq server ?<br>
<br>
</div><a href="http://www.erlang.org/doc/efficiency_guide/profiling.html" target="_blank">http://www.erlang.org/doc/efficiency_guide/profiling.html</a><br>
<br>
I'd like to point you that it's not RabbitMQ that seem to need profiling but your<br>
client.<br>
<br>
Try MulticastMain that ships with the RabbitMQ Java client or amqpc [1]<br>
to see what kind of thoughput your RabbitMQ node can offer with really fast<br>
clients.<br>
<br>
1. <a href="https://github.com/gocardless/amqpc" target="_blank">https://github.com/gocardless/amqpc</a><br>
--<br>
MK<br>
<br>
<br>_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
<br></blockquote></div><br></div>