[rabbitmq-discuss] Throughput observation with RabbitMQ-3.1.3 and Erlang R16B01 - Single Node and Cluster Node

Fri Aug 16 10:33:44 BST 2013

Hi Again,

1) Regarding Overhead point:
You mentioned AMQP is a binary protocol. Could you give some importance of
this type of protocol with overhead ? Do you mean that it will have less
overhead than what I am saying ?

I might have made some mistake since I am new to this world. So please
guide me.

In my wireshark traces, I see following frames for every message I send. I
have publisher confirm and consumer ack enabled with prefetch_count not
set.

content header = 88 bytes ( does not show any direct info so not sure)
basic.publish frame = 85 bytes ( shows details of queue, exchange, routing
key etc)
content body = 137 bytes ( in this my payload is 100 bytes of string + 37
bytes might be checksum etc.)
basic.ack = 87 bytes (shows delivery tag number)

so from this I interpreted as 297 bytes  non-payload + 100 bytes of actual
payload = 397 total bytes for every message on wire.

Is this not normal ?

My publisher and consumer code looks like this.

2) Regarding Interrupts and context switches point:
I agree and understand what you said about context switch but I wanted to
bring this up since I felt my CPU was spending lot of time in doing context
switches and that could be one of the reason that I don't get any higher
throughput even if I increase number of producer and receiver.

This logic might be completely wrong but good to ask experts in this field
about this context switch increase.

3) Regarding changing to faster clients like Java or amqpc:
My current concern is not that I don't have sufficinet throughout, I would
like to understand given the fact later project need to use python based
client and not java based client, would like to understand how system
resources are used by rabbitmq.

Could you please guide me on

1) How to measure load on the disk, when running as high as I can run.

Thank you again !

Best Regards,
Priyanki.

On Thu, Aug 15, 2013 at 12:06 PM, Michael Klishin <mklishin at gopivotal.com>wrote:

> Priyanki Vashi:
>
> > 1) I have a quite high bandwidth between my client (producer & receiver)
> and server. Its around 9.2 Gbps. I have confirmed this by running iperf
> utility. But the maximum bandwidth I can reach is only 160 mbps. (I have
> pika client and single rabbitmq DISK type)
>
> This means your client cannot publish faster.
>
> > 2) I then also took network traces using wireshark, and what I observed
> that the AMQP overhead was quite high then the actual payload. If I send
> 100 bytes of string then overhead is 300 bytes so for every message of 100
> bytes I end up sending 400 bytes on wire.
>
> That does not sound right. AMQP 0-9-1 is a binary protocol and for every
> published message
> without custom headers, you should get a few dozens of bytes + message
> payload.
>
> The rest is probably TCP and IP overhead. There is no way to use RabbitMQ
> over UDP.
>
> > Since I have publish confirm/ack enabled so I understand part of this
> but there is also content header and some overhead in content body as well.
> > Is this standard for most of the clients or its specific for
> pika-clients only ?
>
> For a basic.publish with payload, you will get 3 frames:
>
> [basic.publish method frame] [content headers] [content payload]
>
> If the message has no content, it becomes
>
> [basic.publish method frame] [content headers]
>
> with every correct client.
>
> > 3) I then also try to observer how my system resources are used. I am
> new to this but after some googling I learnt a linux command 'dstat
> -dmnsy'. This provides me disk usage, RAM usage, interrupts and context
> switches, network send/recev. What I observer that as I increase no of
> producer/receiver, compare to DISK and RAM usage, its interrupts and
> context switches, which increases rapidly.
> >
> > what does this really mean ?
>
> When you publish rapidly, RabbitMQ stores messages (and some metadata
> about them) in RAM and on disk. Context switches means that OS gives
> various processes (your publisher, RabbitMQ, other processes running) time
> to execute. Flat network traffic rate suggests your publisher cannot
> publish any faster.
>
> > 4) Also, after certain point the value under network send/receive field
> remains constant. Not sure what does this mean also ?
>
> See above.
>
> >
> > I am pasting here one of the observed sample from dstat.
> >
> >    mq10 at mqserver10:~$ sudo rabbitmqadmin list queues vhost name node
> messages message_stats.publish_details.rate
> >
> +-------+------+-----------------+----------+------------------------------------+
> > | vhost | name |      node       | messages |
> message_stats.publish_details.rate |
> >
> +-------+------+-----------------+----------+------------------------------------+
> > | /     | 1    | Moon at mqserver10 | 5        | 3380.4
>           |
> > | /     | 2    | Moon at mqserver10 | 3        | 3649.0
>           |
> > | /     | 3    | Moon at mqserver10 | 6        | 3402.4
>           |
> > | /     | 4    | Moon at mqserver10 | 2        | 3450.8
>           |
> > | /     | 5    | Moon at mqserver10 | 3        | 3444.8
>           |
> >
> +-------+------+-----------------+----------+------------------------------------+
> > mq10 at mqserver10:~$ dstat -dmnsy
> > -dsk/total- ------memory-usage----- -net/total- ----swap--- ---system--
> >  read  writ| used  buff  cach  free| recv  send| used  free| int   csw
> >  100k 8974B| 188M 25.7M  115M 9674M|   0     0 |   0     0 |  15k 9759
> >    0    16k| 189M 25.7M  115M 9673M|6363k 4615k|   0     0 |  22k 6590
> >    0     0 | 189M 25.7M  115M 9673M|6666k 4847k|   0     0 |  22k 7846
> >    0     0 | 189M 25.7M  115M 9673M|6293k 4500k|   0     0 |  21k 7518
> >    0     0 | 189M 25.7M  115M 9674M|7049k 5121k|   0     0 |  22k 5936
> >    0     0 | 188M 25.7M  115M 9674M|6947k 5081k|   0     0 |  23k 6309
> >    0     0 | 188M 25.7M  115M 9674M|6387k 4741k|   0     0 |  22k 7020
> >    0     0 | 188M 25.7M  115M 9674M|6482k 4739k|   0     0 |  22k 7094
> >    0     0 | 189M 25.7M  115M 9673M|5642k 3976k|   0     0 |  18k 7586
> >    0     0 | 189M 25.7M  115M 9673M|6246k 4487k|   0     0 |  21k 7497
> >    0     0 | 189M 25.7M  115M 9673M|6631k 4827k|   0     0 |  21k 5924
> >    0     0 | 189M 25.7M  115M 9673M|6615k 4722k|   0     0 |  21k 5448
> >    0     0 | 190M 25.7M  115M 9672M|5677k 4051k|   0     0 |  20k 5757
> >    0     0 | 190M 25.7M  115M 9672M|6007k 4284k|   0     0 |  20k 6581
> >    0     0 | 190M 25.7M  115M 9672M|6983k 5052k|   0     0 |  23k 5704
> >    0     0 | 190M 25.7M  115M 9672M|6318k 4539k|   0     0 |  22k 6208
> >    0     0 | 190M 25.7M  115M 9672M|6769k 4862k|   0     0 |  23k 5952
> >    0     0 | 189M 25.7M  115M 9673M|6334k 4586k|   0     0 |  22k 6518
> >    0     0 | 191M 25.7M  115M 9671M|7019k 5089k|   0     0 |  23k 5937 ^C
>
> This suggests your publisher publishes at a roughly constant rate.
> Which, given spare network capacity (per your own words) suggests
> that the publisher cannot go any faster.
>
> > Also, what is the good profiler to run for rabbitmq server ?
>
> http://www.erlang.org/doc/efficiency_guide/profiling.html
>
> I'd like to point you that it's not RabbitMQ that seem to need profiling
> but your
> client.
>
> Try MulticastMain that ships with the RabbitMQ Java client or amqpc [1]
> to see what kind of thoughput your RabbitMQ node can offer with really fast
> clients.
>
> 1. https://github.com/gocardless/amqpc
> --
> MK
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130816/b637f2ce/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Asyn_Pub_class.py
Type: application/octet-stream
Size: 14631 bytes
Desc: not available
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130816/b637f2ce/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Asyn_Pub_1.py
Type: application/octet-stream
Size: 307 bytes
Desc: not available
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130816/b637f2ce/attachment-0001.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Asyn_Cons_1.py
Type: application/octet-stream
Size: 265 bytes
Desc: not available
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130816/b637f2ce/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Asyn_Cons_class.py
Type: application/octet-stream
Size: 13986 bytes
Desc: not available
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130816/b637f2ce/attachment-0003.obj>