[rabbitmq-discuss] Transfer rate improvement above 45mb/second?

[Aaron Westendorf]

The AMQP protocol is not very friendly to fast (de)serializing as many
of its types are encoded as "type | byte_length | bytes", which means
that one has to encode full data structures and frames before knowing
what `byte_length` is, at which point you have to go back to that
point and write the size. That is, you can't stream directly to/from a
socket, you have to maintain at least one buffer for your entire
frame, and possibly several if you aren't set up to use a single
buffer per frame.  You can see the problem here:

https://github.com/agoragames/haigha/blob/68ed0408dd9e88327b202259a9d4d12da571241d/haigha/writer.py#L169
https://github.com/agoragames/haigha/blob/9034e3608a98a3d724be7ce1503a70091293435d/haigha/frames/header_frame.py#L113

I consider this a major oversight in the protocol and a fundamental design flaw.

You may also be encountering limits of the hardware configuration,
which I described in this post:

http://blog.agoragames.com/blog/2011/06/24/of-penguins-rabbits-and-buses/

I think I saw that Rabbit 2.7.1 and Erlang R15B both have improvements
when it comes to shared read and write locks between threads, so if
you're not running the absolute latest, look into that. I haven't
repeated my tests in that blog post using the latest software so the
results may be different now.

I am not a Windows user and so I have little of value to offer in
diagnosing the problem beyond the application level.

-Aaron




On Tue, Mar 6, 2012 at 12:11 AM, Matt <matt.wolf74 at gmail.com> wrote:
> Thats sounds very unsatisfactory simply because I am not sure it addresses
> the real problem and makes me wonder  how efficient/inefficient the AMQP
> parser really is. If I send large byte arrays what needs to be parsed? My
> understanding is that each message gets parsed to AMQP but not the content
> of the message body. So, I fail to understand how parsing makes a difference
> whether I send a byte array of size 10 bytes versus a message that contains
> a body that holds a 10 megabyte byte array. I do not see why the bus limits
> me to 45mb/second. My binary reader moves about 200mb/second, reading from a
> physical storage into memory, including the parsing of the byte arrays to C#
> objects, thus the raw speed should be even a lot higher. There must be
> something else going on that limits RabbitMQ to go beyond the 45mb/second.
> The order of messages is important thus I cannot run more channels in
> parallel. Can anyone please chime in so I understand what I am overlooking
> here? Thanks a lot.
>
>
> On Tuesday, March 6, 2012 1:36:53 AM UTC+9, Simon MacMullen wrote:
>>
>> I suspect you're probably limited by the speed of the AMQP parser in
>> this case. Unfortunately the only thing you can really do to make it run
>> faster is turn on HiPE compilation or buy a a faster machine. And HiPE
>> compilation only works on Unix I'm afraid.
>>
>> If you can afford to lose ordering guarantees you could push the
>> messages through more than one connection / queue, and thus take
>> advantage of more cores.
>>
>> Cheers, Simon
>>
>> --
>> Simon MacMullen
>> RabbitMQ, VMware
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>



-- 
Aaron Westendorf
Senior Software Engineer
Agora Games
359 Broadway
Troy, NY 12180
Phone: 518.268.1000
aaron at agoragames.com
www.agoragames.com