[rabbitmq-discuss] RabbitMQ experience

[Valentino Volonghi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


On Jan 24, 2009, at 4:42 AM, Martin Sustrik wrote:

> Matthias,
>
>>>> RabbitMQ/stomp (np)     4475/360    362/33    175/17    85/8
>>>> (persistent)            3920/371    375/35    180/18    88/8
>
> The interesting thing about the test is that the overall throughput
> corresponds to message size in a very straightforward way. The network
> wasn't exhausted (unless very low-bandwidth connection was used or a
> very unefficient format for encoding the message body) so it seems  
> that
> the main bottleneck has to do with processing the body of the message.
> Excessive copying may be the cause, or maybe a slow binary->text- 
> >binary
> conversions)...


A lot of help comes from using zlib:compress() on the payload before  
sending
it. I've measured about 4400 persistent messages per second on a simple
dual processor using the erlang client and rabbitmq 1.5.0 with a  
payload of
about 4K compressed. And that value is actually the req/sec that I was  
able to
serve from a mochiweb that was using rabbitmq to log requests. Another  
probable
reason for the increased speed I'm seeing is the use of the erlang  
client that sits
in the same process and doesn't incur in tcp performance hit on the  
wire.

Actually I'm VERY satisfied performance wise with RabbitMQ and really  
the
only thing missing to make it the perfect store-and-forward solution
is page-to-disk.

My tests were done on a Core 2 Duo 2.4Ghz with 4GB of memory on
a Gigabit network (used by the python consumer that was able to get  
more than
150msg/sec, including decompression), but this value is probably also  
changed
by the shovel that runs on the source rabbitmq that batches 1000  
messages together
before sending them to a destination rabbitmq.

I should probably run more rigorous tests and see what happens again,  
it would
be very interesting for me too. With 1KB messages though I remember  
seeing
about 8000 msg/sec on the producer side (just pushed in rabbitmq, not  
queued)
with about 4000 msg/sec on the consumer side (consumed). So it could  
indeed
be an 'issue' with serialization as the body of the message grows.

On EC2 instead it's totally different story :), unfortunately the  
shared nature of
the machines makes them pretty slow in almost all cases, a medium high- 
cpu
instance barely reaches 1500 req/sec in the 'same' configuration (the  
one with
mochiweb) and only for short times but the ability to scale on demand  
could be
put to good use :).

I'm not sure if this is useful to anybody, hope so.

- --
Valentino Volonghi aka Dialtone
Now running MacOS X 10.5
Home Page: http://www.twisted.it
http://www.adroll.com

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iEYEARECAAYFAkl87WkACgkQ9Llz28widGXK4wCbBvKRu/D5C2e72f6djV0Ju4cH
EYEAoJxN0adnuwUHZexIJ0cm8NYmr4ZP
=sBVb
-----END PGP SIGNATURE-----