[rabbitmq-discuss] RabbitMQ on EC2 performance challenge

[Jaymin Shah]

What could be performance expectations of RabbitMQ on EC2? Would appreciate
sharing experience here.


I am trying to do some performance test of RabbitMQ on aws EC2. I have 3
separate EC2 instance running for RabbitMQ, Publisher and consumer/worker.


The scenario I have is that Publisher pushes JSON string (approx 165-200
bytes) to exchange type direct with durable set to true and bind queue with
durable set to true (i.e. both in persistent mode). Consumer/worker is
running on separate box - keeps pulling messages. (Moving forward these
messages at worker are expected to be persisted in MongoDB and Publisher
would be replaced with Restful  service using REST easy)


To keep things simple I have simulated this scenario by using Multicast
sample code. I had split multicast code in to two separate java file namely
“Producer” and “Worker” to run each on separate box. I have used “c1.mediam”
EC2 with Ubuntu server v11.4 32 bit for running producer and consumer and
“m1.large” with Ubuntu server v11.4 64 bit for RabbitMQ.


I am able to achieve a throughput of 3-5k messages per second i.e. keeping
study message push rate to 5K. (This concur
withhttp://www.rabbitmq.com/faq.html#performance-latency
)


Further, when I increase the push rate to 10-12k messages per second.
Consumer’s ability to consume messages drops to 1-2k messages per second and
it generates backlog (Many time it goes below 800 messages per second too).


With above scenario, I have following questions and would appreciate
thoughts/suggestion to improve throughput of consumer as well.  (NOTE: all
the messages in my scenario are expected to similar type giving no
opportunity to group them for setting routing therefore may need some kind
of load-balancer approach)


1) This performance is observed with one rabbitMQ server, one exchange and
one queue. Is anything further can be configured, fine-tuned to improvise
throughput to more than 5k with persistent mode.


2) I do understand, clustering could be another option. However, I need to
set cluster based on incoming load and I may not get message grouping /
identity to define routing (since messages are expected to be just log
description). Can I have clustering following load balancing option for
worker/consumer?


3) I am expected to process several hundred thousand requests per second. I
would appreciate sharing some experience and approach to achieve this.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110804/9ae7c51f/attachment.htm>