[rabbitmq-discuss] RabbitMQ on EC2 performance challenge

[Marek Majkowski]

On Thu, Aug 4, 2011 at 10:40, Jaymin Shah <sjaymin at gmail.com> wrote:
> What could be performance expectations of RabbitMQ on EC2? Would appreciate
> sharing experience here.
>
> I am trying to do some performance test of RabbitMQ on aws EC2. I have 3
> separate EC2 instance running for RabbitMQ, Publisher and consumer/worker.
>
> The scenario I have is that Publisher pushes JSON string (approx 165-200
> bytes) to exchange type direct with durable set to true and bind queue with
> durable set to true (i.e. both in persistent mode). Consumer/worker is
> running on separate box - keeps pulling messages. (Moving forward these
> messages at worker are expected to be persisted in MongoDB and Publisher
> would be replaced with Restful  service using REST easy)
>
> To keep things simple I have simulated this scenario by using Multicast
> sample code. I had split multicast code in to two separate java file namely
> “Producer” and “Worker” to run each on separate box. I have used “c1.mediam”
> EC2 with Ubuntu server v11.4 32 bit for running producer and consumer and
> “m1.large” with Ubuntu server v11.4 64 bit for RabbitMQ.

Sounds sensible.

> I am able to achieve a throughput of 3-5k messages per second i.e. keeping
> study message push rate to 5K. (This concur with
> http://www.rabbitmq.com/faq.html#performance-latency)
>
> Further, when I increase the push rate to 10-12k messages per second.
> Consumer’s ability to consume messages drops to 1-2k messages per second and
> it generates backlog (Many time it goes below 800 messages per second too).
>
> With above scenario, I have following questions and would appreciate
> thoughts/suggestion to improve throughput of consumer as well.  (NOTE: all
> the messages in my scenario are expected to similar type giving no
> opportunity to group them for setting routing therefore may need some kind
> of load-balancer approach)
>
> 1) This performance is observed with one rabbitMQ server, one exchange and
> one queue. Is anything further can be configured, fine-tuned to improvise
> throughput to more than 5k with persistent mode.

Oh, you're sending persistent messages. Can you repeat the test for the
transient messages?

Why are you using persistence? I'm not sure what's the point,
as you're running on ec2, you have no guarantees whatsoever,
ebs may die, your local storage may die.

BTW, are you using ebs or just local ec2 disk?

> 2) I do understand, clustering could be another option. However, I need to
> set cluster based on incoming load and I may not get message grouping /
> identity to define routing (since messages are expected to be just log
> description). Can I have clustering following load balancing option for
> worker/consumer?

Clustering can increase performance only in a very few cases.
You must use multiple queues, configure clients to connect
to different nodes and have a good connectivity between nodes.
Clustering is a bit like setting few separate rabbitmq servers
that are able to exchange messages between them
(and obviously they share a namespace).

> 3) I am expected to process several hundred thousand requests per second. I
> would appreciate sharing some experience and approach to achieve this.

You should think about setting few rabbitmq instances (using clustering or
as completely separate instances) and using at least one queue
on every rabbit.

Cheers,
   Marek