[rabbitmq-discuss] Scaling rabbitmq on ec2

[Roman Kournjaev]

Hi

I hope someone can help me out with some advice , cause I am struggling for 
quite a while now.

I have some ad server application running on AWS ec2. The server are 
tomcats that can scale , with the load right now I have 10 servers running. 
The overall load on the servers is something 1K rps . It produces twice as 
much messages that have to be persisted eventually.

Right now the servers send these messages to rabbitmq instances ( that are 
not connected to a cluster ) , each tomcat has an open connection to every 
rabbitmq broker and just picks one randomly to send a message.

On the other side I have a consumer that consumes all the messages from all 
the brokers.

The brokers are running on a medium instance ( thats 1 cpu and 3.5 GB ram ) 
and the queues are not persistent or mirrored. The issue is that the 
brokers get to 100 cpu at speeds reaching 700 messages/sec. 

I guess connecting the brokers to a cluster will only decrease the 
performance also i will have to configure a tcp load balancer to connect to 
the cluster from tomcats. Also creating a rabbitmq cluster is not trivial 
at all , especially if the brokers are on an auto scaling group.

So for the question : 


   1. I read some benchmarks out there and could not understand whether the 
   700m/sec is slow or not. I can use a bigger instance with more CPU , but 
   will the load grow linear ? 
   2. Do you spot anything wrong in the architecture ? Can you think of an 
   overall better approach in the message chain ?


Thanks
Roman



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20131231/91206f58/attachment.html>