[rabbitmq-discuss] Scaling rabbitmq on ec2

Mon Jan 6 08:22:05 GMT 2014

Anyone ? There must be at least someone in that group that could answer my 
question ...

On Tuesday, December 31, 2013 11:20:58 AM UTC+2, Roman Kournjaev wrote:
>
> Hi
>
> I hope someone can help me out with some advice , cause I am struggling 
> for quite a while now.
>
> I have some ad server application running on AWS ec2. The server are 
> tomcats that can scale , with the load right now I have 10 servers running. 
> The overall load on the servers is something 1K rps . It produces twice as 
> much messages that have to be persisted eventually.
>
> Right now the servers send these messages to rabbitmq instances ( that are 
> not connected to a cluster ) , each tomcat has an open connection to every 
> rabbitmq broker and just picks one randomly to send a message.
>
> On the other side I have a consumer that consumes all the messages from 
> all the brokers.
>
> The brokers are running on a medium instance ( thats 1 cpu and 3.5 GB ram 
> ) and the queues are not persistent or mirrored. The issue is that the 
> brokers get to 100 cpu at speeds reaching 700 messages/sec. 
>
> I guess connecting the brokers to a cluster will only decrease the 
> performance also i will have to configure a tcp load balancer to connect to 
> the cluster from tomcats. Also creating a rabbitmq cluster is not trivial 
> at all , especially if the brokers are on an auto scaling group.
>
> So for the question : 
>
>
>    1. I read some benchmarks out there and could not understand whether 
>    the 700m/sec is slow or not. I can use a bigger instance with more CPU , 
>    but will the load grow linear ? 
>    2. Do you spot anything wrong in the architecture ? Can you think of 
>    an overall better approach in the message chain ?
>
>
> Thanks
> Roman
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140106/b0e09984/attachment.html>