[rabbitmq-discuss] Scaling rabbitmq on ec2
kournjaev at gmail.com
Tue Dec 31 09:20:58 GMT 2013
I hope someone can help me out with some advice , cause I am struggling for
quite a while now.
I have some ad server application running on AWS ec2. The server are
tomcats that can scale , with the load right now I have 10 servers running.
The overall load on the servers is something 1K rps . It produces twice as
much messages that have to be persisted eventually.
Right now the servers send these messages to rabbitmq instances ( that are
not connected to a cluster ) , each tomcat has an open connection to every
rabbitmq broker and just picks one randomly to send a message.
On the other side I have a consumer that consumes all the messages from all
The brokers are running on a medium instance ( thats 1 cpu and 3.5 GB ram )
and the queues are not persistent or mirrored. The issue is that the
brokers get to 100 cpu at speeds reaching 700 messages/sec.
I guess connecting the brokers to a cluster will only decrease the
performance also i will have to configure a tcp load balancer to connect to
the cluster from tomcats. Also creating a rabbitmq cluster is not trivial
at all , especially if the brokers are on an auto scaling group.
So for the question :
1. I read some benchmarks out there and could not understand whether the
700m/sec is slow or not. I can use a bigger instance with more CPU , but
will the load grow linear ?
2. Do you spot anything wrong in the architecture ? Can you think of an
overall better approach in the message chain ?
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the rabbitmq-discuss