[rabbitmq-discuss] Driver connecting to multiple instances in a cluster

Wed Oct 10 19:40:08 BST 2012

Hi all,

The itch:
The rabbitmq driver connects and sends message to only one of the rabbitmq instances in a cluster limiting performance at high speeds due to inter-instance message forwarding. 

The background:
We've been using rabbitmq for a long time now and with the 2.8 release it's become cemented as an integral part of our system. It's great. The thing is, we send a lot of messages through the system - thousands a second. And we use multiple queues over a cluster of rabbitmq servers. The problem with that is that the driver will only connect to one of the rabbitmq instances and rely on the cluster to forward messages to queues hosted on the other instance. 

Let me put it to you this way. Assume we have a cluster (MQ1, MQ2) with two queues (fragments1, fragments2) under same exchange. The client connects to MQ1.

If the client wants to send a message to fragments 1, the message will be sent by client and received and routed by MQ1.

On the other hand, if the client wants to post a message on fragments2, MQ1 will have to forward the message to MQ2 which means the message will be sent from client, received by MQ1, sent by MQ1 and received and routed by MQ2.

This is a neat solution but when running at high speed it becomes a bottleneck. So, we created autobahn which is a jruby transport layer wrapper that automatically connects to all MQ instances and sends the message to the server that hosts the queue. That gave us a massive boost in throughput. Yey us.

So why am I on the figurative soap-box? The problem's solved, right? Why am I not on a beach under an umbrella wearing a stupid-ass hat and sipping on a Bahama-mama? Well.. The throughput improvement was so substantial that I couldn't help feeling this should be the driver's default behavior.

Right?

Is anything like this planned? Are we the only ones who've encountered this? It wasn't very tricky to implement really..

Kind regards,
Srdan

--
Srdan Urban Kvrgic
LABS, Burt

www.burtcorp.com
http://twitter.com/srdan | http://twitter.com/burtcorp
–––––––––––––––––––––––––––––––––––––––––––
The Analytics Platform for Online Media