[rabbitmq-discuss] Driver connecting to multiple instances in a cluster

Wed Oct 10 21:54:10 BST 2012

Hi, Srdan:

Out of curiosity, what sort of benchmarked throughput improvement did you
experience connecting directly to the cluster node on which the destination
queue lived?

Best regards,
Jerry

On Wed, Oct 10, 2012 at 11:40 AM, Srdan Kvrgic <srdan at burtcorp.com> wrote:

> Hi all,
>
> The itch:
> The rabbitmq driver connects and sends message to only one of the rabbitmq
> instances in a cluster limiting performance at high speeds due to
> inter-instance message forwarding.
>
> The background:
> We've been using rabbitmq for a long time now and with the 2.8 release
> it's become cemented as an integral part of our system. It's great. The
> thing is, we send a lot of messages through the system - thousands a
> second. And we use multiple queues over a cluster of rabbitmq servers. The
> problem with that is that the driver will only connect to one of the
> rabbitmq instances and rely on the cluster to forward messages to queues
> hosted on the other instance.
>
> Let me put it to you this way. Assume we have a cluster (MQ1, MQ2) with
> two queues (fragments1, fragments2) under same exchange. The client
> connects to MQ1.
>
> If the client wants to send a message to fragments 1, the message will be
> sent by client and received and routed by MQ1.
>
> On the other hand, if the client wants to post a message on fragments2,
> MQ1 will have to forward the message to MQ2 which means the message will be
> sent from client, received by MQ1, sent by MQ1 and received and routed by
> MQ2.
>
> This is a neat solution but when running at high speed it becomes a
> bottleneck. So, we created autobahn which is a jruby transport layer
> wrapper that automatically connects to all MQ instances and sends the
> message to the server that hosts the queue. That gave us a massive boost in
> throughput. Yey us.
>
> So why am I on the figurative soap-box? The problem's solved, right? Why
> am I not on a beach under an umbrella wearing a stupid-ass hat and sipping
> on a Bahama-mama? Well.. The throughput improvement was so substantial that
> I couldn't help feeling this should be the driver's default behavior.
>
> Right?
>
> Is anything like this planned? Are we the only ones who've encountered
> this? It wasn't very tricky to implement really..
>
> Kind regards,
> Srdan
>
>
> --
> Srdan Urban Kvrgic
> LABS, Burt
>
> www.burtcorp.com
> http://twitter.com/srdan | http://twitter.com/burtcorp
> –––––––––––––––––––––––––––––––––––––––––––
> The Analytics Platform for Online Media
>
>
>
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20121010/e8dc1606/attachment.htm>