[rabbitmq-discuss] Driver connecting to multiple instances in a cluster

[Pablo Molnar]

That seems great, you are saving one hop and load balancing the connections
right?
You have this autobahn product open sourced somewhere? Would be if we
could experiment a little with your work!

Cheers,
Pablo

On Wed, Oct 10, 2012 at 3:40 PM, Srdan Kvrgic <srdan at burtcorp.com> wrote:

> Hi all,
>
> The itch:
> The rabbitmq driver connects and sends message to only one of the rabbitmq
> instances in a cluster limiting performance at high speeds due to
> inter-instance message forwarding.
>
> The background:
> We've been using rabbitmq for a long time now and with the 2.8 release
> it's become cemented as an integral part of our system. It's great. The
> thing is, we send a lot of messages through the system - thousands a
> second. And we use multiple queues over a cluster of rabbitmq servers. The
> problem with that is that the driver will only connect to one of the
> rabbitmq instances and rely on the cluster to forward messages to queues
> hosted on the other instance.
>
> Let me put it to you this way. Assume we have a cluster (MQ1, MQ2) with
> two queues (fragments1, fragments2) under same exchange. The client
> connects to MQ1.
>
> If the client wants to send a message to fragments 1, the message will be
> sent by client and received and routed by MQ1.
>
> On the other hand, if the client wants to post a message on fragments2,
> MQ1 will have to forward the message to MQ2 which means the message will be
> sent from client, received by MQ1, sent by MQ1 and received and routed by
> MQ2.
>
> This is a neat solution but when running at high speed it becomes a
> bottleneck. So, we created autobahn which is a jruby transport layer
> wrapper that automatically connects to all MQ instances and sends the
> message to the server that hosts the queue. That gave us a massive boost in
> throughput. Yey us.
>
> So why am I on the figurative soap-box? The problem's solved, right? Why
> am I not on a beach under an umbrella wearing a stupid-ass hat and sipping
> on a Bahama-mama? Well.. The throughput improvement was so substantial that
> I couldn't help feeling this should be the driver's default behavior.
>
> Right?
>
> Is anything like this planned? Are we the only ones who've encountered
> this? It wasn't very tricky to implement really..
>
> Kind regards,
> Srdan
>
>
> --
> Srdan Urban Kvrgic
> LABS, Burt
>
> www.burtcorp.com
> http://twitter.com/srdan | http://twitter.com/burtcorp
> –––––––––––––––––––––––––––––––––––––––––––
> The Analytics Platform for Online Media
>
>
>
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20121011/761cc6eb/attachment.htm>