Hi, Srdan:<div><br></div><div>Out of curiosity, what sort of benchmarked throughput improvement did you experience connecting directly to the cluster node on which the destination queue lived?<br><br>Best regards,</div><div>
Jerry</div><div><br><div class="gmail_quote">On Wed, Oct 10, 2012 at 11:40 AM, Srdan Kvrgic <span dir="ltr"><<a href="mailto:srdan@burtcorp.com" target="_blank">srdan@burtcorp.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi all,<br>
<br>
The itch:<br>
The rabbitmq driver connects and sends message to only one of the rabbitmq instances in a cluster limiting performance at high speeds due to inter-instance message forwarding.<br>
<br>
The background:<br>
We've been using rabbitmq for a long time now and with the 2.8 release it's become cemented as an integral part of our system. It's great. The thing is, we send a lot of messages through the system - thousands a second. And we use multiple queues over a cluster of rabbitmq servers. The problem with that is that the driver will only connect to one of the rabbitmq instances and rely on the cluster to forward messages to queues hosted on the other instance.<br>
<br>
Let me put it to you this way. Assume we have a cluster (MQ1, MQ2) with two queues (fragments1, fragments2) under same exchange. The client connects to MQ1.<br>
<br>
If the client wants to send a message to fragments 1, the message will be sent by client and received and routed by MQ1.<br>
<br>
On the other hand, if the client wants to post a message on fragments2, MQ1 will have to forward the message to MQ2 which means the message will be sent from client, received by MQ1, sent by MQ1 and received and routed by MQ2.<br>
<br>
This is a neat solution but when running at high speed it becomes a bottleneck. So, we created autobahn which is a jruby transport layer wrapper that automatically connects to all MQ instances and sends the message to the server that hosts the queue. That gave us a massive boost in throughput. Yey us.<br>
<br>
So why am I on the figurative soap-box? The problem's solved, right? Why am I not on a beach under an umbrella wearing a stupid-ass hat and sipping on a Bahama-mama? Well.. The throughput improvement was so substantial that I couldn't help feeling this should be the driver's default behavior.<br>
<br>
Right?<br>
<br>
Is anything like this planned? Are we the only ones who've encountered this? It wasn't very tricky to implement really..<br>
<br>
Kind regards,<br>
Srdan<br>
<br>
<br>
--<br>
Srdan Urban Kvrgic<br>
LABS, Burt<br>
<br>
<a href="http://www.burtcorp.com" target="_blank">www.burtcorp.com</a><br>
<a href="http://twitter.com/srdan" target="_blank">http://twitter.com/srdan</a> | <a href="http://twitter.com/burtcorp" target="_blank">http://twitter.com/burtcorp</a><br>
�������������������������������������������<br>
The Analytics Platform for Online Media<br>
<br>
<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
rabbitmq-discuss mailing list<br>
<a href="mailto:rabbitmq-discuss@lists.rabbitmq.com">rabbitmq-discuss@lists.rabbitmq.com</a><br>
<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</blockquote></div><br></div>