<div dir="ltr">Yeah this is still the issue for us - how would you dynamically increase your consumers easily from say a Java process? Generally, using things like SimpleMessageListenerContainer - it starts up and consumes from a queue "foo.data" - so would we need to start up 8 separate consumer processes? Or would a single bound consumer to that queue handle this when set to 8 concurrent consumers? Or would we need to listen for a shard change event to allow it to do this? I'd also ask about the shovel process - would the shovel just consume from "foo.data" and how would that handle the multiple queues?<br>
<br>Jason</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jul 10, 2014 at 10:41 AM, Alvaro Videla <span dir="ltr"><<a href="mailto:videlalvaro@gmail.com" target="_blank">videlalvaro@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">If you have 8 queues per shard, then each shard will require 8 consumers to get the data out of those 8 queues that lay behind the shard. The plugin will take care of selecting the queue with the least amount of consumers whenever a new basic.consume arrives.</div>
<div class="HOEnZb"><div class="h5">
<div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jul 10, 2014 at 5:00 PM, Jason McIntosh <span dir="ltr"><<a href="mailto:mcintoshj@gmail.com" target="_blank">mcintoshj@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Since I just saw this crop up again on the list, thought I'd ask again on this plugin as it looks danged handy. With geographically distributed data centers, we're using RabbitMQ to replicate data and information to remote data centers. But with the WAN latency, we hit an issue where a single queue would seemingly get backlogged trying to shovel messages across the WAN. SO we used x-consistent-hashes to basically do something like this:<div>
Publish to a fanout exchange "bob"</div><div>bob is bound to "bob.multi"</div><div>bob.multi is an x-consistent-hash bound to 8 queues, "bobs.data.1" through "bobs.data.8". We then shovel "bobs.data.1" through 8 to a remote system that has a single exchange "bob" with a single queue "bobs.data". We have the shovel config over ride the routing key and exchange on the publish side so things land in one spot. If I didn't have to script this logic out I'd be more than happy to switch to using the "shard" exchange. And the goal on the remote side is to have a single consumer (java process) connected to queue "bobs.data" on the remote side, and a single publisher publishing to "bob" on the publish side. I THINK I may have misunderstood the earlier comment where we'd need 1 consumer per shard thing.</div>
<span><font color="#888888">
<div><br></div><div>Jason<br><div><br></div><div><br></div></div></font></span></div><div class="gmail_extra"><div><div><br><br><div class="gmail_quote">On Thu, Apr 10, 2014 at 3:10 PM, Alvaro Videla <span dir="ltr"><<a href="mailto:videlalvaro@gmail.com" target="_blank">videlalvaro@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<div><br>
On Thu, Apr 10, 2014 at 10:07 PM, Jason McIntosh <<a href="mailto:mcintoshj@gmail.com" target="_blank">mcintoshj@gmail.com</a>> wrote:<br>
> The one thing I didn't see in the post was how many consumers you'd need.<br>
> It sounds like you'd need at least N consumers where N is the number of<br>
> nodes in the shard? With the shoveling config, we'd need to dynamically<br>
> grow consumers to adapt to the cluster size it sounds like.<br>
><br>
<br>
</div>Yes, at the moment you would need 1 consumer per shard. So if the<br>
shard-per-node value is 4, then you'd need 4 consumers per node in the<br>
cluster.<br>
<br>
Regards,<br>
<br>
Alvaro<br>
</blockquote></div><br><br clear="all"><div><br></div></div></div><div>-- <br><div dir="ltr">Jason McIntosh<br><a href="https://github.com/jasonmcintosh/" target="_blank">https://github.com/jasonmcintosh/</a><br>
<a href="tel:573-424-7612" value="+15734247612" target="_blank">573-424-7612</a></div>
</div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Jason McIntosh<br><a href="https://github.com/jasonmcintosh/" target="_blank">https://github.com/jasonmcintosh/</a><br>573-424-7612</div>
</div>