[rabbitmq-discuss] Introducing the RabbitMQ Sharding Plugin

[Alvaro Videla]

If you have 8 queues per shard, then each shard will require 8 consumers to
get the data out of those 8 queues that lay behind the shard. The plugin
will take care of selecting the queue with the least amount of consumers
whenever a new basic.consume arrives.


On Thu, Jul 10, 2014 at 5:00 PM, Jason McIntosh <mcintoshj at gmail.com> wrote:

> Since I just saw this crop up again on the list, thought I'd ask again on
> this plugin as it looks danged handy.  With geographically distributed data
> centers, we're using RabbitMQ to replicate data and information to remote
> data centers.  But with the WAN latency, we hit an issue where a single
> queue would seemingly get backlogged trying to shovel messages across the
> WAN.  SO we used x-consistent-hashes to basically do something like this:
> Publish to a fanout exchange "bob"
> bob is bound to "bob.multi"
> bob.multi is an x-consistent-hash bound to 8 queues, "bobs.data.1" through
> "bobs.data.8".  We then shovel "bobs.data.1" through 8 to a remote system
> that has a single exchange "bob" with a single queue "bobs.data".  We have
> the shovel config over ride the routing key and exchange on the publish
> side so things land in one spot.  If I didn't have to script this logic out
> I'd be more than happy to switch to using the "shard" exchange.  And the
> goal on the remote side is to have a single consumer (java process)
> connected to queue "bobs.data" on the remote side, and a single publisher
> publishing to "bob" on the publish side.  I THINK I may have misunderstood
> the earlier comment where we'd need 1 consumer per shard thing.
>
> Jason
>
>
>
>
> On Thu, Apr 10, 2014 at 3:10 PM, Alvaro Videla <videlalvaro at gmail.com>
> wrote:
>
>> Hi,
>>
>> On Thu, Apr 10, 2014 at 10:07 PM, Jason McIntosh <mcintoshj at gmail.com>
>> wrote:
>> >   The one thing I didn't see in the post was how many consumers you'd
>> need.
>> > It sounds like you'd need at least N consumers where N is the number of
>> > nodes in the shard?  With the shoveling config, we'd need to dynamically
>> > grow consumers to adapt to the cluster size it sounds like.
>> >
>>
>> Yes, at the moment you would need 1 consumer per shard. So if the
>> shard-per-node value is 4, then you'd need 4 consumers per node in the
>> cluster.
>>
>> Regards,
>>
>> Alvaro
>>
>
>
>
> --
> Jason McIntosh
> https://github.com/jasonmcintosh/
> 573-424-7612
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140710/33b0dd2d/attachment.html>