[rabbitmq-discuss] Introducing the RabbitMQ Sharding Plugin

Thu Jul 10 16:00:42 BST 2014

Since I just saw this crop up again on the list, thought I'd ask again on
this plugin as it looks danged handy.  With geographically distributed data
centers, we're using RabbitMQ to replicate data and information to remote
data centers.  But with the WAN latency, we hit an issue where a single
queue would seemingly get backlogged trying to shovel messages across the
WAN.  SO we used x-consistent-hashes to basically do something like this:
Publish to a fanout exchange "bob"
bob is bound to "bob.multi"
bob.multi is an x-consistent-hash bound to 8 queues, "bobs.data.1" through
"bobs.data.8".  We then shovel "bobs.data.1" through 8 to a remote system
that has a single exchange "bob" with a single queue "bobs.data".  We have
the shovel config over ride the routing key and exchange on the publish
side so things land in one spot.  If I didn't have to script this logic out
I'd be more than happy to switch to using the "shard" exchange.  And the
goal on the remote side is to have a single consumer (java process)
connected to queue "bobs.data" on the remote side, and a single publisher
publishing to "bob" on the publish side.  I THINK I may have misunderstood
the earlier comment where we'd need 1 consumer per shard thing.

Jason

On Thu, Apr 10, 2014 at 3:10 PM, Alvaro Videla <videlalvaro at gmail.com>
wrote:

> Hi,
>
> On Thu, Apr 10, 2014 at 10:07 PM, Jason McIntosh <mcintoshj at gmail.com>
> wrote:
> >   The one thing I didn't see in the post was how many consumers you'd
> need.
> > It sounds like you'd need at least N consumers where N is the number of
> > nodes in the shard?  With the shoveling config, we'd need to dynamically
> > grow consumers to adapt to the cluster size it sounds like.
> >
>
> Yes, at the moment you would need 1 consumer per shard. So if the
> shard-per-node value is 4, then you'd need 4 consumers per node in the
> cluster.
>
> Regards,
>
> Alvaro
>

-- 
Jason McIntosh
https://github.com/jasonmcintosh/
573-424-7612
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140710/5c8eb401/attachment.html>