[rabbitmq-discuss] Millions of Queues

Wed Feb 18 10:06:23 GMT 2009

Martin, Ryan,

On Wed, Feb 18, 2009 at 6:03 AM, Martin Sustrik <sustrik at imatix.com> wrote:
> Ryan,
>
>>>   To be more clear
>>> about our use case, we want to create a chat room system, where each
>>> user is a member of K rooms.  The way we're thinking of modeling this is
>>> that each room is represented as a (direct) routing key, and each user
>>> has an individual queue which is then bound to K routing keys.  There's
>>> not really any pattern between users and rooms, so if we partitioned the
>>> queues (and thus users) among clusters, any message sent to an
>>> individual room would have to get sent to every cluster on the (small)
>>> chance that a listening user was in that cluster.  It'd work for a
>>> while, but it can't possibly scale indefinitely.  Is there perhaps a
>>> better way to structure this application?
>>>
>>
>> Answering my own question, another way to structure this would be to
>> partition on routing key, such that when sending to or binding to a
>> particular routing key, the client connects to the same cluster.

As Matthias said, this is a good pattern.  You may want to use hashing
to provide a uniform distribution across nodes.

If you look at say Twitter, a solution would work in a similar sort of
way.  Each user has a queue representing the consolidated feed of
people whose tweets they follow.  Each 'follow' relationship is a
binding from that queue to the (direct) exchange.  The keys can just
be user names.  So, each queue will be bound to the exchange N times,
if the queue owner is following N people.

>> The
>> downside of this is that, at the extreme, each chat room participant
>> will have to maintain as many connections as rooms they are in (since at
>> the extreme each chat room would be hosted in its own cluster).  But
>> assuming K (the number of chat rooms a user can be in) is finite, then
>> it's not as bad as replicating every message across the entire cluster.

On option (probably not the best but worth mulling over) is to use the
twitter pattern that I described.  This would mean each chat room
participant has exactly one connection to the cluster.  The price you
pay is that the chat room participant's queue contains messages from
each chat room that they are in.  You would need to disaggregate the
feed at the point of consumption.

> This is a classic example of application that requires pub/sub
> federation. I don't believe it's implemented in RabbitMQ yet, but it
> should be doable.

Martin, you could achieve this using the relay queue in Ben's shovel,
rather than a true federation, right?  This is because the model you
describe is actually  a cascade, with messages propagating in one
direction through a ramified 'branching tree' structure, with view
consistency at every point.  If the branches recombined, then we would
want a true federation.  But they don't afaict.

Although, I am not convinced a federated solution is needed at all.
The cluster should suffice, no?

alexis