<div dir="ltr"><div>Hello RabbitMQ Gurus,</div><div><br></div><div>If you don't mind, I'd love some help trying to arrive at the right design for a user messaging system I am working on. </div><div><br></div><div>
High-Level Requirements</div><div>=======================</div><div>- users can send messages to another user, several users, a group, or everyone</div><div> - users and groups are managed in a proprietary user management system</div>
<div>- no chat rooms-- each user just has a message feed or "timeline" of incoming & outgoing messages</div><div>- message history must be preserved (in a database)</div><div>- if a user is offline, the message should be forwarded to the user's email</div>
<div>- 5000 concurrent users sending a few messages every few minutes (20,000 users total in system).</div><div><br></div><div>Environment</div><div>===========</div><div>The primary interface is a web application (or maybe a few web applications), but future iterations may introduce standalone clients. RabbitMQ is 3.x and likely to be clustered or replicated, and running on same machine(s) as web application servers.</div>
<div><br></div><div>General Approach</div><div>================</div><div>In order to store every message in a database (for historical purposes), all messages are initially sent to a direct exchange with a worker queue. Consumers of the worker queue (let's call them messaging service workers) receive the message and store it to the database. Then they distribute the message to the intended recipients (this is the part I think I need most help with).</div>
<div><br></div><div>Message Distribution Option 1: Direct Exchange w/ Queue Per User</div><div>=================================================================</div><div>Message is sent to a direct exchange with username as the routing key. For every logged in user, the webapp declares a queue and binds with the username.</div>
<div><br></div><div>Issue 1: How to support groups? The messaging service worker could resolve the group to a list of users and post the message multiple times (once for each user/routing key)-- but this would not scale very well. Or we could create individual queues for every group too, but this increases complexity in the web app (and later in our other clients) since they need to determine what groups they need to listen for and who to distribute to.</div>
<div><br></div><div>Issue 2: Is 5000 queues a bit much? Especially when they are all ultimately going to the same consumer (in the webapp case)? This isn't the only use of RabbitMQ, so there will be other queues/exchanges/bindings as well...</div>
<div><br></div><div>Message Distribution Option 2: Topic Exchange w/ Single Queue per Webapp</div><div>========================================================================</div><div>Message is sent to topic exchange with routing key containing ALL addressed users (i.e., "|usera|userb|userc|"). Webapp declares single queue and a separate binding to the exchange for each user (i.e., "#|usera|#", "#|userb|#", etc.). This is similar to Option 1, but with the benefit that each message only has to be sent once.</div>
<div><br></div><div>Issue 1: Routing key can get huge (20000 names if addressed to "everyone" group).</div><div>Issue 2: It just seems like the wrong use of topic exchange.</div><div><br></div><div>Message Distribution Option 3: Fanout Exchange w/ Single Queue per Webapp</div>
<div>=========================================================================</div><div>Message is sent to fanout exchange. Webapp listens on fanout exchange and simply discards all messages for users that aren't logged in.</div>
<div><br></div><div>Issue 1: Doesn't scale well (or even if it does, there is lots of waste filtering on the client side).</div><div>Issue 2: Not at all secure (although, for now, we won't try to solve security).</div>
<div><br></div><div>Message Distribution Option 4: Direct Exchange w/ Single Queue per Webapp</div><div>========================================================================</div><div>Webapp must notify messaging service worker as users log in and out-- and of the single queue on which it wants messages. When a message needs to be distributed, messaging service worker checks to see which users are logged in where. It sends message to the queue of each webapp that has logged in users that the message is addressed to (message itself contains "to:" list so webapp can send to right user UI).</div>
<div><br></div><div>Issue 1: Messaging Service Worker now has a lot more state and processing messages is a lot heavier. Need to make sure it stays in synch with webapp.</div><div><br></div><div>Issue 2: Messaging Service Worker is now doing the routing-- which just seems wrong since that is what RabbitMQ is so good at!</div>
<div><br></div><div>Message Distribution Option 5: Fanout Exchanges for Groups</div><div>==========================================================</div><div>Same as option 1 (queue per user), but also create a fanout exchange for every group. The individual user queues need to be bound to the fanout exchange corresponding to each group they belong to.</div>
<div><br></div><div>Issue 1: Could result in a lot of exchanges (not sure how many groups there are).</div><div><br></div><div>Issue 2: Still seems like a lot of queues too (5000).</div><div><br></div><div>Message Distribution Option 6: Custom Exchange</div>
<div>==============================================</div><div>Create our own exchange that allows consumers to bind with usernames. When a message is posted to the exchange, it knows how to read the "to:" list from it, can even resolve groups, and then sends to the correct queues based on the bindings.</div>
<div><br></div><div>Issue 1: We don't have any Erlang experience.</div><div><br></div><div>Issue 2: Resolving groups in an Erlang custom exchange could be tricky (not sure about the APIs to our User Management).</div>
<div><br></div><div>Something Else</div><div>==============</div><div>I'm more than open to other ideas too!</div><div><br></div><div>BONUS QUESTION</div><div>==============</div><div>Whatever the correct approach is, how do we determine if a user is "offline" and we need to forward the message to their email? If we are using queue-per-user, then we can set the mandatory flag, I guess? If we are using another approach, what then?</div>
<div><br></div><div>Thanks to anyone who can help point me in the right direction (or steer me away from the wrong direction)... I'm not familiar enough yet with RabbitMQ performance characteristics to know what is a bad idea, and not familiar enough yet with its many features to know what is a good idea... ;-)</div>
<div><br></div><div>Thanks again,</div><div>Chris</div><div><br></div></div>