[rabbitmq-discuss] publish/suscribe model with typed consumers

Fri Jun 10 19:03:48 BST 2011

FWIW, my "solution" (pretty basic) to the same problem is to have one
exchange/queue combo per type of message (or job). IOW I use the same queue
for all subscribers to any exchange.

Each consumer ("agent" in me-speak) subscribes to the queues it knows how to
handle, assuming I want it to...I can set up workers anywhere, but some
tasks are best handled in certain places, due to network or hardware issues.

RabbitMQ (at least with v1.7.2...I dunno about others.) does a really neat
round-robin distribution of messages when multiple consumers subscribe to
the same queue. Workers "ack" each message only when they're finished doing
the work (yes, that can create network timeout issues), so there's
near-perfect load-balancing built in. Any time I need to pass more info than
I want to try to push through RabbitMQ, I store it in S3 and pass the
filename.

HTH, but actually I hope someone else has a better idea...

-David

On Fri, Jun 10, 2011 at 5:15 AM, PADIOU Pierre-Marie (MORPHO) <
pierre-marie.padiou at morpho.com> wrote:

>  Hello,
>
>
>
> I am currently struggling with the following use case:
>
> - I am using a publish/subscribe model
>
> - I’ve got many subscribers and I only want one of them to process a
> publisher’s message
>
> - requests can be of many types (say A to Z), and each consumer knows how
> to deal with only a subset of them
>
>
>
> For example :
>
> - consumer 1 can process requests A to M
>
> - consumer 2 can process requests N to Z
>
> - consumer 3 can process requests F to Q
>
>
>
> There is actually far more types than that (say a thousand). For now, the
> only solution I see would be to create as many queues as requests types
> (queueA, queueB, etc), and make consumer listen to the appropriate queues.
>
>
>
> Thing is:
>
> - this implies that consumers create as many threads as queues there are
> listening from (more than a hundred in my case), which I would like to
> avoid.
>
> - or I could make a single thread of a consumer randomly and sequentially
> dequeue messages from all queues assigned to it?
>
>
>
> Is there a better way to handle this?
>
>
> Thanks in advance
>
>
>
> Pierre-Marie
>
>
>
>
>
> #
> " Ce courriel et les documents qui lui sont joints peuvent contenir des informations confidentielles ou ayant un caractère privé. S'ils ne vous sont pas destinés, nous vous signalons qu'il est strictement interdit de les divulguer, de les reproduire ou d'en utiliser de quelque manière que ce soit le contenu. Si ce message vous a été transmis par erreur, merci d'en informer l'expéditeur et de supprimer immédiatement de votre système informatique ce courriel ainsi que tous les documents qui y sont attachés."
> ******
> " This e-mail and any attached documents may contain confidential or proprietary information. If you are not the intended recipient, you are notified that any dissemination, copying of this e-mail and any attachments thereto or use of their contents by any means whatsoever is strictly prohibited. If you have received this e-mail in error, please advise the sender immediately and delete this e-mail and all attached documents from your computer system."
> #
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110610/fbaeec65/attachment.htm>