[rabbitmq-discuss] How to have multiple consumers for a Queue in Ruby

Fri Feb 25 04:55:03 GMT 2011

Thanks for the ideas. I'll probably for now go with breaking down the logic
to consumers which need to be run once and consumers which could be many.
Threading inside the same consumer would be too troublesome and might not
get the same results.

We are using Ruby 1.9 but the GIL still exists.

-Rajat

2011/2/23 Jakub Šťastný <stastny at 101ideas.cz>

> Rajat,
>
> about the second example, why more instances should be a problem? Having
> more subscribers in the same process doesn't have much sense, in my opinion,
> since the reactor pattern doesn't block -> you don't have to wait -> you
> don't need to use threads for parallelisation. Threads can be an advantage,
> but I don't think it'll be a big difference. Plus Ruby has GIL which means
> threads are sort of useless anyway, well, especially if you use Ruby 1.8
> which still has green threads. I'd say if you want to go this route, use
> Kernel#fork, but then you have to be careful about consistency of state.
> Breaking logic down sounds like a much better & cleaner way to go in my
> opinion.
>
> Jakub
>
> http://www.flickr.com/photos/jakub-stastny
> http://twitter.com/botanicus
>
>
>
> 2011/2/23 Rajat Vig <rajat.vig at gmail.com>
>
> Thanks for the suggestions. I'd the same thing in mind about being able to
>> start multiple of those processes.
>>
>> There are 2 different kinds of consumers
>> 1. Get the message and write it to a database or send it out to someplace
>> else.
>> Parallelizing this would be easy. Controlling would be more of a problem
>> here but manageable.
>>
>> 2. There's one large consumer which accepts different kinds of messages.
>> Like start/stop/abandon a game (which are always singular) and messages
>> for the game itself which are many.
>> I don't want to run this one many times since it might just run into
>> conflicts (though I can't see many currently - it does impose a rigid
>> discipline in the code).
>> The alternative that I'm thinking here is to internally re-route by
>> (generated) random keys and have multiple subscribers in the same
>> process. Would that work fine?
>> It not, I think breaking it further to a consumer for all admin messages
>> and multiple consumers for the game plays would be the way forward but it'll
>> be a significant refactor.
>>
>> -Rajat
>>
>> 2011/2/22 Jakub Šťastný <stastny at 101ideas.cz>
>>
>> Hi Rajat, could you maybe provide some more info? In general I'd agree
>>> with Jim, although it depends, there are probably other solutions as well
>>> (like more queues -> more granular design of the app with more consumers).
>>>
>>> Jakub
>>>
>>> http://www.flickr.com/photos/jakub-stastny
>>> http://twitter.com/botanicus
>>>
>>>
>>>
>>> On 22 February 2011 17:43, Rajat Vig <rajat.vig at gmail.com> wrote:
>>>
>>>> Hi
>>>>
>>>> We've written an application using the amqp gem with RabbitMQ.
>>>> Currently the bottleneck we've run into is that if there are too many
>>>> message incoming then processing the queue takes long.
>>>> After reading some messages on the list, I think allowing multiple
>>>> consumers is the way forward.
>>>>
>>>> How do I do that? And are there any pitfalls/gotchas?
>>>>
>>>> -Rajat
>>>>
>>>> _______________________________________________
>>>> rabbitmq-discuss mailing list
>>>> rabbitmq-discuss at lists.rabbitmq.com
>>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110225/d7f0b9ea/attachment.htm>