[rabbitmq-discuss] Problem with multiple consumers on multiple servers !

Sun Mar 31 13:48:32 BST 2013

Hi,

On 30 Mar 2013, at 14:28, tluck wrote:
> Thanks for your quick answer. Don't you think that if I increase prefetch
> count then it may cause redundancy of  the same messages to more then one
> consumer ? I mean the same message/job can be consumed by two or more
> consumers at a time (I read it somewhere and may be I'm wrong).

That's not correct. A dequeued message will be delivered to exactly one consumer. 

> Suppose, if
> I increase prefetch count to 10 then broker won't send messages to any of
> the consumers (which are consuming messages from current queue)  until  all
> 10 messages are consumed ? right ?

I'm not quite sure what you're asking. RabbitMQ dispatches messages once they
enter the queue. It doesn't look at the number of unacknowledged messages for a
consumer, but simply dispatches every n-th message to the n-th consumer. If the
prefetch count is 1, then 1 message will be delivered to the consumer and any 
subsequent messages will be dispatched to the next consumer, and so on.
Only once a consumer starts to send acknowledgements for some (or all) delivered
messages will it become eligible to receive more.

So if you set the prefetch count to 10, you'll be able to receive a maximum of 10
messages per consumer before the broker stops delivering to that consumer until some
acks are received.

If the number of messages in the queue is the same as the prefetch count
of all consumers then one consumer will get all the messages in the
queue (until more messages arrive in the queue). This means that other
consumers will be idle.

> If yes, then how can we tackle the
> problem which can be caused by all this. If one/two consumer(s) take too
> long time to consume their messages then others will go idle (as they have
> to wait others to finish their job).

>> Can anyone please help me to find out, why only one job is being consumed at a time ?

These two points are orthogonal, but really I think the problem boils down to these
comments you made previously:

"Now I've setup multiple consumers and each consumer is listening to a single
queue"

... and ...

"The problem which I've
noticed is that only one message/job from each queue is being consumed at a
time.

So I'm not quite sure what you're trying to achieve here. If you have one consumer
on each queue, then this next comment:

"The problem which I've noticed is that only one message/job from each queue is
being consumed at a time."

That's how a queue works right, it's giving you FIFO semantics. I'm not really sure
what else you're expecting to happen.

> 
> If I have 20K messages and 10 consumers on a single queue then what would be
> an ideal count for prefetch_count ?
> 

It really depends on what you want to do. The prefetch count can be used either as
an optimisation (to make sure that you've received some messages locally that you
can work on as soon as you're ready) or to create a kind of fair dispatch (based on
preventing workers from having messages dispatched to them until they've acknowledged
the previous one(s) already).

Having 10 consumers on a single queue then they'll receive messages in round robin
fashion (unless you use the prefetch count to override that). If you don't bother
with the prefetch count then the broker will continue sending messages (regardless of
acks) to the consumers in this fashion and once it gets an ACK for each message it
will remove it (fully) from the queue.

"The problem which I've noticed is that only one message/job from each queue is being
consumed at a time" - what does this even mean? Are you saying you want each consumer
on a single queue to consume more than one message before messages get delivered to the
next? That's what prefetch count > 1 will do for you and is why I suggested it... Once
n messages (where n=prefetchCount) have been delivered to a consumer, messages will start
getting delivered to other consumers until the original consumer starts sending acks (or
a single ACK using the multiple flag) at which point it will be included in the round
robin dispatch again.

Are you saying you want something else to happen? 

Tim