[rabbitmq-discuss] Proposal: Change consumers Round-Robin behaviour

Thu May 2 13:43:53 BST 2013

We often have empty queues. In fact we strive for that state.

We like the current round robin because it predictably exercises each
consumer on the queue. We monitor the processing history (region,
availability zone, instance, service, etc) of each message through the
messaging pipeline. The pipelines may involve multiple clusters and
federated links. Typically we process multiple replicas of the same
message through alternate regional pipelines that ultimately 'fan-in'.
Because of the round robin predictability we can very quickly find 'holes'
and slowdowns by comparing the histories of replicas at resolution time -
then take appropriate action, e.g. Terminate low-performing instances so
auto-scaling will add/cluster new ones.

Additionally we use certain queues to hold mutex tokens. For any one
token, multiple processes subscribe w ack and one of them gets it and does
its work, maybe sleeps for a while, and then nacks - the next process gets
the token and so on. Predictable, exercises each process in turn, and if a
process fails the next one picks up immediately.

Michael

On 5/2/13 7:53 AM, "Simon MacMullen" <simon at rabbitmq.com> wrote:

>On 02/05/13 12:27, Jesper Louis Andersen wrote:
>> Hi,
>
>Hi.
>
><snip>
>
>> When this happens, we pick a random consumer in the queue and move him
>> to the front. Over time, this "shuffles" the queue into a random order.
>> It is also not going to cost anything on the critical path since we only
>> do it when we have an empty queue and excess workers.
>
>Bear in mind that the "empty queue and excess workers" case *is* on the
>critical path for straight-through messaging (i.e. where the queue is on
>average empty and we are streaming messages through it as fast as
>possible).
>
>> And we are going
>> to do very little work unless the queue has a behaviour where it empties
>> often in which case you get full random distribution on the consumers
>> with this scheme.
>
>Again, this is a common case for some users.
>
>Also: quite a lot of people *want* round-robin behaviour here.
>
>> The background for the proposal is that Round-robin distribution of
>> messages often tend to bad behaviour over time. By adding a bit of
>> randomness to the process, we automatically alleviate a number of
>> determinism-problems and get better distribution of messages over
>> consumers. One could also imagine different distribution schemes, but
>> those will be more expensive in practice compared to this proposal,
>> which should only have a cost when the queue is not under heavy load.
>>
>> * Did I miss anything?
>> * Is this a good or bad idea? And why?
>> * Do we break any rules w.r.t. AMQP by implementing this?
>> * Is priority on the queue going to be harder to implement? (I don't
>> think so, but...)
>
>So I am not (yet) convinced. This would not violate the spec or make
>priorities harder to implement but I'm not convinced that the majority
>of users would see it as a step forward.
>
>Does anybody who currently depends on round-robin want to speak up?
>
>Cheers, Simon
>
>-- 
>Simon MacMullen
>RabbitMQ, VMware
>_______________________________________________
>rabbitmq-discuss mailing list
>rabbitmq-discuss at lists.rabbitmq.com
>https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss