[rabbitmq-discuss] Implementing Prioritised Queues

[James Dear]

> Then the feature would only work for qos channels. Such non-orthogonality is usually a bad idea.

Not too sure about this, but I can't see any reason why you *would*
want prioritization between queues and *wouldn't* want rate limiting
(QoS). With no rate limiting you would just suck up all of both queues
anyway(?), rendering prioritization irrelevant.

> It would probably be easier to implement priority queues instead.

Also I don't see prioritization *between queues* as solving the same
problem as prioritizing *within a single queue*. The former allows
recipient-defined priorities whereas the latter allows sender-defined
priority. The global 1-10 priority feature of AMQP looks kind of
deficient in this respect. Both Alice and Bob should be able to
prioritize their own messages as higher than their colleagues',
allowing them to share their compute resources while not surrendering
burst capacity and not suffering tragedy-of-commons. This isn't
possible with 'global' single-queue priorities, but is easily achieved
with 'subjective' prioritization between queues.

> And yes, that's quite difficult too.

I imagine that prioritization within a single queue is difficult to
implement efficiently because you are (very roughly speaking) doing a
kind of sort operation (or at least some kind of compare) on
potentially millions of items. By prioritizing between queues you are
only comparing between a handful of different queues (in this case
only two).

It's admittedly a bit of a hack, but you could (?) implement this
through the limiter by way of a prefetch *per queue*, similar in style
to Doug's client-side solution. If your low priority queue has a
prefetch of 1 and your high priority queue has a prefetch of 2 (or
higher), then low priority messages will be blocked until all the high
priority messages have gone through.

James

On 14 October 2011 11:42, Matthias Radestock <matthias at rabbitmq.com> wrote:
> James,
>
> On 14/10/11 11:09, James Dear wrote:
>>
>> Oh I see :(
>>
>> How does QoS manage to load-balance multiple queues within a channel?
>> I thought the queue consulted the limiter before delivering a new
>> message
>
> indeed they do, but only when qos is enabled.
>
>> ...in that case could one add a little bit of logic to the
>> limiter to prefer one queue over another?
>
> Then the feature would only work for qos channels. Such non-orthogonality is
> usually a bad idea.
>
> And it's not clear how this would work even in the restricted qos-only case.
> For one, the limiter has no idea whether a queue contains more messages. I
> can think of a few other complications too.
>
> It would probably be easier to implement priority queues instead. And yes,
> that's quite difficult too.
>
>
> Matthias.
>