[rabbitmq-discuss] Priorities support in RabbitMQ

[Holger Hoffstätte]

Michal Chruszcz wrote:
> control order of message processing. Say I have distributed  
> asynchronous data processing service, which uses RabbitMQ for queueing  
> tasks, and would like to prioritize them depending on  data volume  
> (tiny packages are processed first). Priorities seemed to be perfect  
> solution, but unfortunately later I realized it's not supported yet.

It is a common misconception that priorities alone can be used to do this
"properly" - it will only work well if you are accidentally lucky with
your ingress patterns or really don't care about a total lack of
predictable processing latency for larger messages. Since Rabbit does not
page to disk yet, preferring small messages might even make it run out of
memory *sooner* as more and more large messages are held back. Do not
underestimate the effectiveness of randomized workloads. :)
Explicitly moding this with multiple queues and multiple consumers is much
more likely to result in better (both predictable and controllable)
scalability and a well-balanced throughput for both small and large
messages. Spin up multiple consumers for small messages (keeping those
cores busy and even hopefully reducing per-message latency), have one
low-priority consumer chew on larger tasks in the background or even on
another machine. It also gives you the option of having different
durability/persistence options for different priorities.

-h