[rabbitmq-discuss] Reordering messages

Fri Jan 21 15:37:03 GMT 2011

Hi Dimitri,

Dimitri del Marmol <tontondimsum at gmail.com> writes:
>> Put the two categories of messages onto two separate queues, and write
>> your application to preferentially consume from the higher priority
>> queue.
>
> I actually toyed with the idea but I have trouble imagining what the
> listener(s) would look like: should one listener subscribe to both
> queues? If so, it will have to know when to ignore the low priority
> callback (because the high priority queue is not empty)

The details of how you achieve this depend on which client library you
are using, and how your application is organized.  So with more
information someone might be able to give more specific advice.  But
here is the general idea.

As you suggest, your listener should consume from both queues.  You
should consume with no-ack=False (i.e. explicit acks).

In your code, you'll need a data structure to hold pending high-priority
messages (those that have arrived, but not yet been processed), and
another to hold pending low-prority messages.  The basic logic is then
as follows:

When a message arrives (from either queue):
  If a message is already being processed:
    Add the new message to the approprite pending messages data structure
  Else:
    Start processing the message

When processing of a message completes:
  Ack the processed message
  If there are any pending high-priority messages
    Remove one and start processing it
  Else, if there are any pending low-priority messages
    Remove one and start processing it
  Else:
    Nothing to do, so wait until another message arrives

So we obey the priorities because we always look for the high-priority
messages before looking in the low-priority slot.  If there are always
high-priority messages coming in, then we never look at the low-priority
messages, though we may have many of them sitting there un-acked.

How you would code this up depends upon the client library, and whether you
are in a multi-threaded or event-driven environment.

Things are a bit more complicated if you want to have multiple workers,
all obeying the priorities.

David

-- 
David Wragg
Staff Engineer, RabbitMQ
SpringSource, a division of VMware