[rabbitmq-discuss] Scalable multithreaded consumer

[Tony Garnock-Jones]

Hi Karel,

Karel Alfonso wrote:
> I haven’t found in RabbitMQ a mechanism to
> process concurrently the messages received. Is there any suggestion or
> guideline I could follow?

If you mean handing messages received on a consumer off to multiple
worker threads, one approach is to have multiple threads reading from a
single QueueingConsumer. Each thread would, at its heart, do something
like this:

    while (true) {
        QueueingConsumer.Delivery delivery = consumer.nextDelivery();
        // ... handle delivery ...
        synchronized (channel) {
            channel.basicAck(delivery.getEnvelope().getDeliveryTag(),
                             false);
        }
    }

QueueingConsumer uses java.util.concurrent.BlockingQueue internally, so
it is thread-safe. You still need to lock the channel object if you are
accessing it from multiple threads, though - the usual model is for a
single thread to have a single channel, so if they share one, they must
take scrupulous care to not step on each other's toes.

Another approach is to open a channel and a consumer for each thread.
That way, you avoid the locking headaches needed when sharing a channel
between threads, and each thread can manage its own resources. This
approach is also gives better control over the thread/process boundary,
so if at a later date you decide you need to have multiple JVMs, it'll
be easier to move a few worker threads at a time over to the other process.

Regards,
  Tony
-- 
 [][][] Tony Garnock-Jones     | Mob: +44 (0)7905 974 211
   [][] LShift Ltd             | Tel: +44 (0)20 7729 7060
 []  [] http://www.lshift.net/ | Email: tonyg at lshift.net