[rabbitmq-discuss] How to consume from 10, 000 to 100, 000 queues in a single process (java)

Wed Nov 10 12:11:41 GMT 2010

Adam,

On Thu, Nov 4, 2010 at 20:33, Adam Hitchcock <adam.hitchcock at skype.net> wrote:
> Currently I am using a single threaded process that performs
> basic_consume with a custom callback handler (based off the
> ConsumerMain.java example)
>
> I am finding that queues are building up because I'm not able to
> consume fast enough.

If your consumer is too slow, maybe you can run second one?
Can you scale on the consuming side?

> So what are the best practices to consume from
> thousands to hundres of thousands of queues on a single core machine?

Well, if you can you should avoid it. Have you seen exchange-to-exchange
bindings, maybe using them you could create a simpler architecture:
http://www.rabbitmq.com/blog/2010/10/19/exchange-to-exchange-bindings/

If you can't avoid using many queues, make sure they are small and
make sure you have enough resources (plenty of memory, a lot
of erlang process descriptors, and fast CPU).
Also SSD hard drive may help. 64 bit operating system is a requirement
for any setup with more than 2 gigs of RAM.
You should also avoid `touching` the queues too often. Don't use
basic.get, don't run `rabbitmqctl list_queues`. If possible don't
use acks (noack=True, as unacked messages need to be stored in-memory).

And obviously, run the newest RabbitMQ on a recent erlang.

> More details:
> noAck = true
> I am using a single instance of a class extending DefaultConsumer with
> an over ridden handleDelivery().

That sounds okay.

> Does anybody have numbers on what the maximum consume rate is on a
> stock ec2 setup?

No, but it's not that hard to measure. Here you have an example of how to
run our java MulticastMain benchmark:
http://www.rabbitmq.com/ec2.html#testing

Cheers,
  Marek