[rabbitmq-discuss] direct queue throughput

[Matthew Sackman]

Hi Charity,

On Fri, Aug 28, 2009 at 04:51:19PM -0700, Charity Majors wrote:
> We aren't acking at all.

Err, this seems odd. Do you mean that your consumers set noAck to true
when setting up the subscription to the queue? If you're not, and you're
not explicitly acking then Rabbit will be keeping a copy of the message
in memory all the time, because it won't discard messages until it
receives an ack from the consumer for the msg.

Are the queues durable and are the messages being published persistent
(delivery_mode = 2)? If so, this /may/ slow things down depending on
the nature of your consumers.

However, if a) cpu load is very low, and b) rabbitmqctl list_queues
shows that the queues are being kept empty, I rather suspect that
messages aren't getting in to Rabbit fast enough.

> Network load ... I would have to check on that.  But I don't think
> that's the ceiling we're bumping up against.  I can run it with a null
> plugin, so it just pushes the jobs through the rabbitmq queues and
> fakes the mysql responses, and everything stays on the same box.  And
> my throughput rate looks identical to the rates I see testing the
> SSDs.

Yes, I think this backs up my theory - if a null consumer isn't going
any faster, then I would definitely point the finger at the producers.
How big are the messages you're sending? If you're all on the same box,
I find this a bit surprising - I can push above 400M*Bytes*/s over
localhost loopback to and from Rabbit.

First of all though, if you've got noAck false in the consumers then
you *must* explicitly ack messages. I'd correct that and then see where
you are.

Matthew