[rabbitmq-discuss] Consumer Performance Issues?

[Matthias Radestock]

Marv,

On 27/09/11 00:47, cOre dUmPeR wrote:
> One thing I've been seeing that has me a little puzzled is that (for my
> test setup) message rates on the input side are an order of magnitude
> higher that on the output side. Meaning I can push 25K msgs/sec from my
> producer. However, my consumer, only seems to get about 2.5K
> msgs/sec...although both consumer and producer are setup similarly.
>
> Do I have something misconfigured? Or is this expected? I would expect
> *some* difference between producer(s)/consumer(s)...but not an order of
> magnitude.

When the broker is heavily loaded the internal scheduling could well 
result in messages getting accepted at a much higher rate than they are 
delivered to consumers. That said, a 10:1 ratio is quite unusual.

> HW
> ------
> (3) EC2 m1.xlarge boxes - 4 cores, 15GB mem
> 1 producer, 1 consumer, 1 broker
>
> Broker
> ----------
> Basic RabbitMQ broker install. Didnt really play with any options,
> except installing the Management plugins and using a different port than
> default.
>
> Producer
> -------------
> Java Client. 4 Producer threads, each with its own connection and
> channel. Direct exchange, NO persist, NO autodelete.
>
> Consumer
> ---------------
> Java client, 4 consumer threads, each with its own connection and
> channel. AutoAck YES.
 >
> Messages
> -----------------
> JSON bytes, about 150bytes each message.

So you have four producers publishing to the same queue, and four 
consumers consuming off that queue, correct?

If so, the above setup can be approximated quite well with the 
MulticastMain test program that ships with the java client. Run it with 
the options "-x 4 -y 4 -u q -a -s 150"

With RabbitMQ 2.6.1 (if you are running anything older then please try 
this one) running on my desktop machine and limited to 4 cores I see a 
balanced inbound and outbound rate of about 36kHz (NB: MulticastMain 
reports the sending rate independently for each producer; I'm seeing 
~9kHz x 4). It can easily get out of balance though - I've seen producer 
rates as high as 45kHz and consumer rates as low as 12kHz.

> Other Details
> --------------------
> My consumer CPU
> load is almost nothing, SO I dont think its the Consumer holding things
> up. *On the broker side....there is only 1 CPU out of 4 that is doing
> any real work. Is this normal?* With 4 consumer threads...I would expect
> more to be happening on the Broker.  I know when the Producer is pushing
> with 4 threads...the broker is generally very busy. But when I stop the
> producer, and leave the consumer running (again, with 4
> threads)...again, I see only 1 CPU on the broker working.

A queue is effectively a single-threaded resource. Marshalling and 
sending are done in separate threads though, so I would expect to see 
additional cores used for that. In my experiments draining a queue with 
four consumers causes the broker to utilise 2.7 cores.


Regards,

Matthias.