[rabbitmq-discuss] Load balancing with multiple consumers on a single queue

[David.Corcoran at edftrading.com]

Morning Edwin,

Unfortunately the polling can't work for us because a 100ms delay between
each basic.get is too long. A lot of our jobs are only a few ms. But when I
first started using RabbitMq I used basic.get and a 10ms delay with 250
consumers on 250 tcp connections (although we could easily need 500 or
more). At the time we were using Erlang 11b5 and RabbitMQ 1.3 and it didn't
handle it very well. Obviously in our tests the 250 different clients
weren't registering any cpu usage on their machines but RabbitMQ was at
100%.

I stopped the tests and moved onto consumers soon after.

Dave

rabbitmq-discuss-bounces at lists.rabbitmq.com wrote on 31/07/2008 01:38:18:

> I was curious about the cost of polling using basic.get, so I wrote
> a small Erlang client program to start up 500 consumers, on one
> TCP/IP channel, listening on one queue, each polling with a basic.
> get every 100ms. I know this is a high rate, but I wanted to see how
> it behaved.
>
> Maybe my code was naive or inefficient, but I found the polling
> program's Linux process to be taking 85% CPU (and off the chart in
> appmon), while Rabbit's process was about 30% (on a 4-core Intel
> Q6600 box, hence CPU numbers don't add up to 100%). I haven't tried
> it with the "native" client. It seemed to make no difference if
> kernel poll was enabled.
>
> Regards,
> Edwin
>
> Environment: Ubuntu Linux 8.04, 2.4 GHz Q6600, 8 GB RAM, Erlang
> R12B-3, RabbitMQ 1.4.0, RabbitMQ Erlang client as of 2008/06/23.


*********************************************************************
This communication contains confidential information, some or all of which may be privileged. It is for the intended recipient only and others must not disclose, distribute, copy, print or rely on this communication. If an addressing or transmission error has misdirected this communication, please notify the sender by replying to this e-mail and then delete the e-mail. E-mail sent to EDF Trading may be monitored by the company. Thank you. 
EDF Trading Limited
80 Victoria Street, 3rd Floor, Cardinal Place, London, SW1E 5JL
A Company registered in England No. 4255974. 
Switchboard: 020 7061 4000
EDF Trading Markets Limited is a member of the EDF Trading Limited Group and is authorised and regulated by the Financial Services Authority.
VAT number: GB 735 5479 07
*********************************************************************