[rabbitmq-discuss] Consumer stop to receive messages but continue listening queue problem.

[Matthew Sackman]

Hi Gustavo,

On Fri, Mar 12, 2010 at 12:00:52AM -0300, Gustavo Aquino wrote:
> I'm running my RabbitMQ in one server, and have 2 producers in other server
> posting ~2000 messages /s , and have 4 consumers in one server and other 4
> in other.

Ok, so we have 4kHz in, in total? Are the 8 consumers all off the same
queue, or do they all have their own queues? If the former, then the sum
out should also be 4kHz, whereas the latter would suggest the sum out
should be 8 * 4kHz = 32kHz. That's faster than I've seen Rabbit running
before, but then again you do have very meaty hardware.

> When I start the test all running fine.. but my 8 consumers not consuming
> the queue in same speed that we are posting, ok it is the proposal of test,
> all 8 consumers are working harder consuming ~1200 /s when my doubt happen,
> all the time that I run this test it happen, when my queue have a size about
> 200.000 messages all consumers at one of this two consume's machine stop to
> receive messages, all of them, and if I restart all consumers in this
> machine anyone get any messages from Rabbit,all are connected but don't get
> any message, now if instead of restart this consumers I stop the other four
> consumers in the other machine he come back to receive messages...
> 
> Anybody here have any feelings about what is happen ?

That's very interesting. You don't say which client you're using though.
Also, when this happens, what does Rabbit record in the logs?

> My Rabbit server have 20Gb of memory and 4 processors P6 and when this drop
> happen Rabbit are consuming 80% of Proc and 20% of Memory.

I have never used Power6 machines, and I have no idea how well Erlang
(or Rabbit) runs on it, I'm afraid.

> My message size is about 180~200Kbytes

Those are pretty big. If my maths above is right, that means that you're
trying to push (4+32)*1000*200*1024 = 7372,800,000 bytes / sec, which is
6.8GB/s or 55Gb/s. That's enourmous. I suspect maybe my maths is wrong
and you actually have all 8 consumers bound to the same 1 queue?

Matthew