[rabbitmq-discuss] odd consumer behaviour

[David.Corcoran at edftrading.com]

Hi Matthias,

That makes perfect sense. Thanks for your help. I think my brain was a
little fried when I was writing the example. I was actually trying to write
a simple piece of code that exhibited a problem I was having and the
example I gave was obviously wrong.

The problem I was actually having I suspect is related to prefetching. I'll
try and explain it and perhaps you can confirm if it sounds likely?

I'm testing rabbitmq as a new messaging system for our job control. So I
write a job to the queue and it gets picked up by a worker and run.
Unfortunately each job can start more jobs, down to a level of about 3
deep. So with one job runner active this would lead to deadlock, as a job
would start a new job and block waiting for it to finish but no other job
runner would pick it up.

The way we work around this is when a parent job starts its own child jobs
and blocks, a new thread is started with a new channel pointing to the same
queue. Then when the parent job finishes the thread it was running in
finishes too. This way we only have 1 thread doing any work at a time.

In our current system for this to work I have to turn off prefetching by
the clients. Otherwise the first thread can start a job with dependencies,
then block, but the dependent jobs are prefetched into its local buffer
which leads to deadlock. With prefetching off another client can pick up
the job and work on it.

I think I've hit a problem with prefetching and RabbitMQ. Do you think this
might be the case?

Thanks for your help,

Dave




                                                                           
             Matthias                                                      
             Radestock                                                     
             <matthias at lshift.                                          To 
             net>                      David.Corcoran at edftrading.com       
                                                                        cc 
             01/05/2008 19:22          rabbitmq-discuss at lists.rabbitmq.com 
                                                                   Subject 
                                       Re: [rabbitmq-discuss] odd consumer 
                                       behaviour                           
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           





David,

David.Corcoran at edftrading.com wrote:
> 1. Run the HelloClient RabbitMQ example modified slightly so that it
sends
> 20 messages instead of 1.
> 2. Run the HelloServer example but with a slight change to the code. Add
a
> Thread.sleep(10*1000) before the return in handleStringCall().
> 3. Now run a normal HelloServer example, without any sleep.
> What I noticed here is that the normal example doesn't run at full speed.
> It looks like the slow server is slowing down the normal one.
> 5. Kill the slow server (the one with the sleep) and the normal
HelloServer
> now runs at full speed and finishes the rest of the messages within the
> second.

How exactly did you modify the HelloClient? Did you just put the original

             System.out.println(service.stringCall(request));

in a 20-iteration loop?

If so, then you are issuing 20 RPCs sequentially, i.e. the client will
be waiting for the result of the previous call before issuing the next.

Each call will be handled by one of the two HelloServers you are
running. If a call gets handled by the "slow" server then it will take
10 seconds to complete. If it happens to get handled by the fast server
then it will complete nearly instantly. As soon as you stop the slow
server all remaining requests will get handled by the fast server and
hence complete quickly.

So the behaviour you are seeing is to be expected.

Were you trying to construct a load balancing test? If so you'd need to
be issuing RPCs *concurrently*. Also, the RpcServer wasn't designed for
load balancing - it will actually consume messages from the underlying
AMQP queue as fast as it can without waiting for them to be processed.
You'd have to change that.

Regards,

Matthias.



*********************************************************************
This communication contains confidential information, some or all of which may be privileged. It is for the intended recipient only and others must not disclose, distribute, copy, print or rely on this communication. If an addressing or transmission error has misdirected this communication, please notify the sender by replying to this e-mail and then delete the e-mail. E-mail sent to EDF Trading may be monitored by the company. Thank you. 
EDF Trading Limited
80 Victoria Street, 3rd Floor, Cardinal Place, London, SW1E 5JL
A Company registered in England No. 4255974. 
Switchboard: 020 7061 4000
EDF Trading Markets Limited is a member of the EDF Trading Limited Group and is authorised and regulated by the Financial Services Authority.
VAT number: GB 735 5479 07
*********************************************************************