[rabbitmq-discuss] erlang-client problem on OS X (empty_rpc_bottom_half)

[Ben Hood]

Matt,

On Tue, Dec 9, 2008 at 6:37 AM, Matt Stancliff <sysop at mindspring.com> wrote:
> Rabbits,
>
> I've spent the past two days attempting to track down why the erlang-
> client doesn't work for me on OS X.
> Everything works fine on my Linux box.
> The Linux and OS X (10.5.5) installs of everything are performed by
> the same automated scripts.
>
> I'm running erlang (R12B-5), rabbitmq-server (tip: cedf2c7e873d) and
> the erlang-client (tip: a9695aaf5083).

Interesting. I've just tried these revisions (runnning R12B-5 on OS X
10.5.5) and I am unable to reproduce that error.

> =ERROR REPORT==== 8-Dec-2008::22:14:21 ===
> ** Generic server <0.242.0> terminating
> ** Last message in was {send_command,{'channel.flow',false}}
> ** When Server state == {channel_state,1,<0.241.0>,undefined,<0.243.0>,
>                             #Fun<amqp_direct_driver.do.2>,
>                             #Fun<amqp_direct_driver.do.3>,
>                             #Fun<amqp_direct_driver.close_channel.1>,
>                             {[],[]},
>                             {[],[]},
>                             {dict,0,16,16,8,80,48,
>                                 {[],[],[],[],[],[],[],[],[],[],[],[],
> [],[],[],
>                                  []},
>                                 {{[],[],[],[],[],[],[],[],[],[],[],[],
> [],[],
>                                   [],[]}}},
>                             false,undefined,
>                             {dict,0,16,16,8,80,48,
>                                 {[],[],[],[],[],[],[],[],[],[],[],[],
> [],[],[],
>                                  []},
>                                 {{[],[],[],[],[],[],[],[],[],[],[],[],
> [],[],
>                                   [],[]}}}}
> ** Reason for termination ==
> ** empty_rpc_bottom_half
> =====

Can you have a look in the rabbit log to see if there is anything like
this in it:

=INFO REPORT==== 9-Dec-2008::08:54:49 ===
    alarm_handler: {set,{system_memory_high_watermark,[]}}

My first thought is that the client may be receiving a
channel.flow{active=false} notification that it is not able to handle,
but if the client revision is a9695aaf5083 then you are on the 19963
branch which handles that command.

I've tried to channel.flow on before testing this but I still can't
reproduce this symptom.

Furthermore, I can't see how this can occur with that branch, because
the error that you are seeing, empty_rpc_bottom_half, means that the
client has received an RPC response from the server for which it has
no original request, i.e. a phantom RPC repsonse. The channel.flow
command is a reverse RPC, i.e. the server sends the client the RPC and
the client responds with a channel.flow_ok. However, the channel.flow
command has special handling in the client which does not use the
client's RPC queue. So I'm a bit baffled as to how this can happen.

So I think we need to approach this one systematically firstly using
the default branch of the server and the default branch of the client.

Can you first do a make clean run on the server and check the logs for
the alarm handler.

Then do a make clean test_network on the client and report what happens.

Ben