[rabbitmq-discuss] JSON-RPC channel plugin crash on tuple_to_list(Props)

Mon Sep 19 17:15:49 BST 2011

Hello,

I've recently been seeing some intermittent process crashes in the JSON-RPC
channel plugin. Here is an example:

=ERROR REPORT==== 9-Sep-2011::16:35:52 ===
** Generic server <0.13801.1107> terminating
** Last message in was {'EXIT',<0.13799.1107>,
                           {badarg,
                               [{erlang,tuple_to_list,[none]},
                                {rabbit_jsonrpc_channel,do_send_async,3},
                                {rabbit_jsonrpc_channel,handle_info,2},
                                {gen_server,handle_msg,5},
                                {proc_lib,init_p_do_apply,3}]}}
** When Server state == {ch,running,rabbit_jsonrpc_channel,<0.13799.1107>,
                         <0.13799.1107>,undefined,
                         #Fun<rabbit_jsonrpc_channel.0.50269389>,none,
                         {set,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         4,
                         {[],[]},
                         {[],[]},
                         <<"guest">>,<<"/580627">>,
                         <<"amq.gen-gHphjRfHaC0YWeflaEIezg==">>,
                         {dict,1,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],
                            [[<<"amq.ctag-5QgH+lAhBr93FnOG8Y4T6A==">>|
                              {resource,<<"/580627">>,queue,
                               <<"amq.gen-sV5mnds/lpCC6eza+CbMRQ==">>}]],
                            [],[]}}},
                         {dict,0,16,16,8,80,48,
                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
                         <0.13800.1107>,
                         {state,none,undefined}}
** Reason for termination ==
** {badarg,[{erlang,tuple_to_list,[none]},
            {rabbit_jsonrpc_channel,do_send_async,3},
            {rabbit_jsonrpc_channel,handle_info,2},
            {gen_server,handle_msg,5},
            {proc_lib,init_p_do_apply,3}]}

I looked through the RabbitMQ code, and I believe this is getting triggered
when the system in question starts getting enough load that messages are
queued up instead of published immediately as they come in. The decoded
properties get cleared out in rabbit_variable_queue to avoid persisting
extra data, and then later when the message is passed to the json plugin, it
tries to call tuple_to_list on the properties which are now set to the atom
'none'.

(Incidentally, I'm seeing this problem in RabbitMQ 2.1.0, but I checked out
the latest code from hg and the plugin still calls tuple_to_list without
calling ensure_content_decoded first.)

The twist here is that I'm not sure if this is an error anyone would
normally see, but we have written a custom exchange type plugin that creates
messages, calls ensure_content_encoded on them, and then passes them to
rabbit_amqqueue:deliver. The json plugin crash only seems to happen on these
messages that are generated in this way (but like I said, only
intermittently, and most of the time it doesn't seem to cause any issues).
We also have other components of our system that connect to RabbitMQ
directly (not through the json plugin), and there never seem to be any
crashes associated with those connections, despite the fact that they're
dealing with the same kind of messages that do occasionally cause crashes in
the json plugin.

Based on my limited knowledge of the Rabbit code though, it doesn't seem
like we're doing anything wrong in our exchange type plugin, and it seems
like it would make sense for the json plugin to call ensure_content_decoded
before it assumes the content is actually decoded. Could this be considered
a bug in the json plugin? If so, I would be happy to write up a patch and
submit it to whatever the appropriate place might be.

Thanks for your help!
Nick Marino
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20110919/3c72f186/attachment.htm>