Hello,<br><br>I've recently been seeing some intermittent process crashes in the JSON-RPC channel plugin. Here is an example:<br><br>=ERROR REPORT==== 9-Sep-2011::16:35:52 ===<br>** Generic server <0.13801.1107> terminating<br>
** Last message in was {'EXIT',<0.13799.1107>,<br> {badarg,<br> [{erlang,tuple_to_list,[none]},<br> {rabbit_jsonrpc_channel,do_send_async,3},<br>
{rabbit_jsonrpc_channel,handle_info,2},<br> {gen_server,handle_msg,5},<br> {proc_lib,init_p_do_apply,3}]}}<br>** When Server state == {ch,running,rabbit_jsonrpc_channel,<0.13799.1107>,<br>
<0.13799.1107>,undefined,<br> #Fun<rabbit_jsonrpc_channel.0.50269389>,none,<br> {set,0,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},<br>
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},<br> 4,<br> {[],[]},<br> {[],[]},<br> <<"guest">>,<<"/580627">>,<br>
<<"amq.gen-gHphjRfHaC0YWeflaEIezg==">>,<br> {dict,1,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},<br>
{{[],[],[],[],[],[],[],[],[],[],[],[],[],<br> [[<<"amq.ctag-5QgH+lAhBr93FnOG8Y4T6A==">>|<br> {resource,<<"/580627">>,queue,<br>
<<"amq.gen-sV5mnds/lpCC6eza+CbMRQ==">>}]],<br> [],[]}}},<br> {dict,0,16,16,8,80,48,<br> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},<br>
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},<br> <0.13800.1107>,<br> {state,none,undefined}}<br>** Reason for termination == <br>** {badarg,[{erlang,tuple_to_list,[none]},<br>
{rabbit_jsonrpc_channel,do_send_async,3},<br> {rabbit_jsonrpc_channel,handle_info,2},<br> {gen_server,handle_msg,5},<br> {proc_lib,init_p_do_apply,3}]}<br><br>I looked through the RabbitMQ code, and I believe this is getting triggered when the system in question starts getting enough load that messages are queued up instead of published immediately as they come in. The decoded properties get cleared out in rabbit_variable_queue to avoid persisting extra data, and then later when the message is passed to the json plugin, it tries to call tuple_to_list on the properties which are now set to the atom 'none'.<br>
<br>(Incidentally, I'm seeing this problem in RabbitMQ 2.1.0, but I checked out the latest code from hg and the plugin still calls tuple_to_list without calling ensure_content_decoded first.)<br><br>The twist here is that I'm not sure if this is an error anyone would normally see, but we have written a custom exchange type plugin that creates messages, calls ensure_content_encoded on them, and then passes them to rabbit_amqqueue:deliver. The json plugin crash only seems to happen on these messages that are generated in this way (but like I said, only intermittently, and most of the time it doesn't seem to cause any issues). We also have other components of our system that connect to RabbitMQ directly (not through the json plugin), and there never seem to be any crashes associated with those connections, despite the fact that they're dealing with the same kind of messages that do occasionally cause crashes in the json plugin.<br>
<br>Based on my limited knowledge of the Rabbit code though, it doesn't seem like we're doing anything wrong in our exchange type plugin, and it seems like it would make sense for the json plugin to call ensure_content_decoded before it assumes the content is actually decoded. Could this be considered a bug in the json plugin? If so, I would be happy to write up a patch and submit it to whatever the appropriate place might be.<br>
<br>Thanks for your help!<br>Nick Marino<br>