[rabbitmq-discuss] Crash on badmatch

Thu Sep 20 15:55:38 BST 2012

Ouch.

'badmatch' is a rather generic error message - pattern matching did not 
succeed. In the case you've encountered, pattern matching is used as an 
assertion. More specifically, the message store was attempting to merge 
two files, and in doing so found that one of them was not the size it 
expected.

So it's possible you've found a bug. If you see this happening again, 
please zip up all the logs and also the message stores + Mnesia 
themselves (/mnt/drbd0/rabbit at localhost/) since we may need to do quite 
a bit of digging to find out what's happened.

On the other hand, the message store is quite mature code; it hasn't 
changed in a while and it gets a lot of use. So I wouldn't *entirely* 
rule out RAM / HDD corruption as a cause either.

Cheers, Simon

On 20/09/12 15:34, Chris Larsen wrote:
> Hi folks, we had a problem today where the MQ server kept crashing and
> we think it was due to some client pushing in a bad message as we were
> able to keep the server up only after removing access to some users.
> We’re still trying to track down the exact client that was causing the
> issues. However the cause of the crash was listed in the logs as bellow.
> I couldn’t find any references to “badmatch” errors on the list before
> so I was wondering if anyone has seen it?
>
> We’re running Rabbitmq 2.8.5 on erlang 1:15.b.1-1~ubuntu~lucid. Thanks!
>
> =ERROR REPORT==== 20-Sep-2012::12:59:37 ===
>
> ** Generic server <0.229.0> terminating
>
> ** Last message in was {'$gen_cast',{combine,1001,999}}
>
> ** When Server state == {state,
>
> {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> []},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>
> [],[]}}},
>
> [#Fun<rabbit_msg_store.15.106110853>],
>
> {gc_state,
>
> "/mnt/drbd0/rabbit at localhost/msg_store_persistent",
>
> rabbit_msg_store_ets_index,
>
> {state,458823,
>
> "/mnt/drbd0/rabbit at localhost/msg_store_persistent"},
>
> 454726,462920,<0.226.0>}}
>
> ** Reason for termination ==
>
> **
> {{badmatch,{[{msg_location,<<119,180,79,244,242,22,59,63,219,227,203,29,171,
>
> 138,191,149>>,
>
> 1,999,2301,298},
>
> {msg_location,<<15,166,249,210,74,141,92,125,164,106,201,179,
>
> 247,29,128,81>>,
>
> 1,999,4643,297},
>
> {msg_location,<<61,32,10,20,15,47,161,44,66,91,98,119,129,10,
>
> 230,135>>,
>
> 1,999,7239,297},
>
> {msg_location,<<242,222,190,244,108,107,42,177,98,113,130,255,
>
> 211,78,44,242>>,
>
> 1,999,7791,297},
>
> {msg_location,<<107,45,44,203,166,43,87,116,132,109,19,63,113,
>
> 184,191,143>>,
>
> 1,999,9877,297},
>
> ..... { THOUSANDS OF LINES } .....
>
> {msg_location,<<167,233,20,236,134,250,109,211,72,107,129,39,
>
> 57,87,109,67>>,
>
> 1,999,1580282,254}],
>
> 378139}},
>
> [{rabbit_msg_store,combine_files,3,[]},
>
> {rabbit_msg_store_gc,attempt_action,3,[]},
>
> {rabbit_msg_store_gc,handle_cast,2,[]},
>
> {gen_server2,handle_msg,2,[]},
>
> {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,237}]}]}
>
> =ERROR REPORT==== 20-Sep-2012::13:31:15 ===
>
> Error description:
>
> {shutdown,
>
> {gen_server,call,
>
> [{rabbit_amqqueue_sup,rabbit at localhost},
>
> {start_child,
>
> [{amqqueue,
>
> {resource,<<"/exchange/queu">>,queue,
>
> <<"/queue/hostname:1983">>},
>
> true,false,none,[],<0.2434.0>,[],undefined}]},
>
> infinity]}}
>
> Log files (may contain more information):
>
> /var/log/rabbitmq/rabbit at localhost.log
>
> /var/log/rabbitmq/rabbit at localhost-sasl.log
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

-- 
Simon MacMullen
RabbitMQ, VMware