[rabbitmq-discuss] Rabbit Cluster Failure

Simon MacMullen simon at rabbitmq.com
Thu Apr 11 14:04:28 BST 2013


Hi. We haven't seen a bug with that stack trace before, although I guess 
this could be a side effect of something we *have* already seen.

It's probably worth upgrading anyway - there are a few HA bugs which 
have been fixed in newer versions. Also, can you post / send complete 
logs so we can see if anything interesting happened before this?

Cheers, Simon

On 10/04/13 17:51, Farshad Meshkinpour wrote:
> Hello,
> We are running Rabbit 2.8.6 on Erlang R14B04.  We got this error on one
> of 3 rabbit servers in our cluster. After that we observed the followings:
> 1. all 3 rabbit servers went on high cpu usage
> 2. management interface was not functional
> 3. workers were not able to consume from queues
>
> Trying to stop the rabbit server failed (/etc/init.d/rabbitmq-server
> stop) and end up hard kill (kill -9) rabbit process on all nodes of the
> cluster to get the system back.  All mirrored queues contents were lost.
>
> Can someone points us to what may have caused this? Any help is greatly
> appreciated.
>
> Thanks,
>
>
> ERROR REPORT==== 9-Apr-2013::10:56:49 ===
> ** Generic server <0.9085.0> terminating
> ** Last message in was {'$gen_cast',
>                          {delete_and_terminate,
>                           {function_clause,
>                            [{rabbit_amqqueue_process,handle_call,
>                              [{add_on_right,{74,<0.7181.510>}},
>                               {<0.7181.510>,#Ref<0.0.14625.121725>},
>                               {q,
>                                {amqqueue,
>                                 {resource,<<"/fonsyncd/prod/">>,queue,
>                                  <<"Q-SID.16970.fonsyncd">>},
>                                 true,false,none,
>                                 [{<<"x-message-ttl">>,signedint,100000},
>                                  {<<"x-dead-letter-exchange">>,longstr,
>                                   <<"EX-fonsyncd-Exception">>},
>                                  {<<"x-ha-policy">>,longstr,<<"all">>}],
>                                 <0.4403.0>,[],all},
>                                none,false,rabbit_mirror_queue_master,
>                                {state,<0.4404.0>,<0.9085.0>,
>                                 rabbit_variable_queue,
>                                 {vqstate,
>                                  {0,{[],[]}},
>                                  {0,{[],[]}},
>                                  {delta,undefined,0,undefined},
>                                  {0,{[],[]}},
>                                  {0,{[],[]}},
>                                  2114,
>                                  {0,nil},
>                                  undefined,
>                                  {0,nil},
>                                  {qistate,
>
>   "/var/lib/rabbitmq/mnesia/rabbit at rmq1/queues/EYHEI3SQW41F6NFSH8WB2CD91",
>                                   {{dict,0,16,16,8,80,48,
>
>   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                      [],[]},
>
>   {{[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                       [],[],[]}}},
>                                    []},
>                                   #Ref<0.0.25.88612>,0,262144,
>                                   #Fun<rabbit_variable_queue.2.36908642>,
>                                   {0,nil}},
>                                  {{client_msstate,msg_store_persistent,
>
> <<72,22,217,216,236,0,110,187,235,185,163,
>                                      249,92,182,156,56>>,
>                                    {dict,0,16,16,8,80,48,
>
>   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                      [],[]},
>
>   {{[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                       [],[],[]}}},
>                                    {state,696393,
>
>   "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_persistent"},
>                                    rabbit_msg_store_ets_index,
>
> "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_persistent",
>                                    <0.269.0>,700490,692296,704587,708684},
>                                   {client_msstate,msg_store_transient,
>
> <<51,15,67,134,52,182,142,196,131,185,177,5,
>                                      223,15,45,61>>,
>                                    {dict,0,16,16,8,80,48,
>
>   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                      [],[]},
>
>   {{[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                       [],[],[]}}},
>                                    {state,675908,
>
>   "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_transient"},
>                                    rabbit_msg_store_ets_index,
>
> "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_transient",
>                                    <0.264.0>,680005,671795,684102,688199}},
>                                  true,0,
>
> #Fun<rabbit_mirror_queue_slave.1.52898136>,0,
>                                  0,infinity,0,0,0,0,0,
>                                  {rates,
>                                   {{1365,529204,602738},0},
>                                   {{1365,529204,602738},0},
>                                   0.0,0.0,
>                                   {1365,529286,74299}},
>                                  {0,nil},
>                                  {0,nil},
>                                  {0,nil},
>                                  {0,nil},
>                                  0,0,
>                                  {rates,
>                                   {{1365,529204,602738},0},
>                                   {{1365,529204,602738},0},
>                                   0.0,0.0,
>                                   {1365,529286,74299}}},
>                                 0,
>                                 {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                   []},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                    [],[]}}},
>                                 [],
>                                 {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                   []},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                    [],[]}}},
>                                 {set,6,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                   []},
>                                  {{[],
>                                    [<0.17350.499>],
>                                    [<0.17781.499>],
>                                    [],
>                                    [<7525.9758.575>],
>                                    [],
>                                    [<7526.18741.575>,<7526.16733.575>],
>                                    [],[],[],[],[],[],[],[],
>                                    [<7525.13515.575>]}}}},
>                                {[],[]},
>                                undefined,undefined,undefined,undefined,
>                                {state,fine,5000,undefined},
>                                {0,nil},
>                                100000,undefined,1365529282744274,
>                                {dict,6,16,16,8,80,48,
>
>   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                  []},
>                                 {{[],
>                                   [[<0.17350.499>|#Ref<0.0.14563.86478>]],
>                                   [[<0.17781.499>|#Ref<0.0.14563.86169>]],
>                                   [],
>                                   [[<7525.9758.575>|#Ref<0.0.14563.86395>]],
>                                   [],
>                                   [[<7526.16733.575>|#Ref<0.0.14563.86260>],
>
> [<7526.18741.575>|#Ref<0.0.14563.86683>]],
>                                   [],[],[],[],[],[],[],[],
>
>   [[<7525.13515.575>|#Ref<0.0.14563.86620>]]}}},
>                                1,
>                                {{0,nil},{0,nil}},
>                                undefined,
>                                {dict,0,16,16,8,80,48,
>
>   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                  []},
>
>   {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                   []}}},
>                                {resource,<<"/fonsyncd/prod/">>,exchange,
>                                 <<"EX-fonsyncd-Exception">>},
>                                undefined}]},
>                             {gen_server2,handle_msg,2},
>                             {proc_lib,wake_up,3}]}}}
> ** When Server state == {state,
>                              {amqqueue,
>                                  {resource,<<"/fonsyncd/prod/">>,queue,
>                                      <<"Q-SID.16970.fonsyncd">>},
>                                  true,false,none,
>                                  [{<<"x-message-ttl">>,signedint,100000},
>                                   {<<"x-dead-letter-exchange">>,longstr,
>                                    <<"EX-fonsyncd-Exception">>},
>                                   {<<"x-ha-policy">>,longstr,<<"all">>}],
>                                  <0.4403.0>,[],all},
>                              <0.4404.0>,
>                              {dict,6,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                   []},
>                                  {{[],
>                                    [[<0.17350.499>|#Ref<0.0.14563.86480>]],
>                                    [[<0.17781.499>|#Ref<0.0.14563.86174>]],
>                                    [],
>
> [[<7525.9758.575>|#Ref<0.0.14563.86397>]],
>                                    [],
>
> [[<7526.16733.575>|#Ref<0.0.14563.86263>],
>
>   [<7526.18741.575>|#Ref<0.0.14563.86686>]],
>                                    [],[],[],[],[],[],[],[],
>                                    [[<7525.13515.575>|
>                                      #Ref<0.0.14563.86622>]]}}},
>                              #Fun<rabbit_mirror_queue_master.1.37767645>,
>                              #Fun<rabbit_mirror_queue_master.2.24464450>}
> ** Reason for termination ==
> ** {function_clause,
>         [{rabbit_amqqueue_process,handle_call,
>              [{add_on_right,{74,<0.7181.510>}},
>               {<0.7181.510>,#Ref<0.0.14625.121725>},
>               {q,{amqqueue,
>                      {resource,<<"/fonsyncd/prod/">>,queue,
>                          <<"Q-SID.16970.fonsyncd">>},
>                      true,false,none,
>                      [{<<"x-message-ttl">>,signedint,100000},
>                       {<<"x-dead-letter-exchange">>,longstr,
>                        <<"EX-fonsyncd-Exception">>},
>                       {<<"x-ha-policy">>,longstr,<<"all">>}],
>                      <0.4403.0>,[],all},
>                  none,false,rabbit_mirror_queue_master,
>                  {state,<0.4404.0>,<0.9085.0>,rabbit_variable_queue,
>                      {vqstate,
>                          {0,{[],[]}},
>                          {0,{[],[]}},
>                          {delta,undefined,0,undefined},
>                          {0,{[],[]}},
>                          {0,{[],[]}},
>                          2114,
>                          {0,nil},
>                          undefined,
>                          {0,nil},
>                          {qistate,
>
> "/var/lib/rabbitmq/mnesia/rabbit at rmq1/queues/EYHEI3SQW41F6NFSH8WB2CD91",
>                              {{dict,0,16,16,8,80,48,
>
>   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                    [],[]},
>
>   {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                     [],[]}}},
>                               []},
>                              #Ref<0.0.25.88612>,0,262144,
>                              #Fun<rabbit_variable_queue.2.36908642>,
>                              {0,nil}},
>                          {{client_msstate,msg_store_persistent,
>
>   <<72,22,217,216,236,0,110,187,235,185,163,249,92,
>                                 182,156,56>>,
>                               {dict,0,16,16,8,80,48,
>
>   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                    [],[]},
>
>   {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                     [],[]}}},
>                               {state,696393,
>
>   "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_persistent"},
>                               rabbit_msg_store_ets_index,
>
>   "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_persistent",
>                               <0.269.0>,700490,692296,704587,708684},
>                           {client_msstate,msg_store_transient,
>
>   <<51,15,67,134,52,182,142,196,131,185,177,5,223,
>                                 15,45,61>>,
>                               {dict,0,16,16,8,80,48,
>
>   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                    [],[]},
>
>   {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                     [],[]}}},
>                               {state,675908,
>
>   "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_transient"},
>                               rabbit_msg_store_ets_index,
>
>   "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_transient",
>                               <0.264.0>,680005,671795,684102,688199}},
>
> true,0,#Fun<rabbit_mirror_queue_slave.1.52898136>,0,0,
>                          infinity,0,0,0,0,0,
>                          {rates,
>                              {{1365,529204,602738},0},
>                              {{1365,529204,602738},0},
>                              0.0,0.0,
>                              {1365,529286,74299}},
>                          {0,nil},
>                          {0,nil},
>                          {0,nil},
>                          {0,nil},
>                          0,0,
>                          {rates,
>                              {{1365,529204,602738},0},
>                              {{1365,529204,602738},0},
>                              0.0,0.0,
>                              {1365,529286,74299}}},
>                      0,
>                      {dict,0,16,16,8,80,48,
>                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>                      [],
>                      {dict,0,16,16,8,80,48,
>                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>                      {set,6,16,16,8,80,48,
>                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                          {{[],
>                            [<0.17350.499>],
>                            [<0.17781.499>],
>                            [],
>                            [<7525.9758.575>],
>                            [],
>                            [<7526.18741.575>,<7526.16733.575>],
>                            [],[],[],[],[],[],[],[],
>                            [<7525.13515.575>]}}}},
>                  {[],[]},
>                  undefined,undefined,undefined,undefined,
>                  {state,fine,5000,undefined},
>                  {0,nil},
>                  100000,undefined,1365529282744274,
>                  {dict,6,16,16,8,80,48,
>                      {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                      {{[],
>                        [[<0.17350.499>|#Ref<0.0.14563.86478>]],
>                        [[<0.17781.499>|#Ref<0.0.14563.86169>]],
>                        [],
>                        [[<7525.9758.575>|#Ref<0.0.14563.86395>]],
>                        [],
>                        [[<7526.16733.575>|#Ref<0.0.14563.86260>],
>                         [<7526.18741.575>|#Ref<0.0.14563.86683>]],
>                        [],[],[],[],[],[],[],[],
>                        [[<7525.13515.575>|#Ref<0.0.14563.86620>]]}}},
>                  1,
>                  {{0,nil},{0,nil}},
>                  undefined,
>                  {dict,0,16,16,8,80,48,
>                      {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                      {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>                  {resource,<<"/fonsyncd/prod/">>,exchange,
>                      <<"EX-fonsyncd-Exception">>},
>                  undefined}]},
>          {gen_server2,handle_msg,2},
>          {proc_lib,wake_up,3}]}
>
>
> =ERROR REPORT==== 9-Apr-2013::10:56:49 ===
> ** Generic server <0.4403.0> terminating
> ** Last message in was {add_on_right,{74,<0.7181.510>}}
> ** When Server state == {q,
>                           {amqqueue,
>                            {resource,<<"/fonsyncd/prod/">>,queue,
>                             <<"Q-SID.16970.fonsyncd">>},
>                            true,false,none,
>                            [{<<"x-message-ttl">>,signedint,100000},
>                             {<<"x-dead-letter-exchange">>,longstr,
>                              <<"EX-fonsyncd-Exception">>},
>                             {<<"x-ha-policy">>,longstr,<<"all">>}],
>                            <0.4403.0>,[],all},
>                           none,false,rabbit_mirror_queue_master,
>
>   {state,<0.4404.0>,<0.9085.0>,rabbit_variable_queue,
>                            {vqstate,
>                             {0,{[],[]}},
>                             {0,{[],[]}},
>                             {delta,undefined,0,undefined},
>                             {0,{[],[]}},
>                             {0,{[],[]}},
>                             2114,
>                             {0,nil},
>                             undefined,
>                             {0,nil},
>                             {qistate,
>
> "/var/lib/rabbitmq/mnesia/rabbit at rmq1/queues/EYHEI3SQW41F6NFSH8WB2CD91",
>                              {{dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                 []},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                  []}}},
>                               []},
>                              #Ref<0.0.25.88612>,0,262144,
>                              #Fun<rabbit_variable_queue.2.36908642>,
>                              {0,nil}},
>                             {{client_msstate,msg_store_persistent,
>
>   <<72,22,217,216,236,0,110,187,235,185,163,249,92,
>                                 182,156,56>>,
>                               {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                 []},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                  []}}},
>                               {state,696393,
>
> "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_persistent"},
>                               rabbit_msg_store_ets_index,
>
>   "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_persistent",
>                               <0.269.0>,700490,692296,704587,708684},
>                              {client_msstate,msg_store_transient,
>
>   <<51,15,67,134,52,182,142,196,131,185,177,5,223,
>                                 15,45,61>>,
>                               {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                 []},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                  []}}},
>                               {state,675908,
>
> "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_transient"},
>                               rabbit_msg_store_ets_index,
>
>   "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_transient",
>                               <0.264.0>,680005,671795,684102,688199}},
>
>   true,0,#Fun<rabbit_mirror_queue_slave.1.52898136>,
>                             0,0,infinity,0,0,0,0,0,
>                             {rates,
>                              {{1365,529204,602738},0},
>                              {{1365,529204,602738},0},
>                              0.0,0.0,
>                              {1365,529286,74299}},
>                             {0,nil},
>                             {0,nil},
>                             {0,nil},
>                             {0,nil},
>                             0,0,
>                             {rates,
>                              {{1365,529204,602738},0},
>                              {{1365,529204,602738},0},
>                              0.0,0.0,
>                              {1365,529286,74299}}},
>                            0,
>                            {dict,0,16,16,8,80,48,
>
>   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                               []}}},
>                            [],
>                            {dict,0,16,16,8,80,48,
>
>   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                               []}}},
>                            {set,6,16,16,8,80,48,
>
>   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                             {{[],
>                               [<0.17350.499>],
>                               [<0.17781.499>],
>                               [],
>                               [<7525.9758.575>],
>                               [],
>                               [<7526.18741.575>,<7526.16733.575>],
>                               [],[],[],[],[],[],[],[],
>                               [<7525.13515.575>]}}}},
>                           {[],[]},
>                           undefined,undefined,undefined,undefined,
>                           {state,fine,5000,undefined},
>                           {0,nil},
>                           100000,undefined,1365529282744274,
>                           {dict,6,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                            {{[],
>                              [[<0.17350.499>|#Ref<0.0.14563.86478>]],
>                              [[<0.17781.499>|#Ref<0.0.14563.86169>]],
>                              [],
>                              [[<7525.9758.575>|#Ref<0.0.14563.86395>]],
>                              [],
>                              [[<7526.16733.575>|#Ref<0.0.14563.86260>],
>                               [<7526.18741.575>|#Ref<0.0.14563.86683>]],
>                              [],[],[],[],[],[],[],[],
>                              [[<7525.13515.575>|#Ref<0.0.14563.86620>]]}}},
>                           1,
>                           {{0,nil},{0,nil}},
>                           undefined,
>                           {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>                           {resource,<<"/fonsyncd/prod/">>,exchange,
>                            <<"EX-fonsyncd-Exception">>},
>                           undefined}
> ** Reason for termination ==
> ** {function_clause,
>         [{rabbit_amqqueue_process,handle_call,
>              [{add_on_right,{74,<0.7181.510>}},
>               {<0.7181.510>,#Ref<0.0.14625.121725>},
>               {q,{amqqueue,
>                      {resource,<<"/fonsyncd/prod/">>,queue,
>                          <<"Q-SID.16970.fonsyncd">>},
>                      true,false,none,
>                      [{<<"x-message-ttl">>,signedint,100000},
>                       {<<"x-dead-letter-exchange">>,longstr,
>                        <<"EX-fonsyncd-Exception">>},
>                       {<<"x-ha-policy">>,longstr,<<"all">>}],
>                      <0.4403.0>,[],all},
>                  none,false,rabbit_mirror_queue_master,
>                  {state,<0.4404.0>,<0.9085.0>,rabbit_variable_queue,
>                      {vqstate,
>                          {0,{[],[]}},
>                          {0,{[],[]}},
>                          {delta,undefined,0,undefined},
>                          {0,{[],[]}},
>                          {0,{[],[]}},
>                          2114,
>                          {0,nil},
>                          undefined,
>                          {0,nil},
>                          {qistate,
>
> "/var/lib/rabbitmq/mnesia/rabbit at rmq1/queues/EYHEI3SQW41F6NFSH8WB2CD91",
>                              {{dict,0,16,16,8,80,48,
>
>   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                    [],[]},
>
>   {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                     [],[]}}},
>                               []},
>                              #Ref<0.0.25.88612>,0,262144,
>                              #Fun<rabbit_variable_queue.2.36908642>,
>                              {0,nil}},
>                          {{client_msstate,msg_store_persistent,
>
>   <<72,22,217,216,236,0,110,187,235,185,163,249,92,
>                                 182,156,56>>,
>                               {dict,0,16,16,8,80,48,
>
>   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                    [],[]},
>
>   {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                     [],[]}}},
>                               {state,696393,
>
>   "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_persistent"},
>                               rabbit_msg_store_ets_index,
>
>   "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_persistent",
>                               <0.269.0>,700490,692296,704587,708684},
>                           {client_msstate,msg_store_transient,
>
>   <<51,15,67,134,52,182,142,196,131,185,177,5,223,
>                                 15,45,61>>,
>                               {dict,0,16,16,8,80,48,
>
>   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                    [],[]},
>
>   {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                     [],[]}}},
>                               {state,675908,
>
>   "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_transient"},
>                               rabbit_msg_store_ets_index,
>
>   "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_transient",
>                               <0.264.0>,680005,671795,684102,688199}},
>
> true,0,#Fun<rabbit_mirror_queue_slave.1.52898136>,0,0,
>                          infinity,0,0,0,0,0,
>                          {rates,
>                              {{1365,529204,602738},0},
>                              {{1365,529204,602738},0},
>                              0.0,0.0,
>                              {1365,529286,74299}},
>                          {0,nil},
>                          {0,nil},
>                          {0,nil},
>                          {0,nil},
>                          0,0,
>                          {rates,
>                              {{1365,529204,602738},0},
>                              {{1365,529204,602738},0},
>                              0.0,0.0,
>                              {1365,529286,74299}}},
>                      0,
>                      {dict,0,16,16,8,80,48,
>                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>                      [],
>                      {dict,0,16,16,8,80,48,
>                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>                      {set,6,16,16,8,80,48,
>                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                          {{[],
>                            [<0.17350.499>],
>                            [<0.17781.499>],
>                            [],
>                            [<7525.9758.575>],
>                            [],
>                            [<7526.18741.575>,<7526.16733.575>],
>                            [],[],[],[],[],[],[],[],
>                            [<7525.13515.575>]}}}},
>                  {[],[]},
>                  undefined,undefined,undefined,undefined,
>                  {state,fine,5000,undefined},
>                  {0,nil},
>                  100000,undefined,1365529282744274,
>                  {dict,6,16,16,8,80,48,
>                      {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                      {{[],
>                        [[<0.17350.499>|#Ref<0.0.14563.86478>]],
>                        [[<0.17781.499>|#Ref<0.0.14563.86169>]],
>                        [],
>                        [[<7525.9758.575>|#Ref<0.0.14563.86395>]],
>                        [],
>                        [[<7526.16733.575>|#Ref<0.0.14563.86260>],
>                         [<7526.18741.575>|#Ref<0.0.14563.86683>]],
>                        [],[],[],[],[],[],[],[],
>                        [[<7525.13515.575>|#Ref<0.0.14563.86620>]]}}},
>                  1,
>                  {{0,nil},{0,nil}},
>                  undefined,
>                  {dict,0,16,16,8,80,48,
>                      {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                      {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>                  {resource,<<"/fonsyncd/prod/">>,exchange,
>                      <<"EX-fonsyncd-Exception">>},
>                  undefined}]},
>          {gen_server2,handle_msg,2},
>          {proc_lib,wake_up,3}]}
>
> =ERROR REPORT==== 9-Apr-2013::10:56:49 ===
> ** Generic server <0.7181.510> terminating
> ** Last message in was {'$gen_cast',join}
> ** When Server state == {state,
>                              {74,<0.7181.510>},
>                              {{74,<0.7181.510>},undefined},
>                              {{74,<0.7181.510>},undefined},
>                              {resource,<<"/fonsyncd/prod/">>,queue,
>                                  <<"Q-SID.18335.fonsyncd">>},
>
> rabbit_mirror_queue_slave,undefined,-1,undefined,
>                              [<0.7180.510>],
>                              {[],[]},
>                              [],undefined}
> ** Reason for termination ==
> ** {{function_clause,
>          [{rabbit_amqqueue_process,handle_call,
>               [{add_on_right,{74,<0.7181.510>}},
>                {<0.7181.510>,#Ref<0.0.14625.121725>},
>                {q,{amqqueue,
>                       {resource,<<"/fonsyncd/prod/">>,queue,
>                           <<"Q-SID.16970.fonsyncd">>},
>                       true,false,none,
>                       [{<<"x-message-ttl">>,signedint,100000},
>                        {<<"x-dead-letter-exchange">>,longstr,
>                         <<"EX-fonsyncd-Exception">>},
>                        {<<"x-ha-policy">>,longstr,<<"all">>}],
>                       <0.4403.0>,[],all},
>                   none,false,rabbit_mirror_queue_master,
>                   {state,<0.4404.0>,<0.9085.0>,rabbit_variable_queue,
>                       {vqstate,
>                           {0,{[],[]}},
>                           {0,{[],[]}},
>                           {delta,undefined,0,undefined},
>                           {0,{[],[]}},
>                           {0,{[],[]}},
>                           2114,
>                           {0,nil},
>                           undefined,
>                           {0,nil},
>                           {qistate,
>
>   "/var/lib/rabbitmq/mnesia/rabbit at rmq1/queues/EYHEI3SQW41F6NFSH8WB2CD91",
>                               {{dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                     [],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                      [],[]}}},
>                                []},
>                               #Ref<0.0.25.88612>,0,262144,
>                               #Fun<rabbit_variable_queue.2.36908642>,
>                               {0,nil}},
>                           {{client_msstate,msg_store_persistent,
>
> <<72,22,217,216,236,0,110,187,235,185,163,249,92,
>                                  182,156,56>>,
>                                {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                     [],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                      [],[]}}},
>                                {state,696393,
>
> "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_persistent"},
>                                rabbit_msg_store_ets_index,
>
> "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_persistent",
>                                <0.269.0>,700490,692296,704587,708684},
>                            {client_msstate,msg_store_transient,
>
> <<51,15,67,134,52,182,142,196,131,185,177,5,223,
>                                  15,45,61>>,
>                                {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                     [],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                      [],[]}}},
>                                {state,675908,
>
> "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_transient"},
>                                rabbit_msg_store_ets_index,
>
> "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_transient",
>                                <0.264.0>,680005,671795,684102,688199}},
>
>   true,0,#Fun<rabbit_mirror_queue_slave.1.52898136>,0,
>                           0,infinity,0,0,0,0,0,
>                           {rates,
>                               {{1365,529204,602738},0},
>                               {{1365,529204,602738},0},
>                               0.0,0.0,
>                               {1365,529286,74299}},
>                           {0,nil},
>                           {0,nil},
>                           {0,nil},
>                           {0,nil},
>                           0,0,
>                           {rates,
>                               {{1365,529204,602738},0},
>                               {{1365,529204,602738},0},
>                               0.0,0.0,
>                               {1365,529286,74299}}},
>                       0,
>                       {dict,0,16,16,8,80,48,
>                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>   {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>                       [],
>                       {dict,0,16,16,8,80,48,
>                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>   {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>                       {set,6,16,16,8,80,48,
>                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                           {{[],
>                             [<0.17350.499>],
>                             [<0.17781.499>],
>                             [],
>                             [<7525.9758.575>],
>                             [],
>                             [<7526.18741.575>,<7526.16733.575>],
>                             [],[],[],[],[],[],[],[],
>                             [<7525.13515.575>]}}}},
>                   {[],[]},
>                   undefined,undefined,undefined,undefined,
>                   {state,fine,5000,undefined},
>                   {0,nil},
>                   100000,undefined,1365529282744274,
>                   {dict,6,16,16,8,80,48,
>                       {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                       {{[],
>                         [[<0.17350.499>|#Ref<0.0.14563.86478>]],
>                         [[<0.17781.499>|#Ref<0.0.14563.86169>]],
>                         [],
>                         [[<7525.9758.575>|#Ref<0.0.14563.86395>]],
>                         [],
>                         [[<7526.16733.575>|#Ref<0.0.14563.86260>],
>                          [<7526.18741.575>|#Ref<0.0.14563.86683>]],
>                         [],[],[],[],[],[],[],[],
>                         [[<7525.13515.575>|#Ref<0.0.14563.86620>]]}}},
>                   1,
>                   {{0,nil},{0,nil}},
>                   undefined,
>                   {dict,0,16,16,8,80,48,
>                       {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                       {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>                   {resource,<<"/fonsyncd/prod/">>,exchange,
>                       <<"EX-fonsyncd-Exception">>},
>                   undefined}]},
>           {gen_server2,handle_msg,2},
>           {proc_lib,wake_up,3}]},
>
> {gen_server2,call,[<0.4403.0>,{add_on_right,{74,<0.7181.510>}},infinity]}}
>
>
>
> ERROR REPORT==== 9-Apr-2013::10:56:49 ===
> ** Generic server <0.9085.0> terminating
> ** Last message in was {'$gen_cast',
>                          {delete_and_terminate,
>                           {function_clause,
>                            [{rabbit_amqqueue_process,handle_call,
>                              [{add_on_right,{74,<0.7181.510>}},
>                               {<0.7181.510>,#Ref<0.0.14625.121725>},
>                               {q,
>                                {amqqueue,
>                                 {resource,<<"/fonsyncd/prod/">>,queue,
>                                  <<"Q-SID.16970.fonsyncd">>},
>                                 true,false,none,
>                                 [{<<"x-message-ttl">>,signedint,100000},
>                                  {<<"x-dead-letter-exchange">>,longstr,
>                                   <<"EX-fonsyncd-Exception">>},
>                                  {<<"x-ha-policy">>,longstr,<<"all">>}],
>                                 <0.4403.0>,[],all},
>                                none,false,rabbit_mirror_queue_master,
>                                {state,<0.4404.0>,<0.9085.0>,
>                                 rabbit_variable_queue,
>                                 {vqstate,
>                                  {0,{[],[]}},
>                                  {0,{[],[]}},
>                                  {delta,undefined,0,undefined},
>                                  {0,{[],[]}},
>                                  {0,{[],[]}},
>                                  2114,
>                                  {0,nil},
>                                  undefined,
>                                  {0,nil},
>                                  {qistate,
>
>   "/var/lib/rabbitmq/mnesia/rabbit at rmq1/queues/EYHEI3SQW41F6NFSH8WB2CD91",
>                                   {{dict,0,16,16,8,80,48,
>
>   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                      [],[]},
>
>   {{[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                       [],[],[]}}},
>                                    []},
>                                   #Ref<0.0.25.88612>,0,262144,
>                                   #Fun<rabbit_variable_queue.2.36908642>,
>                                   {0,nil}},
>                                  {{client_msstate,msg_store_persistent,
>
> <<72,22,217,216,236,0,110,187,235,185,163,
>                                      249,92,182,156,56>>,
>                                    {dict,0,16,16,8,80,48,
>
>   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                      [],[]},
>
>   {{[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                       [],[],[]}}},
>                                    {state,696393,
>
>   "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_persistent"},
>                                    rabbit_msg_store_ets_index,
>
> "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_persistent",
>                                    <0.269.0>,700490,692296,704587,708684},
>                                   {client_msstate,msg_store_transient,
>
> <<51,15,67,134,52,182,142,196,131,185,177,5,
>                                      223,15,45,61>>,
>                                    {dict,0,16,16,8,80,48,
>
>   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                      [],[]},
>
>   {{[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                       [],[],[]}}},
>                                    {state,675908,
>
>   "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_transient"},
>                                    rabbit_msg_store_ets_index,
>
> "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_transient",
>                                    <0.264.0>,680005,671795,684102,688199}},
>                                  true,0,
>
> #Fun<rabbit_mirror_queue_slave.1.52898136>,0,
>                                  0,infinity,0,0,0,0,0,
>                                  {rates,
>                                   {{1365,529204,602738},0},
>                                   {{1365,529204,602738},0},
>                                   0.0,0.0,
>                                   {1365,529286,74299}},
>                                  {0,nil},
>                                  {0,nil},
>                                  {0,nil},
>                                  {0,nil},
>                                  0,0,
>                                  {rates,
>                                   {{1365,529204,602738},0},
>                                   {{1365,529204,602738},0},
>                                   0.0,0.0,
>                                   {1365,529286,74299}}},
>                                 0,
>                                 {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                   []},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                    [],[]}}},
>                                 [],
>                                 {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                   []},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                    [],[]}}},
>                                 {set,6,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                   []},
>                                  {{[],
>                                    [<0.17350.499>],
>                                    [<0.17781.499>],
>                                    [],
>                                    [<7525.9758.575>],
>                                    [],
>                                    [<7526.18741.575>,<7526.16733.575>],
>                                    [],[],[],[],[],[],[],[],
>                                    [<7525.13515.575>]}}}},
>                                {[],[]},
>                                undefined,undefined,undefined,undefined,
>                                {state,fine,5000,undefined},
>                                {0,nil},
>                                100000,undefined,1365529282744274,
>                                {dict,6,16,16,8,80,48,
>
>   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                  []},
>                                 {{[],
>                                   [[<0.17350.499>|#Ref<0.0.14563.86478>]],
>                                   [[<0.17781.499>|#Ref<0.0.14563.86169>]],
>                                   [],
>                                   [[<7525.9758.575>|#Ref<0.0.14563.86395>]],
>                                   [],
>                                   [[<7526.16733.575>|#Ref<0.0.14563.86260>],
>
> [<7526.18741.575>|#Ref<0.0.14563.86683>]],
>                                   [],[],[],[],[],[],[],[],
>
>   [[<7525.13515.575>|#Ref<0.0.14563.86620>]]}}},
>                                1,
>                                {{0,nil},{0,nil}},
>                                undefined,
>                                {dict,0,16,16,8,80,48,
>
>   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                  []},
>
>   {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                   []}}},
>                                {resource,<<"/fonsyncd/prod/">>,exchange,
>                                 <<"EX-fonsyncd-Exception">>},
>                                undefined}]},
>                             {gen_server2,handle_msg,2},
>                             {proc_lib,wake_up,3}]}}}
> ** When Server state == {state,
>                              {amqqueue,
>                                  {resource,<<"/fonsyncd/prod/">>,queue,
>                                      <<"Q-SID.16970.fonsyncd">>},
>                                  true,false,none,
>                                  [{<<"x-message-ttl">>,signedint,100000},
>                                   {<<"x-dead-letter-exchange">>,longstr,
>                                    <<"EX-fonsyncd-Exception">>},
>                                   {<<"x-ha-policy">>,longstr,<<"all">>}],
>                                  <0.4403.0>,[],all},
>                              <0.4404.0>,
>                              {dict,6,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                   []},
>                                  {{[],
>                                    [[<0.17350.499>|#Ref<0.0.14563.86480>]],
>                                    [[<0.17781.499>|#Ref<0.0.14563.86174>]],
>                                    [],
>
> [[<7525.9758.575>|#Ref<0.0.14563.86397>]],
>                                    [],
>
> [[<7526.16733.575>|#Ref<0.0.14563.86263>],
>
>   [<7526.18741.575>|#Ref<0.0.14563.86686>]],
>                                    [],[],[],[],[],[],[],[],
>                                    [[<7525.13515.575>|
>                                      #Ref<0.0.14563.86622>]]}}},
>                              #Fun<rabbit_mirror_queue_master.1.37767645>,
>                              #Fun<rabbit_mirror_queue_master.2.24464450>}
> ** Reason for termination ==
> ** {function_clause,
>         [{rabbit_amqqueue_process,handle_call,
>              [{add_on_right,{74,<0.7181.510>}},
>               {<0.7181.510>,#Ref<0.0.14625.121725>},
>               {q,{amqqueue,
>                      {resource,<<"/fonsyncd/prod/">>,queue,
>                          <<"Q-SID.16970.fonsyncd">>},
>                      true,false,none,
>                      [{<<"x-message-ttl">>,signedint,100000},
>                       {<<"x-dead-letter-exchange">>,longstr,
>                        <<"EX-fonsyncd-Exception">>},
>                       {<<"x-ha-policy">>,longstr,<<"all">>}],
>                      <0.4403.0>,[],all},
>                  none,false,rabbit_mirror_queue_master,
>                  {state,<0.4404.0>,<0.9085.0>,rabbit_variable_queue,
>                      {vqstate,
>                          {0,{[],[]}},
>                          {0,{[],[]}},
>                          {delta,undefined,0,undefined},
>                          {0,{[],[]}},
>                          {0,{[],[]}},
>                          2114,
>                          {0,nil},
>                          undefined,
>                          {0,nil},
>                          {qistate,
>
> "/var/lib/rabbitmq/mnesia/rabbit at rmq1/queues/EYHEI3SQW41F6NFSH8WB2CD91",
>                              {{dict,0,16,16,8,80,48,
>
>   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                    [],[]},
>
>   {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                     [],[]}}},
>                               []},
>                              #Ref<0.0.25.88612>,0,262144,
>                              #Fun<rabbit_variable_queue.2.36908642>,
>                              {0,nil}},
>                          {{client_msstate,msg_store_persistent,
>
>   <<72,22,217,216,236,0,110,187,235,185,163,249,92,
>                                 182,156,56>>,
>                               {dict,0,16,16,8,80,48,
>
>   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                    [],[]},
>
>   {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                     [],[]}}},
>                               {state,696393,
>
>   "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_persistent"},
>                               rabbit_msg_store_ets_index,
>
>   "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_persistent",
>                               <0.269.0>,700490,692296,704587,708684},
>                           {client_msstate,msg_store_transient,
>
>   <<51,15,67,134,52,182,142,196,131,185,177,5,223,
>                                 15,45,61>>,
>                               {dict,0,16,16,8,80,48,
>
>   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                    [],[]},
>
>   {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                     [],[]}}},
>                               {state,675908,
>
>   "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_transient"},
>                               rabbit_msg_store_ets_index,
>
>   "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_transient",
>                               <0.264.0>,680005,671795,684102,688199}},
>
> true,0,#Fun<rabbit_mirror_queue_slave.1.52898136>,0,0,
>                          infinity,0,0,0,0,0,
>                          {rates,
>                              {{1365,529204,602738},0},
>                              {{1365,529204,602738},0},
>                              0.0,0.0,
>                              {1365,529286,74299}},
>                          {0,nil},
>                          {0,nil},
>                          {0,nil},
>                          {0,nil},
>                          0,0,
>                          {rates,
>                              {{1365,529204,602738},0},
>                              {{1365,529204,602738},0},
>                              0.0,0.0,
>                              {1365,529286,74299}}},
>                      0,
>                      {dict,0,16,16,8,80,48,
>                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>                      [],
>                      {dict,0,16,16,8,80,48,
>                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>                      {set,6,16,16,8,80,48,
>                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                          {{[],
>                            [<0.17350.499>],
>                            [<0.17781.499>],
>                            [],
>                            [<7525.9758.575>],
>                            [],
>                            [<7526.18741.575>,<7526.16733.575>],
>                            [],[],[],[],[],[],[],[],
>                            [<7525.13515.575>]}}}},
>                  {[],[]},
>                  undefined,undefined,undefined,undefined,
>                  {state,fine,5000,undefined},
>                  {0,nil},
>                  100000,undefined,1365529282744274,
>                  {dict,6,16,16,8,80,48,
>                      {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                      {{[],
>                        [[<0.17350.499>|#Ref<0.0.14563.86478>]],
>                        [[<0.17781.499>|#Ref<0.0.14563.86169>]],
>                        [],
>                        [[<7525.9758.575>|#Ref<0.0.14563.86395>]],
>                        [],
>                        [[<7526.16733.575>|#Ref<0.0.14563.86260>],
>                         [<7526.18741.575>|#Ref<0.0.14563.86683>]],
>                        [],[],[],[],[],[],[],[],
>                        [[<7525.13515.575>|#Ref<0.0.14563.86620>]]}}},
>                  1,
>                  {{0,nil},{0,nil}},
>                  undefined,
>                  {dict,0,16,16,8,80,48,
>                      {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                      {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>                  {resource,<<"/fonsyncd/prod/">>,exchange,
>                      <<"EX-fonsyncd-Exception">>},
>                  undefined}]},
>          {gen_server2,handle_msg,2},
>          {proc_lib,wake_up,3}]}
>
>
> =ERROR REPORT==== 9-Apr-2013::10:56:49 ===
> ** Generic server <0.4403.0> terminating
> ** Last message in was {add_on_right,{74,<0.7181.510>}}
> ** When Server state == {q,
>                           {amqqueue,
>                            {resource,<<"/fonsyncd/prod/">>,queue,
>                             <<"Q-SID.16970.fonsyncd">>},
>                            true,false,none,
>                            [{<<"x-message-ttl">>,signedint,100000},
>                             {<<"x-dead-letter-exchange">>,longstr,
>                              <<"EX-fonsyncd-Exception">>},
>                             {<<"x-ha-policy">>,longstr,<<"all">>}],
>                            <0.4403.0>,[],all},
>                           none,false,rabbit_mirror_queue_master,
>
>   {state,<0.4404.0>,<0.9085.0>,rabbit_variable_queue,
>                            {vqstate,
>                             {0,{[],[]}},
>                             {0,{[],[]}},
>                             {delta,undefined,0,undefined},
>                             {0,{[],[]}},
>                             {0,{[],[]}},
>                             2114,
>                             {0,nil},
>                             undefined,
>                             {0,nil},
>                             {qistate,
>
> "/var/lib/rabbitmq/mnesia/rabbit at rmq1/queues/EYHEI3SQW41F6NFSH8WB2CD91",
>                              {{dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                 []},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                  []}}},
>                               []},
>                              #Ref<0.0.25.88612>,0,262144,
>                              #Fun<rabbit_variable_queue.2.36908642>,
>                              {0,nil}},
>                             {{client_msstate,msg_store_persistent,
>
>   <<72,22,217,216,236,0,110,187,235,185,163,249,92,
>                                 182,156,56>>,
>                               {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                 []},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                  []}}},
>                               {state,696393,
>
> "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_persistent"},
>                               rabbit_msg_store_ets_index,
>
>   "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_persistent",
>                               <0.269.0>,700490,692296,704587,708684},
>                              {client_msstate,msg_store_transient,
>
>   <<51,15,67,134,52,182,142,196,131,185,177,5,223,
>                                 15,45,61>>,
>                               {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                 []},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                  []}}},
>                               {state,675908,
>
> "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_transient"},
>                               rabbit_msg_store_ets_index,
>
>   "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_transient",
>                               <0.264.0>,680005,671795,684102,688199}},
>
>   true,0,#Fun<rabbit_mirror_queue_slave.1.52898136>,
>                             0,0,infinity,0,0,0,0,0,
>                             {rates,
>                              {{1365,529204,602738},0},
>                              {{1365,529204,602738},0},
>                              0.0,0.0,
>                              {1365,529286,74299}},
>                             {0,nil},
>                             {0,nil},
>                             {0,nil},
>                             {0,nil},
>                             0,0,
>                             {rates,
>                              {{1365,529204,602738},0},
>                              {{1365,529204,602738},0},
>                              0.0,0.0,
>                              {1365,529286,74299}}},
>                            0,
>                            {dict,0,16,16,8,80,48,
>
>   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                               []}}},
>                            [],
>                            {dict,0,16,16,8,80,48,
>
>   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                               []}}},
>                            {set,6,16,16,8,80,48,
>
>   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                             {{[],
>                               [<0.17350.499>],
>                               [<0.17781.499>],
>                               [],
>                               [<7525.9758.575>],
>                               [],
>                               [<7526.18741.575>,<7526.16733.575>],
>                               [],[],[],[],[],[],[],[],
>                               [<7525.13515.575>]}}}},
>                           {[],[]},
>                           undefined,undefined,undefined,undefined,
>                           {state,fine,5000,undefined},
>                           {0,nil},
>                           100000,undefined,1365529282744274,
>                           {dict,6,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                            {{[],
>                              [[<0.17350.499>|#Ref<0.0.14563.86478>]],
>                              [[<0.17781.499>|#Ref<0.0.14563.86169>]],
>                              [],
>                              [[<7525.9758.575>|#Ref<0.0.14563.86395>]],
>                              [],
>                              [[<7526.16733.575>|#Ref<0.0.14563.86260>],
>                               [<7526.18741.575>|#Ref<0.0.14563.86683>]],
>                              [],[],[],[],[],[],[],[],
>                              [[<7525.13515.575>|#Ref<0.0.14563.86620>]]}}},
>                           1,
>                           {{0,nil},{0,nil}},
>                           undefined,
>                           {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>                           {resource,<<"/fonsyncd/prod/">>,exchange,
>                            <<"EX-fonsyncd-Exception">>},
>                           undefined}
> ** Reason for termination ==
> ** {function_clause,
>         [{rabbit_amqqueue_process,handle_call,
>              [{add_on_right,{74,<0.7181.510>}},
>               {<0.7181.510>,#Ref<0.0.14625.121725>},
>               {q,{amqqueue,
>                      {resource,<<"/fonsyncd/prod/">>,queue,
>                          <<"Q-SID.16970.fonsyncd">>},
>                      true,false,none,
>                      [{<<"x-message-ttl">>,signedint,100000},
>                       {<<"x-dead-letter-exchange">>,longstr,
>                        <<"EX-fonsyncd-Exception">>},
>                       {<<"x-ha-policy">>,longstr,<<"all">>}],
>                      <0.4403.0>,[],all},
>                  none,false,rabbit_mirror_queue_master,
>                  {state,<0.4404.0>,<0.9085.0>,rabbit_variable_queue,
>                      {vqstate,
>                          {0,{[],[]}},
>                          {0,{[],[]}},
>                          {delta,undefined,0,undefined},
>                          {0,{[],[]}},
>                          {0,{[],[]}},
>                          2114,
>                          {0,nil},
>                          undefined,
>                          {0,nil},
>                          {qistate,
>
> "/var/lib/rabbitmq/mnesia/rabbit at rmq1/queues/EYHEI3SQW41F6NFSH8WB2CD91",
>                              {{dict,0,16,16,8,80,48,
>
>   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                    [],[]},
>
>   {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                     [],[]}}},
>                               []},
>                              #Ref<0.0.25.88612>,0,262144,
>                              #Fun<rabbit_variable_queue.2.36908642>,
>                              {0,nil}},
>                          {{client_msstate,msg_store_persistent,
>
>   <<72,22,217,216,236,0,110,187,235,185,163,249,92,
>                                 182,156,56>>,
>                               {dict,0,16,16,8,80,48,
>
>   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                    [],[]},
>
>   {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                     [],[]}}},
>                               {state,696393,
>
>   "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_persistent"},
>                               rabbit_msg_store_ets_index,
>
>   "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_persistent",
>                               <0.269.0>,700490,692296,704587,708684},
>                           {client_msstate,msg_store_transient,
>
>   <<51,15,67,134,52,182,142,196,131,185,177,5,223,
>                                 15,45,61>>,
>                               {dict,0,16,16,8,80,48,
>
>   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                    [],[]},
>
>   {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                     [],[]}}},
>                               {state,675908,
>
>   "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_transient"},
>                               rabbit_msg_store_ets_index,
>
>   "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_transient",
>                               <0.264.0>,680005,671795,684102,688199}},
>
> true,0,#Fun<rabbit_mirror_queue_slave.1.52898136>,0,0,
>                          infinity,0,0,0,0,0,
>                          {rates,
>                              {{1365,529204,602738},0},
>                              {{1365,529204,602738},0},
>                              0.0,0.0,
>                              {1365,529286,74299}},
>                          {0,nil},
>                          {0,nil},
>                          {0,nil},
>                          {0,nil},
>                          0,0,
>                          {rates,
>                              {{1365,529204,602738},0},
>                              {{1365,529204,602738},0},
>                              0.0,0.0,
>                              {1365,529286,74299}}},
>                      0,
>                      {dict,0,16,16,8,80,48,
>                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>                      [],
>                      {dict,0,16,16,8,80,48,
>                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>                      {set,6,16,16,8,80,48,
>                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                          {{[],
>                            [<0.17350.499>],
>                            [<0.17781.499>],
>                            [],
>                            [<7525.9758.575>],
>                            [],
>                            [<7526.18741.575>,<7526.16733.575>],
>                            [],[],[],[],[],[],[],[],
>                            [<7525.13515.575>]}}}},
>                  {[],[]},
>                  undefined,undefined,undefined,undefined,
>                  {state,fine,5000,undefined},
>                  {0,nil},
>                  100000,undefined,1365529282744274,
>                  {dict,6,16,16,8,80,48,
>                      {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                      {{[],
>                        [[<0.17350.499>|#Ref<0.0.14563.86478>]],
>                        [[<0.17781.499>|#Ref<0.0.14563.86169>]],
>                        [],
>                        [[<7525.9758.575>|#Ref<0.0.14563.86395>]],
>                        [],
>                        [[<7526.16733.575>|#Ref<0.0.14563.86260>],
>                         [<7526.18741.575>|#Ref<0.0.14563.86683>]],
>                        [],[],[],[],[],[],[],[],
>                        [[<7525.13515.575>|#Ref<0.0.14563.86620>]]}}},
>                  1,
>                  {{0,nil},{0,nil}},
>                  undefined,
>                  {dict,0,16,16,8,80,48,
>                      {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                      {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>                  {resource,<<"/fonsyncd/prod/">>,exchange,
>                      <<"EX-fonsyncd-Exception">>},
>                  undefined}]},
>          {gen_server2,handle_msg,2},
>          {proc_lib,wake_up,3}]}
>
> =ERROR REPORT==== 9-Apr-2013::10:56:49 ===
> ** Generic server <0.7181.510> terminating
> ** Last message in was {'$gen_cast',join}
> ** When Server state == {state,
>                              {74,<0.7181.510>},
>                              {{74,<0.7181.510>},undefined},
>                              {{74,<0.7181.510>},undefined},
>                              {resource,<<"/fonsyncd/prod/">>,queue,
>                                  <<"Q-SID.18335.fonsyncd">>},
>
> rabbit_mirror_queue_slave,undefined,-1,undefined,
>                              [<0.7180.510>],
>                              {[],[]},
>                              [],undefined}
> ** Reason for termination ==
> ** {{function_clause,
>          [{rabbit_amqqueue_process,handle_call,
>               [{add_on_right,{74,<0.7181.510>}},
>                {<0.7181.510>,#Ref<0.0.14625.121725>},
>                {q,{amqqueue,
>                       {resource,<<"/fonsyncd/prod/">>,queue,
>                           <<"Q-SID.16970.fonsyncd">>},
>                       true,false,none,
>                       [{<<"x-message-ttl">>,signedint,100000},
>                        {<<"x-dead-letter-exchange">>,longstr,
>                         <<"EX-fonsyncd-Exception">>},
>                        {<<"x-ha-policy">>,longstr,<<"all">>}],
>                       <0.4403.0>,[],all},
>                   none,false,rabbit_mirror_queue_master,
>                   {state,<0.4404.0>,<0.9085.0>,rabbit_variable_queue,
>                       {vqstate,
>                           {0,{[],[]}},
>                           {0,{[],[]}},
>                           {delta,undefined,0,undefined},
>                           {0,{[],[]}},
>                           {0,{[],[]}},
>                           2114,
>                           {0,nil},
>                           undefined,
>                           {0,nil},
>                           {qistate,
>
>   "/var/lib/rabbitmq/mnesia/rabbit at rmq1/queues/EYHEI3SQW41F6NFSH8WB2CD91",
>                               {{dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                     [],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                      [],[]}}},
>                                []},
>                               #Ref<0.0.25.88612>,0,262144,
>                               #Fun<rabbit_variable_queue.2.36908642>,
>                               {0,nil}},
>                           {{client_msstate,msg_store_persistent,
>
> <<72,22,217,216,236,0,110,187,235,185,163,249,92,
>                                  182,156,56>>,
>                                {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                     [],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                      [],[]}}},
>                                {state,696393,
>
> "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_persistent"},
>                                rabbit_msg_store_ets_index,
>
> "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_persistent",
>                                <0.269.0>,700490,692296,704587,708684},
>                            {client_msstate,msg_store_transient,
>
> <<51,15,67,134,52,182,142,196,131,185,177,5,223,
>                                  15,45,61>>,
>                                {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                     [],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                      [],[]}}},
>                                {state,675908,
>
> "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_transient"},
>                                rabbit_msg_store_ets_index,
>
> "/var/lib/rabbitmq/mnesia/rabbit at rmq1/msg_store_transient",
>                                <0.264.0>,680005,671795,684102,688199}},
>
>   true,0,#Fun<rabbit_mirror_queue_slave.1.52898136>,0,
>                           0,infinity,0,0,0,0,0,
>                           {rates,
>                               {{1365,529204,602738},0},
>                               {{1365,529204,602738},0},
>                               0.0,0.0,
>                               {1365,529286,74299}},
>                           {0,nil},
>                           {0,nil},
>                           {0,nil},
>                           {0,nil},
>                           0,0,
>                           {rates,
>                               {{1365,529204,602738},0},
>                               {{1365,529204,602738},0},
>                               0.0,0.0,
>                               {1365,529286,74299}}},
>                       0,
>                       {dict,0,16,16,8,80,48,
>                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>   {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>                       [],
>                       {dict,0,16,16,8,80,48,
>                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
>   {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>                       {set,6,16,16,8,80,48,
>                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                           {{[],
>                             [<0.17350.499>],
>                             [<0.17781.499>],
>                             [],
>                             [<7525.9758.575>],
>                             [],
>                             [<7526.18741.575>,<7526.16733.575>],
>                             [],[],[],[],[],[],[],[],
>                             [<7525.13515.575>]}}}},
>                   {[],[]},
>                   undefined,undefined,undefined,undefined,
>                   {state,fine,5000,undefined},
>                   {0,nil},
>                   100000,undefined,1365529282744274,
>                   {dict,6,16,16,8,80,48,
>                       {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                       {{[],
>                         [[<0.17350.499>|#Ref<0.0.14563.86478>]],
>                         [[<0.17781.499>|#Ref<0.0.14563.86169>]],
>                         [],
>                         [[<7525.9758.575>|#Ref<0.0.14563.86395>]],
>                         [],
>                         [[<7526.16733.575>|#Ref<0.0.14563.86260>],
>                          [<7526.18741.575>|#Ref<0.0.14563.86683>]],
>                         [],[],[],[],[],[],[],[],
>                         [[<7525.13515.575>|#Ref<0.0.14563.86620>]]}}},
>                   1,
>                   {{0,nil},{0,nil}},
>                   undefined,
>                   {dict,0,16,16,8,80,48,
>                       {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                       {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
>                   {resource,<<"/fonsyncd/prod/">>,exchange,
>                       <<"EX-fonsyncd-Exception">>},
>                   undefined}]},
>           {gen_server2,handle_msg,2},
>           {proc_lib,wake_up,3}]},
>
> {gen_server2,call,[<0.4403.0>,{add_on_right,{74,<0.7181.510>}},infinity]}}
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>


-- 
Simon MacMullen
RabbitMQ, VMware


More information about the rabbitmq-discuss mailing list