[rabbitmq-discuss] debugging lost queue on rabbit server switchover

Mon Mar 24 01:39:41 GMT 2014

Hi,

I'm looking for some help with debugging what seems to be a rabbit 
server issue.

I've got a rabbit server being used for OpenStack, and it's set up in a 
active-standby config using pacemaker.

Yesterday we had some guys doing testing by triggering a controlled 
switchover from active to standby and back, multiple times.

One of our other servers started showing some odd behaviour, and when we 
dug into it the problem seemed to be that the queue that it thought it 
was using didn't seem to exist on the rabbit server anymore.

# The message trace of the RPC request shows the required reply queue of 
the following.

reply_5e65eb22a7bd4bd98a5bc6b7c354fa5b

21:16:02.279850 00:1e:67:51:57:09 > 00:1e:67:54:aa:39, ethertype IPv4 
(0x0800), length 678: (tos 0x0, ttl 64, id 21886, offset 0, flags [DF], 
proto TCP (6), length 664)
     192.168.204.194.47946 > 192.168.204.2.5672: Flags [P.], cksum 
0x1ca1 (incorrect -> 0xcadb), seq 33263:33875, ack 2765, win 123, 
options [nop,nop,TS val 72448895 ecr 74959841], length 612
      0x0000:  4500 0298 557e 4000 4006 c8cb c0a8 ccc2  E...U~@. at .......
      0x0010:  c0a8 cc02 bb4a 1628 dd1a 91e2 01f0 e9ef  .....J.(........
      0x0020:  8018 007b 1ca1 0000 0101 080a 0451 7b7f  ...{.........Q{.
      0x0030:  0477 cbe1 0300 0100 0002 5c7b 226f 736c  .w........\{"osl
      0x0040:  6f2e 6d65 7373 6167 6522 3a20 227b 5c22  o.message":."{\"
      0x0050:  5f63 6f6e 7465 7874 5f72 6f6c 6573 5c22  _context_roles\"
      0x0060:  3a20 5b5c 2261 646d 696e 5c22 5d2c 205c  :.[\"admin\"],.\
      0x0070:  225f 6d73 675f 6964 5c22 3a20 5c22 6532  "_msg_id\":.\"e2
      0x0080:  6135 3465 6630 6466 3334 3436 3132 6133  a54ef0df344612a3
      0x0090:  3363 3464 6431 6231 6334 3238 3164 5c22  3c4dd1b1c4281d\"
      0x00a0:  2c20 5c22 5f63 6f6e 7465 7874 5f72 6561  ,.\"_context_rea
      0x00b0:  645f 6465 6c65 7465 645c 223a 205c 226e  d_deleted\":.\"n
      0x00c0:  6f5c 222c 205c 225f 7265 706c 795f 715c  o\",.\"_reply_q\
      0x00d0:  223a 205c 2272 6570 6c79 5f35 6536 3565  ":.\"reply_5e65e
      0x00e0:  6232 3261 3762 6434 6264 3938 6135 6263  b22a7bd4bd98a5bc
      0x00f0:  3662 3763 3335 3466 6135 625c 222c 205c  6b7c354fa5b\",.\

However, rabbitmq does not have any information for this queue

rabbitmqadmin -H 192.168.204.2 list queues name
| reply_09e9cf08bfd84f8f903b527080ac28fc                           |
| reply_1d7e7b12580e46109c5a01e54b5cfbbf                           |
| reply_215c6a00191f480daafa9daa18949700                           |
| reply_22f18026daad41219463ce64fae3d33f                           |
| reply_2d2486e8b9264f8a98086ec59ddb24a8                           |
| reply_2fc4fac0fa3d4345a2bd9d6313694a97                           |
| reply_3022cb2f5cae45de897fac4a6bf6ac45                           |
| reply_33636e6f24bd429a8f91776b493c8c8f                           |
| reply_42fd8c79cb3440c4b35134b973dba709                           |
| reply_484bf7df103a44a6ba0860198eb642e8                           |
| reply_515d88f6ebdc4c3cb1e46064b900cd56                           |
| reply_63f15fb7c91d4bd7b6803e85fd61788a                           |
| reply_6716118331874dc4a5aa65b001546647                           |
| reply_67382c459e4849c28d80a02a4c9e3086                           |
| reply_690f93f4379d49a9bedef3a872b02f5b                           |
| reply_6b665fbbd5c3487f9d829e4f461c1ab3                           |
| reply_6c9aa49582f347d28fa3c928a71083e9                           |
| reply_6f4dcbf71d864038872177514633387f                           |
| reply_81f3d1b37c7e4cdfb064c8b28321e4c1                           |
| reply_84391d9200b34b64bede108723decb13                           |
| reply_9536e8be4ca4440ea894486814b60f6d                           |
| reply_96ab63f2ba0a4fe69adabbd70e974dfb                           |
| reply_9ebe0711815a4808951f78b2a1b2cd10                           |
| reply_b18ed1503d7943ed90390ddd2392244a                           |
| reply_ba6a767f92554a6ab9cc532d77d50d33                           |
| reply_be244933bfba4ec58b922c549f2ebdc8                           |
| reply_be79b68595394fb886bfc2109a925c5c                           |
| reply_e6cc599265e240f0a95164c6ecf48989                           |
| reply_ef1dbac27378427584820be544080fd1                           |

What's the best way for me to figure out how it got into this state?

Thanks,
Chris