[rabbitmq-discuss] Queue disappears during high volume federation
Matt Pietrek
mpietrek at skytap.com
Tue Aug 14 01:25:42 BST 2012
I'm hitting an issue where a queue disappears from underneath me in a high
volume federated scenario.
Background: I'm on RabbitMQ 2.8.5 on top of Ubuntu 10.04.
I have three brokers bi-directionally federated similarly to the "Small
complete Graph" example in http://www.rabbitmq.com/federation.html. My
exchange is called "Skytap", and is a topic exchange. One of the routing
key 'words' directs messages to the appropriate broker.
In my example, I have a single master sending "requests" to two slaves
which reply back to the master, simulating an RPC scenario. A client app on
the 'master' broker reads from an exclusive queue called 'master", while
client apps on the slave brokers read from an exclusive queue called
"slave'.
In the low volume case, everything works as expected: A steady stream of
request messages seen on the slaves and a steady stream of reply messages
seen on the master.
However, if I write "request" messages as fast as possible, in short order
I see the 'master' queue disappear. (I know this because I have mnesia
logging turned on.)
Here's the mnesia trace from the master client app startup:
--------
Mnesia(rabbit at mq1): write performed by {tid,16820,<0.154.0>} on record:
{rabbit_queue,{resource,<<"/">>,queue,<<"master">>},
false,false,<0.3392.0>,[],<0.3398.0>,[],undefined}
Mnesia(rabbit at mq1): write performed by {tid,16820,<0.154.0>} on record:
{rabbit_route,{binding,{resource,<<"/">>,exchange,<<>>},
<<"master">>,
{resource,<<"/">>,queue,<<"master">>},
[]},
const}
Mnesia(rabbit at mq1): write performed by {tid,16820,<0.154.0>} on record:
{rabbit_reverse_route,
{reverse_binding,
{resource,<<"/">>,queue,<<"master">>},
<<"master">>,
{resource,<<"/">>,exchange,<<>>},
[]},
const}
Mnesia(rabbit at mq1): write performed by {tid,16821,<0.153.0>} on record:
{rabbit_topic_trie_node,
{trie_node,
{resource,<<"/">>,exchange,<<"skytap">>},
<<167,162,233,102,38,77,235,139,202,148,84,98,202,70,145,209>>},
0,3}
Mnesia(rabbit at mq1): write performed by {tid,16821,<0.153.0>} on record:
{rabbit_exchange_serial,{resource,<<"/">>,exchange,<<"skytap">>},16}
Mnesia(rabbit at mq1): write performed by {tid,16821,<0.153.0>} on record:
{rabbit_reverse_route,
{reverse_binding,
{resource,<<"/">>,queue,<<"master">>},
<<"master.mgt">>,
{resource,<<"/">>,exchange,<<"skytap">>},
[]},
const}
Mnesia(rabbit at mq1): write performed by {tid,16821,<0.153.0>} on record:
{rabbit_topic_trie_binding,
{trie_binding,
{resource,<<"/">>,exchange,<<"skytap">>},
<<167,162,233,102,38,77,235,139,202,148,84,98,202,70,145,209>>,
{resource,<<"/">>,queue,<<"master">>}},
const}
Mnesia(rabbit at mq1): write performed by {tid,16821,<0.153.0>} on record:
{rabbit_route,{binding,{resource,<<"/">>,exchange,<<"skytap">>},
<<"master.mgt">>,
{resource,<<"/">>,queue,<<"master">>},
[]},
const}
--------
And then some time later, after a bunch of successful requests/replies, I
get this:
--------
Mnesia(rabbit at mq1): write performed by {tid,16822,<0.154.0>} on record:
{rabbit_topic_trie_node,
{trie_node,
{resource,<<"/">>,exchange,<<"skytap">>},
<<167,162,233,102,38,77,235,139,202,148,84,98,202,70,145,209>>},
0,2}
Mnesia(rabbit at mq1): delete_object performed by {tid,16822,<0.154.0>} on
record:
{rabbit_semi_durable_route,
{binding,
{resource,<<"/">>,exchange,<<>>},
<<"master">>,
{resource,<<"/">>,queue,<<"master">>},
[]},
const}
Mnesia(rabbit at mq1): write performed by {tid,16822,<0.154.0>} on record:
{rabbit_exchange_serial,{resource,<<"/">>,exchange,<<"skytap">>},17}
Mnesia(rabbit at mq1): delete performed by {tid,16822,<0.154.0>} on record:
{rabbit_queue,{resource,<<"/">>,queue,<<"master">>}}
Mnesia(rabbit at mq1): delete_object performed by {tid,16822,<0.154.0>} on
record:
{rabbit_reverse_route,
{reverse_binding,
{resource,<<"/">>,queue,<<"master">>},
<<"master.mgt">>,
{resource,<<"/">>,exchange,<<"skytap">>},
[]},
const}
Mnesia(rabbit at mq1): delete_object performed by {tid,16822,<0.154.0>} on
record:
{rabbit_topic_trie_binding,
{trie_binding,
{resource,<<"/">>,exchange,<<"skytap">>},
<<167,162,233,102,38,77,235,139,202,148,84,98,202,70,145,209>>,
{resource,<<"/">>,queue,<<"master">>}},
const}
Mnesia(rabbit at mq1): delete_object performed by {tid,16822,<0.154.0>} on
record:
{rabbit_semi_durable_route,
{binding,
{resource,<<"/">>,exchange,<<"skytap">>},
<<"master.mgt">>,
{resource,<<"/">>,queue,<<"master">>},
[]},
const}
Mnesia(rabbit at mq1): delete_object performed by {tid,16822,<0.154.0>} on
record:
{rabbit_route,{binding,{resource,<<"/">>,exchange,<<>>},
<<"master">>,
{resource,<<"/">>,queue,<<"master">>},
[]},
const}
Mnesia(rabbit at mq1): delete_object performed by {tid,16822,<0.154.0>} on
record:
{rabbit_route,{binding,{resource,<<"/">>,exchange,<<"skytap">>},
<<"master.mgt">>,
{resource,<<"/">>,queue,<<"master">>},
[]},
const}
Mnesia(rabbit at mq1): delete_object performed by {tid,16822,<0.154.0>} on
record:
{rabbit_reverse_route,
{reverse_binding,
{resource,<<"/">>,queue,<<"master">>},
<<"master">>,
{resource,<<"/">>,exchange,<<>>},
[]},
const}
Mnesia(rabbit at mq1): delete_object performed by {tid,16822,<0.154.0>} on
record:
{rabbit_durable_route,
{binding,
{resource,<<"/">>,exchange,<<"skytap">>},
<<"master.mgt">>,
{resource,<<"/">>,queue,<<"master">>},
[]},
const}
Mnesia(rabbit at mq1): delete_object performed by {tid,16822,<0.154.0>} on
record:
{rabbit_durable_route,
{binding,
{resource,<<"/">>,exchange,<<>>},
<<"master">>,
{resource,<<"/">>,queue,<<"master">>},
[]},
const}
Mnesia(rabbit at mq1): delete performed by {tid,16822,<0.154.0>} on record:
{rabbit_durable_queue,{resource,<<"/">>,queue,<<"master">>}}
--------
The rabbit at mq1.log file shows nothing of any interest. Just the expected
AMQP connection accepting/closing.
Anything else useful I can provide?
Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120813/bf2e6449/attachment.htm>
More information about the rabbitmq-discuss
mailing list