[rabbitmq-discuss] queue/exchange creation/deletion replication serialization

[Ray Pekowski]

Hi RabbitMQ community.  This is my first post.  I have been discussing a
cluster serialization problem with OpenStack RPC's usage of RabbitMQ on the
OpenStack developers mailing list with the generic subject line of
"RabbitMQ Scaling".

The relevant code versions I'm using are:
rabbitmq-server : 2.7.1.Oubuntu4
python-amqplib : 1.0.0+ds-1
python-kombu : 1.4.3-1

I read somewhere that Kombu only uses the 0-8 level protocol, but I'm not
sure how to confirm that.

The OpenStack RPC usage pattern and the problem are shown in the following
table:

No cluster  Cluster of 3   AMQP Action
(in ms)     (in ms)
0.647       1.704          Type: Method (1) Channel: 1 Class: Channel (20)
Method: Open (10)
0.451       0.468          Type: Method (1) Channel: 1 Class: Channel (20)
Method: Open-Ok (11)
1.484       1.459          Type: Method (1) Channel: 1 Class: Exchange (40)
Method: Declare (10) Exchange: XXXXXXXXXX Type: direct
0.431       *2.612*          Type: Method (1) Channel: 1 Class: Exchange
(40) Method: Declare-Ok (11)
0.612       0.612          Type: Method (1) Channel: 1 Class: Queue (50)
Method: Declare (10) Queue: XXXXXXXXXX
0.86        *4.369*          Type: Method (1) Channel: 1 Class: Queue (50)
Method: Declare-Ok (11) Queue: XXXXXXXXXX
0.64        0.723          Type: Method (1) Channel: 1 Class: Queue (50)
Method: Bind (20) Queue: XXXXXXXXXX Exchange: XXXXXXXXXX Routing-Key:
XXXXXXXXXX
0.622       *3.34*          Type: Method (1) Channel: 1 Class: Queue (50)
Method: Bind-Ok (21)
0.758       0.731          Type: Method (1) Channel: 1 Class: Exchange (40)
Method: Declare (10) Exchange: nova Type: topic
0.193       0.194          Type: Method (1) Channel: 1 Class: Exchange (40)
Method: Declare-Ok (11)
0.864       0.886          Type: Method (1) Channel: 1 Class: Basic (60)
Method: Publish (40) Exchange: nova Routing-Key: perfsvc1
0.029       0.034          Type: Content header (2) Channel: 1
0.067       0.068          Type: Method (1) Channel: 1 Class: Basic (60)
Method: Consume (20) Queue: XXXXXXXXXX
0.607       1.186          Type: Method (1) Channel: 1 Class: Basic (60)
Method: Consume-Ok (21)
8.408       9.461          Type: Method (1) Channel: 1 Class: Channel (20)
Method: Close (40)
1.8         *10.47*          Type: Method (1) Channel: 1 Class: Channel
(20) Method: Close-Ok (41)

The numbers in the first and second columns are for the time between the
prior action and the start of the action listed on that row and rows are
listed in real order of occurrence.  The "declares", bind and close are all
significantly longer.  The queues and exchanges are auto delete, so the
close results in the deletion of the queue and exchange.

This is from a wireshark trace of a single producer and a single consumer
from the RabbbitMQ server.  RabbitMQ is taking significantly longer to do
something in the clustered case and it seems likely that "the something" is
most likely the replication of the creation and destruction of the queue
and exchange.

Is it known that the dynamic creation and deletion of queues and exchanges
is costly in a clustered environment?  Is it known that the replication of
queues and exchanges in a cluster environment is a serialization point that
bottlenecks RabbitMQ scalability for this dynamic queue/exchange use
pattern?

Ray
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20121120/55fb91d6/attachment.htm>