[rabbitmq-discuss] RabbitMQ scaling with OpenStack

[Ask Solem]

On Feb 5, 2013, at 5:50 PM, Ray Pekowski <pekowski at gmail.com> wrote:

> Ask Solem-3 wrote
>>> On Jan 22, 2013, at 1:58 PM, Ask Solem 
>> <ask at rabbitmq.com>
>> wrote:
>> 
>> I've spent some time reviewing the code, but it's really hard to follow,
>> and it's
>> reinventing a lot of things that Kombu already supports.   Having two sets
>> of
>> implementations means that we both have to debug and maintain it, this
>> code could
>> definitely be reduced by a lot.
> 
> Thank you for taking the time to review the code.  Yes, it is confusing and
> perhaps impossible to fully understand without also reading amqp.py
> (https://github.com/openstack/nova/blob/master/nova/openstack/common/rpc/amqp.py),
> which is a common module between the kombu and Qpid implementations for
> OpenStack.

Yeah, but it seems that you are re-implementing things already in Kombu on top of it,
which maybe could have been moved into the qpid implementation.

E.g. the reconnecting at failover, there could be improvements added to kombu
that you would not be able to take advantage of.


> 
> 
> Ask Solem-3 wrote
>> One thing that I'm unable to understand is whether it's creating a channel
>> or not
>> (other than the Connection.default_channel), and if so if it creates a
>> channel for
>> every request.  This would be bad since creating channels is very
>> expensive (at least in amqplib/py-amqp).
> 
> It does not create a new channel on each request.  The channel object is
> held in a impl_kombu.Connection class instance and instances of
> impl_kombu.Connection are pooled.  You can find the pool implementation in
> amqp.py that I mentioned earlier.

Right, I saw the pool, but it was not obvious how channels are handled.
Kombu also comes with a connection pool, and you can pass a connection
to any method requiring a channel and it will use a 'default channel' that is closed
with the connection.
> 
> 
> Ask Solem-3 wrote
>> Also, it seems that the reply messages are persistent, do they have to be?
> 
> The reply messages are RPC replies are implemented in the DirectPublisher
> class which has {'durable': False} for its exchange and queue options.  The
> RPC call is done via the TopicPublisher class and it has the durable option
> that defaults to False, but can be overridden using a OpenStack nova.conf
> file configuration setting.

But durable is not the same as being persistent, which depends on the delivery_mode
property of the message.  Persistent messages are default in Kombu, but you can
change that by using .publish(delivery_mode=1)
> 
> BTW, one of the questions that has come up is whether it makes sense for the
> RPC reply queues to mirrored.  Is HA generally thought of as important for
> RPC replies?  I just am not sure.  Our current implementation has mirroring
> turned off for reply queues, but we are trying to figure out if that was
> just an oversight.
> 

If the RPC replies must be available from multiple processes (e.g. any process can
request the reply for a request by id) then I think you should consider storing the
replies in a database instead.

If not I recommend that you use one queue per client for RPC replies, this will
minimize the number of queues used for replies, and optimally they should
be non-persistent and not mirrored, but that depends on the application
(openstack could leave this configurable for those who need it)