[rabbitmq-discuss] Does SSD improve rabbitmq's performance

[Matthias Radestock]

Graeme,

On 20/11/13 20:21, Graeme N wrote:
> I'll try to generate a synthetic workload on disk based systems and
> graph the change in rabbitmq delivery performance. Certainly we know
> that apps like Riak, with its universal append-only datastore, see a
> much smaller margin of advantage with SSDs over spinning disks
> compared to what we're seeing with RabbitMQ on our disk vs SSD
> nodes. However, I understand what its data store is doing a lot
> better than I understand what RabbitMQ is doing.

There is quite a difference in the nature of the workload for a queue vs
cache vs general store. See
http://www.rabbitmq.com/blog/2011/01/20/rabbitmq-backing-stores-databases-and-disks/ 
for a discussion on that subject.

> - Does RabbitMQ have a compaction step like bitcask or innodb, where
> after writing a certain amount of append-only work, it coalesces this
> to a more efficient on disk format?

Yes, the per-queue persistent storage - which only stores index
information (i.e. what messages are in what in position in a given
queue) has an append-only journal which gets compacted to so-called
segment files. See the essay near the beginning of
http://hg.rabbitmq.com/rabbitmq-server/file/default/src/rabbit_queue_index.erl 
for the gory details.

The main msg_store (where the actual messages are stored) doesn't have 
anything compactable, beyond what's done by the gc (see below).

> - How are deletes handled on queues that don't fully drain? If I
> increase the message store size to 500 MiB, but the queues they
> represent are never fully emptied, will they just grow to max queue
> size and stay there like InnoDB on disk files?

The msg_store has a gc hat has a hard upper bound on the amount of
garbage left around. See the essay near the beginning of
http://hg.rabbitmq.com/rabbitmq-server/file/default/src/rabbit_msg_store.erl 
for the gory details.

Matthias.