[rabbitmq-discuss] Is RabbitMQ indeed "hype ware"?

[Matthew Sackman]

On Sat, Aug 29, 2009 at 03:00:34PM -0600, Jason J. W. Williams wrote:
> Also, if you have to queue to disk to keep from keeling over, there's  
> something very inappropriate about the sizing of your setup. Not to  
> say disk queuing doesn't have a place, but if you're so overloaded  
> that it's required you're not that long from AMQ seizing up too due to  
> inability to service requests because of disk latency.

This raises some interesting points.

A disk-backed queue is always going to be slower than a queue held
solely in memory. Unless you substantially underclock your RAM.

If you have a low-rate queue which you want to run such that messages
in it are stored on disk so that they can be recovered in the event of
failure, then that's one good use case of a disk queue.

If you have a very bursty, frequently idle, but sometimes very busy
producer and bunch of consumers that can consume at a steady rate, then
this is potentially another good use - the disk queue will ensure that
messages don't need to be held in memory (coming soon in Rabbit 1.7!),
so the broker won't explode (provided the burst isn't so big that it
runs out of RAM before messages can be written out to disk), and can
deliver messages to the consumers as they are able to handle them.
Obviously, whether this works depends on just how bursty the producers
are, how much data they try to flood the broker with and the rate the
broker (and consumers) can handle that data at.

But the simple fact is that if you have a producer which is pumping out
a steady stream of above 70MB/s, you're beyond most hard discs at that
point so you need to start looking at RAID 0 or other more exotic
solutions. At the end of the day, no matter how smart the programmers
are, if the bottleneck is your harddisc, the broker *won't keep up*.
Also note that writing messages out to disk is a substantially
sequential activity.

At this point, mechanics of hard discs and the cost of operating system
calls read their heads. Fancy SSDs *still* only write sequentially at up
to 70MB/s and in practise, this rate drops off once the drive has been
filled once over. High quality rotating media has higher sequential
write rates. (That said, SSD optimisation seems to be a trade off
between sequential write rate, and random write latency. There are some
drives that can write sequentially very quickly, but are very very slow
at small random writes).

It's no surprise that the easiest way of saturating a disk is to write
in page-size units to it. If your messages happen to be that size, then
good for you. If your messages are very small, then the cost of
thousands of OS calls to write small messages is going to be your
bottleneck and you will struggle to saturate disks (yes, batching writes
helps here, but eg if you're doing transactions and are really unlucky,
each transaction commit /can/ result in an fsync. This is not going to be
fast).

Matthew