[rabbitmq-discuss] rabbit MQ high-through put stock quotes

[Matthias Radestock]

Mark,

smittycb10 wrote:
> My idea for a future system is to have the same server simply push the
> messages to a rabbit MQ Broker or Brokers and then have all clients simply
> listen to whatever queues they need to get there jobs done. I need to design
> for 100,000 (one hundred thousand) messages to made available for consuming
> per second. Most consumers will only consume a fraction of these messages,
> although some of our server pieces may need to subscribe to the whole
> universe of incoming messages. Messages can and will be batched on the
> publishing side.

I recommend treating the "all messages to a few consumers" and "each
message to a small subset of lots of consumers" cases separately.

If you don't mind, I'd like to focus on the first part, since that one
is easier to set up. Once you have that working we can then move on to
the second part.

To get a 100KHz stream of messages distributed to a handful of
consumers, I'd suggest batching them based on time, e.g. 1 millisecond's
worth of messages, and send each batch as a single message to a fanout
exchange to which all consumers have bound a queue.

On the consuming side, use a no-ack consumer (and, consequently, do not
explicitly acknowledge the messages) for best performance.

Also, let me just re-iterate that for meaningful benchmark results it is
important to throttle the producer, e.g. send messages at a fixed rate.
If you do the time-based batching above you'll have good control over
that. Then monitor the rate at which messages are received by consumers,
and slowly increase the producing rate until the consumption rate starts
falling behind, at which point you have found the limits of that
particular setup.

Also, make sure you run the producers and consumers on a different
machine to the broker, and monitor their CPU usage in case they are
maxing out the CPU (and thus skew the results).


Regards,

Matthias.