[rabbitmq-discuss] What is the currently fastest way to stream messages?

Thu Apr 12 12:11:19 BST 2012

On 12/04/12 07:30, Matt wrote:
> I am working on a project with one producer, one consumer that need to
> pass messages over the localhost (inter-process). I am targeting C#
> .Net4.5(4.0 also fine). I look for a way to published raw byte[] (size
> of each 16 bytes) messages, but I can chunk up the byte[] into larger
> sizes and send.

Chunking would definitely help; there is a certain byte overhead per 
message that is much more than 16 bytes, and there will be some routing 
work per message as well.

> The order of the sent messages must be preserved.

OK. Otherwise I would suggest parallelising.

> is there a way to go brokerless for throughput optimization,

Not with AMQP.

> can amqp message parsing be turned off (I
> read its a major bottleneck and design flaw how the amqp parser works),

I'm not sure what this would mean. We don't parse AMQP message bodies, 
they're just binaries. But we parse the AMQP byte stream, we kind of 
have to.

Although it's a bottleneck I'm not sure I'd call it a flaw. As long as 
there's a protocol we will have to parse it, and IMHO AMQP is not so 
horrible for parsing (opinions may differ).

To remove parsing overhead it may be worth looking at Tony's new UDP 
exchange in raw mode, although (a) I don't know if he's been optimising 
it and (b) it's UDP.

> can I omit queues

If consumers keep up they will be mostly omitted anyway. Messages still 
have to pass through the queue process, but it doesn't enqueue / dequeue 
in this case.

You could set the immediate flag to ensure messages are never enqueued 
*but* since this is defined to send a basic.return whenever a message is 
dropped it has to do some synchronous stuff and is usually slower.

The next release will allow setting a message TTL of 0, which will 
ensure messages are never enqueued.

> will turning off ackq speed things up?

Definitely.

> Basically
> anything than can get my throughput up is highly welcomed.

Other ideas: Enable HiPE (not available on Windows though) to make 
everything faster. Route via a fanout exchange for the cheapest routing 
logic.

But batching messages will help the most I suspect.

As always, the next release will contain some more performance 
enhancements...

> I understand
> I will not be able to reach ZeroMQs throughput rates because its a
> brokerless solution but I like RabbitMQ and its mature enough to run in
> production and this pub/sub pattern is only one of the many other
> message patterns that I use and where throughputs are not critical. But
> this byte array stream is throughput critical and I would appreciate any
> comments how to tune this.

Hopefully this gives you a clue. But RabbitMQ will never get as fast as 
ZeroMQ - ZeroMQ is just so minimal.

Cheers, Simon

-- 
Simon MacMullen
RabbitMQ, VMware