[rabbitmq-discuss] What is the currently fastest way to stream messages?
simon at rabbitmq.com
Thu Apr 12 12:11:19 BST 2012
On 12/04/12 07:30, Matt wrote:
> I am working on a project with one producer, one consumer that need to
> pass messages over the localhost (inter-process). I am targeting C#
> .Net4.5(4.0 also fine). I look for a way to published raw byte (size
> of each 16 bytes) messages, but I can chunk up the byte into larger
> sizes and send.
Chunking would definitely help; there is a certain byte overhead per
message that is much more than 16 bytes, and there will be some routing
work per message as well.
> The order of the sent messages must be preserved.
OK. Otherwise I would suggest parallelising.
> is there a way to go brokerless for throughput optimization,
Not with AMQP.
> can amqp message parsing be turned off (I
> read its a major bottleneck and design flaw how the amqp parser works),
I'm not sure what this would mean. We don't parse AMQP message bodies,
they're just binaries. But we parse the AMQP byte stream, we kind of
Although it's a bottleneck I'm not sure I'd call it a flaw. As long as
there's a protocol we will have to parse it, and IMHO AMQP is not so
horrible for parsing (opinions may differ).
To remove parsing overhead it may be worth looking at Tony's new UDP
exchange in raw mode, although (a) I don't know if he's been optimising
it and (b) it's UDP.
> can I omit queues
If consumers keep up they will be mostly omitted anyway. Messages still
have to pass through the queue process, but it doesn't enqueue / dequeue
in this case.
You could set the immediate flag to ensure messages are never enqueued
*but* since this is defined to send a basic.return whenever a message is
dropped it has to do some synchronous stuff and is usually slower.
The next release will allow setting a message TTL of 0, which will
ensure messages are never enqueued.
> will turning off ackq speed things up?
> anything than can get my throughput up is highly welcomed.
Other ideas: Enable HiPE (not available on Windows though) to make
everything faster. Route via a fanout exchange for the cheapest routing
But batching messages will help the most I suspect.
As always, the next release will contain some more performance
> I understand
> I will not be able to reach ZeroMQs throughput rates because its a
> brokerless solution but I like RabbitMQ and its mature enough to run in
> production and this pub/sub pattern is only one of the many other
> message patterns that I use and where throughputs are not critical. But
> this byte array stream is throughput critical and I would appreciate any
> comments how to tune this.
Hopefully this gives you a clue. But RabbitMQ will never get as fast as
ZeroMQ - ZeroMQ is just so minimal.
More information about the rabbitmq-discuss