[rabbitmq-discuss] Publisher throttled way too early
simon at rabbitmq.com
Mon Apr 16 11:36:24 BST 2012
On 13/04/12 21:23, François Beausoleil wrote:
> Hi all!
> My publisher is barely publishing 200 messages per second,
> with another 600 coming from the rest of the system. In total, I
> haven't breached 900 messages/second.
> As you can see, my publisher is essentially always blocked.
Well, the blocking is flicking on and off rapidly, so it's essentially
rate-limited rather than blocked. But still.
2.7.1 and earlier would accept message as fast as they could be read off
the socket, indefinitely. Hence your ballooning memory use. 2.8.1 will
only accept messages as fast as they can be handled. It's not
inconceivable that this code is buggy, but it's fairly simple and has
received quite a bit of testing, so I suspect it's OK.
So therefore I think that your system can really only handle that many
messages per second. Are they persistent, or are your queues big enough
that non-persistent messages page out? That sort of performance sounds
more like the sort of thing you would see when IO-bound.
> What I'm trying to achieve is replay history from a set of archive
> files. I run an event-sourced system, and I found the need to replay
> my history. A 3.5 GiB file is currently planned to finish in 6h, and
> I have more than 300 3.5 GiB files to replay.
If you need to absorb spikes in message rate then you can increase the
size of the OS-supplied TCP buffers, but it sounds like you really need
> I'm using Akka 1.1.3, RabbitMQ 2.8.1, Scala 22.214.171.124, OpenJDK-6,
> Ubuntu 11.10, all on Rackspace's virtual machines.
> Did I forget something somewhere? 200 messages per second seems very
> paltry compared to the 2000 I had on 2.7.
It depends - if 2.8.1 is putting 200/900 msg/s on disc, and 2.7.1 is
buffering 2000 msg/s in memory, then it's not too surprising.
More information about the rabbitmq-discuss