[rabbitmq-discuss] RabbitMQ - strange synchronization behavior

[Simon MacMullen]

Please keep rabbitmq-discuss on CC.

I have just synced 2M x 512 byte messages in 50 seconds on my 
workstation with paging disabled. So the sync mechanism itself is not 
necessarily slow.

But that's with a) a fast network b) all messages in RAM on both nodes. 
Once the queue is big enough that it's paged out, synchronisation slows 
right down. On a slower machine (2010 MBP), the same test took 9 min 36 
sec with much paging.

Obviously paging will slow things down, although it does look like the 
slowdown is disproportionate (it took 3min to load the messages in the 
first place, which also required paging). So there might be room for 
improvement there.

But 7 hours is way too much! Are you sure the messages were 512 bytes? 
Were there any unacknowledged messages? Is your disk particularly slow?

Cheers, Simon

On 26/09/2013 9:55AM, Peter Sokolowski wrote:
> I have made a few tests [the same hardware/software configuration as earlier]:
> a) synchronization of 200.000 messages: takes about 1 minute
> b) synchronization of 1.000.000 messages: takes about 1 hour
> c) synchronization of 2.000.000 messages: takes about 7 hours
>
> During these tests I observed CPU utilization about 100% (on restarted
> node[Slave], the other[Master] had only 1-2%)
>
> I don't understand why synchronization of 2.000.000 takes so long
> [each message=512 bytes]? Is this behavior normal too? Is there any
> chance to speed up synchronization?
>
> 2013/9/26, Simon MacMullen <simon at rabbitmq.com>:
>> On 25/09/2013 9:34PM, Peter Sokolowski wrote:
>>> When I restart one of the nodes(by "stop_app" and "start_app" from the
>>> commandline)  the whole cluster is blocked. It doesn't accept any
>>> publish and receive operations during synchronization stage. When
>>> synchronization is finished everything get back to normal - cluster
>>> accepts publish and receive operations.
>>> Is this behavior normal?
>>
>> Yes.
>>
>> "If a queue is set to automatically synchronise it will synchronise
>> whenever a new slave joins - becoming unresponsive until it has done so."
>>      -- http://www.rabbitmq.com/ha.html#unsynchronised-slaves
>>
>> While any given queue is (explicitly) syncing it will not do anything
>> else; publishes to it will block. Therefore it's a reasonable idea to
>> only explicitly sync queues that are not busy, if a queue is being
>> actively published to and consumed from then it will become synchronised
>> after a while anyway, explicit syncing is not needed. We envisaged
>> explicit syncing being used more for queues that are not very active.
>>
>> And yes, it would be nice if explicit syncing could happen in the
>> background while other things happened. Let's just say that it would be
>> complicated to do that...
>>
>> Cheers, Simon
>>
>> --
>> Simon MacMullen
>> RabbitMQ, Pivotal
>>

-- 
Simon MacMullen
RabbitMQ, Pivotal