[rabbitmq-discuss] Per-Connection Flow Control - RMQ 2.8.1

[DawgTool]

Hi Simon,

The Flow-Control in 2.8.1 has been a huge failure for my UAT systems.
The blocking is completely takes over the system, in some cases it takes 
several minutes before RabbitMQ is able to process my disconnected channels.

I really need some guidance, my issues so far; (messages are non-persistent)
Flow-Control kicks in heavy around 3k msq/sec and is almost unusable at 
5k msg/sec
When using TTLs on queues, when memory exceeds 1gb for a queue, blocking 
start taking seconds while purging the queue.
When using TTLs on queues, memory above 1gb per queue and mirroring 
(ha:true) blocking completely locks up the system and take several 
minutes to recover.

None of my machines are running hot, average CPU is about 90 to 110% 
(smp:enable), no io wait issues, network is underutilized, memory less 
then 10% consumed.

last last_ state recv_oct recv_cnt   sendsend send
blocked blocked                                        oct cnt pend
by age
flow    28.036103    blocked    114346544    199970    384    4    0
flow    28.04571     blocked    114310503    200332    384    4    0
flow    28.033909    blocked    114317988    199906    384    4    0
flow    28.034395    blocked    114245740    199822    384    4    0
flow    28.046772    blocked    114198573    199882    384    4    0
flow    28.034031    blocked    114254993    200164    384    4    0
flow    28.034131    blocked     30595831     40077    384    4    0
flow    28.033161    blocked     30580062     40261    384    4    0
flow    28.035666    blocked     30605155     40355    384    4    0
flow    28.035648    blocked     30638405     40347    384    4    0
flow    28.394708    blocked     30560023     40286    384    4    0
flow    28.04654     blocked    114268244    200232    384    4    0
flow    28.045263    blocked    114245987    200025    384    4    0
flow    28.035341    blocked    114262211    199989    384    4    0
flow    28.035473    blocked    114319531    200172    384    4    0
flow    28.044786    blocked     30614203     40132    384    4    0
flow    28.401974    blocked     30588179     40063    384    4    0
flow    28.046696    blocked     30624773     40239    384    4    0
flow    28.037019    blocked     30617997     40074    384    4    0
flow    28.381844    blocked     30581067     40084    384    4    0



On 3/26/12 6:25 AM, Simon MacMullen wrote:
> Hi.
>
> There's no way to monitor what the flow control is doing.
>
> I would assume that with 7k msg/s on 2.7.1 you see memory use steadily 
> increasing? The flow control should just be preventing (significant 
> numbers of) messages being queued up anywhere else than in queues. The 
> idea is to ensure that whatever performance RabbitMQ is giving you is 
> sustainable, rather than letting it absorb more messages in the short 
> term at the cost of working itself into a hole that is difficult to 
> get out of.
>
> If you need the ability to absorb spikes in publishing rates then you 
> can increase the size of TCP buffers on the server.
>
> Cheers, Simon
>
> On 23/03/12 20:03, DawgTool wrote:
>> Hi All,
>>
>> Wondering is anyone having issues with the Per-Connection Flow Control
>> in 2.8.1?
>> Under 2.7.1 was able to push about 4.5MB/s per connection (about
>> ~7000msg/s).
>> Under 2.8.1 I am hitting the flow control constantly at about 2.4MB/s
>> per connection.
>> Clean installed 2.8.1 on the same hardware as 2.7.1.
>>
>> Is there someway to configure/monitor what is causing the Flow Control
>> to be tripped?
>>
>> Thanks in Advance.
>>
>>
>>
>> Config: (both 2.7.1/2.8.1)
>> Erlang R15B (erts-5.9) [source] [64-bit] [smp:4:4] [async-threads:30]
>> [hipe] [kernel-poll:true]
>>
>> ==> enabled_plugins <==
>> [rabbitmq_management,rabbitmq_management_agent].
>>
>> ==> rabbitmq.config <==
>> [
>> {rabbit, [{vm_memory_high_watermark, 0.6},
>> {collect_statistics_interval, 5000},
>> {hipe_compile, true}
>> ]
>> },
>> {mnesia, [{dc_dump_limit, 40},
>> {dump_log_write_threshold, 50000},
>> {no_table_loaders, 20},
>> {send_compressed, 9},
>> {snmp, true}
>> ]
>> },
>> {rabbitmq_management, [ {http_log_dir,
>> "/data/rabbitmq/dc001/rabbit-mgmt"} ] },
>> {rabbitmq_management_agent, [ {force_fine_statistics, true} ] }
>> ].
>>
>> ==> rabbitmq-env.conf <==
>> NODENAME=dc001
>> BASE=/data/rabbitmq/dc001
>> MNESIA_BASE=/data/rabbitmq/dc001/mnesia
>> LOG_BASE=/data/rabbitmq/dc001/log
>> SERVER_START_ARGS="+K true -smp enable"
>>
>>
>>
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>