[rabbitmq-discuss] Memory not flushing to disk

[Rodrian, Logan P (IS)]

I am having an issue where I am pushing data to queues and after some point, a particular queue stop batching the in-memory data to disk, it appears.  In the included data below, the queue in question is queue E.  As seen from the backing_queue_status, queue E has {ram_msg_count,396922} and {persistent_count,2572735}.  In the case of queue C, which contains more messages, none of them are residing in memory.

I have a rabbitmq.config as follows:
[
  {rabbit, [ {hipe_compile, false},
             {collect_statistics_interval, 10000},
             {disk_free_limit, 50000000},
             {tcp_listeners, [5678]}
            ] },
  {rabbitmq_management_agent, [ {force_fine_statistics, false}
                              ] }
].

It appears that at some point the flushing stops and then once the vm_memory_high_watermark is hit, I get a message informing me.

My question is two-fold:
  1) Has this been seen before/is there a solution already out there?
  2) Is there a configuration parameter that tells rabbitmq to write messages immediately to disk instead of buffering to memory first?




DATA

Queues on /:
name    durable  msg_ready    msg_unackd   messages    consumers    active_consumers    memory
A       true     962          0            962         0            0                   230848
B       true     0            0            0           1            1                   22240
C       true     7688286      0            7688286     0            0                   973696
D       true     3847206      0            3847206     0            0                   4120112
E       true     2572735      0            2572735     0            0                   367917336
F       true     9334         0            9334        0            0                   4120112


backing_queue_status
A
[{q1,0}, {q2,0}, {delta,{delta,0,0,0}}, {q3,962}, {q4,0}, {len,962}, {pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {next_seq_id,962}, {persistent_count,0}, {avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, {avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}]

B
[{q1,0}, {q2,0}, {delta,{delta,0,0,0}}, {q3,0}, {q4,0}, {len,0}, {pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {next_seq_id,2}, {persistent_count,0}, {avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, {avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}]

C
[{q1,0}, {q2,15}, {delta,{delta,16384,7688256,7704640}}, {q3,15}, {q4,0}, {len,7688286}, {pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {next_seq_id,7704655}, {persistent_count,7688286}, {avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, {avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}]

D
[{q1,0}, {q2,38}, {delta,{delta,16384,3830784,3847168}}, {q3,16384}, {q4,0}, {len,3847206}, {pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {next_seq_id,3847206}, {persistent_count,0}, {avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, {avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}]

E
[{q1,0}, {q2,45}, {delta,{delta,1671936,2138780,3810716}}, {q3,36988}, {q4,396922}, {len,2572735}, {pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,396922}, {ram_ack_count,0}, {next_seq_id,3810761}, {persistent_count,2572735}, {avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, {avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}]

F
[{q1,0}, {q2,0}, {delta,{delta,undefined,0,undefined}}, {q3,9334}, {q4,0}, {len,9334}, {pending_acks,0}, {target_ram_count,infinity}, {ram_msg_count,0}, {ram_ack_count,0}, {next_seq_id,17426}, {persistent_count,9334}, {avg_ingress_rate,0.0}, {avg_egress_rate,0.0}, {avg_ack_ingress_rate,0.0}, {avg_ack_egress_rate,0.0}]



Logan Rodrian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130422/a2553631/attachment.htm>