[rabbitmq-discuss] problem with new persiter with 1000 topics/queues

[alex chen]

Hi,

we are testing the new persiter (built from last week's default branch). when 
there are 1000 queues with backed up messages, starting the consumers would 
cause some queues to disappear.  here are the steps to reproduce the problem:

1) create 1000 queues for 1000 topics.
2) publish messages to these topics.  each message size is 20 KB.
3) when total number of queued messages bytes reaches 10GB, start 1000 consumers 
with 1000 processes.

The broker would start having problem when several consumers start consuming 
messages very fast, which causes other consumers error in login or 
basic_consume.  then some queues would disappear and the publishers would get 
no_route errors.

there was no problem if the num of topics/queues is less than 100. our machine 
has 8GB mem and 4 disks RAID.  we could get around 50MB/sec publish rate and 30 
MB/sec consume rate.
 
Could someone help to look into this problem? 
thanks.

-alex