[rabbitmq-discuss] problem with new persiter with 1000 topics/queues

Sat Aug 14 03:15:42 BST 2010

Alex,

thanks for your problem report ...

alex chen wrote:
> we are testing the new persister (built from last week's default branch).

What's the hg revision id?

And what O/S and version of Erlang are you running this on?

> when there are 1000 queues with backed up messages, starting the
> consumers would cause some queues to disappear.  here are the steps
> to reproduce the problem: [...]

Can you post the code for this test?

> 1) create 1000 queues for 1000 topics.
> 2) publish messages to these topics.  each message size is 20 KB.
> 3) when total number of queued messages bytes reaches 10GB, start 1000 consumers 
> with 1000 processes.

Does that mean you have 1000+ connections?

What is the file descriptor limit for the user account running the 
rabbit server?

> The broker would start having problem when several consumers start consuming 
> messages very fast, which causes other consumers error in login or 
> basic_consume.  then some queues would disappear and the publishers would get 
> no_route errors.

Are there any interesting messages in the rabbit@<node>.log or 
rabbit@<node>-sasl.log? Can you post the contents of these log files? 
(If they are large, please upload them somewhere we can see them.)

Also, please post the output of
   rabbitmqctl list_connections
   rabbitmqctl list_channels
   rabbitmqctl list_exchanges
   rabbitmqctl list_bindings
   rabbitmqctl list_queues

(again, if the output is large please upload it somewhere).

Regards,

Matthias.