[rabbitmq-discuss] Java client Channel.Close blocking indefinitely

[Iain Hull]

Hi Matthias,

On 01/09/11 09:06, Matthias Radestock wrote:
> You have 28323 Erlang processes. That suggests you have lots of
channels 
> or lots of queues. Each of these has a memory footprint. If your app 
> really needs that many channels/queues then you need to run rabbit on
a 
> machine with more memory, or set up a cluster to divide the 
> channels/queues between multiple nodes.
My test is designed to add more queues until Rabbit stops working, at
this stage it has created 32,000 there are 16 threads with a Channel
each trying to send and receive messages to a section of these queues at
any one time until all queues have been sent to.

The purpose of my testing is to discover the upper limits of RabbitMQ
and what happens when they are breached.  Now that I know that channels
add to the memory footprint of the server I will have to modify my test.


Our proposed production environment will have two rabbit servers, with a
complete set of queues on each. The client code handles failover and
load balancing (we could not wait for RabbitMQ HA).  Producers select a
rabbit server at random and use a single channel to talk to a group of
queues.  Consumers receive a messages from all queues on all rabbit
server, and uses a different channel for each.  The consumers will be
running in a clustered ESB so there could be multiple consumer nodes
(e.g. 2 or 4).  We are using lots of queues, the application is
multi-tenanted and each tenant must use its own queue to ensure fairness
between tenants.  We expect most of these queues to be quite most of the
time with bursts of activity on a small number of tenants at a time, but
one very busy tenant cannot be allowed to starve another.

Thus to support 32 logical queues for 1000 tenants I will require 32000
RabbitMQ queues on two servers.  If there are 4 ESB nodes there will be
at least 32000 * 4 channels on each server.  The production servers will
be a lot larger than my laptop, probably 24Gb of ram, and I will start
testing with these once they arrive.

My current goal it to discover the parameters that contribute to maxing
out the server.  I will now reduce the number of queues and increase the
channels to find the new breaking point, until the hardware arrives.

On 01/09/11 09:06, Matthias Radestock wrote:
> [as an aside, it is curious that the limit for the erlang processes is

> reported as 32k - the default rabbit config sets it to 1M; have you 
> changed the config / start up scripts?]
That is strange I am currently running 2.6.0 on windows without a config
file.  I also have 8 Gb of ram but I am running 32 bit erlang, so maybe
that explains the low 819.2MB high watermark.  Production will running
on CentOS with 24 Gb or 32 Gb of ram and I expect to increase the
vm_memory_high_watermark to use most of this.

Regards,
Iain.