[rabbitmq-discuss] Problems with rabbit and hoping I can get some help

[Dathan Pattishall]

CPU is rather Idle,
6% CPU system on average
8% CPU User on average

No CPU wait IO


so plenty of CPU / plenty of Memory


On Tue, Jan 31, 2012 at 1:13 PM, Jerry Kuch <jerryk at vmware.com> wrote:

> Hi, Dathan...
>
> For dropping messages, you might consider setting message TTLs, but that
> may not give you quite what you want in all cases.
>
> What does the CPU consumption of your Rabbit node look like when you're
> seeing these pauses?  If you wait, do they relent, with things getting
> moving again?
>
> Best regards,
> Jerry
>
> ----- Original Message -----
> From: "Dathan Pattishall" <dathan at schoolfeed.com>
> To: "Jerry Kuch" <jerryk at vmware.com>
> Cc: rabbitmq-discuss at lists.rabbitmq.com
> Sent: Tuesday, January 31, 2012 1:05:39 PM
> Subject: Re: [rabbitmq-discuss] Problems with rabbit and hoping I can get
> some help
>
> Hi Jerry,
>
>
> I neglected to mentioned that I am not hitting the memory-based flow
> control according to my memory alarm stat from
> rabbitmqadmin.py list nodes
>
>
> What threshold would the TCP back pressure logic hit? I assume when the
> memory limit is reached rabbit pushes back on the publishers? Rabbit did
> not use more then 2G of RAM out of the 5G allowed for it, if that is the
> case.
>
>
> Also is there a way to tell rabbit drop the messages instead of block?
>
>
>
> On Tue, Jan 31, 2012 at 12:56 PM, Jerry Kuch < jerryk at vmware.com > wrote:
>
>
> Hi, Dathan...
>
> What are your consumers doing with the published messages? If you use
> rabbitmqctl or
> the management plugin to look at what's going on in your queues, do you
> see messages
> accumulating but not being delivered? Or delivered but not ACKed? If
> messages are
> building up (either undelivered or unACKed) faster than consumers are
> draining them,
> you might be hitting memory-based flow control, which will use TCP back
> pressure to
> stop the publishers.
>
> See here for more information:
>
> http://www.rabbitmq.com/memory.html
>
> To get an idea whether this is happening to you, check out your queue
> contents as
> suggested above, and see if memory alarms are being set in your rabbit
> logs.../
>
> Best regards,
> Jerry
>
>
> ----- Original Message -----
> From: "Dathan Pattishall" < dathan at schoolfeed.com >
> To: rabbitmq-discuss at lists.rabbitmq.com
> Sent: Tuesday, January 31, 2012 12:52:55 PM
> Subject: [rabbitmq-discuss] Problems with rabbit and hoping I can get some
> help
>
>
>
>
> Let me first describe my setup.
>
> root at webnode1]# rabbitmqadmin.py show overview
>
> +--------------------+-----------------+--------------------+------------------+
> | management_version | node | statistics_db_node | statistics_level |
>
> +--------------------+-----------------+--------------------+------------------+
> | 2.7.1 | rabbit at webnode1 | rabbit at webnode1 | fine |
>
> +--------------------+-----------------+--------------------+------------------+
>
>
>
> Rabbit MQ's producers comes from PHP 5.3.8
> http://www.php.net/manual/en/book.amqp.php . Each apache process could
> produce a rabbit message, I am producing around 1000 messages a second on
> c1.xtralarge instance at ec2.
>
>
>
> My erlang version is
>
>
> /usr/local/bin/erl -v
> Erlang R15B (erts-5.9) [source] [64-bit] [smp:8:8] [async-threads:0]
> [hipe] [kernel-poll:false]
>
>
>
>
> The PROBLEM:
>
>
> After about 40 mins of rabbit accepting messages all connections block
> causing a rather bad error on the front ends killing traffic. Turning
> rabbit off and restarting the web servers forces a recovery.
>
>
> Stats from Rabbit:
>
>
> Roughly 5000 queues are made
> Roughly 3600 exchanges are made
> Each exchange can have at most 1200 queues bound to it.
> Each Queue is setup for autodelete and so is the exchanges with delivery
> type 1.
> All data passed is JSON
>
>
> The consumer is NODE and its keeping up with the consumption
>
>
> RabbitMQ memlimit is around 5.3G
> RabbitMQ mem used hits around 1.9G when it freezes produces
> RabbitMQ proc used hits around 220K
> RabbitMQ fd_total is 50K
> RabbitMQ socks_total is around 45K and Socks used is 4K
> mem_ets hists 100M // not sure what this is
>
>
> Any idea what is going on? What limit am I hitting? Why does RabbitMQ
> block? How can I detect that I am about to hit a block state? Any
> suggestions or request of additional data would be great.
>
>
>
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120131/34a924e9/attachment.htm>