[rabbitmq-discuss] rabbitmq gets high load

[Tim Watson]

Hey Doug

On 18 Jan 2013, at 03:42, Doug <doug at hacks.perl.sh> wrote:
> 
> This is the info in sasl log (many of these):
> 

Do they all look the same (talking about channel_sup_sup and chanel_sup with the same errors?

> =SUPERVISOR REPORT==== 17-Jan-2013::17:00:31 ===
>     Supervisor: {<0.31128.1705>,
>                                           rabbit_channel_sup_sup}
>     Context:    shutdown_error
>     Reason:     noproc
>     Offender:   [{pid,<0.31130.1705>},
>                  {name,channel_sup},
>                  {mfa,{rabbit_channel_sup,start_link,[]}},
>                  {restart_type,temporary},
>                  {shutdown,infinity},
>                  {child_type,supervisor}]
> 
> 
> =SUPERVISOR REPORT==== 17-Jan-2013::17:02:41 ===
>     Supervisor: {<0.30250.1707>,
>                                           rabbit_channel_sup_sup}
>     Context:    shutdown_error
>     Reason:     shutdown
>     Offender:   [{pid,<0.30252.1707>},
>                  {name,channel_sup},
>                  {mfa,{rabbit_channel_sup,start_link,[]}},
>                  {restart_type,temporary},
>                  {shutdown,infinity},
>                  {child_type,supervisor}]
> 
> 
> And this is the status report:
> 
> Status of node 'testmq-master at ubuntu' ...
> [{pid,17994},
> {running_applications,[{rabbit,"RabbitMQ","2.8.2"},
>                        {os_mon,"CPO  CXC 138 46","2.2.9"},
>                        {sasl,"SASL  CXC 138 11","2.2.1"},
>                        {mnesia,"MNESIA  CXC 138 12","4.7"},
>                        {stdlib,"ERTS  CXC 138 10","1.18.1"},
>                        {kernel,"ERTS  CXC 138 10","2.15.1"}]},
> {os,{unix,linux}},
> {erlang_version,"Erlang R15B01 (erts-5.9.1) [source] [64-bit]
> [smp:8:8] [async-threads:30] [kernel-poll:true]\n"},
> {memory,[{total,394953040},
>          {processes,313197287},
>          {processes_used,313056817},
>          {system,81755753},
>          {atom,1788105},
>          {atom_used,1770587},
>          {binary,16938376},
>          {code,11936253},
>          {ets,5830120}]},
> {vm_memory_high_watermark,0.6},
> {vm_memory_limit,10114781184},
> {disk_free_limit,16857968640},
> {disk_free,206313144320},
> {file_descriptors,[{total_limit,65435},
>                    {total_used,151},
>                    {sockets_limit,58889},
>                    {sockets_used,137}]},
> {processes,[{limit,1048576},{used,2867}]},
> {run_queue,13},
> {uptime,8705279}]
> ...done.
> 
> 
> 
> 2013/1/17 Tim Watson <tim at rabbitmq.com>:
>> Hi Doug
>> 
>> On 17 Jan 2013, at 04:29, Doug wrote:
>> 
>>> PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>> 17994 rabbitmq  20   0 11.6g  10g 1764 S  367 66.2 416834:07 beam.smp
>>> 
>>> As you see, rabbitmq process has lots of system load (367% CPU, 66.2% memory).
>> 
>> Certainly looks like it. That level of CPU utilisation is a bit surprising.
>> 
>>> It even can't list the queue now:
>>> 
>>> root at ubuntu:~# rabbitmqctl -p /pay list_queues
>>> Listing queues ...
>>> ^C
>>> BREAK: (a)bort (c)ontinue (p)roc info (i)nfo (l)oaded
>>>      (v)ersion (k)ill (D)b-tables (d)istribution
>>> 
>>> 
>>> So how to handle this case? Thanks.
>> 
>> You've not given us much to go on here. Normally I'd ask you to run the output of `rabbitmqctl report` but that's likely to lock up just as list_queues is doing. Can you post the regular as sasl rabbit logs please? If they're too bit to post, a /paste/ url will do fine and of course you may wish to scrub the logs to remove any private information if that's relevant.
>> 
>> Also what kind of /usage/ is this rabbit subject to at the time? Is it serving a large number of producers, consumers, both? Have you looked at any stats around the time this starts happening in order to see what the broker is up to?
>> 
>> Cheers,
>> Tim
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss at lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss