[rabbitmq-discuss] rabbitmq-server process consumes 100% of 1 CPU (rabbitmq 2.8.4 on Erlang R14B04)

Anthony Harris anthony at lshift.net
Wed Jul 11 13:48:25 BST 2012


Hi,

We've run into what appears to be the same issue as was reported
14-Jun-2012 in "rabbitmq 2.8.2 beam process consumes 100% of one cpu in
tight epoll_wait loop" with the following system configuration:
* RabbitMQ 2.8.4
* Erlang R14B04

There was no activity on the instance, but there were ~1400 queues left
open since starting (durable queues persisted through reboot).

We observed the following:
* CPU usage immediately jumped to more than 100% CPU utilisation (up to
300% on another affected four-way host)
* responsiveness of rabbitmqctl deteriorated quickly (up to 5 seconds
for a status report)

The output of etop (against the affected node) is listed at the end of
this email.

Whilst we were collecting the data below however, we also found that
etop would emit a huge volume of messages like this:

Erlang top got garbage {trace_ts,<4806.155.0>,out,
                                 {lists,foldl,3},
                                 {1342,6858,451064}}
The output was sufficiently voluminous to create ~40MB of output in a
minute or so.

When we ran strace over the beam.smp process, filtering out calls to
sched_yield, epoll_wait and futex, we noticed that the process seemed to
be continuously opening and closing mnesia journal files correlating to
it's queues (see attached file). To our eyes, this seems to correlate
with the activity of the file_handle_cache process in the etop output,
but don't know enough to say if that is definitely the case.

We've seen similar results on as system with the configuration:
* RabbitMQ 2.7.0
* Erlang R14B03

On that system, however, we observed only 100% CPU utilisation (on 1
core).  That behaviour more closely matches the symptoms observed in
http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2012-June/020620.html

We also observed the same pattern of output from strace (EAGAIN from a
pipe, calls to epoll_wait in a tight loop).

We could spend a lot longer trying to trace the cause, but we'd greatly
appreciate if you could look into this, as it seems to not be an
isolated incident.


Regards,
Anthony


(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:44:39
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache
36681********80398152       0 gen_server2:loop/1
(etop at ci-slave-lin5)1> <4806.219.0>   proc_lib:init_p/5       6420
3468715  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5
144******** 7628376       0 gen_server:loop/6
(etop at ci-slave-lin5)1> <4806.1603.0>  proc_lib:init_p/5        107
3483195  109264       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.912.0>   proc_lib:init_p/5        100
3463759  196928       2 gen:do_call/4
(etop at ci-slave-lin5)1> <4806.1462.0>  proc_lib:init_p/5         93
3499173  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.331.0>   proc_lib:init_p/5         92
3509679  163416       0 gen:do_call/4
(etop at ci-slave-lin5)1> <4806.220.0>   proc_lib:init_p/5         88
3493388  284400       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1281.0>  proc_lib:init_p/5         82
3497526  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1597.0>  proc_lib:init_p/5         81
3525102  284400       0 gen_server2:process_
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:44:44
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     603362
6342434********     292 lists:foreach/2
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      94480
1281752 7628376       0 gen_server:loop/6
(etop at ci-slave-lin5)1> <4806.1517.0>  proc_lib:init_p/5      25444
3440  286992      80 gen:do_call/4
(etop at ci-slave-lin5)1> <4806.1462.0>  proc_lib:init_p/5      14122
10484  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.251.0>   proc_lib:init_p/5       7711
1721  287504      96 gen:do_call/4
(etop at ci-slave-lin5)1> <4806.199.0>   rabbit_memory_monito    6030
15909  129672       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.167.0>   rabbit_event            5913
11527   42224       0 gen_event:fetch_msg/
(etop at ci-slave-lin5)1> <4806.446.0>   proc_lib:init_p/5       5702
27798  196832       0 rabbit_variable_queu
(etop at ci-slave-lin5)1> <4806.699.0>   proc_lib:init_p/5       5562
27048  263728       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.314.0>   proc_lib:init_p/5       5473
27142  284432       1 gen_server2:process_
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:44:49
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     354652
3760046********     136 lists:foldl/3
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      75034
1307713 4715224       8 proplists:get_value/
(etop at ci-slave-lin5)1> <4806.1462.0>  proc_lib:init_p/5      57904
10483  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1080.0>  proc_lib:init_p/5      15074
32069  196832       0 file_handle_cache:pu
(etop at ci-slave-lin5)1> <4806.1516.0>  proc_lib:init_p/5      12215
32832  284400       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.848.0>   proc_lib:init_p/5      11437
32976  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.475.0>   proc_lib:init_p/5      11142
33599  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.801.0>   proc_lib:init_p/5      10391
32639  284400       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.857.0>   proc_lib:init_p/5      10268
31693  196864       1 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.866.0>   proc_lib:init_p/5       8475
32597  163416       1 gen_server2:process_
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:44:55
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     369157
3326079********     675 lists:foldl/3
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      35631
1333882 7628560       5 proplists:get_value/
(etop at ci-slave-lin5)1> <4806.1462.0>  proc_lib:init_p/5      33387
10131  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.530.0>   proc_lib:init_p/5      15506
39052  284400       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.778.0>   proc_lib:init_p/5      15427
39348  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.785.0>   proc_lib:init_p/5      14875
40295  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.770.0>   proc_lib:init_p/5      14395
10096  284400       0 file_handle_cache:pu
(etop at ci-slave-lin5)1> <4806.1084.0>  proc_lib:init_p/5      13049
40911  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1324.0>  proc_lib:init_p/5      12875
39092  196864       1 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.740.0>   proc_lib:init_p/5      12281
40218  176160       0 file_handle_cache:pu
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:45:00
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     381061
3264868********     303 lists:foreach/2
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      89767
1360720 7628784      12 proplists:get_value/
(etop at ci-slave-lin5)1> <4806.1462.0>  proc_lib:init_p/5      33557
9419  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1247.0>  proc_lib:init_p/5      21772
45062  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1586.0>  proc_lib:init_p/5      15283
9522  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1046.0>  proc_lib:init_p/5      12113
41757  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.809.0>   proc_lib:init_p/5      11990
45574  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.727.0>   proc_lib:init_p/5      11673
9425  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1123.0>  proc_lib:init_p/5      11220
8404  163832      13 gen:do_call/4
(etop at ci-slave-lin5)1> <4806.710.0>   proc_lib:init_p/5      10278
38415  176192       1 gen_server2:process_
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:45:05
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     373685
4510295********      73 lists:foreach/2
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      60559
1370669 4714944       0 gen_server:loop/6
(etop at ci-slave-lin5)1> <4806.1462.0>  proc_lib:init_p/5      56902
5711  177472      40 gen:do_call/4
(etop at ci-slave-lin5)1> <4806.1072.0>  proc_lib:init_p/5      11439
9061  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.715.0>   proc_lib:init_p/5      10288
39610  163416       1 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.857.0>   proc_lib:init_p/5       9311
9060  163416       1 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1110.0>  proc_lib:init_p/5       8930
21719  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.809.0>   proc_lib:init_p/5       8710
9249  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1131.0>  proc_lib:init_p/5       8332
23505  284400       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1187.0>  proc_lib:init_p/5       8121
9134  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:45:10
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     608264
639759941258408      85 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1586.0>  proc_lib:init_p/5      60486
10673  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      36301
1319523 4714944       0 gen_server:loop/6
(etop at ci-slave-lin5)1> <4806.1095.0>  proc_lib:init_p/5       9594
10753  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.825.0>   proc_lib:init_p/5       9220
10855  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.778.0>   proc_lib:init_p/5       8858
10895  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.791.0>   proc_lib:init_p/5       7175
1711  200192     104 gen:do_call/4
(etop at ci-slave-lin5)1> <4806.199.0>   rabbit_memory_monito    6389
17622  129672       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.167.0>   rabbit_event            5468
12555   34328       0 gen_event:fetch_msg/
(etop at ci-slave-lin5)1> <4806.1220.0>  proc_lib:init_p/5       5331
23649  426088       0 gen_server2:process_
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:45:16
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     474511
402239458220816    1089 lists:foreach/2
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      31989
1319504 7628472       3 gen_server:loop/6
(etop at ci-slave-lin5)1> <4806.1586.0>  proc_lib:init_p/5      17757
11395  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.766.0>   proc_lib:init_p/5       9903
28407  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1467.0>  proc_lib:init_p/5       9435
11388  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.511.0>   proc_lib:init_p/5       8705
11209  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1318.0>  proc_lib:init_p/5       8139
11380  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1465.0>  proc_lib:init_p/5       8047
11396  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.750.0>   proc_lib:init_p/5       7996
31235  122040       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.851.0>   proc_lib:init_p/5       7305
 0  291600     224 gen:do_call/4
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:45:21
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     371042
337973358206400    1044 lists:foreach/2
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      83579
1317230 7628376       0 gen_server:loop/6
(etop at ci-slave-lin5)1> <4806.308.0>   proc_lib:init_p/5      13238
37066  142712       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.736.0>   proc_lib:init_p/5      12227
37530  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1229.0>  proc_lib:init_p/5       9065
37208  284400       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.914.0>   proc_lib:init_p/5       8477
10076  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1490.0>  proc_lib:init_p/5       8389
10124  284400       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.376.0>   proc_lib:init_p/5       8271
35172  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.388.0>   proc_lib:init_p/5       7964
36146  284400       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.787.0>   proc_lib:init_p/5       7347
10148  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:45:26
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     361567
344879058168176      74 file_handle_cache:'-
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      28719
1356844 4715328      12 gen_server:loop/6
(etop at ci-slave-lin5)1> <4806.1517.0>  proc_lib:init_p/5      13391
10036  284400       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.654.0>   proc_lib:init_p/5      10458
38215  176192       1 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.413.0>   proc_lib:init_p/5      10155
41685  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.452.0>   proc_lib:init_p/5       8858
10287  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1047.0>  proc_lib:init_p/5       8098
32497  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.496.0>   proc_lib:init_p/5       7991
37605  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1458.0>  proc_lib:init_p/5       7824
30971  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.221.0>   proc_lib:init_p/5       7706
39544  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:45:31
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     420465
460800491164200       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5     109975
1365688 4714944       0 gen_server:loop/6
(etop at ci-slave-lin5)1> <4806.1517.0>  proc_lib:init_p/5      28722
9319  284432       1 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.969.0>   proc_lib:init_p/5      14990
33568  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.199.0>   rabbit_memory_monito   13184
21404  129672       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.369.0>   proc_lib:init_p/5      11974
36802  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.747.0>   proc_lib:init_p/5      11410
35004  196864       1 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1325.0>  proc_lib:init_p/5      10215
34080  263728       0 file_handle_cache:pu
(etop at ci-slave-lin5)1> <4806.933.0>   proc_lib:init_p/5      10068
9536  196832       0 file_handle_cache:pu
(etop at ci-slave-lin5)1> <4806.1048.0>  proc_lib:init_p/5       7904
9489  284400       0 gen_server2:process_
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:45:37
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     712420
650484191205136     553 lists:foreach/2
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      83277
1324824 7628456       2 lists:'-concat/1-fun
(etop at ci-slave-lin5)1> <4806.1126.0>  proc_lib:init_p/5      19381
10400  101368       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1517.0>  proc_lib:init_p/5      15301
10389  284464       2 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.856.0>   proc_lib:init_p/5      12669
10408  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.858.0>   proc_lib:init_p/5      11553
10405  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1329.0>  proc_lib:init_p/5      11163
25310  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.904.0>   proc_lib:init_p/5      10211
21848  284400       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.631.0>   proc_lib:init_p/5      10166
26370  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.949.0>   proc_lib:init_p/5      10095
25815  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:45:42
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     401679
414815191213376    1011 lists:foreach/2
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      87918
1411556 2173056      16 rabbit_mgmt_format:f
(etop at ci-slave-lin5)1> <4806.858.0>   proc_lib:init_p/5      49060
11191  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.793.0>   proc_lib:init_p/5      23977
29512  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1208.0>  proc_lib:init_p/5      15211
27978  284432       1 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1316.0>  proc_lib:init_p/5      15067
11011  196864       1 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1517.0>  proc_lib:init_p/5      13764
11192  284432       1 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.895.0>   proc_lib:init_p/5      12133
28624  284432       1 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1464.0>  proc_lib:init_p/5      11427
11094  163416       1 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1231.0>  proc_lib:init_p/5      10371
11229  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:45:47
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     308855
352530291216168     987 lists:foreach/2
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      94950
1659062 4117008      68 rabbit_mgmt_db:pget/
(etop at ci-slave-lin5)1> <4806.1200.0>  proc_lib:init_p/5      14179
38622  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.840.0>   proc_lib:init_p/5       9142
38574  263728       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.463.0>   proc_lib:init_p/5       8800
10681  109296       1 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1322.0>  proc_lib:init_p/5       8769
38403  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1210.0>  proc_lib:init_p/5       8310
10668  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1333.0>  proc_lib:init_p/5       7589
37860  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.167.0>   rabbit_event            6704
12760   42288       2 gen_event:fetch_msg/
(etop at ci-slave-lin5)1> <4806.1128.0>  proc_lib:init_p/5       6683
38314  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:45:53
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     272243
3182835********      56 lists:foreach/2
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      70426
1291511 8238048      96 rabbit_mgmt_db:is_de
(etop at ci-slave-lin5)1> <4806.579.0>   proc_lib:init_p/5      16089
43407  101368       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.578.0>   proc_lib:init_p/5      12557
41333  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.253.0>   proc_lib:init_p/5      10648
41098  263728       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1517.0>  proc_lib:init_p/5      10296
9327  284400       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.593.0>   proc_lib:init_p/5       8863
34641  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.682.0>   proc_lib:init_p/5       8538
15815  284400       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1371.0>  proc_lib:init_p/5       8053
9344  109264       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.925.0>   proc_lib:init_p/5       8043
41693  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:45:58
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     398161
3901770********     830 lists:foreach/2
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      59160
2078217 4714976       1 gen_server:loop/6
(etop at ci-slave-lin5)1> <4806.1517.0>  proc_lib:init_p/5      19017
4528  286096      52 gen:do_call/4
(etop at ci-slave-lin5)1> <4806.1161.0>  proc_lib:init_p/5      18239
9059  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1368.0>  proc_lib:init_p/5      18136
9042  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.407.0>   proc_lib:init_p/5      17377
33134  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1546.0>  proc_lib:init_p/5      11297
9050  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.826.0>   proc_lib:init_p/5      10419
8971  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.315.0>   proc_lib:init_p/5      10140
8986  176192       1 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1389.0>  proc_lib:init_p/5       8969
35056  284400       0 gen_server2:process_
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:46:03
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     579763
6033273********     274 lists:foreach/2
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5     102043
1284933 7638512     119 rabbit_mgmt_format:t
(etop at ci-slave-lin5)1> <4806.787.0>   proc_lib:init_p/5      52241
23479  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1161.0>  proc_lib:init_p/5      28945
5084  177824      51 gen:do_call/4
(etop at ci-slave-lin5)1> <4806.1422.0>  proc_lib:init_p/5       9509
92  201696     151 gen:do_call/4
(etop at ci-slave-lin5)1> <4806.1307.0>  proc_lib:init_p/5       8631
9577  284400       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1392.0>  proc_lib:init_p/5       8505
25367  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.167.0>   rabbit_event            8022
13152   55000       0 gen_event:fetch_msg/
(etop at ci-slave-lin5)1> <4806.684.0>   proc_lib:init_p/5       7884
9603  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.199.0>   rabbit_memory_monito    7638
19398  118760       0 gen_server2:process_
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:46:08
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     466390
4306327********       0 gen_server2:handle_c
(etop at ci-slave-lin5)1> <4806.787.0>   proc_lib:init_p/5      48740
10396  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      31582
1349050 7628376       0 gen_server:loop/6
(etop at ci-slave-lin5)1> <4806.283.0>   proc_lib:init_p/5      14945
10580  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1311.0>  proc_lib:init_p/5      12190
25418  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.530.0>   proc_lib:init_p/5       9814
10507   88592       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.405.0>   proc_lib:init_p/5       9448
10497  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1061.0>  proc_lib:init_p/5       8796
10423  109264       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.888.0>   proc_lib:init_p/5       8726
10514  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.859.0>   proc_lib:init_p/5       7814
4463  199072      69 gen:do_call/4
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:46:13
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     430687
3898666********     805 gen_server2:in/3
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      82691
1356565 7628376       0 gen_server:loop/6
(etop at ci-slave-lin5)1> <4806.1578.0>  proc_lib:init_p/5      21030
33457  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1024.0>  proc_lib:init_p/5      11178
33308  284400       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.711.0>   proc_lib:init_p/5      10609
34540  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.787.0>   proc_lib:init_p/5      10541
11296  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.408.0>   proc_lib:init_p/5      10385
34241  142712       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.809.0>   proc_lib:init_p/5       9364
11314  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.336.0>   proc_lib:init_p/5       9111
11285  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1187.0>  proc_lib:init_p/5       8609
11305  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:46:19
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     287339
3311890********      11 gen_server2:handle_m
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      63553
1319976 7628656       8 proplists:get_value/
(etop at ci-slave-lin5)1> <4806.530.0>   proc_lib:init_p/5      23944
9876   63040       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.933.0>   proc_lib:init_p/5      19043
38142  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1586.0>  proc_lib:init_p/5      15096
9947  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.846.0>   proc_lib:init_p/5      14685
40333  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1448.0>  proc_lib:init_p/5      13976
9970  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1594.0>  proc_lib:init_p/5      10233
38359  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1175.0>  proc_lib:init_p/5       9865
41515  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1299.0>  proc_lib:init_p/5       9156
9951  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:46:24
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     328106
3701791********    1419 lists:foreach/2
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      35864
1348673 4714944       0 gen_server:loop/6
(etop at ci-slave-lin5)1> <4806.412.0>   proc_lib:init_p/5      20773
36341  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1586.0>  proc_lib:init_p/5      16916
9423  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1231.0>  proc_lib:init_p/5      12504
34956  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.591.0>   proc_lib:init_p/5      11053
31353  142712       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1025.0>  proc_lib:init_p/5       9272
9658   88592       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1060.0>  proc_lib:init_p/5       9082
40103  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.389.0>   proc_lib:init_p/5       8408
36100  284400       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1524.0>  proc_lib:init_p/5       8091
40976  142712       0 gen_server2:process_
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:46:29
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     525547
5490099********      17 lists:foldl/3
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      57527
1348039 4714944       0 gen_server:loop/6
(etop at ci-slave-lin5)1> <4806.1219.0>  proc_lib:init_p/5      15380
10038  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1517.0>  proc_lib:init_p/5      12887
22194  263728       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.787.0>   proc_lib:init_p/5      12530
10048  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.938.0>   proc_lib:init_p/5      12103
22718  101368       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1386.0>  proc_lib:init_p/5      10706
21631  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.249.0>   proc_lib:init_p/5      10472
25174  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1126.0>  proc_lib:init_p/5      10469
24285  109264       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.560.0>   proc_lib:init_p/5       9538
21873  263728       0 gen_server2:process_
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:46:34
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     533337
519511658199768     754 lists:foreach/2
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      76066
1311722 4714944       0 gen_server:loop/6
(etop at ci-slave-lin5)1> <4806.1180.0>  proc_lib:init_p/5      58063
10021  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1517.0>  proc_lib:init_p/5      50387
10059  284432       1 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1126.0>  proc_lib:init_p/5      27804
10048  142712       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.745.0>   proc_lib:init_p/5      14248
25234  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.167.0>   rabbit_event           12832
11290   42256       1 gen_event:fetch_msg/
(etop at ci-slave-lin5)1> <4806.1004.0>  proc_lib:init_p/5      11063
25274  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1504.0>  proc_lib:init_p/5       8816
10136  284400       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1294.0>  proc_lib:init_p/5       8590
6372  164792      43 gen:do_call/4
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:46:39
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     314909
374099458172336      25 lists:foldl/3
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      89405
1298368 4715040       3 gen_server:loop/6
(etop at ci-slave-lin5)1> <4806.1126.0>  proc_lib:init_p/5      35143
10031  101368       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1386.0>  proc_lib:init_p/5      16413
10058  163416       1 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.858.0>   proc_lib:init_p/5      15065
9937  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1407.0>  proc_lib:init_p/5      11821
30788  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.935.0>   proc_lib:init_p/5      11022
29984  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1069.0>  proc_lib:init_p/5      10525
29881  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1250.0>  proc_lib:init_p/5       8800
9972  142744       1 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1217.0>  proc_lib:init_p/5       8143
10052  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:46:44
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     355578
344041358180832     317 lists:foreach/2
(etop at ci-slave-lin5)1> <4806.858.0>   proc_lib:init_p/5      43828
10239  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1195.0>  proc_lib:init_p/5      28740
36075  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      20712
1337810 7628376       0 gen_server:loop/6
(etop at ci-slave-lin5)1> <4806.1438.0>  proc_lib:init_p/5      19897
35354  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1227.0>  proc_lib:init_p/5      12504
37949  122040       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.242.0>   proc_lib:init_p/5      12279
35276  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1472.0>  proc_lib:init_p/5      11629
37350  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1424.0>  proc_lib:init_p/5      11420
36034  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.410.0>   proc_lib:init_p/5      11033
37948  284400       0 gen_server2:process_
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:46:49
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     336489
390343764345656       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      94420
1353540 7628376       0 gen_server:loop/6
(etop at ci-slave-lin5)1> <4806.858.0>   proc_lib:init_p/5      30558
9420  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.257.0>   proc_lib:init_p/5      11436
41548  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.728.0>   proc_lib:init_p/5      10861
34225  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.450.0>   proc_lib:init_p/5      10222
37727  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.175.0>   os_cmd_port_creator     9965
 6    5576       0 os:start_port_srv_lo
(etop at ci-slave-lin5)1> <4806.1297.0>  proc_lib:init_p/5       9850
12579  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.752.0>   proc_lib:init_p/5       8960
9410  196864       1 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1380.0>  proc_lib:init_p/5       8685
9590  284400       0 gen_server2:process_
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:46:54
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     413618
421532491216016    1021 gen_server2:drain/1
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      45104
1349170 4715008       2 gen_server:loop/6
(etop at ci-slave-lin5)1> <4806.1517.0>  proc_lib:init_p/5      25616
9493  284400       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.699.0>   proc_lib:init_p/5      11537
26576  284400       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.259.0>   proc_lib:init_p/5      10517
24809  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1321.0>  proc_lib:init_p/5       9348
34490  263728       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.315.0>   proc_lib:init_p/5       9121
20500  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.605.0>   proc_lib:init_p/5       8443
9401  176352       5 gen:do_call/4
(etop at ci-slave-lin5)1> <4806.199.0>   rabbit_memory_monito    7231
19696  110864       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1333.0>  proc_lib:init_p/5       6438
34163  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:47:00
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     467057
561699391178064     117 lists:foreach/2
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5     102801
1326080 4714944       0 gen_server:loop/6
(etop at ci-slave-lin5)1> <4806.1344.0>  proc_lib:init_p/5      24217
9676  196864       1 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1517.0>  proc_lib:init_p/5      20705
4234  265776      63 gen:do_call/4
(etop at ci-slave-lin5)1> <4806.1458.0>  proc_lib:init_p/5      14414
9890  196864       0 gen:do_call/4
(etop at ci-slave-lin5)1> <4806.354.0>   proc_lib:init_p/5      14346
7936  176864      21 gen:do_call/4
(etop at ci-slave-lin5)1> <4806.882.0>   proc_lib:init_p/5      11355
25534  284400       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.979.0>   proc_lib:init_p/5      10430
9096  197120       8 gen:do_call/4
(etop at ci-slave-lin5)1> <4806.864.0>   proc_lib:init_p/5       8201
9599  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.167.0>   rabbit_event            6490
12320   67808       1 gen_event:fetch_msg/
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:47:05
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     452238
440276191187776     418 lists:foreach/2
(etop at ci-slave-lin5)1> <4806.228.0>   proc_lib:init_p/5      63052
10845  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      45829
1301427 7628376       0 gen_server:loop/6
(etop at ci-slave-lin5)1> <4806.641.0>   proc_lib:init_p/5      14573
10808  196864       1 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1215.0>  proc_lib:init_p/5      10299
29131  109296       1 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.324.0>   proc_lib:init_p/5       7168
10918  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1443.0>  proc_lib:init_p/5       6647
27389  122040       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.293.0>   proc_lib:init_p/5       5614
31171  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.722.0>   proc_lib:init_p/5       5211
27052  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.216.0>   proc_lib:init_p/5       5083
22204  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:47:10
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     358625
354421291212936     451 lists:foreach/2
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      33084
1304706 7628376       0 gen_server:loop/6
(etop at ci-slave-lin5)1> <4806.787.0>   proc_lib:init_p/5      30809
10389  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.300.0>   proc_lib:init_p/5      13210
35853  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1219.0>  proc_lib:init_p/5      11653
33969  176192       1 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1472.0>  proc_lib:init_p/5      10920
10332  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.920.0>   proc_lib:init_p/5      10246
35673  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.641.0>   proc_lib:init_p/5      10105
10130  163544       4 gen:do_call/4
(etop at ci-slave-lin5)1> <4806.542.0>   proc_lib:init_p/5       9890
35877  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.780.0>   proc_lib:init_p/5       9228
31279  109264       0 gen_server2:process_
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:47:15
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     335100
3392312********     456 lists:foreach/2
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      54952
1355733 7628376       0 gen_server:loop/6
(etop at ci-slave-lin5)1> <4806.1079.0>  proc_lib:init_p/5      17985
38010  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.416.0>   proc_lib:init_p/5      14833
39924  263728       0 file_handle_cache:wi
(etop at ci-slave-lin5)1> <4806.391.0>   proc_lib:init_p/5      14123
9587  109264       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1318.0>  proc_lib:init_p/5      13096
36218  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.956.0>   proc_lib:init_p/5      11993
40052  163416       1 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.261.0>   proc_lib:init_p/5      10772
39489  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.247.0>   proc_lib:init_p/5       9997
37634  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1462.0>  proc_lib:init_p/5       9536
6235  198144      40 gen:do_call/4
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:47:21
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     375103
4091283********    1179 rabbit_amqqueue:set_
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      59507
1363580 7628632       8 gen_server:loop/6
(etop at ci-slave-lin5)1> <4806.1109.0>  proc_lib:init_p/5      19595
34180  284400       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1244.0>  proc_lib:init_p/5      16177
37798  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.787.0>   proc_lib:init_p/5      14892
9575  196864       1 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.451.0>   proc_lib:init_p/5      11988
38706  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.716.0>   proc_lib:init_p/5      11104
39224  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1580.0>  proc_lib:init_p/5      10632
9719  196864       1 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.780.0>   proc_lib:init_p/5       9966
9696  142712       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.260.0>   proc_lib:init_p/5       9384
36353  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:47:26
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     567269
5844914********       2 gen_server2:loop/1
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      45727
1339452 7628376       0 gen_server:loop/6
(etop at ci-slave-lin5)1> <4806.1072.0>  proc_lib:init_p/5      14708
10479  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.779.0>   proc_lib:init_p/5      11396
24223  284400       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1602.0>  proc_lib:init_p/5      10849
25499  109264       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.783.0>   proc_lib:init_p/5       8785
10498   67920       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1419.0>  proc_lib:init_p/5       8272
10597  263728       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1221.0>  proc_lib:init_p/5       7388
9078  197408      17 gen:do_call/4
(etop at ci-slave-lin5)1> <4806.310.0>   proc_lib:init_p/5       6403
23908  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.167.0>   rabbit_event            5282
13848   42224       0 gen_event:fetch_msg/
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:47:31
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     490370
4785283********     145 timer:now_diff/2
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      94384
1304098 7628504       4 gen_server:loop/6
(etop at ci-slave-lin5)1> <4806.787.0>   proc_lib:init_p/5      22252
10809  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1060.0>  proc_lib:init_p/5      11349
10772  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1072.0>  proc_lib:init_p/5       9479
10755  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.858.0>   proc_lib:init_p/5       9451
25206  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1149.0>  proc_lib:init_p/5       8466
10703  196832       0 file_handle_cache:pu
(etop at ci-slave-lin5)1> <4806.1022.0>  proc_lib:init_p/5       7014
 0  170328     216 gen:do_call/4
(etop at ci-slave-lin5)1> <4806.1586.0>  proc_lib:init_p/5       6088
10743  196896       2 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1167.0>  proc_lib:init_p/5       5912
2080  287568      98 gen:do_call/4
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:47:36
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     359279
3668969********     882 lists:foreach/2
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5     100054
1314192 4714976       1 gen_server:loop/6
(etop at ci-slave-lin5)1> <4806.1551.0>  proc_lib:init_p/5      17155
10653  263728       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.887.0>   proc_lib:init_p/5      15044
34552   67920       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.858.0>   proc_lib:init_p/5      14432
10582  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1261.0>  proc_lib:init_p/5      12231
35035  142744       1 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.876.0>   proc_lib:init_p/5       8872
30935  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1519.0>  proc_lib:init_p/5       7573
10554  176160       0 file_handle_cache:pu
(etop at ci-slave-lin5)1> <4806.1266.0>  proc_lib:init_p/5       7128
31939  109264       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1422.0>  proc_lib:init_p/5       6998
34228   88592       0 gen_server2:process_
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:47:41
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     320963
3568928********     338 lists:foreach/2
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      87663
1344643 4714944       0 gen_server:loop/6
(etop at ci-slave-lin5)1> <4806.858.0>   proc_lib:init_p/5      26699
9685  163384       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1517.0>  proc_lib:init_p/5      16175
9798  284400       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.421.0>   proc_lib:init_p/5      11819
9608  196928       2 gen:do_call/4
(etop at ci-slave-lin5)1> <4806.218.0>   proc_lib:init_p/5       9553
9669  284400       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.915.0>   proc_lib:init_p/5       9341
9704  122040       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.288.0>   proc_lib:init_p/5       9156
9685  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1357.0>  proc_lib:init_p/5       8442
9809  196864       1 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.361.0>   proc_lib:init_p/5       7763
9767  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:47:47
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     361707
3844289********     108 lists:foreach/2
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      43057
1365655 7628376       0 gen_server:loop/6
(etop at ci-slave-lin5)1> <4806.338.0>   proc_lib:init_p/5      37397
9826   88592       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.826.0>   proc_lib:init_p/5      17902
25391  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.903.0>   proc_lib:init_p/5      12779
39566  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1466.0>  proc_lib:init_p/5      10386
34370  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.481.0>   proc_lib:init_p/5       9112
9688  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1504.0>  proc_lib:init_p/5       8971
9600  263728       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1399.0>  proc_lib:init_p/5       8804
9594  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.949.0>   proc_lib:init_p/5       8531
31619  176160       0 file_handle_cache:pu
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>  'rabbit at ci-slave-lin5'
                          11:47:52
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1> Pid            Name or Initial Func    Time
Reds  Memory    MsgQ Current Function
(etop at ci-slave-lin5)1>
----------------------------------------------------------------------------------------
(etop at ci-slave-lin5)1> <4806.155.0>   file_handle_cache     507389
5004720********     652 lists:foreach/2
(etop at ci-slave-lin5)1> <4806.944.0>   proc_lib:init_p/5      79844
9865  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1667.0>  proc_lib:init_p/5      33807
1340135 7628376       0 gen_server:loop/6
(etop at ci-slave-lin5)1> <4806.506.0>   proc_lib:init_p/5      14991
27482  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1364.0>  proc_lib:init_p/5      12434
 0  169272     183 gen:do_call/4
(etop at ci-slave-lin5)1> <4806.218.0>   proc_lib:init_p/5      11053
9849  263728       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1481.0>  proc_lib:init_p/5      10933
9785  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1469.0>  proc_lib:init_p/5      10213
9774  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.969.0>   proc_lib:init_p/5       9770
9873  176160       0 gen_server2:process_
(etop at ci-slave-lin5)1> <4806.1181.0>  proc_lib:init_p/5       9531
9765  196832       0 gen_server2:process_
(etop at ci-slave-lin5)1>
========================================================================================
(etop at ci-slave-lin5)1>
(etop at ci-slave-lin5)1>
-------------- next part --------------
[pid 27404] close(515)                  = 0
[pid 27405] close(622)                  = 0
[pid 27418] munmap(0x7fe45a828000, 974848) = 0
[pid 27418] mmap(NULL, 1572864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe45a796000
[pid 27409] stat("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/5H5K8QOLA0KN5T7GDR018MEEW/journal.jif", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
[pid 27409] open("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/5H5K8QOLA0KN5T7GDR018MEEW/journal.jif", O_RDWR|O_CREAT, 0666) = 424
[pid 27418] munmap(0x7fe448413000, 13463552) = 0
[pid 27418] munmap(0x7fe4989d6000, 974848) = 0
[pid 27407] close(624)                  = 0
[pid 27410] stat("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/4G87PPYQ0THSMGCWZJAPIMMG/journal.jif", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
[pid 27410] open("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/4G87PPYQ0THSMGCWZJAPIMMG/journal.jif", O_RDWR|O_CREAT, 0666) = 425
[pid 27419] mmap(NULL, 13463552, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe448413000
[pid 27411] stat("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/1MMKWVVBHX09XSNFA9WG02PRP/journal.jif", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
[pid 27411] open("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/1MMKWVVBHX09XSNFA9WG02PRP/journal.jif", O_RDWR|O_CREAT, 0666) = 426
[pid 27406] close(623)                  = 0
[pid 27412] stat("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/3BG1IP0TEK8BGZI6AMNS33S0H/journal.jif", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
[pid 27412] open("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/3BG1IP0TEK8BGZI6AMNS33S0H/journal.jif", O_RDWR|O_CREAT, 0666) = 427
[pid 27413] stat("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/5N52RIGNR8LZJ581O9P0W595M/journal.jif", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
[pid 27413] open("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/5N52RIGNR8LZJ581O9P0W595M/journal.jif", O_RDWR|O_CREAT, 0666) = 428
[pid 27408] close(625)                  = 0
[pid 27414] stat("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/1Y0KVTNBOPV5ENE7HWE7FANF1/journal.jif", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
[pid 27414] open("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/1Y0KVTNBOPV5ENE7HWE7FANF1/journal.jif", O_RDWR|O_CREAT, 0666) = 429
[pid 27409] close(626)                  = 0
[pid 27415] stat("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/73PV9BVSZIWW7Q2ZSE9I6BH2U/journal.jif", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
[pid 27415] open("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/73PV9BVSZIWW7Q2ZSE9I6BH2U/journal.jif", O_RDWR|O_CREAT, 0666) = 430
[pid 27416] stat("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/5NZWQY3OE6SS7S7XCHRBT6VV2/journal.jif", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
[pid 27416] open("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/5NZWQY3OE6SS7S7XCHRBT6VV2/journal.jif", O_RDWR|O_CREAT, 0666) = 431
[pid 27388] stat("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/8J125FN8OUOYF0XSPNBNWLFS3/journal.jif", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
[pid 27388] open("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/8J125FN8OUOYF0XSPNBNWLFS3/journal.jif", O_RDWR|O_CREAT, 0666) = 432
[pid 27387] stat("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/180EBH0Y63ND7RJT3V8G0ME73/journal.jif", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
[pid 27387] open("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/180EBH0Y63ND7RJT3V8G0ME73/journal.jif", O_RDWR|O_CREAT, 0666) = 433
[pid 27389] stat("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/M3JJOTHHCHSYRVPY4YWTWSKP/journal.jif", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
[pid 27389] open("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/M3JJOTHHCHSYRVPY4YWTWSKP/journal.jif", O_RDWR|O_CREAT, 0666) = 434
[pid 27410] close(627)                  = 0
[pid 27411] close(516)                  = 0
[pid 27413] close(629)                  = 0
[pid 27390] stat("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/BR0B92B3PSL683X3OQIL1GHQB/journal.jif", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
[pid 27390] open("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/BR0B92B3PSL683X3OQIL1GHQB/journal.jif", O_RDWR|O_CREAT, 0666) = 435
[pid 27419] mmap(NULL, 974848, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe4989d6000
[pid 27418] write(4, "!", 1)            = 1
[pid 27419] read(3, "!", 32)            = 1
[pid 27419] read(3, 0x7fe49ab232e0, 32) = -1 EAGAIN (Resource temporarily unavailable)
[pid 27391] stat("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/1ABYJL08VJ8CVF4Y10SV689SF/journal.jif", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
[pid 27391] open("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/1ABYJL08VJ8CVF4Y10SV689SF/journal.jif", O_RDWR|O_CREAT, 0666) = 436
[pid 27392] stat("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/8X46WGNX2UAC3XDT1JJ43O50P/journal.jif", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
[pid 27392] open("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/8X46WGNX2UAC3XDT1JJ43O50P/journal.jif", O_RDWR|O_CREAT, 0666) = 437
[pid 27419] munmap(0x7fe444d2d000, 13463552) = 0
[pid 27419] munmap(0x7fe45a796000, 1572864) = 0
[pid 27419] mmap(NULL, 100306944, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe437201000
[pid 27393] stat("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/4ZZ1M45UCGT7WRNCFBXMHPLBJ/journal.jif", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
[pid 27393] open("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/4ZZ1M45UCGT7WRNCFBXMHPLBJ/journal.jif", O_RDWR|O_CREAT, 0666) = 438
[pid 27412] close(628)                  = 0
[pid 27419] write(4, "!", 1)            = 1
[pid 27418] read(3, "!", 32)            = 1
[pid 27418] read(3, 0x7fe49b3242e0, 32) = -1 EAGAIN (Resource temporarily unavailable)
[pid 27394] stat("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/AD3IR47D0AH483EVVTVLBTIH3/journal.jif", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
[pid 27394] open("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/AD3IR47D0AH483EVVTVLBTIH3/journal.jif", O_RDWR|O_CREAT, 0666) = 439
[pid 27395] stat("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/BOXM3H4SCZDZADB4E5YPFW2RL/journal.jif", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
[pid 27395] open("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/BOXM3H4SCZDZADB4E5YPFW2RL/journal.jif", O_RDWR|O_CREAT, 0666) = 440
[pid 27396] stat("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/29572I5GN1BFLUI6HUOK67A97/journal.jif", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
[pid 27396] open("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/29572I5GN1BFLUI6HUOK67A97/journal.jif", O_RDWR|O_CREAT, 0666) = 441
[pid 27397] stat("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/C6TW9PD0E194ESWYTZILUGQ4H/journal.jif", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
[pid 27397] open("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/C6TW9PD0E194ESWYTZILUGQ4H/journal.jif", O_RDWR|O_CREAT, 0666) = 442
[pid 27419] write(4, "!", 1 <unfinished ...>
[pid 27418] read(3, "!", 32)            = 1
[pid 27418] read(3, 0x7fe49b3242e0, 32) = -1 EAGAIN (Resource temporarily unavailable)
[pid 27419] <... write resumed> )       = 1
[pid 27418] write(4, "!", 1)            = 1
[pid 27419] read(3, "!", 32)            = 1
[pid 27419] read(3, 0x7fe49ab232e0, 32) = -1 EAGAIN (Resource temporarily unavailable)
[pid 27419] write(4, "!", 1)            = 1
[pid 27419] read(3, "!", 32)            = 1
[pid 27419] read(3, 0x7fe49ab232e0, 32) = -1 EAGAIN (Resource temporarily unavailable)
[pid 27419] write(4, "!", 1)            = 1
[pid 27418] read(3, "!", 32)            = 1
[pid 27418] read(3, 0x7fe49b3242e0, 32) = -1 EAGAIN (Resource temporarily unavailable)
[pid 27398] stat("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/2XOYI61BMOL6WOBJBN9L1LYLF/journal.jif", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
[pid 27398] open("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/2XOYI61BMOL6WOBJBN9L1LYLF/journal.jif", O_RDWR|O_CREAT, 0666) = 443
[pid 27418] write(4, "!", 1)            = 1
[pid 27419] read(3, "!", 32)            = 1
[pid 27419] read(3, 0x7fe49ab232e0, 32) = -1 EAGAIN (Resource temporarily unavailable)
[pid 27399] stat("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/9GLI52BSC3ZYBP03SWJ8PUFYZ/journal.jif", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
[pid 27399] open("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/9GLI52BSC3ZYBP03SWJ8PUFYZ/journal.jif", O_RDWR|O_CREAT, 0666) = 444
[pid 27418] munmap(0x7fe43a2fc000, 48947200) = 0
[pid 27418] mmap(NULL, 1572864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe45a796000
[pid 27400] stat("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/AR6K5E9O5KBRZ2M7INFUY928X/journal.jif", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
[pid 27400] open("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/AR6K5E9O5KBRZ2M7INFUY928X/journal.jif", O_RDWR|O_CREAT, 0666) = 445
[pid 27401] stat("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/CCBHE5A8W1714FPEV3UMJGIW1/journal.jif", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
[pid 27401] open("/var/lib/rabbitmq/mnesia/rabbit at ci-slave-lin5/queues/CCBHE5A8W1714FPEV3UMJGIW1/journal.jif", O_RDWR|O_CREAT, 0666) = 446


More information about the rabbitmq-discuss mailing list