[rabbitmq-discuss] beam process using 100% cpu and lots of memory
Joel Heenan
joelh at planetjoel.com
Tue Feb 16 00:15:05 GMT 2010
Here is the memory stats over the period of the test, it looks like
either a memory leak or an overwhelmed queue
05:30:01 AM 1763568 2430736 57.95 69700 1600004
524224 56 0.01 28
05:40:02 AM 1726492 2467812 58.84 71920 1602944
524224 56 0.01 28
05:50:01 AM 1685448 2508856 59.82 74044 1600452
524224 56 0.01 28
06:00:01 AM 1642048 2552256 60.85 76196 1603092
524224 56 0.01 28
06:10:01 AM 1705784 2488520 59.33 78352 1605564
524224 56 0.01 28
06:20:01 AM 1637576 2556728 60.96 80492 1603172
524224 56 0.01 28
06:30:01 AM 1640544 2553760 60.89 82628 1605768
524224 56 0.01 28
06:40:01 AM 1583628 2610676 62.24 84812 1608044
524224 56 0.01 28
06:50:01 AM 1549156 2645148 63.07 86988 1609996
524224 56 0.01 28
07:00:02 AM 1511584 2682720 63.96 89096 1608384
524224 56 0.01 28
07:10:01 AM 1540352 2653952 63.28 91232 1610184
524224 56 0.01 28
07:20:01 AM 1459132 2735172 65.21 93348 1612432
524224 56 0.01 28
07:30:01 AM 1451352 2742952 65.40 95508 1610720
524224 56 0.01 28
07:40:01 AM 1432380 2761924 65.85 97584 1612848
524224 56 0.01 28
07:50:01 AM 1428536 2765768 65.94 99688 1614412
524224 56 0.01 28
08:00:01 AM 1368272 2826032 67.38 101796 1613104
524224 56 0.01 28
08:10:01 AM 1465116 2729188 65.07 103880 1615952
524224 56 0.01 28
08:20:01 AM 1297096 2897208 69.07 105996 1616300
524224 56 0.01 28
08:20:01 AM kbmemfree kbmemused %memused kbbuffers kbcached
kbswpfree kbswpused %swpused kbswpcad
08:30:01 AM 1430024 2764280 65.91 108100 1614892
524224 56 0.01 28
08:40:01 AM 1434116 2760188 65.81 110192 1616732
524224 56 0.01 28
08:50:01 AM 1355624 2838680 67.68 112292 1618260
524224 56 0.01 28
09:00:01 AM 1384888 2809416 66.98 114392 1616824
524224 56 0.01 28
09:10:01 AM 1322020 2872284 68.48 116512 1618640
524224 56 0.01 28
09:20:01 AM 1350416 2843888 67.80 118588 1620196
524224 56 0.01 28
09:30:01 AM 1288168 2906136 69.29 120552 1618764
524224 56 0.01 28
09:40:01 AM 1316688 2877616 68.61 122464 1621272
524224 56 0.01 28
09:50:01 AM 1264980 2929324 69.84 124332 1621928
524224 56 0.01 28
10:00:01 AM 1208312 2985992 71.19 126224 1620688
524224 56 0.01 28
10:10:01 AM 1248612 2945692 70.23 128116 1622132
524224 56 0.01 28
10:20:01 AM 1203352 2990952 71.31 129988 1623552
524224 56 0.01 28
10:30:01 AM 1150900 3043404 72.56 131956 1624752
524224 56 0.01 28
10:40:01 AM 1206052 2988252 71.25 133784 1623780
524224 56 0.01 28
10:50:01 AM 1165628 3028676 72.21 135696 1625208
524224 56 0.01 28
11:00:01 AM 1116516 3077788 73.38 137752 1626512
524224 56 0.01 28
11:10:01 AM 1083664 3110640 74.16 139676 1625464
524224 56 0.01 28
11:20:02 AM 1136612 3057692 72.90 141476 1626764
524224 56 0.01 28
11:30:01 AM 1086816 3107488 74.09 143212 1628084
524224 56 0.01 28
11:40:01 AM 1064444 3129860 74.62 144904 1627028
524224 56 0.01 28
11:50:01 AM 1022780 3171524 75.62 146432 1628300
524224 56 0.01 28
12:00:01 PM 994012 3200292 76.30 147836 1629516
524224 56 0.01 28
12:10:01 PM 959664 3234640 77.12 149232 1628632
524224 56 0.01 28
12:20:01 PM 1020424 3173880 75.67 150804 1629820
524224 56 0.01 28
12:30:01 PM 995128 3199176 76.27 152356 1631076
524224 56 0.01 28
12:40:01 PM 964624 3229680 77.00 153756 1630080
524224 56 0.01 28
12:50:01 PM 941808 3252496 77.55 155244 1631252
524224 56 0.01 28
01:00:01 PM 1073620 3120684 74.40 156788 1632452
524224 56 0.01 28
01:10:01 PM 955820 3238484 77.21 158220 1634520
524224 56 0.01 28
01:20:01 PM 845956 3348348 79.83 159824 1632592
524224 56 0.01 28
01:30:01 PM 1102916 3091388 73.70 161404 1599240
524224 56 0.01 28
01:40:02 PM 1121392 3072912 73.26 162788 1600240
524224 56 0.01 28
01:50:01 PM 1009040 3185264 75.94 164124 1599448
524224 56 0.01 28
02:00:01 PM 1012024 3182280 75.87 165396 1600556
524224 56 0.01 28
02:10:01 PM 1057160 3137144 74.80 167168 1601672
524224 56 0.01 28
02:20:01 PM 948412 3245892 77.39 168644 1600828
524224 56 0.01 28
02:30:01 PM 976064 3218240 76.73 170444 1601896
524224 56 0.01 28
02:40:01 PM 1020828 3173476 75.66 172188 1602980
524224 56 0.01 28
02:50:01 PM 915180 3279124 78.18 173896 1602140
524224 56 0.01 28
03:00:01 PM 943700 3250604 77.50 175272 1603268
524224 56 0.01 28
03:10:01 PM 991440 3202864 76.36 176732 1604232
524224 56 0.01 28
03:20:01 PM 887776 3306528 78.83 178040 1603504
524224 56 0.01 28
03:30:01 PM 919024 3275280 78.09 179368 1604472
524224 56 0.01 28
03:40:01 PM 814492 3379812 80.58 180856 1605532
524224 56 0.01 28
03:50:02 PM 867440 3326864 79.32 182364 1604792
524224 56 0.01 28
04:00:01 PM 901044 3293260 78.52 184036 1605716
524224 56 0.01 28
04:10:01 PM 798116 3396188 80.97 185396 1606836
524224 56 0.01 28
04:20:01 PM 853428 3340876 79.65 187016 1606068
524224 56 0.01 28
04:30:01 PM 889140 3305164 78.80 188584 1606852
524224 56 0.01 28
04:40:01 PM 787460 3406844 81.23 190032 1608104
524224 56 0.01 28
04:40:01 PM kbmemfree kbmemused %memused kbbuffers kbcached
kbswpfree kbswpused %swpused kbswpcad
04:50:01 PM 843260 3351044 79.90 191384 1608968
524224 56 0.01 28
05:00:01 PM 745672 3448632 82.22 192864 1608036
524224 56 0.01 28
05:10:01 PM 783120 3411184 81.33 194204 1609228
524224 56 0.01 28
05:20:01 PM 685656 3508648 83.65 195528 1610068
524224 56 0.01 28
05:30:01 PM 745176 3449128 82.23 196644 1609552
524224 56 0.01 28
05:40:01 PM 785104 3409200 81.28 197992 1610360
524224 56 0.01 28
05:50:01 PM 688508 3505796 83.58 199256 1611276
524224 56 0.01 28
06:00:01 PM 750508 3443796 82.11 200444 1610540
524224 56 0.01 28
06:10:01 PM 655276 3539028 84.38 202000 1611308
524224 56 0.01 28
06:20:01 PM 696816 3497488 83.39 203356 1612440
524224 56 0.01 28
06:30:01 PM 605924 3588380 85.55 204868 1611632
524224 56 0.01 28
06:40:01 PM 664700 3529604 84.15 206040 1612872
524224 56 0.01 28
06:50:01 PM 708472 3485832 83.11 207052 1613604
524224 56 0.01 28
07:00:01 PM 612384 3581920 85.40 208612 1612908
524224 56 0.01 28
07:10:01 PM 681564 3512740 83.75 209772 1613916
524224 56 0.01 28
07:20:01 PM 589928 3604376 85.94 210796 1614784
524224 56 0.01 28
07:30:01 PM 638660 3555644 84.77 212148 1613904
524224 56 0.01 28
07:40:01 PM 548512 3645792 86.92 213160 1614812
524224 56 0.01 28
07:50:01 PM 613240 3581064 85.38 214260 1615740
524224 56 0.01 28
08:00:01 PM 524828 3669476 87.49 215192 1616544
524224 56 0.01 28
08:10:01 PM 574180 3620124 86.31 216300 1615872
524224 56 0.01 28
08:20:01 PM 486140 3708164 88.41 217480 1616728
524224 56 0.01 28
08:30:01 PM 552480 3641824 86.83 218604 1617680
524224 56 0.01 28
08:40:01 PM 467788 3726516 88.85 219584 1617024
524224 56 0.01 28
08:50:01 PM 517512 3676792 87.66 220584 1617820
524224 56 0.01 28
09:00:01 PM 431332 3762972 89.72 221584 1618672
524224 56 0.01 28
09:10:01 PM 501888 3692416 88.03 222616 1618176
524224 56 0.01 28
09:20:02 PM 416196 3778108 90.08 223896 1619036
524224 56 0.01 28
09:30:01 PM 467664 3726640 88.85 225136 1619768
524224 56 0.01 28
09:40:01 PM 385452 3808852 90.81 226184 1619152
524224 56 0.01 28
09:50:01 PM 455264 3739040 89.15 227420 1619992
524224 56 0.01 28
10:00:01 PM 371688 3822616 91.14 228248 1620744
524224 56 0.01 28
10:10:01 PM 427364 3766940 89.81 229184 1620160
524224 56 0.01 28
10:20:01 PM 344532 3849772 91.79 230028 1620968
524224 56 0.01 28
10:30:02 PM 416328 3777976 90.07 230912 1621764
524224 56 0.01 28
10:40:01 PM 336596 3857708 91.97 231980 1621176
524224 56 0.01 28
10:50:01 PM 410344 3783960 90.22 233020 1603676
524224 56 0.01 28
On Tue, Feb 16, 2010 at 10:51 AM, Joel Heenan <joelh at planetjoel.com> wrote:
> We are having a situation where when we put lots of data through a
> RabbitMQ queue we sometimes we see a beam process running using 100%
> cpu and up to 1.5GB (RSS) of memory. The only way to resolve this
> situation is to restart RabbitMQ.
>
> We are trying to trace down the exact conditions that cause the
> problem, and I hope to also post up an strace and lsof of what it is
> doing during this time and log messages.
>
> I could not find anything particularly interesting in the rabbitmq.log file.
>
> We are talking a very large number of connections here, 1807334
> connections in 17 hours, or 29/second.
>
> We started using RabbitMQ Stomp Server 1.6.0-5 then moved to version
> latest public umbrella (as of 24 hours ago). OS is Centos 5.4 x86_64
> Xen domU with 4GB of RAM and 1VCPU.
>
> We are using persistent mode on the queues, currently testing with
> non-persistent.
>
> Joel
>
More information about the rabbitmq-discuss
mailing list