[rabbitmq-discuss] RabbitMQ running at 100% CPU.

Thu Mar 13 14:17:26 GMT 2008

Matthias,

Thank you for your suggestion. There is no strace on Mac OS X, but I  
did find a way to see what the C-program was doing (see below). Most  
of the threads are just waiting (as expected) but a single one seems  
always to be in some kind of tcp_send_error. I haven't yet found a way  
to see if this infinite loop is in the erlang runtime system or the  
error actually reaches rabbit. Anyway, just a status. I'll try to work  
out what I can - Rabbit is still running :-)

Michael

     1002 Thread_2503
       340 writev$UNIX2003
         340 writev$UNIX2003
       299 kevent
         299 kevent
       257 tcp_inet_drv_output
         197 getpeername$UNIX2003
           197 getpeername$UNIX2003
         42 tcp_send_error
           32 inet_reply_error
             16 error_atom
               8 __tolower
                 8 __tolower
               4 pthread_getspecific
                 4 pthread_getspecific
               3 error_atom
               1 dyld_stub_pthread_getspecific
                 1 dyld_stub_pthread_getspecific
             3 hash_put
               3 hash_put
             3 strlen
               3 strlen
             2 atom_hash
               2 atom_hash
             2 driver_mk_atom
               2 driver_mk_atom
             2 dyld_stub___tolower
               2 dyld_stub___tolower
             1 am_atom_put
               1 am_atom_put
             1 erl_errno_id
               1 erl_errno_id
             1 index_put
               1 index_put
             1 inet_reply_error
           4 inet_reply_error_am
             4 inet_reply_error_am
           4 tcp_send_error
           1 __error
             1 __error
           1 driver_send_term
             1 driver_send_term
         8 tcp_inet_drv_output
         4 cerror
           3 cthread_set_errno_self
             3 __error
               3 __error
           1 cerror
         2 _sysenter_trap
           2 _sysenter_trap
         2 inet_reply_error
           2 inet_reply_error
         1 __error
           1 __error
         1 dyld_stub___error
           1 dyld_stub___error
       18 check_async_ready
         6 __spin_lock
           6 __spin_lock
         5 pthread_mutex_unlock
           5 pthread_mutex_unlock
         4 pthread_mutex_lock
           4 pthread_mutex_lock
         2 check_async_ready
         1 spin_lock
           1 spin_lock
       14 erts_deliver_time
         9 gettimeofday
           9 __gettimeofday
             5 __gettimeofday
             4 __nanotime
               4 __nanotime
         4 erts_deliver_time
         1 __commpage_gettimeofday
           1 __commpage_gettimeofday
       13 erts_time_remaining
         9 gettimeofday
           8 __gettimeofday
             7 __nanotime
               7 __nanotime
             1 __gettimeofday
           1 gettimeofday
         4 erts_time_remaining
       9 erts_poll_wait_kp
         7 erts_poll_wait_kp
         2 kevent
           2 kevent
       8 0x38830014
         3 bcmp
           3 bcmp
         3 hash_put
           3 hash_put
         2 atom_cmp
           2 atom_cmp
       8 cerror
         4 cthread_set_errno_self
           3 cthread_set_errno_self
           1 __error
             1 __error
         3 cerror
         1 dyld_stub___error
           1 dyld_stub___error
       7 sweep_proc_bins
         7 sweep_proc_bins
       6 erl_sys_schedule
         6 erl_sys_schedule
       4 schedule
         4 schedule
       3 erts_check_io_kp
         3 erts_check_io_kp
       2 __error
         2 __error
       2 dyld_stub___error
         2 dyld_stub___error
       2 process_main
         2 process_main
       1 _sysenter_trap
         1 _sysenter_trap
       1 copy_shallow
         1 copy_shallow
       1 copy_struct
         1 copy_struct
       1 db_get_hash
         1 db_get_hash
       1 driver_peekq
         1 driver_peekq
       1 dyld_stub_gettimeofday
         1 dyld_stub_gettimeofday
       1 erts_check_io_interrupt_kp
         1 erts_check_io_interrupt_kp
       1 erts_poll_interrupt_kp
         1 erts_poll_interrupt_kp
       1 free_message_buffer
         1 erts_cleanup_mso
           1 erts_cleanup_mso
       1 tcp_inet_drv_input
         1 tcp_recv
           1 tcp_deliver
             1 deq_async
               1 deq_async_w_tmo
                 1 deq_async_w_tmo
On Mar 13, 2008, at 10:49 , Matthias Radestock wrote:

> Michael,
>
> Michael Arnoldus wrote:
>> I've kept it running. I'm sure if I restart erlang everything will   
>> just run perfectly, but I'd rather use this incident to try to  
>> find  the problem if it's possible. However i'm running out of  
>> things to  try, so any suggestions will be welcome.
>
> Try running strace on the Erlang process, to see whether it's stuck  
> in some busy loop making system calls.
>
>> If nobody has any other suggestions I'd be inclined to upgrade to  
>> the  latest version of erlang. Does anybody have any experince  
>> using this  with Rabbit?
>
> Several people have been running with R12B in their development  
> environments without any problems. It should be fine, but we haven't  
> done any thorough testing with it.
>
>
> Matthias.