High CPU load but only few sessions
Hello, since a few days I observe a high CPU load of my FS server, but I have no idea what it could be. There are only a few sessions running and there is only a few log activity. 2 days ago I restarted FS, but no change. The top command shows this: top - 15:02:33 up 106 days, 30 min, 4 users, load average: 0.24, 0.35, 0.42 Tasks: 190 total, 1 running, 189 sleeping, 0 stopped, 0 zombie Cpu(s): 7.2%us, 12.2%sy, 0.0%ni, 80.0%id, 0.2%wa, 0.2%hi, 0.2%si, 0.0%st Mem: 4151776k total, 4003664k used, 148112k free, 414708k buffers Swap: 15623204k total, 88k used, 15623116k free, 2021412k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 14048 ippbx 20 0 1555m 1.2g 10m S 43 30.5 333:11.03 freeswitch 14049 ippbx 20 0 1555m 1.2g 10m S 0 30.5 5:06.88 freeswitch 14054 ippbx 20 0 1555m 1.2g 10m S 0 30.5 4:14.38 freeswitch 14055 ippbx 20 0 1555m 1.2g 10m S 0 30.5 4:58.50 freeswitch 14057 ippbx 20 0 1555m 1.2g 10m S 0 30.5 13:05.20 freeswitch 20511 ippbx 20 0 1555m 1.2g 10m S 0 30.5 0:00.14 freeswitch so only one process (PID: 14048) is causing that load. It's not the parent process (the initial FS startup process) as ps -elf shows: ippbx@ippbx-prod-node0:~/ippbx.prod$ ps -eLf | grep frees ippbx 14033 1 14033 0 28 Mar23 ? 00:00:01 bin/freeswitch -nc ippbx 14033 1 14034 0 28 Mar23 ? 00:00:08 bin/freeswitch -nc ippbx 14033 1 14035 0 28 Mar23 ? 00:03:39 bin/freeswitch -nc ippbx 14033 1 14036 0 28 Mar23 ? 00:00:07 bin/freeswitch -nc ippbx 14033 1 14037 0 28 Mar23 ? 00:00:00 bin/freeswitch -nc ippbx 14033 1 14038 0 28 Mar23 ? 00:00:00 bin/freeswitch -nc ippbx 14033 1 14039 0 28 Mar23 ? 00:00:02 bin/freeswitch -nc ippbx 14033 1 14042 0 28 Mar23 ? 00:03:41 bin/freeswitch -nc ippbx 14033 1 14043 0 28 Mar23 ? 00:00:03 bin/freeswitch -nc ippbx 14033 1 14044 0 28 Mar23 ? 00:00:01 bin/freeswitch -nc ippbx 14033 1 14045 0 28 Mar23 ? 00:00:25 bin/freeswitch -nc ippbx 14033 1 14046 0 28 Mar23 ? 00:01:20 bin/freeswitch -nc ippbx 14033 1 14047 0 28 Mar23 ? 00:05:32 bin/freeswitch -nc ippbx 14033 1 14048 7 28 Mar23 ? 05:33:35 bin/freeswitch -nc ippbx 14033 1 14049 0 28 Mar23 ? 00:05:07 bin/freeswitch -nc ippbx 14033 1 14050 0 28 Mar23 ? 00:01:01 bin/freeswitch -nc ippbx 14033 1 14051 0 28 Mar23 ? 00:25:43 bin/freeswitch -nc ippbx 14033 1 14052 0 28 Mar23 ? 00:00:01 bin/freeswitch -nc ippbx 14033 1 14054 0 28 Mar23 ? 00:04:14 bin/freeswitch -nc ippbx 14033 1 14055 0 28 Mar23 ? 00:04:58 bin/freeswitch -nc ippbx 14033 1 14056 0 28 Mar23 ? 00:06:23 bin/freeswitch -nc ippbx 14033 1 14057 0 28 Mar23 ? 00:13:05 bin/freeswitch -nc ippbx 14033 1 14058 0 28 Mar23 ? 00:00:00 bin/freeswitch -nc ippbx 14033 1 14059 0 28 Mar23 ? 00:00:00 bin/freeswitch -nc ippbx 14033 1 20518 0 28 15:02 ? 00:00:00 bin/freeswitch -nc ippbx 14033 1 20519 0 28 15:02 ? 00:00:00 bin/freeswitch -nc ippbx 14033 1 20521 0 28 15:02 ? 00:00:00 bin/freeswitch -nc ippbx 14033 1 20522 0 28 15:02 ? 00:00:00 bin/freeswitch -nc ippbx 20526 19854 20526 0 1 15:03 pts/0 00:00:00 grep frees Doing a strace on PID 14048 prints tons of "epoll_wait(21, {}, 4, 0) = 0" lines on the screen, which eats all of my desktop pc's cpu power :/ So can a developer say what this is, or what and how should I debug to find out the cause of this? Can I shot it down via kill or "kill -9" without crashing FS totally? regards helmut
gcore -o fs [pid here] gdb /path/to/fs core.file thread apply all bt then look for the thread and show me the backtrace. Math On 26-Mar-09, at 10:12 AM, Helmut Kuper wrote:
Hello,
since a few days I observe a high CPU load of my FS server, but I have no idea what it could be. There are only a few sessions running and there is only a few log activity. 2 days ago I restarted FS, but no change. The top command shows this:
top - 15:02:33 up 106 days, 30 min, 4 users, load average: 0.24, 0.35, 0.42 Tasks: 190 total, 1 running, 189 sleeping, 0 stopped, 0 zombie Cpu(s): 7.2%us, 12.2%sy, 0.0%ni, 80.0%id, 0.2%wa, 0.2%hi, 0.2%si, 0.0%st Mem: 4151776k total, 4003664k used, 148112k free, 414708k buffers Swap: 15623204k total, 88k used, 15623116k free, 2021412k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 14048 ippbx 20 0 1555m 1.2g 10m S 43 30.5 333:11.03 freeswitch 14049 ippbx 20 0 1555m 1.2g 10m S 0 30.5 5:06.88 freeswitch 14054 ippbx 20 0 1555m 1.2g 10m S 0 30.5 4:14.38 freeswitch 14055 ippbx 20 0 1555m 1.2g 10m S 0 30.5 4:58.50 freeswitch 14057 ippbx 20 0 1555m 1.2g 10m S 0 30.5 13:05.20 freeswitch 20511 ippbx 20 0 1555m 1.2g 10m S 0 30.5 0:00.14 freeswitch
so only one process (PID: 14048) is causing that load. It's not the parent process (the initial FS startup process) as ps -elf shows:
ippbx@ippbx-prod-node0:~/ippbx.prod$ ps -eLf | grep frees ippbx 14033 1 14033 0 28 Mar23 ? 00:00:01 bin/freeswitch -nc ippbx 14033 1 14034 0 28 Mar23 ? 00:00:08 bin/freeswitch -nc ippbx 14033 1 14035 0 28 Mar23 ? 00:03:39 bin/freeswitch -nc ippbx 14033 1 14036 0 28 Mar23 ? 00:00:07 bin/freeswitch -nc ippbx 14033 1 14037 0 28 Mar23 ? 00:00:00 bin/freeswitch -nc ippbx 14033 1 14038 0 28 Mar23 ? 00:00:00 bin/freeswitch -nc ippbx 14033 1 14039 0 28 Mar23 ? 00:00:02 bin/freeswitch -nc ippbx 14033 1 14042 0 28 Mar23 ? 00:03:41 bin/freeswitch -nc ippbx 14033 1 14043 0 28 Mar23 ? 00:00:03 bin/freeswitch -nc ippbx 14033 1 14044 0 28 Mar23 ? 00:00:01 bin/freeswitch -nc ippbx 14033 1 14045 0 28 Mar23 ? 00:00:25 bin/freeswitch -nc ippbx 14033 1 14046 0 28 Mar23 ? 00:01:20 bin/freeswitch -nc ippbx 14033 1 14047 0 28 Mar23 ? 00:05:32 bin/freeswitch -nc ippbx 14033 1 14048 7 28 Mar23 ? 05:33:35 bin/freeswitch -nc ippbx 14033 1 14049 0 28 Mar23 ? 00:05:07 bin/freeswitch -nc ippbx 14033 1 14050 0 28 Mar23 ? 00:01:01 bin/freeswitch -nc ippbx 14033 1 14051 0 28 Mar23 ? 00:25:43 bin/freeswitch -nc ippbx 14033 1 14052 0 28 Mar23 ? 00:00:01 bin/freeswitch -nc ippbx 14033 1 14054 0 28 Mar23 ? 00:04:14 bin/freeswitch -nc ippbx 14033 1 14055 0 28 Mar23 ? 00:04:58 bin/freeswitch -nc ippbx 14033 1 14056 0 28 Mar23 ? 00:06:23 bin/freeswitch -nc ippbx 14033 1 14057 0 28 Mar23 ? 00:13:05 bin/freeswitch -nc ippbx 14033 1 14058 0 28 Mar23 ? 00:00:00 bin/freeswitch -nc ippbx 14033 1 14059 0 28 Mar23 ? 00:00:00 bin/freeswitch -nc ippbx 14033 1 20518 0 28 15:02 ? 00:00:00 bin/freeswitch -nc ippbx 14033 1 20519 0 28 15:02 ? 00:00:00 bin/freeswitch -nc ippbx 14033 1 20521 0 28 15:02 ? 00:00:00 bin/freeswitch -nc ippbx 14033 1 20522 0 28 15:02 ? 00:00:00 bin/freeswitch -nc ippbx 20526 19854 20526 0 1 15:03 pts/0 00:00:00 grep frees
Doing a strace on PID 14048 prints tons of "epoll_wait(21, {}, 4, 0) = 0" lines on the screen, which eats all of my desktop pc's cpu power :/
So can a developer say what this is, or what and how should I debug to find out the cause of this? Can I shot it down via kill or "kill -9" without crashing FS totally?
regards helmut
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Hi Mathieu, thx for the help :) The epoll_wait function on PID 14048 listens on fd 21 which points to "/anon_inode:[eventpoll]" On 26.03.2009 15:26, Mathieu Rene wrote:
thread apply all bt Here the output:
Core was generated by `/opt/app/voip/ippbx.prod/bin/freeswitch'. [New process 14034] [New process 14035] [New process 14036] [New process 14037] [New process 14038] [New process 14039] [New process 14042] [New process 14043] [New process 14044] [New process 14045] [New process 14046] [New process 14047] [New process 14048] [New process 14049] [New process 14050] [New process 14051] [New process 14052] [New process 14054] [New process 14055] [New process 14056] [New process 14057] [New process 14058] [New process 14059] [New process 14033] #0 0xb7ee6410 in __kernel_vsyscall () (gdb) (gdb) (gdb) (gdb) (gdb) (gdb) (gdb) (gdb) (gdb) (gdb) (gdb) (gdb) (gdb) (gdb) (gdb) (gdb) (gdb) (gdb) (gdb) (gdb) (gdb) thread apply all bt Thread 24 (process 14033): #0 0xb7ee6410 in __kernel_vsyscall () #1 0xb7c44881 in select () from /lib/tls/i686/cmov/libc.so.6 #2 0xb7e00a59 in apr_sleep (t=1000000) at time/unix/time.c:246 #3 0xb7ddf09e in do_sleep (t=4294966782) at src/switch_time.c:109 #4 0xb7d9e334 in switch_core_runtime_loop (bg=1) at src/switch_core.c:656 #5 0x0804a459 in main (argc=2, argv=0xbf9d27a4) at src/switch.c:666 Thread 23 (process 14059): #0 0xb7ee6410 in __kernel_vsyscall () #1 0xb7c41c07 in poll () from /lib/tls/i686/cmov/libc.so.6 #2 0xb36ef7e1 in ?? () from /opt/app/voip/ippbx.prod/mod/mod_xml_rpc.so #3 0xb36e2612 in ChanSwitchAccept (chanSwitchP=0x81cce68, channelPP=0xab1f00e0, channelInfoPP=0xab1f00dc, errorP=0xab1f00e4) at ../../../../libs/xmlrpc-c/lib/abyss/src/chanswitch.c:149 #4 0xb36ee351 in ServerRun (serverP=0xb372092c) at ../../../../libs/xmlrpc-c/lib/abyss/src/server.c:908 #5 0xb36df892 in mod_xml_rpc_runtime () at mod_xml_rpc.c:867 #6 0xb7da8473 in switch_loadable_module_exec (thread=0x80c38a8, obj=0x80c3698) at src/switch_loadable_module.c:94 #7 0xb7dfeeb6 in dummy_worker (opaque=0x80c38a8) at threadproc/unix/thread.c:138 #8 0xb7cf94fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #9 0xb7c4be5e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 22 (process 14058): #0 0xb7ee6410 in __kernel_vsyscall () #1 0xb7d00bb8 in accept () from /lib/tls/i686/cmov/libpthread.so.0 #2 0xb7dfdbcd in apr_socket_accept (new=0xab9f134c, sock=0x82639e8, connection_context=0x66a76500) at network_io/unix/sockets.c:187 #3 0xb7d7cf4b in switch_socket_accept (new_sock=0xab9f134c, sock=0x82639e8, pool=0x66a76500) at src/switch_apr.c:664 #4 0xb328ff2c in mod_event_socket_runtime () at mod_event_socket.c:2293 #5 0xb7da8473 in switch_loadable_module_exec (thread=0x80c3640, obj=0x80c3430) at src/switch_loadable_module.c:94 #6 0xb7dfeeb6 in dummy_worker (opaque=0x80c3640) at threadproc/unix/thread.c:138 #7 0xb7cf94fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #8 0xb7c4be5e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 21 (process 14057): #0 0xb7ee6410 in __kernel_vsyscall () #1 0xb7c44881 in select () from /lib/tls/i686/cmov/libc.so.6 #2 0xb7e00a59 in apr_sleep (t=1000) at time/unix/time.c:246 #3 0xb7ddf09e in do_sleep (t=4294966782) at src/switch_time.c:109 #4 0xb7de0c25 in softtimer_runtime () at src/switch_time.c:459 #5 0xb7da8473 in switch_loadable_module_exec (thread=0x80c33d8, obj=0x80c31c8) at src/switch_loadable_module.c:94 #6 0xb7dfeeb6 in dummy_worker (opaque=0x80c33d8) at threadproc/unix/thread.c:138 #7 0xb7cf94fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #8 0xb7c4be5e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 20 (process 14056): #0 0xb7ee6410 in __kernel_vsyscall () #1 0xb7cfdaa5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/cmov/libpthread.so.0 #2 0xb7df8f6a in apr_thread_cond_wait (cond=0x80c4760, mutex=0x80c4730) at locks/unix/thread_cond.c:68 #3 0xb7d7d8b4 in switch_thread_cond_wait (cond=0x80c4760, mutex=0x80c4730) at src/switch_apr.c:359 #4 0xb7ddf2c8 in timer_next (timer=0xaccae0f8) at src/switch_time.c:335 #5 0xb7d8ad7c in switch_core_timer_next (timer=0x26f180a8) at src/switch_core_timer.c:76 ---Type <return> to continue, or q <return> to quit--- #6 0xad4e67dc in ?? () from /opt/app/voip/ippbx.prod/mod/mod_local_stream.so #7 0xb7dfeeb6 in dummy_worker (opaque=0xb272b2e0) at threadproc/unix/thread.c:138 #8 0xb7cf94fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #9 0xb7c4be5e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 19 (process 14055): #0 0xb7ee6410 in __kernel_vsyscall () #1 0xb7cfdaa5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/cmov/libpthread.so.0 #2 0xb7df8f6a in apr_thread_cond_wait (cond=0x80c4760, mutex=0x80c4730) at locks/unix/thread_cond.c:68 #3 0xb7d7d8b4 in switch_thread_cond_wait (cond=0x80c4760, mutex=0x80c4730) at src/switch_apr.c:359 #4 0xb7ddf2c8 in timer_next (timer=0xad4af0f8) at src/switch_time.c:335 #5 0xb7d8ad7c in switch_core_timer_next (timer=0x26f180a5) at src/switch_core_timer.c:76 #6 0xad4e67dc in ?? () from /opt/app/voip/ippbx.prod/mod/mod_local_stream.so #7 0xb7dfeeb6 in dummy_worker (opaque=0xb27272d0) at threadproc/unix/thread.c:138 #8 0xb7cf94fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #9 0xb7c4be5e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 18 (process 14054): #0 0xb7ee6410 in __kernel_vsyscall () #1 0xb7cfdaa5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/cmov/libpthread.so.0 #2 0xb7df8f6a in apr_thread_cond_wait (cond=0x80c4760, mutex=0x80c4730) at locks/unix/thread_cond.c:68 #3 0xb7d7d8b4 in switch_thread_cond_wait (cond=0x80c4760, mutex=0x80c4730) at src/switch_apr.c:359 #4 0xb7ddf2c8 in timer_next (timer=0xadf700f8) at src/switch_time.c:335 #5 0xb7d8ad7c in switch_core_timer_next (timer=0x26f180a6) at src/switch_core_timer.c:76 #6 0xad4e67dc in ?? () from /opt/app/voip/ippbx.prod/mod/mod_local_stream.so #7 0xb7dfeeb6 in dummy_worker (opaque=0xb27232c0) at threadproc/unix/thread.c:138 #8 0xb7cf94fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #9 0xb7c4be5e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 17 (process 14052): #0 0xb7ee6410 in __kernel_vsyscall () #1 0xb7c44881 in select () from /lib/tls/i686/cmov/libc.so.6 #2 0xb7e00a59 in apr_sleep (t=1000000) at time/unix/time.c:246 #3 0xb7ddf09e in do_sleep (t=4294966782) at src/switch_time.c:109 #4 0xaedcce06 in ?? () from /opt/app/voip/ippbx.prod/mod/mod_fifo.so #5 0xb7dfeeb6 in dummy_worker (opaque=0xaea4cb90) at threadproc/unix/thread.c:138 #6 0xb7cf94fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #7 0xb7c4be5e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 16 (process 14051): #0 0xb7ee6410 in __kernel_vsyscall () #1 0xb7c44881 in select () from /lib/tls/i686/cmov/libc.so.6 #2 0xb7e00a59 in apr_sleep (t=100000) at time/unix/time.c:246 #3 0xb7ddf09e in do_sleep (t=4294966782) at src/switch_time.c:109 #4 0xb315892f in sofia_presence_event_thread_run (thread=0x80f1bb0, obj=0x0) at sofia_presence.c:664 #5 0xb7dfeeb6 in dummy_worker (opaque=0x80f1bb0) at threadproc/unix/thread.c:138 #6 0xb7cf94fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #7 0xb7c4be5e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 15 (process 14050): #0 0xb7ee6410 in __kernel_vsyscall () ---Type <return> to continue, or q <return> to quit--- #1 0xb7c41c07 in poll () from /lib/tls/i686/cmov/libc.so.6 #2 0xb280bf00 in wanpipe_wait (zchan=0x8146fc8, flags=0xafe0ef8c, to=100) at src/ozmod/ozmod_wanpipe/ozmod_wanpipe.c:255 #3 0xafeecb17 in zap_channel_wait (zchan=0x64, flags=0xafe0ef8c, to=100) at src/zap_io.c:1488 #4 0xafebfb54 in zap_isdn_run (me=0x8112ab0, obj=0x811ff58) at src/ozmod/ozmod_isdn/ozmod_isdn.c:1736 #5 0xafef747a in thread_launch (args=0x8112ab0) at src/zap_threadmutex.c:74 #6 0xb7cf94fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #7 0xb7c4be5e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 14 (process 14049): #0 0xb7ee6410 in __kernel_vsyscall () #1 0xb7c44881 in select () from /lib/tls/i686/cmov/libc.so.6 #2 0xb7e00a59 in apr_sleep (t=10000) at time/unix/time.c:246 #3 0xb7ddf09e in do_sleep (t=4294966782) at src/switch_time.c:109 #4 0xb313e306 in sofia_profile_worker_thread_run (thread=0x81004c8, obj=0x80ffb38) at sofia.c:656 #5 0xb7dfeeb6 in dummy_worker (opaque=0x81004c8) at threadproc/unix/thread.c:138 #6 0xb7cf94fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #7 0xb7c4be5e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 13 (process 14048): #0 0xb7ee6410 in __kernel_vsyscall () #1 0xb7c4c676 in epoll_wait () from /lib/tls/i686/cmov/libc.so.6 #2 0xb321998d in su_epoll_port_wait_events (self=0x8107f70, tout=1000) at su_epoll_port.c:495 #3 0xb3220218 in su_base_port_run (self=0x8107f70) at su_base_port.c:349 #4 0xb321af8f in su_port_run (self=0x8107f70) at su_port.h:326 #5 0xb321af6c in su_root_run (self=0x8107770) at su_root.c:819 #6 0xb320af0a in su_pthread_port_clone_main (varg=0xb16fd098) at su_pthread_port.c:324 #7 0xb7cf94fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #8 0xb7c4be5e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 12 (process 14047): #0 0xb7ee6410 in __kernel_vsyscall () #1 0xb7c4c676 in epoll_wait () from /lib/tls/i686/cmov/libc.so.6 #2 0xb321998d in su_epoll_port_wait_events (self=0x80fbaa8, tout=1000) at su_epoll_port.c:495 #3 0xb3220369 in su_base_port_step (self=0x80fbaa8, tout=1000) at su_base_port.c:467 #4 0xb321b0b5 in su_port_step (self=0x80fbaa8, tout=1000) at su_port.h:340 #5 0xb321b085 in su_root_step (self=0x80fba70, tout=1000) at su_root.c:858 #6 0xb3148442 in sofia_profile_thread_run (thread=0x81003e8, obj=0x80ffb38) at sofia.c:831 #7 0xb7dfeeb6 in dummy_worker (opaque=0x81003e8) at threadproc/unix/thread.c:138 #8 0xb7cf94fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #9 0xb7c4be5e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 11 (process 14046): #0 0xb7ee6410 in __kernel_vsyscall () #1 0xb7c44881 in select () from /lib/tls/i686/cmov/libc.so.6 #2 0xb7e00a59 in apr_sleep (t=10000) at time/unix/time.c:246 #3 0xb7ddf09e in do_sleep (t=4294966782) at src/switch_time.c:109 #4 0xb313e306 in sofia_profile_worker_thread_run (thread=0x80f5030, obj=0x80f4280) at sofia.c:656 #5 0xb7dfeeb6 in dummy_worker (opaque=0x80f5030) at threadproc/unix/thread.c:138 #6 0xb7cf94fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #7 0xb7c4be5e in clone () from /lib/tls/i686/cmov/libc.so.6 ---Type <return> to continue, or q <return> to quit--- Thread 10 (process 14045): #0 0xb7ee6410 in __kernel_vsyscall () #1 0xb7c4c676 in epoll_wait () from /lib/tls/i686/cmov/libc.so.6 #2 0xb321998d in su_epoll_port_wait_events (self=0x80fd1a0, tout=1000) at su_epoll_port.c:495 #3 0xb3220218 in su_base_port_run (self=0x80fd1a0) at su_base_port.c:349 #4 0xb321af8f in su_port_run (self=0x80fd1a0) at su_port.h:326 #5 0xb321af6c in su_root_run (self=0x80fef98) at su_root.c:819 #6 0xb320af0a in su_pthread_port_clone_main (varg=0xb30ac098) at su_pthread_port.c:324 #7 0xb7cf94fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #8 0xb7c4be5e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 9 (process 14044): #0 0xb7ee6410 in __kernel_vsyscall () #1 0xb7c4c676 in epoll_wait () from /lib/tls/i686/cmov/libc.so.6 #2 0xb321998d in su_epoll_port_wait_events (self=0x80e0e80, tout=1000) at su_epoll_port.c:495 #3 0xb3220369 in su_base_port_step (self=0x80e0e80, tout=1000) at su_base_port.c:467 #4 0xb321b0b5 in su_port_step (self=0x80e0e80, tout=1000) at su_port.h:340 #5 0xb321b085 in su_root_step (self=0x80f8528, tout=1000) at su_root.c:858 #6 0xb3148442 in sofia_profile_thread_run (thread=0x80f4f50, obj=0x80f4280) at sofia.c:831 #7 0xb7dfeeb6 in dummy_worker (opaque=0x80f4f50) at threadproc/unix/thread.c:138 #8 0xb7cf94fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #9 0xb7c4be5e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 8 (process 14043): #0 0xb7ee6410 in __kernel_vsyscall () #1 0xb7c44881 in select () from /lib/tls/i686/cmov/libc.so.6 #2 0xb7e00a59 in apr_sleep (t=500000) at time/unix/time.c:246 #3 0xb7ddf09e in do_sleep (t=4294966782) at src/switch_time.c:109 #4 0xb7da20a4 in switch_scheduler_task_thread (thread=0x80be590, obj=0x0) at src/switch_scheduler.c:171 #5 0xb7dfeeb6 in dummy_worker (opaque=0x80be590) at threadproc/unix/thread.c:138 #6 0xb7cf94fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #7 0xb7c4be5e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 7 (process 14042): #0 0xb7ee6410 in __kernel_vsyscall () #1 0xb7cfdaa5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/cmov/libpthread.so.0 #2 0xb7df8f6a in apr_thread_cond_wait (cond=0x80c4760, mutex=0x80c4730) at locks/unix/thread_cond.c:68 #3 0xb7d7d8b4 in switch_thread_cond_wait (cond=0x80c4760, mutex=0x80c4730) at src/switch_apr.c:359 #4 0xb7ddf4d6 in switch_cond_next () at src/switch_time.c:203 #5 0xb7d91b5b in switch_core_sql_thread (thread=0xb3782ae8, obj=0x0) at src/switch_core_sqldb.c:220 #6 0xb7dfeeb6 in dummy_worker (opaque=0xb3782ae8) at threadproc/unix/thread.c:138 #7 0xb7cf94fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #8 0xb7c4be5e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 6 (process 14039): #0 0xb7ee6410 in __kernel_vsyscall () #1 0xb7cfdaa5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/cmov/libpthread.so.0 #2 0xb7df8f6a in apr_thread_cond_wait (cond=0xb70ddc60, mutex=0xb70ddc30) at locks/unix/thread_cond.c:68 #3 0xb7defcfc in apr_queue_pop (queue=0xb70ddc00, data=0xb4fe93a8) at misc/apr_queue.c:276 #4 0xb7d7c884 in switch_queue_pop (queue=0xb70ddc00, data=0xb4fe93a8) at src/switch_apr.c:879 #5 0xb7dd7bed in log_thread (t=0xb504bae0, obj=0x0) at src/switch_log.c:213 ---Type <return> to continue, or q <return> to quit--- #6 0xb7dfeeb6 in dummy_worker (opaque=0xb504bae0) at threadproc/unix/thread.c:138 #7 0xb7cf94fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #8 0xb7c4be5e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 5 (process 14038): #0 0xb7ee6410 in __kernel_vsyscall () #1 0xb7cfdaa5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/cmov/libpthread.so.0 #2 0xb7df8f6a in apr_thread_cond_wait (cond=0xb7140b38, mutex=0xb7140b08) at locks/unix/thread_cond.c:68 #3 0xb7defcfc in apr_queue_pop (queue=0xb7140ad8, data=0xb58783a8) at misc/apr_queue.c:276 #4 0xb7d7c884 in switch_queue_pop (queue=0xb7140ad8, data=0xb58783a8) at src/switch_apr.c:879 #5 0xb7dae2d0 in switch_event_thread (thread=0x8068228, obj=0xb7140ad8) at src/switch_event.c:284 #6 0xb7dfeeb6 in dummy_worker (opaque=0x8068228) at threadproc/unix/thread.c:138 #7 0xb7cf94fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #8 0xb7c4be5e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 4 (process 14037): #0 0xb7ee6410 in __kernel_vsyscall () #1 0xb7cfdaa5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/cmov/libpthread.so.0 #2 0xb7df8f6a in apr_thread_cond_wait (cond=0xb71a3b38, mutex=0xb71a3b08) at locks/unix/thread_cond.c:68 #3 0xb7defcfc in apr_queue_pop (queue=0xb71a3ad8, data=0xb60793a8) at misc/apr_queue.c:276 #4 0xb7d7c884 in switch_queue_pop (queue=0xb71a3ad8, data=0xb60793a8) at src/switch_apr.c:879 #5 0xb7dae2d0 in switch_event_thread (thread=0x8068208, obj=0xb71a3ad8) at src/switch_event.c:284 #6 0xb7dfeeb6 in dummy_worker (opaque=0x8068208) at threadproc/unix/thread.c:138 #7 0xb7cf94fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #8 0xb7c4be5e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 3 (process 14036): #0 0xb7ee6410 in __kernel_vsyscall () #1 0xb7cfdaa5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/cmov/libpthread.so.0 #2 0xb7df8f6a in apr_thread_cond_wait (cond=0x805e548, mutex=0x805e518) at locks/unix/thread_cond.c:68 #3 0xb7defcfc in apr_queue_pop (queue=0x805e4e8, data=0xb687a3a8) at misc/apr_queue.c:276 #4 0xb7d7c884 in switch_queue_pop (queue=0x805e4e8, data=0xb687a3a8) at src/switch_apr.c:879 #5 0xb7dae2d0 in switch_event_thread (thread=0x80681e8, obj=0x805e4e8) at src/switch_event.c:284 #6 0xb7dfeeb6 in dummy_worker (opaque=0x80681e8) at threadproc/unix/thread.c:138 #7 0xb7cf94fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #8 0xb7c4be5e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 2 (process 14035): #0 0xb7ee6410 in __kernel_vsyscall () #1 0xb7cfdaa5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/cmov/libpthread.so.0 #2 0xb7df8f6a in apr_thread_cond_wait (cond=0xb70ddb38, mutex=0xb70ddb08) at locks/unix/thread_cond.c:68 #3 0xb7defcfc in apr_queue_pop (queue=0xb70ddad8, data=0xb707b3a8) at misc/apr_queue.c:276 #4 0xb7d7c884 in switch_queue_pop (queue=0xb70ddad8, data=0xb707b3a8) at src/switch_apr.c:879 #5 0xb7daf069 in switch_event_dispatch_thread (thread=0x80681c8, obj=0xb70ddad8) at src/switch_event.c:241 #6 0xb7dfeeb6 in dummy_worker (opaque=0x80681c8) at threadproc/unix/thread.c:138 #7 0xb7cf94fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #8 0xb7c4be5e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 1 (process 14034): #0 0xb7ee6410 in __kernel_vsyscall () #1 0xb7c44881 in select () from /lib/tls/i686/cmov/libc.so.6 ---Type <return> to continue, or q <return> to quit--- #2 0xb7e00a59 in apr_sleep (t=100000) at time/unix/time.c:246 #3 0xb7ddf09e in do_sleep (t=4294966782) at src/switch_time.c:109 #4 0xb7d8d52b in pool_thread (thread=0xb79d6da8, obj=0x0) at src/switch_core_memory.c:423 #5 0xb7dfeeb6 in dummy_worker (opaque=0xb79d6da8) at threadproc/unix/thread.c:138 #6 0xb7cf94fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #7 0xb7c4be5e in clone () from /lib/tls/i686/cmov/libc.so.6
Before we go any further... what SVN rev are you on? And by heavy load what does your load average say? /b On Mar 26, 2009, at 10:01 AM, Helmut Kuper wrote:
Hi Mathieu,
thx for the help :)
The epoll_wait function on PID 14048 listens on fd 21 which points to "/anon_inode:[eventpoll]"
Brian West brian@freeswitch.org -- Meet us a ClueCon! http://www.cluecon.com
Hello Brian, On 26.03.2009 16:07, Brian West wrote:
Before we go any further... what SVN rev are you on? And by heavy load what does your load average say?
I'm using "FreeSWITCH Version 1.0.trunk (12347M)" my average load is currently this: top - 16:22:00 up 106 days, 1:49, 3 users, load average: 0.64, 0.76, 0.73 FS status is: freeswitch@internal> status UP 0 years, 3 days, 0 hours, 54 minutes, 0 seconds, 407 milliseconds, 119 microseconds 2029 session(s) since startup 4 session(s) 0/30 regards helmut
Hello, today I killed that special thread via "kill -9" a simple kill didn't helped. Unfortunately this led to a normal shutdown of FS although I killed not the parent process. :( After restart of FS the server has a normal load again. regards and a nice weekend Helmut
kill -9 on a thread will kill the process which kills freeswitch. On Fri, Mar 27, 2009 at 10:09 AM, Helmut Kuper <helmut.kuper@ewetel.de>wrote:
Hello,
today I killed that special thread via "kill -9" a simple kill didn't helped. Unfortunately this led to a normal shutdown of FS although I killed not the parent process. :(
After restart of FS the server has a normal load again.
regards and a nice weekend Helmut
-- -Rupa
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello, I still have this problem. From the day of starting up freeswitch two threads are consuming slowly more and more CPU power. In parallel FS virtual and physical memory usage grows slowly as well. FS is up for 6 days now and served 3160 sessions. Virtual memory usage has grown from 200MB to 1.1GB (18,1%) and is still growing. CPU is mostly around 25% with lowest of 17% and a maximum of 50% (all on a 32 bit 4 CPU core system) and is still growing. There are two FS-Threads with nearly same CPU usage of currently around 20% each (I used htop for this): strace Thread 1 (I guess this is the sofia/sip thread): epoll_wait(21, {}, 4, 0) = 0 epoll_wait(21, {}, 4, 0) = 0 epoll_wait(21, {{EPOLLIN, {u32=2, u64=2}}}, 4, 0) = 1 ioctl(24, FIONREAD, [268]) = 0 recvmsg(24, {msg_name(16)={sa_family=AF_INET, sin_port=htons(1068), sin_addr=inet_addr("85.16.245.249")}, msg_iov(1)=[{"SIP/2.0 200 Ok\r\nVia: SIP/2.0/UDP"..., 268}], msg_controllen=0, msg_flags=0}, 0) = 268 gettimeofday({1239012809, 343580}, NULL) = 0 gettimeofday({1239012809, 343645}, NULL) = 0 epoll_wait(21, {}, 4, 0) = 0 epoll_wait(21, {}, 4, 0) = 0 ... strace Thread 2: select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout) select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout) select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout) select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout) select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout) select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout) ... I use FreeSWITCH Version 1.0.trunk (12347M) regards helmut -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (MingW32) iD8DBQFJ2ddp4tZeNddg3dwRAlkXAJ9fIwpJw6u18JPhFC4hzB+0Z1iAbgCfW7AE dnrmpXDLVOnWtjwFKMoVw48= =zzZ9 -----END PGP SIGNATURE-----
Hi Mathieu I straced process 14047 as well and I found that it also epolls a anon_inode. But the process' epoll call lokk slightly different that 14048's: epoll_wait(13, {}, 4, 1000) = 0 Here a timeout is given. 14048 is not using a timeout (0). Maybe this helps you ... regards Helmut On 26.03.2009 16:01, Helmut Kuper wrote:
Hi Mathieu,
thx for the help :)
The epoll_wait function on PID 14048 listens on fd 21 which points to "/anon_inode:[eventpoll]"
participants (4)
-
Brian West
-
Helmut Kuper
-
Mathieu Rene
-
Rupa Schomaker