Hello List. I’m using a Freeswitch server as a B2BUA transcoder. I have a a HP Proliant DL380 G7 with 2 Quad Cores and 8 Gig memory. I’m using Centos 5.5 64bits [root@transcoder sngtc]# more /etc/issue CentOS release 5.5 (Final) Kernel \r on an \m [root@transcoder sngtc]# uname -a Linux transcoder.redvoiss.net 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux I also have two transcoding cards Sangoma D100e. The problem is that I’m having weird spikes of “load average” cpu every 20 minutes (aprox), without any reason, reaching near of 12.5 of load average for this machine. I have an average of 200 simultaneous calls (all doing transcoding from G711 to G729) and I don’t have any crontab job running. What could be the cause of these spikes? I have been trying to optimize the configuration but the cpu spikes persists. Can someone help me to find the problem here? Thanks in advance. * * *Ricardo Martinez.-*
On Mon, Oct 22, 2012 at 8:37 AM, Ricardo Martinez <rmartinez@redvoiss.net> wrote:
The problem is that I’m having weird spikes of “load average” cpu every 20 minutes (aprox), without any reason, reaching near of 12.5 of load average for this machine.
What have you done to verify that it happens 'without any reason'? Gabe
Hi Gabe. I'm mean between the spikes there is nothing different as regarding to the calls... the cps maintains the same.. there are no scripts running or crontabs running, the simultaneous calls are the same also. So, for some reason the "load average" start to grow (every 20 or 25 minutes) without any clear cause. I have debug with gdb the moment the server is with 12.5 of "load average" here : http://pastebin.freeswitch.org/20112 Can we detect a problem from here? Regards, Ricardo. -----Mensaje original----- De: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] En nombre de Gabriel Gunderson Enviado el: lunes, 22 de octubre de 2012 13:31 Para: FreeSWITCH Users Help Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes On Mon, Oct 22, 2012 at 8:37 AM, Ricardo Martinez <rmartinez@redvoiss.net> wrote:
The problem is that I'm having weird spikes of "load average" cpu every 20 minutes (aprox), without any reason, reaching near of 12.5 of load average for this machine.
What have you done to verify that it happens 'without any reason'? Gabe _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Do you have anything like Java running that could be doing garbage collection? What other processes are running on the machine? Does TOP show which process is causing the spike? Guillermo On Mon, Oct 22, 2012 at 3:29 PM, Ricardo Martinez <rmartinez@redvoiss.net>wrote:
Hi Gabe. I'm mean between the spikes there is nothing different as regarding to the calls... the cps maintains the same.. there are no scripts running or crontabs running, the simultaneous calls are the same also. So, for some reason the "load average" start to grow (every 20 or 25 minutes) without any clear cause. I have debug with gdb the moment the server is with 12.5 of "load average" here : http://pastebin.freeswitch.org/20112
Can we detect a problem from here?
Regards, Ricardo.
-----Mensaje original----- De: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] En nombre de Gabriel Gunderson Enviado el: lunes, 22 de octubre de 2012 13:31 Para: FreeSWITCH Users Help Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes
On Mon, Oct 22, 2012 at 8:37 AM, Ricardo Martinez <rmartinez@redvoiss.net> wrote:
The problem is that I'm having weird spikes of "load average" cpu every 20 minutes (aprox), without any reason, reaching near of 12.5 of load average for this machine.
What have you done to verify that it happens 'without any reason'?
Gabe
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Guillermo Ruiz Camauer
AS far as i can see there is no java running. This is the top at the moment of the spike : top - 15:48:29 up 17 days, 14:55, 3 users, load average: 14.07, 8.87, 6.48 Tasks: 158 total, 1 running, 157 sleeping, 0 stopped, 0 zombie Cpu(s): 12.6%us, 9.0%sy, 0.0%ni, 66.1%id, 2.7%wa, 0.3%hi, 9.2%si, 0.0%st Mem: 8171500k total, 8041556k used, 129944k free, 177688k buffers Swap: 10223608k total, 0k used, 10223608k free, 6951988k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31772 root 15 0 870m 481m 6880 S 212.4 6.0 9523:51 freeswitch 4210 root 19 0 36344 2940 1444 S 1.0 0.0 56:21.08 sngtc_server 29586 root 15 0 12740 1132 808 R 0.3 0.0 0:00.02 top 1 root 15 0 10348 704 588 S 0.0 0.0 0:01.13 init 2 root RT -5 0 0 0 S 0.0 0.0 0:00.41 migration/0 3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0 4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0 5 root RT -5 0 0 0 S 0.0 0.0 0:00.83 migration/1 6 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1 7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1 8 root RT -5 0 0 0 S 0.0 0.0 0:00.14 migration/2 9 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/2 10 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/2 11 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/3 12 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/3 13 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/3 14 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/4 15 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/4 16 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/4 17 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/5 18 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/5 19 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/5 20 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/6 Regards, Ricardo.- *De:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *En nombre de *Guillermo Ruiz Camauer *Enviado el:* lunes, 22 de octubre de 2012 15:38 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes Do you have anything like Java running that could be doing garbage collection? What other processes are running on the machine? Does TOP show which process is causing the spike? Guillermo On Mon, Oct 22, 2012 at 3:29 PM, Ricardo Martinez <rmartinez@redvoiss.net> wrote: Hi Gabe. I'm mean between the spikes there is nothing different as regarding to the calls... the cps maintains the same.. there are no scripts running or crontabs running, the simultaneous calls are the same also. So, for some reason the "load average" start to grow (every 20 or 25 minutes) without any clear cause. I have debug with gdb the moment the server is with 12.5 of "load average" here : http://pastebin.freeswitch.org/20112 Can we detect a problem from here? Regards, Ricardo. -----Mensaje original----- De: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] En nombre de Gabriel Gunderson Enviado el: lunes, 22 de octubre de 2012 13:31 Para: FreeSWITCH Users Help Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes On Mon, Oct 22, 2012 at 8:37 AM, Ricardo Martinez <rmartinez@redvoiss.net> wrote:
The problem is that I'm having weird spikes of "load average" cpu every 20 minutes (aprox), without any reason, reaching near of 12.5 of load average for this machine.
What have you done to verify that it happens 'without any reason'? Gabe _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Guillermo Ruiz Camauer
This is the gdb from the proccess consuming more cpu : (gdb) thread 7 [Switching to thread 7 (Thread 31781)]#0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 (gdb) bt #0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 #1 0x00002aede19276de in unixSync (id=0x2aaaac726380, dataOnly=<value optimized out>) at ./src/os_unix.c:1229 #2 0x00002aede1951178 in syncJournal (pPager=0x1a1c200) at ./src/pager.c:2229 #3 0x00002aede1953b65 in sqlite3pager_sync (pPager=0x1a1c200, zMaster=0x0, nTrunc=0) at ./src/pager.c:3770 #4 0x00002aede193d07b in sqlite3BtreeSync (p=<value optimized out>, zMaster=0x0) at ./src/btree.c:6520 #5 0x00002aede1931f15 in sqlite3VdbeHalt (p=0x2aaaad17ac70) at ./src/vdbeaux.c:1062 #6 0x00002aede1962613 in sqlite3VdbeExec (p=0x2aaaad17ac70) at ./src/vdbe.c:2360 #7 0x00002aede193036c in sqlite3Step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:236 #8 sqlite3_step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:289 #9 0x00002aede1938978 in sqlite3_exec (db=0x1a1f070, zSql=0x2aede199e717 "COMMIT", xCallback=0, pArg=0x0, pzErrMsg=0x4178bed0) at ./src/legacy.c:78 #10 0x00002aede18ae0c1 in switch_core_db_exec (db=0x1a1f070, sql=0x2aede199e717 "COMMIT", callback=0, data=0x0, errmsg=0x4178bf38) at src/switch_core_db.c:93 #11 0x00002aede1897c18 in switch_cache_db_execute_sql_real (dbh=0x1a2dce0, sql=0x2aede199e717 "COMMIT", err=0x0) at src/switch_core_sqldb.c:445 #12 0x00002aede18985c3 in switch_cache_db_persistant_execute_trans (dbh=0x1a2dce0, sql=0x5f03bd0 "delete from calls where (caller_uuid='e64319c5-7f91-47a1-9561-42da291e499a' or callee_uuid='e64319c5-7f91-47a1-9561-42da291e499a') and hostname=' transcoder.red.net';\ndelete from calls where (call"..., retries=1) at src/switch_core_sqldb.c:787 #13 0x00002aede189bf1c in switch_core_sql_thread (thread=<value optimized out>, obj=<value optimized out>) at src/switch_core_sqldb.c:1026 #14 0x0000003c03c0673d in start_thread () from /lib64/libpthread.so.0 #15 0x0000003c030d3f6d in clone () from /lib64/libc.so.6 *De:* Ricardo Martinez [mailto:rmartinez@redvoiss.net] *Enviado el:* lunes, 22 de octubre de 2012 15:50 *Para:* 'FreeSWITCH Users Help' *Asunto:* RE: [Freeswitch-users] CPU Spikes every 20 minutes AS far as i can see there is no java running. This is the top at the moment of the spike : top - 15:48:29 up 17 days, 14:55, 3 users, load average: 14.07, 8.87, 6.48 Tasks: 158 total, 1 running, 157 sleeping, 0 stopped, 0 zombie Cpu(s): 12.6%us, 9.0%sy, 0.0%ni, 66.1%id, 2.7%wa, 0.3%hi, 9.2%si, 0.0%st Mem: 8171500k total, 8041556k used, 129944k free, 177688k buffers Swap: 10223608k total, 0k used, 10223608k free, 6951988k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31772 root 15 0 870m 481m 6880 S 212.4 6.0 9523:51 freeswitch 4210 root 19 0 36344 2940 1444 S 1.0 0.0 56:21.08 sngtc_server 29586 root 15 0 12740 1132 808 R 0.3 0.0 0:00.02 top 1 root 15 0 10348 704 588 S 0.0 0.0 0:01.13 init 2 root RT -5 0 0 0 S 0.0 0.0 0:00.41 migration/0 3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0 4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0 5 root RT -5 0 0 0 S 0.0 0.0 0:00.83 migration/1 6 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1 7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1 8 root RT -5 0 0 0 S 0.0 0.0 0:00.14 migration/2 9 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/2 10 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/2 11 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/3 12 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/3 13 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/3 14 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/4 15 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/4 16 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/4 17 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/5 18 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/5 19 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/5 20 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/6 Regards, Ricardo.- *De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Guillermo Ruiz Camauer *Enviado el:* lunes, 22 de octubre de 2012 15:38 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes Do you have anything like Java running that could be doing garbage collection? What other processes are running on the machine? Does TOP show which process is causing the spike? Guillermo On Mon, Oct 22, 2012 at 3:29 PM, Ricardo Martinez <rmartinez@redvoiss.net> wrote: Hi Gabe. I'm mean between the spikes there is nothing different as regarding to the calls... the cps maintains the same.. there are no scripts running or crontabs running, the simultaneous calls are the same also. So, for some reason the "load average" start to grow (every 20 or 25 minutes) without any clear cause. I have debug with gdb the moment the server is with 12.5 of "load average" here : http://pastebin.freeswitch.org/20112 Can we detect a problem from here? Regards, Ricardo. -----Mensaje original----- De: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] En nombre de Gabriel Gunderson Enviado el: lunes, 22 de octubre de 2012 13:31 Para: FreeSWITCH Users Help Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes On Mon, Oct 22, 2012 at 8:37 AM, Ricardo Martinez <rmartinez@redvoiss.net> wrote:
The problem is that I'm having weird spikes of "load average" cpu every 20 minutes (aprox), without any reason, reaching near of 12.5 of load average for this machine.
What have you done to verify that it happens 'without any reason'? Gabe _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Guillermo Ruiz Camauer
Could it be just a normal sqlite running a vacuum? Sqlite does this every so often or if its calling a sync.... Also freeswitch with a load avg of 12 for a short period of time isnt anything to really worry about. Due to the number of threads that¹s possible with FS (ie: 1 per call leg) its not unusual to see a box humming along and the load avg well above 100 in my experience... This is due to the number of threads waiting in the run queue when it was sampled... Now, that¹s not saying that this is not specifically indicative or a problem but something else to think about On 10/22/12 1:52 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote:
This is the gdb from the proccess consuming more cpu : (gdb) thread 7 [Switching to thread 7 (Thread 31781)]#0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 (gdb) bt #0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 #1 0x00002aede19276de in unixSync (id=0x2aaaac726380, dataOnly=<value optimized out>) at ./src/os_unix.c:1229 #2 0x00002aede1951178 in syncJournal (pPager=0x1a1c200) at ./src/pager.c:2229 #3 0x00002aede1953b65 in sqlite3pager_sync (pPager=0x1a1c200, zMaster=0x0, nTrunc=0) at ./src/pager.c:3770 #4 0x00002aede193d07b in sqlite3BtreeSync (p=<value optimized out>, zMaster=0x0) at ./src/btree.c:6520 #5 0x00002aede1931f15 in sqlite3VdbeHalt (p=0x2aaaad17ac70) at ./src/vdbeaux.c:1062 #6 0x00002aede1962613 in sqlite3VdbeExec (p=0x2aaaad17ac70) at ./src/vdbe.c:2360 #7 0x00002aede193036c in sqlite3Step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:236 #8 sqlite3_step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:289 #9 0x00002aede1938978 in sqlite3_exec (db=0x1a1f070, zSql=0x2aede199e717 "COMMIT", xCallback=0, pArg=0x0, pzErrMsg=0x4178bed0) at ./src/legacy.c:78 #10 0x00002aede18ae0c1 in switch_core_db_exec (db=0x1a1f070, sql=0x2aede199e717 "COMMIT", callback=0, data=0x0, errmsg=0x4178bf38) at src/switch_core_db.c:93 #11 0x00002aede1897c18 in switch_cache_db_execute_sql_real (dbh=0x1a2dce0, sql=0x2aede199e717 "COMMIT", err=0x0) at src/switch_core_sqldb.c:445 #12 0x00002aede18985c3 in switch_cache_db_persistant_execute_trans (dbh=0x1a2dce0, sql=0x5f03bd0 "delete from calls where (caller_uuid='e64319c5-7f91-47a1-9561-42da291e499a' or callee_uuid='e64319c5-7f91-47a1-9561-42da291e499a') and hostname='transcoder.red.net <http://transcoder.red.net> ';\ndelete from calls where (call"..., retries=1) at src/switch_core_sqldb.c:787 #13 0x00002aede189bf1c in switch_core_sql_thread (thread=<value optimized out>, obj=<value optimized out>) at src/switch_core_sqldb.c:1026 #14 0x0000003c03c0673d in start_thread () from /lib64/libpthread.so.0 #15 0x0000003c030d3f6d in clone () from /lib64/libc.so.6
De: Ricardo Martinez [mailto:rmartinez@redvoiss.net] Enviado el: lunes, 22 de octubre de 2012 15:50 Para: 'FreeSWITCH Users Help' Asunto: RE: [Freeswitch-users] CPU Spikes every 20 minutes AS far as i can see there is no java running. This is the top at the moment of the spike : top - 15:48:29 up 17 days, 14:55, 3 users, load average: 14.07, 8.87, 6.48 Tasks: 158 total, 1 running, 157 sleeping, 0 stopped, 0 zombie Cpu(s): 12.6%us, 9.0%sy, 0.0%ni, 66.1%id, 2.7%wa, 0.3%hi, 9.2%si, 0.0%st Mem: 8171500k total, 8041556k used, 129944k free, 177688k buffers Swap: 10223608k total, 0k used, 10223608k free, 6951988k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31772 root 15 0 870m 481m 6880 S 212.4 6.0 9523:51 freeswitch 4210 root 19 0 36344 2940 1444 S 1.0 0.0 56:21.08 sngtc_server 29586 root 15 0 12740 1132 808 R 0.3 0.0 0:00.02 top 1 root 15 0 10348 704 588 S 0.0 0.0 0:01.13 init 2 root RT -5 0 0 0 S 0.0 0.0 0:00.41 migration/0 3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0 4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0 5 root RT -5 0 0 0 S 0.0 0.0 0:00.83 migration/1 6 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1 7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1 8 root RT -5 0 0 0 S 0.0 0.0 0:00.14 migration/2 9 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/2 10 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/2 11 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/3 12 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/3 13 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/3 14 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/4 15 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/4 16 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/4 17 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/5 18 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/5 19 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/5 20 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/6 Regards, Ricardo.- De: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] En nombre de Guillermo Ruiz Camauer Enviado el: lunes, 22 de octubre de 2012 15:38 Para: FreeSWITCH Users Help Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes Do you have anything like Java running that could be doing garbage collection?
What other processes are running on the machine? Does TOP show which process is causing the spike?
Guillermo
On Mon, Oct 22, 2012 at 3:29 PM, Ricardo Martinez <rmartinez@redvoiss.net> wrote: Hi Gabe. I'm mean between the spikes there is nothing different as regarding to the calls... the cps maintains the same.. there are no scripts running or crontabs running, the simultaneous calls are the same also. So, for some reason the "load average" start to grow (every 20 or 25 minutes) without any clear cause. I have debug with gdb the moment the server is with 12.5 of "load average" here : http://pastebin.freeswitch.org/20112
Can we detect a problem from here?
Regards, Ricardo.
-----Mensaje original----- De: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] En nombre de Gabriel Gunderson Enviado el: lunes, 22 de octubre de 2012 13:31 Para: FreeSWITCH Users Help Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes
On Mon, Oct 22, 2012 at 8:37 AM, Ricardo Martinez <rmartinez@redvoiss.net> wrote:
The problem is that I'm having weird spikes of "load average" cpu every 20 minutes (aprox), without any reason, reaching near of 12.5 of load average for this machine.
What have you done to verify that it happens 'without any reason'?
Gabe
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Ken http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org irc.freenode.net #freeswitch
Re: [Freeswitch-users] CPU Spikes every 20 minutes Hi Ken. Maybe that is the problem. I was sure that the “load average” can’t go more than #cores in your system. In my case I have 8 cores, so a “load average” of 12 is high from my point of view. Where did you get the number 100 for this parameter? Anyway, what is running a sqlite in freeswitch. Does this have to do with the core.db file? Or with the cdr_sqlite module?. Thanks Ricardo.- *De:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *En nombre de *Ken Rice *Enviado el:* lunes, 22 de octubre de 2012 16:18 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes Could it be just a normal sqlite running a vacuum? Sqlite does this every so often or if its calling a sync.... Also freeswitch with a load avg of 12 for a short period of time isnt anything to really worry about. Due to the number of threads that’s possible with FS (ie: 1 per call leg) its not unusual to see a box humming along and the load avg well above 100 in my experience... This is due to the number of threads waiting in the run queue when it was sampled... Now, that’s not saying that this is not specifically indicative or a problem but something else to think about On 10/22/12 1:52 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: This is the gdb from the proccess consuming more cpu : (gdb) thread 7 [Switching to thread 7 (Thread 31781)]#0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 (gdb) bt #0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 #1 0x00002aede19276de in unixSync (id=0x2aaaac726380, dataOnly=<value optimized out>) at ./src/os_unix.c:1229 #2 0x00002aede1951178 in syncJournal (pPager=0x1a1c200) at ./src/pager.c:2229 #3 0x00002aede1953b65 in sqlite3pager_sync (pPager=0x1a1c200, zMaster=0x0, nTrunc=0) at ./src/pager.c:3770 #4 0x00002aede193d07b in sqlite3BtreeSync (p=<value optimized out>, zMaster=0x0) at ./src/btree.c:6520 #5 0x00002aede1931f15 in sqlite3VdbeHalt (p=0x2aaaad17ac70) at ./src/vdbeaux.c:1062 #6 0x00002aede1962613 in sqlite3VdbeExec (p=0x2aaaad17ac70) at ./src/vdbe.c:2360 #7 0x00002aede193036c in sqlite3Step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:236 #8 sqlite3_step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:289 #9 0x00002aede1938978 in sqlite3_exec (db=0x1a1f070, zSql=0x2aede199e717 "COMMIT", xCallback=0, pArg=0x0, pzErrMsg=0x4178bed0) at ./src/legacy.c:78 #10 0x00002aede18ae0c1 in switch_core_db_exec (db=0x1a1f070, sql=0x2aede199e717 "COMMIT", callback=0, data=0x0, errmsg=0x4178bf38) at src/switch_core_db.c:93 #11 0x00002aede1897c18 in switch_cache_db_execute_sql_real (dbh=0x1a2dce0, sql=0x2aede199e717 "COMMIT", err=0x0) at src/switch_core_sqldb.c:445 #12 0x00002aede18985c3 in switch_cache_db_persistant_execute_trans (dbh=0x1a2dce0, sql=0x5f03bd0 "delete from calls where (caller_uuid='e64319c5-7f91-47a1-9561-42da291e499a' or callee_uuid='e64319c5-7f91-47a1-9561-42da291e499a') and hostname=' transcoder.red.net <http://transcoder.red.net> ';\ndelete from calls where (call"..., retries=1) at src/switch_core_sqldb.c:787 #13 0x00002aede189bf1c in switch_core_sql_thread (thread=<value optimized out>, obj=<value optimized out>) at src/switch_core_sqldb.c:1026 #14 0x0000003c03c0673d in start_thread () from /lib64/libpthread.so.0 #15 0x0000003c030d3f6d in clone () from /lib64/libc.so.6 *De:* Ricardo Martinez [mailto:rmartinez@redvoiss.net<rmartinez@redvoiss.net>] *Enviado el:* lunes, 22 de octubre de 2012 15:50 *Para:* 'FreeSWITCH Users Help' *Asunto:* RE: [Freeswitch-users] CPU Spikes every 20 minutes AS far as i can see there is no java running. This is the top at the moment of the spike : top - 15:48:29 up 17 days, 14:55, 3 users, load average: 14.07, 8.87, 6.48 Tasks: 158 total, 1 running, 157 sleeping, 0 stopped, 0 zombie Cpu(s): 12.6%us, 9.0%sy, 0.0%ni, 66.1%id, 2.7%wa, 0.3%hi, 9.2%si, 0.0%st Mem: 8171500k total, 8041556k used, 129944k free, 177688k buffers Swap: 10223608k total, 0k used, 10223608k free, 6951988k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31772 root 15 0 870m 481m 6880 S 212.4 6.0 9523:51 freeswitch 4210 root 19 0 36344 2940 1444 S 1.0 0.0 56:21.08 sngtc_server 29586 root 15 0 12740 1132 808 R 0.3 0.0 0:00.02 top 1 root 15 0 10348 704 588 S 0.0 0.0 0:01.13 init 2 root RT -5 0 0 0 S 0.0 0.0 0:00.41 migration/0 3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0 4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0 5 root RT -5 0 0 0 S 0.0 0.0 0:00.83 migration/1 6 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1 7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1 8 root RT -5 0 0 0 S 0.0 0.0 0:00.14 migration/2 9 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/2 10 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/2 11 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/3 12 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/3 13 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/3 14 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/4 15 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/4 16 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/4 17 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/5 18 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/5 19 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/5 20 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/6 Regards, Ricardo.- *De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Guillermo Ruiz Camauer *Enviado el:* lunes, 22 de octubre de 2012 15:38 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes Do you have anything like Java running that could be doing garbage collection? What other processes are running on the machine? Does TOP show which process is causing the spike? Guillermo On Mon, Oct 22, 2012 at 3:29 PM, Ricardo Martinez <rmartinez@redvoiss.net> wrote: Hi Gabe. I'm mean between the spikes there is nothing different as regarding to the calls... the cps maintains the same.. there are no scripts running or crontabs running, the simultaneous calls are the same also. So, for some reason the "load average" start to grow (every 20 or 25 minutes) without any clear cause. I have debug with gdb the moment the server is with 12.5 of "load average" here : http://pastebin.freeswitch.org/20112 Can we detect a problem from here? Regards, Ricardo. -----Mensaje original----- De: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] En nombre de Gabriel Gunderson Enviado el: lunes, 22 de octubre de 2012 13:31 Para: FreeSWITCH Users Help Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes On Mon, Oct 22, 2012 at 8:37 AM, Ricardo Martinez <rmartinez@redvoiss.net> wrote:
The problem is that I'm having weird spikes of "load average" cpu every 20 minutes (aprox), without any reason, reaching near of 12.5 of load average for this machine.
What have you done to verify that it happens 'without any reason'? Gabe _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Ken *http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org *irc.freenode.net #freeswitch
Try this out if is not already configured this way: http://wiki.freeswitch.org/index.php?title=FreeSWITCH_DB_In_RAMdrive&redirec... Guillermo On Mon, Oct 22, 2012 at 4:30 PM, Ricardo Martinez <rmartinez@redvoiss.net>wrote:
Hi Ken.
Maybe that is the problem. I was sure that the “load average” can’t go more than #cores in your system. In my case I have 8 cores, so a “load average” of 12 is high from my point of view. Where did you get the number 100 for this parameter?
Anyway, what is running a sqlite in freeswitch. Does this have to do with the core.db file? Or with the cdr_sqlite module?.
Thanks
Ricardo.-
*De:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *En nombre de *Ken Rice *Enviado el:* lunes, 22 de octubre de 2012 16:18
*Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes
Could it be just a normal sqlite running a vacuum?
Sqlite does this every so often or if its calling a sync....
Also freeswitch with a load avg of 12 for a short period of time isnt anything to really worry about. Due to the number of threads that’s possible with FS (ie: 1 per call leg) its not unusual to see a box humming along and the load avg well above 100 in my experience... This is due to the number of threads waiting in the run queue when it was sampled... Now, that’s not saying that this is not specifically indicative or a problem but something else to think about
On 10/22/12 1:52 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote:
This is the gdb from the proccess consuming more cpu :
(gdb) thread 7 [Switching to thread 7 (Thread 31781)]#0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 (gdb) bt #0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 #1 0x00002aede19276de in unixSync (id=0x2aaaac726380, dataOnly=<value optimized out>) at ./src/os_unix.c:1229 #2 0x00002aede1951178 in syncJournal (pPager=0x1a1c200) at ./src/pager.c:2229 #3 0x00002aede1953b65 in sqlite3pager_sync (pPager=0x1a1c200, zMaster=0x0, nTrunc=0) at ./src/pager.c:3770 #4 0x00002aede193d07b in sqlite3BtreeSync (p=<value optimized out>, zMaster=0x0) at ./src/btree.c:6520 #5 0x00002aede1931f15 in sqlite3VdbeHalt (p=0x2aaaad17ac70) at ./src/vdbeaux.c:1062 #6 0x00002aede1962613 in sqlite3VdbeExec (p=0x2aaaad17ac70) at ./src/vdbe.c:2360 #7 0x00002aede193036c in sqlite3Step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:236 #8 sqlite3_step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:289 #9 0x00002aede1938978 in sqlite3_exec (db=0x1a1f070, zSql=0x2aede199e717 "COMMIT", xCallback=0, pArg=0x0, pzErrMsg=0x4178bed0) at ./src/legacy.c:78 #10 0x00002aede18ae0c1 in switch_core_db_exec (db=0x1a1f070, sql=0x2aede199e717 "COMMIT", callback=0, data=0x0, errmsg=0x4178bf38) at src/switch_core_db.c:93 #11 0x00002aede1897c18 in switch_cache_db_execute_sql_real (dbh=0x1a2dce0, sql=0x2aede199e717 "COMMIT", err=0x0) at src/switch_core_sqldb.c:445 #12 0x00002aede18985c3 in switch_cache_db_persistant_execute_trans (dbh=0x1a2dce0, sql=0x5f03bd0 "delete from calls where (caller_uuid='e64319c5-7f91-47a1-9561-42da291e499a' or callee_uuid='e64319c5-7f91-47a1-9561-42da291e499a') and hostname=' transcoder.red.net <http://transcoder.red.net> ';\ndelete from calls where (call"..., retries=1) at src/switch_core_sqldb.c:787 #13 0x00002aede189bf1c in switch_core_sql_thread (thread=<value optimized out>, obj=<value optimized out>) at src/switch_core_sqldb.c:1026 #14 0x0000003c03c0673d in start_thread () from /lib64/libpthread.so.0 #15 0x0000003c030d3f6d in clone () from /lib64/libc.so.6
*De:* Ricardo Martinez [mailto:rmartinez@redvoiss.net<rmartinez@redvoiss.net>]
*Enviado el:* lunes, 22 de octubre de 2012 15:50 *Para:* 'FreeSWITCH Users Help' *Asunto:* RE: [Freeswitch-users] CPU Spikes every 20 minutes
AS far as i can see there is no java running. This is the top at the moment of the spike :
top - 15:48:29 up 17 days, 14:55, 3 users, load average: 14.07, 8.87, 6.48 Tasks: 158 total, 1 running, 157 sleeping, 0 stopped, 0 zombie Cpu(s): 12.6%us, 9.0%sy, 0.0%ni, 66.1%id, 2.7%wa, 0.3%hi, 9.2%si, 0.0%st Mem: 8171500k total, 8041556k used, 129944k free, 177688k buffers Swap: 10223608k total, 0k used, 10223608k free, 6951988k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31772 root 15 0 870m 481m 6880 S 212.4 6.0 9523:51 freeswitch 4210 root 19 0 36344 2940 1444 S 1.0 0.0 56:21.08 sngtc_server 29586 root 15 0 12740 1132 808 R 0.3 0.0 0:00.02 top 1 root 15 0 10348 704 588 S 0.0 0.0 0:01.13 init 2 root RT -5 0 0 0 S 0.0 0.0 0:00.41 migration/0 3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0 4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0 5 root RT -5 0 0 0 S 0.0 0.0 0:00.83 migration/1 6 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1 7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
8 root RT -5 0 0 0 S 0.0 0.0 0:00.14 migration/2 9 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/2 10 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/2 11 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/3 12 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/3 13 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/3 14 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/4 15 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/4 16 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/4 17 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/5 18 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/5 19 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/5 20 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/6
Regards, Ricardo.-
*De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Guillermo Ruiz Camauer *Enviado el:* lunes, 22 de octubre de 2012 15:38 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes
Do you have anything like Java running that could be doing garbage collection?
What other processes are running on the machine? Does TOP show which process is causing the spike?
Guillermo
On Mon, Oct 22, 2012 at 3:29 PM, Ricardo Martinez <rmartinez@redvoiss.net> wrote: Hi Gabe. I'm mean between the spikes there is nothing different as regarding to the calls... the cps maintains the same.. there are no scripts running or crontabs running, the simultaneous calls are the same also. So, for some reason the "load average" start to grow (every 20 or 25 minutes) without any clear cause. I have debug with gdb the moment the server is with 12.5 of "load average" here : http://pastebin.freeswitch.org/20112
Can we detect a problem from here?
Regards, Ricardo.
-----Mensaje original----- De: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] En nombre de Gabriel Gunderson Enviado el: lunes, 22 de octubre de 2012 13:31 Para: FreeSWITCH Users Help Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes
On Mon, Oct 22, 2012 at 8:37 AM, Ricardo Martinez <rmartinez@redvoiss.net> wrote:
The problem is that I'm having weird spikes of "load average" cpu every 20 minutes (aprox), without any reason, reaching near of 12.5 of load average for this machine.
What have you done to verify that it happens 'without any reason'?
Gabe
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Ken *http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org *irc.freenode.net #freeswitch
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Guillermo Ruiz Camauer
HI Guillero, I will try that and let you know the results. One final question, the link you gave me says : 1) In /etc/fstab, add a mountpoint and save. tmpfs /usr/local/freeswitch/ramdb tmpfs defaults 0 0 2) Make sure you also create the directory. mkdir /usr/local/freeswitch/ramdisk mount -a 3) Then, in freeswitch/conf/autoload_configs/switch.conf.xml uncomment core-db-name and set it to the path you created in fstab. <param name="core-db-name" value="/usr/local/freeswitch/ramdisk/core.db" /> 4) Restart FreeSWITCH The directory for the fstab file is : /usr/local/freeswitch/ramdb The directory to mount is : /usr/local/freeswitch/ramdisk Is this a typo error ? or it is ok? Thanks Ricardo.- *De:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *En nombre de *Guillermo Ruiz Camauer *Enviado el:* lunes, 22 de octubre de 2012 16:35 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes Try this out if is not already configured this way: http://wiki.freeswitch.org/index.php?title=FreeSWITCH_DB_In_RAMdrive&redirec... Guillermo On Mon, Oct 22, 2012 at 4:30 PM, Ricardo Martinez <rmartinez@redvoiss.net> wrote: Hi Ken. Maybe that is the problem. I was sure that the “load average” can’t go more than #cores in your system. In my case I have 8 cores, so a “load average” of 12 is high from my point of view. Where did you get the number 100 for this parameter? Anyway, what is running a sqlite in freeswitch. Does this have to do with the core.db file? Or with the cdr_sqlite module?. Thanks Ricardo.- *De:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *En nombre de *Ken Rice *Enviado el:* lunes, 22 de octubre de 2012 16:18 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes Could it be just a normal sqlite running a vacuum? Sqlite does this every so often or if its calling a sync.... Also freeswitch with a load avg of 12 for a short period of time isnt anything to really worry about. Due to the number of threads that’s possible with FS (ie: 1 per call leg) its not unusual to see a box humming along and the load avg well above 100 in my experience... This is due to the number of threads waiting in the run queue when it was sampled... Now, that’s not saying that this is not specifically indicative or a problem but something else to think about On 10/22/12 1:52 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: This is the gdb from the proccess consuming more cpu : (gdb) thread 7 [Switching to thread 7 (Thread 31781)]#0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 (gdb) bt #0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 #1 0x00002aede19276de in unixSync (id=0x2aaaac726380, dataOnly=<value optimized out>) at ./src/os_unix.c:1229 #2 0x00002aede1951178 in syncJournal (pPager=0x1a1c200) at ./src/pager.c:2229 #3 0x00002aede1953b65 in sqlite3pager_sync (pPager=0x1a1c200, zMaster=0x0, nTrunc=0) at ./src/pager.c:3770 #4 0x00002aede193d07b in sqlite3BtreeSync (p=<value optimized out>, zMaster=0x0) at ./src/btree.c:6520 #5 0x00002aede1931f15 in sqlite3VdbeHalt (p=0x2aaaad17ac70) at ./src/vdbeaux.c:1062 #6 0x00002aede1962613 in sqlite3VdbeExec (p=0x2aaaad17ac70) at ./src/vdbe.c:2360 #7 0x00002aede193036c in sqlite3Step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:236 #8 sqlite3_step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:289 #9 0x00002aede1938978 in sqlite3_exec (db=0x1a1f070, zSql=0x2aede199e717 "COMMIT", xCallback=0, pArg=0x0, pzErrMsg=0x4178bed0) at ./src/legacy.c:78 #10 0x00002aede18ae0c1 in switch_core_db_exec (db=0x1a1f070, sql=0x2aede199e717 "COMMIT", callback=0, data=0x0, errmsg=0x4178bf38) at src/switch_core_db.c:93 #11 0x00002aede1897c18 in switch_cache_db_execute_sql_real (dbh=0x1a2dce0, sql=0x2aede199e717 "COMMIT", err=0x0) at src/switch_core_sqldb.c:445 #12 0x00002aede18985c3 in switch_cache_db_persistant_execute_trans (dbh=0x1a2dce0, sql=0x5f03bd0 "delete from calls where (caller_uuid='e64319c5-7f91-47a1-9561-42da291e499a' or callee_uuid='e64319c5-7f91-47a1-9561-42da291e499a') and hostname=' transcoder.red.net <http://transcoder.red.net> ';\ndelete from calls where (call"..., retries=1) at src/switch_core_sqldb.c:787 #13 0x00002aede189bf1c in switch_core_sql_thread (thread=<value optimized out>, obj=<value optimized out>) at src/switch_core_sqldb.c:1026 #14 0x0000003c03c0673d in start_thread () from /lib64/libpthread.so.0 #15 0x0000003c030d3f6d in clone () from /lib64/libc.so.6 *De:* Ricardo Martinez [mailto:rmartinez@redvoiss.net<rmartinez@redvoiss.net>] *Enviado el:* lunes, 22 de octubre de 2012 15:50 *Para:* 'FreeSWITCH Users Help' *Asunto:* RE: [Freeswitch-users] CPU Spikes every 20 minutes AS far as i can see there is no java running. This is the top at the moment of the spike : top - 15:48:29 up 17 days, 14:55, 3 users, load average: 14.07, 8.87, 6.48 Tasks: 158 total, 1 running, 157 sleeping, 0 stopped, 0 zombie Cpu(s): 12.6%us, 9.0%sy, 0.0%ni, 66.1%id, 2.7%wa, 0.3%hi, 9.2%si, 0.0%st Mem: 8171500k total, 8041556k used, 129944k free, 177688k buffers Swap: 10223608k total, 0k used, 10223608k free, 6951988k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31772 root 15 0 870m 481m 6880 S 212.4 6.0 9523:51 freeswitch 4210 root 19 0 36344 2940 1444 S 1.0 0.0 56:21.08 sngtc_server 29586 root 15 0 12740 1132 808 R 0.3 0.0 0:00.02 top 1 root 15 0 10348 704 588 S 0.0 0.0 0:01.13 init 2 root RT -5 0 0 0 S 0.0 0.0 0:00.41 migration/0 3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0 4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0 5 root RT -5 0 0 0 S 0.0 0.0 0:00.83 migration/1 6 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1 7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1 8 root RT -5 0 0 0 S 0.0 0.0 0:00.14 migration/2 9 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/2 10 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/2 11 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/3 12 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/3 13 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/3 14 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/4 15 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/4 16 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/4 17 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/5 18 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/5 19 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/5 20 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/6 Regards, Ricardo.- *De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Guillermo Ruiz Camauer *Enviado el:* lunes, 22 de octubre de 2012 15:38 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes Do you have anything like Java running that could be doing garbage collection? What other processes are running on the machine? Does TOP show which process is causing the spike? Guillermo On Mon, Oct 22, 2012 at 3:29 PM, Ricardo Martinez <rmartinez@redvoiss.net> wrote: Hi Gabe. I'm mean between the spikes there is nothing different as regarding to the calls... the cps maintains the same.. there are no scripts running or crontabs running, the simultaneous calls are the same also. So, for some reason the "load average" start to grow (every 20 or 25 minutes) without any clear cause. I have debug with gdb the moment the server is with 12.5 of "load average" here : http://pastebin.freeswitch.org/20112 Can we detect a problem from here? Regards, Ricardo. -----Mensaje original----- De: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] En nombre de Gabriel Gunderson Enviado el: lunes, 22 de octubre de 2012 13:31 Para: FreeSWITCH Users Help Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes On Mon, Oct 22, 2012 at 8:37 AM, Ricardo Martinez <rmartinez@redvoiss.net> wrote:
The problem is that I'm having weird spikes of "load average" cpu every 20 minutes (aprox), without any reason, reaching near of 12.5 of load average for this machine.
What have you done to verify that it happens 'without any reason'? Gabe _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Ken *http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org *irc.freenode.net #freeswitch _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Guillermo Ruiz Camauer
I am on CentOS 6.3. in /etc/fstab I have: tmpfs /dev/shm tmpfs defaults 0 0 in freeswitch/conf/autoload_configs/switch.conf.xml I have: <param name="core-db-name" value="/dev/shm/core.db" /> Guillermo On Mon, Oct 22, 2012 at 4:40 PM, Ricardo Martinez <rmartinez@redvoiss.net>wrote:
HI Guillero,
I will try that and let you know the results.
One final question, the link you gave me says :
1) In /etc/fstab, add a mountpoint and save.
tmpfs /usr/local/freeswitch/ramdb tmpfs defaults 0 0
2) Make sure you also create the directory.
mkdir /usr/local/freeswitch/ramdisk
mount -a
3) Then, in freeswitch/conf/autoload_configs/switch.conf.xml uncomment core-db-name and set it to the path you created in fstab.
<param name="core-db-name" value="/usr/local/freeswitch/ramdisk/core.db" />
4) Restart FreeSWITCH
The directory for the fstab file is :
/usr/local/freeswitch/ramdb
The directory to mount is :
/usr/local/freeswitch/ramdisk
Is this a typo error ? or it is ok?
Thanks
Ricardo.-
*De:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *En nombre de *Guillermo Ruiz Camauer *Enviado el:* lunes, 22 de octubre de 2012 16:35
*Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes
Try this out if is not already configured this way: http://wiki.freeswitch.org/index.php?title=FreeSWITCH_DB_In_RAMdrive&redirec...
Guillermo
On Mon, Oct 22, 2012 at 4:30 PM, Ricardo Martinez <rmartinez@redvoiss.net> wrote:
Hi Ken.
Maybe that is the problem. I was sure that the “load average” can’t go more than #cores in your system. In my case I have 8 cores, so a “load average” of 12 is high from my point of view. Where did you get the number 100 for this parameter?
Anyway, what is running a sqlite in freeswitch. Does this have to do with the core.db file? Or with the cdr_sqlite module?.
Thanks
Ricardo.-
*De:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *En nombre de *Ken Rice *Enviado el:* lunes, 22 de octubre de 2012 16:18
*Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes
Could it be just a normal sqlite running a vacuum?
Sqlite does this every so often or if its calling a sync....
Also freeswitch with a load avg of 12 for a short period of time isnt anything to really worry about. Due to the number of threads that’s possible with FS (ie: 1 per call leg) its not unusual to see a box humming along and the load avg well above 100 in my experience... This is due to the number of threads waiting in the run queue when it was sampled... Now, that’s not saying that this is not specifically indicative or a problem but something else to think about
On 10/22/12 1:52 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote:
This is the gdb from the proccess consuming more cpu :
(gdb) thread 7 [Switching to thread 7 (Thread 31781)]#0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 (gdb) bt #0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 #1 0x00002aede19276de in unixSync (id=0x2aaaac726380, dataOnly=<value optimized out>) at ./src/os_unix.c:1229 #2 0x00002aede1951178 in syncJournal (pPager=0x1a1c200) at ./src/pager.c:2229 #3 0x00002aede1953b65 in sqlite3pager_sync (pPager=0x1a1c200, zMaster=0x0, nTrunc=0) at ./src/pager.c:3770 #4 0x00002aede193d07b in sqlite3BtreeSync (p=<value optimized out>, zMaster=0x0) at ./src/btree.c:6520 #5 0x00002aede1931f15 in sqlite3VdbeHalt (p=0x2aaaad17ac70) at ./src/vdbeaux.c:1062 #6 0x00002aede1962613 in sqlite3VdbeExec (p=0x2aaaad17ac70) at ./src/vdbe.c:2360 #7 0x00002aede193036c in sqlite3Step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:236 #8 sqlite3_step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:289 #9 0x00002aede1938978 in sqlite3_exec (db=0x1a1f070, zSql=0x2aede199e717 "COMMIT", xCallback=0, pArg=0x0, pzErrMsg=0x4178bed0) at ./src/legacy.c:78 #10 0x00002aede18ae0c1 in switch_core_db_exec (db=0x1a1f070, sql=0x2aede199e717 "COMMIT", callback=0, data=0x0, errmsg=0x4178bf38) at src/switch_core_db.c:93 #11 0x00002aede1897c18 in switch_cache_db_execute_sql_real (dbh=0x1a2dce0, sql=0x2aede199e717 "COMMIT", err=0x0) at src/switch_core_sqldb.c:445 #12 0x00002aede18985c3 in switch_cache_db_persistant_execute_trans (dbh=0x1a2dce0, sql=0x5f03bd0 "delete from calls where (caller_uuid='e64319c5-7f91-47a1-9561-42da291e499a' or callee_uuid='e64319c5-7f91-47a1-9561-42da291e499a') and hostname=' transcoder.red.net <http://transcoder.red.net> ';\ndelete from calls where (call"..., retries=1) at src/switch_core_sqldb.c:787 #13 0x00002aede189bf1c in switch_core_sql_thread (thread=<value optimized out>, obj=<value optimized out>) at src/switch_core_sqldb.c:1026 #14 0x0000003c03c0673d in start_thread () from /lib64/libpthread.so.0 #15 0x0000003c030d3f6d in clone () from /lib64/libc.so.6
*De:* Ricardo Martinez [mailto:rmartinez@redvoiss.net<rmartinez@redvoiss.net>]
*Enviado el:* lunes, 22 de octubre de 2012 15:50 *Para:* 'FreeSWITCH Users Help' *Asunto:* RE: [Freeswitch-users] CPU Spikes every 20 minutes
AS far as i can see there is no java running. This is the top at the moment of the spike :
top - 15:48:29 up 17 days, 14:55, 3 users, load average: 14.07, 8.87, 6.48 Tasks: 158 total, 1 running, 157 sleeping, 0 stopped, 0 zombie Cpu(s): 12.6%us, 9.0%sy, 0.0%ni, 66.1%id, 2.7%wa, 0.3%hi, 9.2%si, 0.0%st Mem: 8171500k total, 8041556k used, 129944k free, 177688k buffers Swap: 10223608k total, 0k used, 10223608k free, 6951988k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31772 root 15 0 870m 481m 6880 S 212.4 6.0 9523:51 freeswitch 4210 root 19 0 36344 2940 1444 S 1.0 0.0 56:21.08 sngtc_server 29586 root 15 0 12740 1132 808 R 0.3 0.0 0:00.02 top 1 root 15 0 10348 704 588 S 0.0 0.0 0:01.13 init 2 root RT -5 0 0 0 S 0.0 0.0 0:00.41 migration/0 3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0 4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0 5 root RT -5 0 0 0 S 0.0 0.0 0:00.83 migration/1 6 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1 7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
8 root RT -5 0 0 0 S 0.0 0.0 0:00.14 migration/2 9 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/2 10 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/2 11 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/3 12 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/3 13 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/3 14 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/4 15 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/4 16 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/4 17 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/5 18 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/5 19 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/5 20 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/6
Regards, Ricardo.-
*De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Guillermo Ruiz Camauer *Enviado el:* lunes, 22 de octubre de 2012 15:38 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes
Do you have anything like Java running that could be doing garbage collection?
What other processes are running on the machine? Does TOP show which process is causing the spike?
Guillermo
On Mon, Oct 22, 2012 at 3:29 PM, Ricardo Martinez <rmartinez@redvoiss.net> wrote: Hi Gabe. I'm mean between the spikes there is nothing different as regarding to the calls... the cps maintains the same.. there are no scripts running or crontabs running, the simultaneous calls are the same also. So, for some reason the "load average" start to grow (every 20 or 25 minutes) without any clear cause. I have debug with gdb the moment the server is with 12.5 of "load average" here : http://pastebin.freeswitch.org/20112
Can we detect a problem from here?
Regards, Ricardo.
-----Mensaje original----- De: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] En nombre de Gabriel Gunderson Enviado el: lunes, 22 de octubre de 2012 13:31 Para: FreeSWITCH Users Help Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes
On Mon, Oct 22, 2012 at 8:37 AM, Ricardo Martinez <rmartinez@redvoiss.net> wrote:
The problem is that I'm having weird spikes of "load average" cpu every 20 minutes (aprox), without any reason, reaching near of 12.5 of load average for this machine.
What have you done to verify that it happens 'without any reason'?
Gabe
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Ken *http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org *irc.freenode.net #freeswitch
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Guillermo Ruiz Camauer
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Guillermo Ruiz Camauer
Load Average is the number of threads/processes waiting in the kernel¹s run queue at sample time... Supposedly number of cores is the how much your load avg can get to before your boxes is exhausted but this isnt exactly true on modern hardware with highly multi-threaded software like FreeSWITCH... Where did I get the 100+ number? From looking at top¹ or w¹ on a box to see what the current load avg is... Sqlite is used all over FreeSWITCH from the coredb to mod_sofia and everything in between.... When you type show channels¹ from fs_cli it doesn¹t query the structures in RAM to show you that output, it pulls it from the sqlite db... That being said, you can test quickly to see if sqlite is hammering the disk at some point by shifting freeswitch/db dir to a ram drive. Example code for linux such as Centos or Debian assuming freeswitch is installed to /usr/local/freeswitch Shutdown freeswitch rm rf /usr/local/freeswitch/db/* mount t tmpfs tmpfs /usr/local/freeswitch/db Startup freeswitch Once you have done that stick it back in rotation... Note this will not affect anything but voicemail if you have voicemail running on that box and not using an external DB via mod_voicemail¹s DSN settings you¹ll lose the voicemail DB... (I would assume this is not the case since you are using this primarily as a Transcoder) K On 10/22/12 2:30 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote:
Hi Ken. Maybe that is the problem. I was sure that the ³load average² can¹t go more than #cores in your system. In my case I have 8 cores, so a ³load average² of 12 is high from my point of view. Where did you get the number 100 for this parameter? Anyway, what is running a sqlite in freeswitch. Does this have to do with the core.db file? Or with the cdr_sqlite module?. Thanks Ricardo.-
De: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] En nombre de Ken Rice Enviado el: lunes, 22 de octubre de 2012 16:18 Para: FreeSWITCH Users Help Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes Could it be just a normal sqlite running a vacuum?
Sqlite does this every so often or if its calling a sync....
Also freeswitch with a load avg of 12 for a short period of time isnt anything to really worry about. Due to the number of threads that¹s possible with FS (ie: 1 per call leg) its not unusual to see a box humming along and the load avg well above 100 in my experience... This is due to the number of threads waiting in the run queue when it was sampled... Now, that¹s not saying that this is not specifically indicative or a problem but something else to think about
On 10/22/12 1:52 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: This is the gdb from the proccess consuming more cpu : (gdb) thread 7 [Switching to thread 7 (Thread 31781)]#0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 (gdb) bt #0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 #1 0x00002aede19276de in unixSync (id=0x2aaaac726380, dataOnly=<value optimized out>) at ./src/os_unix.c:1229 #2 0x00002aede1951178 in syncJournal (pPager=0x1a1c200) at ./src/pager.c:2229 #3 0x00002aede1953b65 in sqlite3pager_sync (pPager=0x1a1c200, zMaster=0x0, nTrunc=0) at ./src/pager.c:3770 #4 0x00002aede193d07b in sqlite3BtreeSync (p=<value optimized out>, zMaster=0x0) at ./src/btree.c:6520 #5 0x00002aede1931f15 in sqlite3VdbeHalt (p=0x2aaaad17ac70) at ./src/vdbeaux.c:1062 #6 0x00002aede1962613 in sqlite3VdbeExec (p=0x2aaaad17ac70) at ./src/vdbe.c:2360 #7 0x00002aede193036c in sqlite3Step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:236 #8 sqlite3_step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:289 #9 0x00002aede1938978 in sqlite3_exec (db=0x1a1f070, zSql=0x2aede199e717 "COMMIT", xCallback=0, pArg=0x0, pzErrMsg=0x4178bed0) at ./src/legacy.c:78 #10 0x00002aede18ae0c1 in switch_core_db_exec (db=0x1a1f070, sql=0x2aede199e717 "COMMIT", callback=0, data=0x0, errmsg=0x4178bf38) at src/switch_core_db.c:93 #11 0x00002aede1897c18 in switch_cache_db_execute_sql_real (dbh=0x1a2dce0, sql=0x2aede199e717 "COMMIT", err=0x0) at src/switch_core_sqldb.c:445 #12 0x00002aede18985c3 in switch_cache_db_persistant_execute_trans (dbh=0x1a2dce0, sql=0x5f03bd0 "delete from calls where (caller_uuid='e64319c5-7f91-47a1-9561-42da291e499a' or callee_uuid='e64319c5-7f91-47a1-9561-42da291e499a') and hostname='transcoder.red.net <http://transcoder.red.net> <http://transcoder.red.net> ';\ndelete from calls where (call"..., retries=1) at src/switch_core_sqldb.c:787 #13 0x00002aede189bf1c in switch_core_sql_thread (thread=<value optimized out>, obj=<value optimized out>) at src/switch_core_sqldb.c:1026 #14 0x0000003c03c0673d in start_thread () from /lib64/libpthread.so.0 #15 0x0000003c030d3f6d in clone () from /lib64/libc.so.6
De: Ricardo Martinez [mailto:rmartinez@redvoiss.net] Enviado el: lunes, 22 de octubre de 2012 15:50 Para: 'FreeSWITCH Users Help' Asunto: RE: [Freeswitch-users] CPU Spikes every 20 minutes AS far as i can see there is no java running. This is the top at the moment of the spike : top - 15:48:29 up 17 days, 14:55, 3 users, load average: 14.07, 8.87, 6.48 Tasks: 158 total, 1 running, 157 sleeping, 0 stopped, 0 zombie Cpu(s): 12.6%us, 9.0%sy, 0.0%ni, 66.1%id, 2.7%wa, 0.3%hi, 9.2%si, 0.0%st Mem: 8171500k total, 8041556k used, 129944k free, 177688k buffers Swap: 10223608k total, 0k used, 10223608k free, 6951988k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31772 root 15 0 870m 481m 6880 S 212.4 6.0 9523:51 freeswitch 4210 root 19 0 36344 2940 1444 S 1.0 0.0 56:21.08 sngtc_server 29586 root 15 0 12740 1132 808 R 0.3 0.0 0:00.02 top 1 root 15 0 10348 704 588 S 0.0 0.0 0:01.13 init 2 root RT -5 0 0 0 S 0.0 0.0 0:00.41 migration/0 3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0 4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0 5 root RT -5 0 0 0 S 0.0 0.0 0:00.83 migration/1 6 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1 7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1 8 root RT -5 0 0 0 S 0.0 0.0 0:00.14 migration/2 9 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/2 10 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/2 11 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/3 12 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/3 13 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/3 14 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/4 15 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/4 16 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/4 17 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/5 18 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/5 19 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/5 20 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/6 Regards, Ricardo.- De: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] En nombre de Guillermo Ruiz Camauer Enviado el: lunes, 22 de octubre de 2012 15:38 Para: FreeSWITCH Users Help Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes Do you have anything like Java running that could be doing garbage collection?
What other processes are running on the machine? Does TOP show which process is causing the spike?
Guillermo
On Mon, Oct 22, 2012 at 3:29 PM, Ricardo Martinez <rmartinez@redvoiss.net> wrote: Hi Gabe. I'm mean between the spikes there is nothing different as regarding to the calls... the cps maintains the same.. there are no scripts running or crontabs running, the simultaneous calls are the same also. So, for some reason the "load average" start to grow (every 20 or 25 minutes) without any clear cause. I have debug with gdb the moment the server is with 12.5 of "load average" here : http://pastebin.freeswitch.org/20112
Can we detect a problem from here?
Regards, Ricardo.
-----Mensaje original----- De: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] En nombre de Gabriel Gunderson Enviado el: lunes, 22 de octubre de 2012 13:31 Para: FreeSWITCH Users Help Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes
On Mon, Oct 22, 2012 at 8:37 AM, Ricardo Martinez <rmartinez@redvoiss.net> wrote:
The problem is that I'm having weird spikes of "load average" cpu every 20 minutes (aprox), without any reason, reaching near of 12.5 of load average for this machine.
What have you done to verify that it happens 'without any reason'?
Gabe
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Ken http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org irc.freenode.net #freeswitch
Re: [Freeswitch-users] CPU Spikes every 20 minutes Hi Ken. I mounted the db directory in RAM as you mention, but the “load average” still is high. You can see this here : Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 132G 8.6G 117G 7% / proc 0 0 0 - /proc sysfs 0 0 0 - /sys devpts 0 0 0 - /dev/pts /dev/cciss/c0d0p1 104M 20M 79M 21% /boot tmpfs 4.2G 0 4.2G 0% /dev/shm none 0 0 0 - /proc/sys/fs/binfmt_misc sunrpc 0 0 0 - /var/lib/nfs/rpc_pipefs *tmpfs 4.2G 1.3M 4.2G 1% /usr/local/freeswitch/db* This is a resume from uptime and total calls for my freeswitch : 17:20:07 up 17 days, Total Calls : 16:26, 3 users, load average: 4.77, 4.50, 4.66 , Total Calls : 209, Total sessions: 440 17:20:12 up 17 days, Total Calls : 16:27, 3 users, load average: 6.79, 4.92, 4.80 , Total Calls : 245, Total sessions: 510 17:20:17 up 17 days, Total Calls : 16:27, 3 users, load average: 6.25, 4.84, 4.77 , Total Calls : 235, Total sessions: 488 17:20:22 up 17 days, Total Calls : 16:27, 3 users, load average: 6.63, 4.94, 4.81 , Total Calls : 233, Total sessions: 484 17:20:27 up 17 days, Total Calls : 16:27, 3 users, load average: 6.42, 4.92, 4.80 , Total Calls : 226, Total sessions: 470 17:20:32 up 17 days, Total Calls : 16:27, 3 users, load average: 5.90, 4.84, 4.78 , Total Calls : 218, Total sessions: 454 17:20:37 up 17 days, Total Calls : 16:27, 3 users, load average: 5.43, 4.76, 4.75 , Total Calls : 198, Total sessions: 416 17:20:43 up 17 days, Total Calls : 16:27, 3 users, load average: 4.99, 4.68, 4.72 , Total Calls : 183, Total sessions: 384 17:20:48 up 17 days, Total Calls : 16:27, 3 users, load average: 4.59, 4.60, 4.70 , Total Calls : 172, Total sessions: 366 17:20:53 up 17 days, Total Calls : 16:27, 3 users, load average: 4.31, 4.54, 4.68 , Total Calls : 184, Total sessions: 388 17:20:58 up 17 days, Total Calls : 16:27, 3 users, load average: 3.96, 4.47, 4.65 , Total Calls : 207, Total sessions: 442 17:21:03 up 17 days, Total Calls : 16:27, 3 users, load average: 4.04, 4.48, 4.65 , Total Calls : 203, Total sessions: 436 17:21:08 up 17 days, Total Calls : 16:27, 3 users, load average: 3.72, 4.40, 4.63 , Total Calls : 198, Total sessions: 418 17:21:13 up 17 days, Total Calls : 16:28, 3 users, load average: 3.47, 4.32, 4.60 , Total Calls : 198, Total sessions: 422 17:21:18 up 17 days, Total Calls : 16:28, 3 users, load average: 4.23, 4.47, 4.65 , Total Calls : 194, Total sessions: 410 17:21:23 up 17 days, Total Calls : 16:28, 3 users, load average: 8.30, 5.31, 4.92 , Total Calls : 211, Total sessions: 450 17:21:28 up 17 days, Total Calls : 16:28, 3 users, load average: 8.11, 5.32, 4.92 , Total Calls : 251, Total sessions: 536 17:21:33 up 17 days, Total Calls : 16:28, 3 users, load average: 8.74, 5.49, 4.98 , Total Calls : 250, Total sessions: 530 17:21:38 up 17 days, Total Calls : 16:28, 3 users, load average: 9.48, 5.70, 5.05 , Total Calls : 236, Total sessions: 496 17:21:43 up 17 days, Total Calls : 16:28, 3 users, load average: 9.45, 5.76, 5.07 , Total Calls : 233, Total sessions: 492 17:21:48 up 17 days, Total Calls : 16:28, 3 users, load average: 9.57, 5.84, 5.10 , Total Calls : 235, Total sessions: 496 17:21:53 up 17 days, Total Calls : 16:28, 3 users, load average: 10.24, 6.04, 5.17 , Total Calls : 222, Total sessions: 466 17:21:58 up 17 days, Total Calls : 16:28, 3 users, load average: 10.46, 6.16, 5.21 , Total Calls : 215, Total sessions: 452 17:22:03 up 17 days, Total Calls : 16:28, 3 users, load average: 11.07, 6.36, 5.28 , Total Calls : 208, Total sessions: 430 17:22:08 up 17 days, Total Calls : 16:28, 3 users, load average: 11.22, 6.47, 5.32 , Total Calls : 202, Total sessions: 422 17:22:14 up 17 days, Total Calls : 16:29, 3 users, load average: 11.52, 6.61, 5.37 , Total Calls : 240, Total sessions: 500 17:22:19 up 17 days, Total Calls : 16:29, 3 users, load average: 11.96, 6.78, 5.44 , Total Calls : 239, Total sessions: 498 17:22:24 up 17 days, Total Calls : 16:29, 3 users, load average: 14.29, 7.35, 5.63 , Total Calls : 250, Total sessions: 516 17:22:29 up 17 days, Total Calls : 16:29, 3 users, load average: 13.78, 7.36, 5.64 , Total Calls : 236, Total sessions: 490 17:22:34 up 17 days, Total Calls : 16:29, 3 users, load average: 12.92, 7.29, 5.63 , Total Calls : 209, Total sessions: 436 17:22:39 up 17 days, Total Calls : 16:29, 3 users, load average: 11.97, 7.18, 5.60 , Total Calls : 180, Total sessions: 372 17:22:44 up 17 days, Total Calls : 16:29, 3 users, load average: 11.01, 7.06, 5.57 , Total Calls : 161, Total sessions: 342 17:22:49 up 17 days, Total Calls : 16:29, 3 users, load average: 11.57, 7.24, 5.64 , Total Calls : 175, Total sessions: 370 17:22:54 up 17 days, Total Calls : 16:29, 3 users, load average: 11.12, 7.22, 5.64 , Total Calls : 181, Total sessions: 382 17:22:59 up 17 days, Total Calls : 16:29, 3 users, load average: 11.19, 7.30, 5.67 , Total Calls : 178, Total sessions: 366 17:23:04 up 17 days, Total Calls : 16:29, 3 users, load average: 11.10, 7.35, 5.69 , Total Calls : 173, Total sessions: 358 17:23:09 up 17 days, Total Calls : 16:30, 3 users, load average: 11.57, 7.51, 5.76 , Total Calls : 182, Total sessions: 378 17:23:14 up 17 days, Total Calls : 16:30, 3 users, load average: 11.36, 7.53, 5.77 , Total Calls : 187, Total sessions: 386 17:23:19 up 17 days, Total Calls : 16:30, 3 users, load average: 12.21, 7.77, 5.86 , Total Calls : 190, Total sessions: 402 17:23:24 up 17 days, Total Calls : 16:30, 3 users, load average: 12.04, 7.81, 5.88 , Total Calls : 239, Total sessions: 496 17:23:29 up 17 days, Total Calls : 16:30, 3 users, load average: 12.27, 7.93, 5.93 , Total Calls : 257, Total sessions: 532 17:23:34 up 17 days, Total Calls : 16:30, 3 users, load average: 12.73, 8.09, 5.99 , Total Calls : 242, Total sessions: 506 17:23:39 up 17 days, Total Calls : 16:30, 3 users, load average: 12.67, 8.16, 6.03 , Total Calls : 233, Total sessions: 482 17:23:45 up 17 days, Total Calls : 16:30, 3 users, load average: 12.38, 8.17, 6.04 , Total Calls : 238, Total sessions: 492 17:23:50 up 17 days, Total Calls : 16:30, 3 users, load average: 12.67, 8.30, 6.10 , Total Calls : 236, Total sessions: 482 17:23:55 up 17 days, Total Calls : 16:30, 3 users, load average: 13.09, 8.46, 6.16 , Total Calls : 222, Total sessions: 458 17:24:00 up 17 days, Total Calls : 16:30, 3 users, load average: 12.37, 8.39, 6.15 , Total Calls : 221, Total sessions: 470 17:24:05 up 17 days, Total Calls : 16:30, 3 users, load average: 11.78, 8.33, 6.14 , Total Calls : 200, Total sessions: 422 17:24:10 up 17 days, Total Calls : 16:31, 3 users, load average: 13.64, 8.77, 6.30 , Total Calls : 214, Total sessions: 452 17:24:15 up 17 days, Total Calls : 16:31, 3 users, load average: 13.98, 8.93, 6.36 , Total Calls : 255, Total sessions: 536 17:24:20 up 17 days, Total Calls : 16:31, 3 users, load average: 14.23, 9.06, 6.42 , Total Calls : 252, Total sessions: 524 17:24:25 up 17 days, Total Calls : 16:31, 3 users, load average: 15.73, 9.46, 6.56 , Total Calls : 260, Total sessions: 536 17:24:30 up 17 days, Total Calls : 16:31, 3 users, load average: 15.03, 9.42, 6.56 , Total Calls : 246, Total sessions: 508 17:24:35 up 17 days, Total Calls : 16:31, 3 users, load average: 15.11, 9.52, 6.61 , Total Calls : 230, Total sessions: 478 17:24:40 up 17 days, Total Calls : 16:31, 3 users, load average: 14.94, 9.58, 6.64 , Total Calls : 194, Total sessions: 408 17:24:45 up 17 days, Total Calls : 16:31, 3 users, load average: 14.54, 9.59, 6.66 , Total Calls : 189, Total sessions: 396 17:24:50 up 17 days, Total Calls : 16:31, 3 users, load average: 13.94, 9.55, 6.66 , Total Calls : 202, Total sessions: 438 17:24:55 up 17 days, Total Calls : 16:31, 3 users, load average: 13.30, 9.49, 6.66 , Total Calls : 221, Total sessions: 470 17:25:00 up 17 days, Total Calls : 16:31, 3 users, load average: 12.88, 9.46, 6.67 , Total Calls : 220, Total sessions: 478 17:25:05 up 17 days, Total Calls : 16:31, 3 users, load average: 14.25, 9.80, 6.79 , Total Calls : 219, Total sessions: 480 17:25:10 up 17 days, Total Calls : 16:32, 3 users, load average: 13.67, 9.76, 6.79 , Total Calls : 222, Total sessions: 490 17:25:16 up 17 days, Total Calls : 16:32, 3 users, load average: 13.05, 9.69, 6.79 , Total Calls : 214, Total sessions: 470 17:25:21 up 17 days, Total Calls : 16:32, 3 users, load average: 13.53, 9.85, 6.85 , Total Calls : 210, Total sessions: 450 17:25:26 up 17 days, Total Calls : 16:32, 3 users, load average: 13.81, 9.97, 6.91 , Total Calls : 250, Total sessions: 524 17:25:31 up 17 days, Total Calls : 16:32, 3 users, load average: 12.70, 9.80, 6.87 , Total Calls : 258, Total sessions: 536 17:25:36 up 17 days, Total Calls : 16:32, 3 users, load average: 11.68, 9.64, 6.83 , Total Calls : 255, Total sessions: 528 17:25:41 up 17 days, Total Calls : 16:32, 3 users, load average: 10.83, 9.49, 6.80 , Total Calls : 246, Total sessions: 524 17:25:46 up 17 days, Total Calls : 16:32, 3 users, load average: 10.04, 9.35, 6.77 , Total Calls : 262, Total sessions: 564 17:25:51 up 17 days, Total Calls : 16:32, 3 users, load average: 9.32, 9.21, 6.74 , Total Calls : 238, Total sessions: 510 17:25:56 up 17 days, Total Calls : 16:32, 3 users, load average: 8.57, 9.06, 6.70 , Total Calls : 220, Total sessions: 476 17:26:01 up 17 days, Total Calls : 16:32, 3 users, load average: 7.88, 8.91, 6.67 , Total Calls : 228, Total sessions: 498 What else could be happening. Did I mention that I’m using two transcoding cards from Sangoma? Regards, Ricardo.- *De:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *En nombre de *Ken Rice *Enviado el:* lunes, 22 de octubre de 2012 16:50 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes Load Average is the number of threads/processes waiting in the kernel’s run queue at sample time... Supposedly number of cores is the how much your load avg can get to before your boxes is exhausted but this isnt exactly true on modern hardware with highly multi-threaded software like FreeSWITCH... Where did I get the 100+ number? From looking at ‘top’ or ‘w’ on a box to see what the current load avg is... Sqlite is used all over FreeSWITCH from the coredb to mod_sofia and everything in between.... When you type ‘show channels’ from fs_cli it doesn’t query the structures in RAM to show you that output, it pulls it from the sqlite db... That being said, you can test quickly to see if sqlite is hammering the disk at some point by shifting freeswitch/db dir to a ram drive. Example code for linux such as Centos or Debian assuming freeswitch is installed to /usr/local/freeswitch Shutdown freeswitch rm –rf /usr/local/freeswitch/db/* mount –t tmpfs tmpfs /usr/local/freeswitch/db Startup freeswitch Once you have done that stick it back in rotation... Note this will not affect anything but voicemail if you have voicemail running on that box and not using an external DB via mod_voicemail’s DSN settings you’ll lose the voicemail DB... (I would assume this is not the case since you are using this primarily as a Transcoder) K On 10/22/12 2:30 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: Hi Ken. Maybe that is the problem. I was sure that the “load average” can’t go more than #cores in your system. In my case I have 8 cores, so a “load average” of 12 is high from my point of view. Where did you get the number 100 for this parameter? Anyway, what is running a sqlite in freeswitch. Does this have to do with the core.db file? Or with the cdr_sqlite module?. Thanks Ricardo.- *De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Ken Rice *Enviado el:* lunes, 22 de octubre de 2012 16:18 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes Could it be just a normal sqlite running a vacuum? Sqlite does this every so often or if its calling a sync.... Also freeswitch with a load avg of 12 for a short period of time isnt anything to really worry about. Due to the number of threads that’s possible with FS (ie: 1 per call leg) its not unusual to see a box humming along and the load avg well above 100 in my experience... This is due to the number of threads waiting in the run queue when it was sampled... Now, that’s not saying that this is not specifically indicative or a problem but something else to think about On 10/22/12 1:52 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: This is the gdb from the proccess consuming more cpu : (gdb) thread 7 [Switching to thread 7 (Thread 31781)]#0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 (gdb) bt #0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 #1 0x00002aede19276de in unixSync (id=0x2aaaac726380, dataOnly=<value optimized out>) at ./src/os_unix.c:1229 #2 0x00002aede1951178 in syncJournal (pPager=0x1a1c200) at ./src/pager.c:2229 #3 0x00002aede1953b65 in sqlite3pager_sync (pPager=0x1a1c200, zMaster=0x0, nTrunc=0) at ./src/pager.c:3770 #4 0x00002aede193d07b in sqlite3BtreeSync (p=<value optimized out>, zMaster=0x0) at ./src/btree.c:6520 #5 0x00002aede1931f15 in sqlite3VdbeHalt (p=0x2aaaad17ac70) at ./src/vdbeaux.c:1062 #6 0x00002aede1962613 in sqlite3VdbeExec (p=0x2aaaad17ac70) at ./src/vdbe.c:2360 #7 0x00002aede193036c in sqlite3Step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:236 #8 sqlite3_step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:289 #9 0x00002aede1938978 in sqlite3_exec (db=0x1a1f070, zSql=0x2aede199e717 "COMMIT", xCallback=0, pArg=0x0, pzErrMsg=0x4178bed0) at ./src/legacy.c:78 #10 0x00002aede18ae0c1 in switch_core_db_exec (db=0x1a1f070, sql=0x2aede199e717 "COMMIT", callback=0, data=0x0, errmsg=0x4178bf38) at src/switch_core_db.c:93 #11 0x00002aede1897c18 in switch_cache_db_execute_sql_real (dbh=0x1a2dce0, sql=0x2aede199e717 "COMMIT", err=0x0) at src/switch_core_sqldb.c:445 #12 0x00002aede18985c3 in switch_cache_db_persistant_execute_trans (dbh=0x1a2dce0, sql=0x5f03bd0 "delete from calls where (caller_uuid='e64319c5-7f91-47a1-9561-42da291e499a' or callee_uuid='e64319c5-7f91-47a1-9561-42da291e499a') and hostname=' transcoder.red.net <http://transcoder.red.net> <http://transcoder.red.net> ';\ndelete from calls where (call"..., retries=1) at src/switch_core_sqldb.c:787 #13 0x00002aede189bf1c in switch_core_sql_thread (thread=<value optimized out>, obj=<value optimized out>) at src/switch_core_sqldb.c:1026 #14 0x0000003c03c0673d in start_thread () from /lib64/libpthread.so.0 #15 0x0000003c030d3f6d in clone () from /lib64/libc.so.6 *De:* Ricardo Martinez [mailto:rmartinez@redvoiss.net<rmartinez@redvoiss.net>] *Enviado el:* lunes, 22 de octubre de 2012 15:50 *Para:* 'FreeSWITCH Users Help' *Asunto:* RE: [Freeswitch-users] CPU Spikes every 20 minutes AS far as i can see there is no java running. This is the top at the moment of the spike : top - 15:48:29 up 17 days, 14:55, 3 users, load average: 14.07, 8.87, 6.48 Tasks: 158 total, 1 running, 157 sleeping, 0 stopped, 0 zombie Cpu(s): 12.6%us, 9.0%sy, 0.0%ni, 66.1%id, 2.7%wa, 0.3%hi, 9.2%si, 0.0%st Mem: 8171500k total, 8041556k used, 129944k free, 177688k buffers Swap: 10223608k total, 0k used, 10223608k free, 6951988k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31772 root 15 0 870m 481m 6880 S 212.4 6.0 9523:51 freeswitch 4210 root 19 0 36344 2940 1444 S 1.0 0.0 56:21.08 sngtc_server 29586 root 15 0 12740 1132 808 R 0.3 0.0 0:00.02 top 1 root 15 0 10348 704 588 S 0.0 0.0 0:01.13 init 2 root RT -5 0 0 0 S 0.0 0.0 0:00.41 migration/0 3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0 4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0 5 root RT -5 0 0 0 S 0.0 0.0 0:00.83 migration/1 6 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1 7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1 8 root RT -5 0 0 0 S 0.0 0.0 0:00.14 migration/2 9 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/2 10 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/2 11 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/3 12 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/3 13 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/3 14 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/4 15 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/4 16 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/4 17 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/5 18 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/5 19 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/5 20 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/6 Regards, Ricardo.- *De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Guillermo Ruiz Camauer *Enviado el:* lunes, 22 de octubre de 2012 15:38 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes Do you have anything like Java running that could be doing garbage collection? What other processes are running on the machine? Does TOP show which process is causing the spike? Guillermo On Mon, Oct 22, 2012 at 3:29 PM, Ricardo Martinez <rmartinez@redvoiss.net> wrote: Hi Gabe. I'm mean between the spikes there is nothing different as regarding to the calls... the cps maintains the same.. there are no scripts running or crontabs running, the simultaneous calls are the same also. So, for some reason the "load average" start to grow (every 20 or 25 minutes) without any clear cause. I have debug with gdb the moment the server is with 12.5 of "load average" here : http://pastebin.freeswitch.org/20112 Can we detect a problem from here? Regards, Ricardo. -----Mensaje original----- De: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] En nombre de Gabriel Gunderson Enviado el: lunes, 22 de octubre de 2012 13:31 Para: FreeSWITCH Users Help Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes On Mon, Oct 22, 2012 at 8:37 AM, Ricardo Martinez <rmartinez@redvoiss.net> wrote:
The problem is that I'm having weird spikes of "load average" cpu every 20 minutes (aprox), without any reason, reaching near of 12.5 of load average for this machine.
What have you done to verify that it happens 'without any reason'? Gabe _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Ken *http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org *irc.freenode.net #freeswitch
How many concurrent calls are you doing on that box? On 10/22/12 3:33 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote:
Hi Ken. I mounted the db directory in RAM as you mention, but the ³load average² still is high. You can see this here : Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 132G 8.6G 117G 7% / proc 0 0 0 - /proc sysfs 0 0 0 - /sys devpts 0 0 0 - /dev/pts /dev/cciss/c0d0p1 104M 20M 79M 21% /boot tmpfs 4.2G 0 4.2G 0% /dev/shm none 0 0 0 - /proc/sys/fs/binfmt_misc sunrpc 0 0 0 - /var/lib/nfs/rpc_pipefs tmpfs 4.2G 1.3M 4.2G 1% /usr/local/freeswitch/db This is a resume from uptime and total calls for my freeswitch : 17:20:07 up 17 days, Total Calls : 16:26, 3 users, load average: 4.77, 4.50, 4.66 , Total Calls : 209, Total sessions: 440 17:20:12 up 17 days, Total Calls : 16:27, 3 users, load average: 6.79, 4.92, 4.80 , Total Calls : 245, Total sessions: 510 17:20:17 up 17 days, Total Calls : 16:27, 3 users, load average: 6.25, 4.84, 4.77 , Total Calls : 235, Total sessions: 488 17:20:22 up 17 days, Total Calls : 16:27, 3 users, load average: 6.63, 4.94, 4.81 , Total Calls : 233, Total sessions: 484 17:20:27 up 17 days, Total Calls : 16:27, 3 users, load average: 6.42, 4.92, 4.80 , Total Calls : 226, Total sessions: 470 17:20:32 up 17 days, Total Calls : 16:27, 3 users, load average: 5.90, 4.84, 4.78 , Total Calls : 218, Total sessions: 454 17:20:37 up 17 days, Total Calls : 16:27, 3 users, load average: 5.43, 4.76, 4.75 , Total Calls : 198, Total sessions: 416 17:20:43 up 17 days, Total Calls : 16:27, 3 users, load average: 4.99, 4.68, 4.72 , Total Calls : 183, Total sessions: 384 17:20:48 up 17 days, Total Calls : 16:27, 3 users, load average: 4.59, 4.60, 4.70 , Total Calls : 172, Total sessions: 366 17:20:53 up 17 days, Total Calls : 16:27, 3 users, load average: 4.31, 4.54, 4.68 , Total Calls : 184, Total sessions: 388 17:20:58 up 17 days, Total Calls : 16:27, 3 users, load average: 3.96, 4.47, 4.65 , Total Calls : 207, Total sessions: 442 17:21:03 up 17 days, Total Calls : 16:27, 3 users, load average: 4.04, 4.48, 4.65 , Total Calls : 203, Total sessions: 436 17:21:08 up 17 days, Total Calls : 16:27, 3 users, load average: 3.72, 4.40, 4.63 , Total Calls : 198, Total sessions: 418 17:21:13 up 17 days, Total Calls : 16:28, 3 users, load average: 3.47, 4.32, 4.60 , Total Calls : 198, Total sessions: 422 17:21:18 up 17 days, Total Calls : 16:28, 3 users, load average: 4.23, 4.47, 4.65 , Total Calls : 194, Total sessions: 410 17:21:23 up 17 days, Total Calls : 16:28, 3 users, load average: 8.30, 5.31, 4.92 , Total Calls : 211, Total sessions: 450 17:21:28 up 17 days, Total Calls : 16:28, 3 users, load average: 8.11, 5.32, 4.92 , Total Calls : 251, Total sessions: 536 17:21:33 up 17 days, Total Calls : 16:28, 3 users, load average: 8.74, 5.49, 4.98 , Total Calls : 250, Total sessions: 530 17:21:38 up 17 days, Total Calls : 16:28, 3 users, load average: 9.48, 5.70, 5.05 , Total Calls : 236, Total sessions: 496 17:21:43 up 17 days, Total Calls : 16:28, 3 users, load average: 9.45, 5.76, 5.07 , Total Calls : 233, Total sessions: 492 17:21:48 up 17 days, Total Calls : 16:28, 3 users, load average: 9.57, 5.84, 5.10 , Total Calls : 235, Total sessions: 496 17:21:53 up 17 days, Total Calls : 16:28, 3 users, load average: 10.24, 6.04, 5.17 , Total Calls : 222, Total sessions: 466 17:21:58 up 17 days, Total Calls : 16:28, 3 users, load average: 10.46, 6.16, 5.21 , Total Calls : 215, Total sessions: 452 17:22:03 up 17 days, Total Calls : 16:28, 3 users, load average: 11.07, 6.36, 5.28 , Total Calls : 208, Total sessions: 430 17:22:08 up 17 days, Total Calls : 16:28, 3 users, load average: 11.22, 6.47, 5.32 , Total Calls : 202, Total sessions: 422 17:22:14 up 17 days, Total Calls : 16:29, 3 users, load average: 11.52, 6.61, 5.37 , Total Calls : 240, Total sessions: 500 17:22:19 up 17 days, Total Calls : 16:29, 3 users, load average: 11.96, 6.78, 5.44 , Total Calls : 239, Total sessions: 498 17:22:24 up 17 days, Total Calls : 16:29, 3 users, load average: 14.29, 7.35, 5.63 , Total Calls : 250, Total sessions: 516 17:22:29 up 17 days, Total Calls : 16:29, 3 users, load average: 13.78, 7.36, 5.64 , Total Calls : 236, Total sessions: 490 17:22:34 up 17 days, Total Calls : 16:29, 3 users, load average: 12.92, 7.29, 5.63 , Total Calls : 209, Total sessions: 436 17:22:39 up 17 days, Total Calls : 16:29, 3 users, load average: 11.97, 7.18, 5.60 , Total Calls : 180, Total sessions: 372 17:22:44 up 17 days, Total Calls : 16:29, 3 users, load average: 11.01, 7.06, 5.57 , Total Calls : 161, Total sessions: 342 17:22:49 up 17 days, Total Calls : 16:29, 3 users, load average: 11.57, 7.24, 5.64 , Total Calls : 175, Total sessions: 370 17:22:54 up 17 days, Total Calls : 16:29, 3 users, load average: 11.12, 7.22, 5.64 , Total Calls : 181, Total sessions: 382 17:22:59 up 17 days, Total Calls : 16:29, 3 users, load average: 11.19, 7.30, 5.67 , Total Calls : 178, Total sessions: 366 17:23:04 up 17 days, Total Calls : 16:29, 3 users, load average: 11.10, 7.35, 5.69 , Total Calls : 173, Total sessions: 358 17:23:09 up 17 days, Total Calls : 16:30, 3 users, load average: 11.57, 7.51, 5.76 , Total Calls : 182, Total sessions: 378 17:23:14 up 17 days, Total Calls : 16:30, 3 users, load average: 11.36, 7.53, 5.77 , Total Calls : 187, Total sessions: 386 17:23:19 up 17 days, Total Calls : 16:30, 3 users, load average: 12.21, 7.77, 5.86 , Total Calls : 190, Total sessions: 402 17:23:24 up 17 days, Total Calls : 16:30, 3 users, load average: 12.04, 7.81, 5.88 , Total Calls : 239, Total sessions: 496 17:23:29 up 17 days, Total Calls : 16:30, 3 users, load average: 12.27, 7.93, 5.93 , Total Calls : 257, Total sessions: 532 17:23:34 up 17 days, Total Calls : 16:30, 3 users, load average: 12.73, 8.09, 5.99 , Total Calls : 242, Total sessions: 506 17:23:39 up 17 days, Total Calls : 16:30, 3 users, load average: 12.67, 8.16, 6.03 , Total Calls : 233, Total sessions: 482 17:23:45 up 17 days, Total Calls : 16:30, 3 users, load average: 12.38, 8.17, 6.04 , Total Calls : 238, Total sessions: 492 17:23:50 up 17 days, Total Calls : 16:30, 3 users, load average: 12.67, 8.30, 6.10 , Total Calls : 236, Total sessions: 482 17:23:55 up 17 days, Total Calls : 16:30, 3 users, load average: 13.09, 8.46, 6.16 , Total Calls : 222, Total sessions: 458 17:24:00 up 17 days, Total Calls : 16:30, 3 users, load average: 12.37, 8.39, 6.15 , Total Calls : 221, Total sessions: 470 17:24:05 up 17 days, Total Calls : 16:30, 3 users, load average: 11.78, 8.33, 6.14 , Total Calls : 200, Total sessions: 422 17:24:10 up 17 days, Total Calls : 16:31, 3 users, load average: 13.64, 8.77, 6.30 , Total Calls : 214, Total sessions: 452 17:24:15 up 17 days, Total Calls : 16:31, 3 users, load average: 13.98, 8.93, 6.36 , Total Calls : 255, Total sessions: 536 17:24:20 up 17 days, Total Calls : 16:31, 3 users, load average: 14.23, 9.06, 6.42 , Total Calls : 252, Total sessions: 524 17:24:25 up 17 days, Total Calls : 16:31, 3 users, load average: 15.73, 9.46, 6.56 , Total Calls : 260, Total sessions: 536 17:24:30 up 17 days, Total Calls : 16:31, 3 users, load average: 15.03, 9.42, 6.56 , Total Calls : 246, Total sessions: 508 17:24:35 up 17 days, Total Calls : 16:31, 3 users, load average: 15.11, 9.52, 6.61 , Total Calls : 230, Total sessions: 478 17:24:40 up 17 days, Total Calls : 16:31, 3 users, load average: 14.94, 9.58, 6.64 , Total Calls : 194, Total sessions: 408 17:24:45 up 17 days, Total Calls : 16:31, 3 users, load average: 14.54, 9.59, 6.66 , Total Calls : 189, Total sessions: 396 17:24:50 up 17 days, Total Calls : 16:31, 3 users, load average: 13.94, 9.55, 6.66 , Total Calls : 202, Total sessions: 438 17:24:55 up 17 days, Total Calls : 16:31, 3 users, load average: 13.30, 9.49, 6.66 , Total Calls : 221, Total sessions: 470 17:25:00 up 17 days, Total Calls : 16:31, 3 users, load average: 12.88, 9.46, 6.67 , Total Calls : 220, Total sessions: 478 17:25:05 up 17 days, Total Calls : 16:31, 3 users, load average: 14.25, 9.80, 6.79 , Total Calls : 219, Total sessions: 480 17:25:10 up 17 days, Total Calls : 16:32, 3 users, load average: 13.67, 9.76, 6.79 , Total Calls : 222, Total sessions: 490 17:25:16 up 17 days, Total Calls : 16:32, 3 users, load average: 13.05, 9.69, 6.79 , Total Calls : 214, Total sessions: 470 17:25:21 up 17 days, Total Calls : 16:32, 3 users, load average: 13.53, 9.85, 6.85 , Total Calls : 210, Total sessions: 450 17:25:26 up 17 days, Total Calls : 16:32, 3 users, load average: 13.81, 9.97, 6.91 , Total Calls : 250, Total sessions: 524 17:25:31 up 17 days, Total Calls : 16:32, 3 users, load average: 12.70, 9.80, 6.87 , Total Calls : 258, Total sessions: 536 17:25:36 up 17 days, Total Calls : 16:32, 3 users, load average: 11.68, 9.64, 6.83 , Total Calls : 255, Total sessions: 528 17:25:41 up 17 days, Total Calls : 16:32, 3 users, load average: 10.83, 9.49, 6.80 , Total Calls : 246, Total sessions: 524 17:25:46 up 17 days, Total Calls : 16:32, 3 users, load average: 10.04, 9.35, 6.77 , Total Calls : 262, Total sessions: 564 17:25:51 up 17 days, Total Calls : 16:32, 3 users, load average: 9.32, 9.21, 6.74 , Total Calls : 238, Total sessions: 510 17:25:56 up 17 days, Total Calls : 16:32, 3 users, load average: 8.57, 9.06, 6.70 , Total Calls : 220, Total sessions: 476 17:26:01 up 17 days, Total Calls : 16:32, 3 users, load average: 7.88, 8.91, 6.67 , Total Calls : 228, Total sessions: 498 What else could be happening. Did I mention that I¹m using two transcoding cards from Sangoma? Regards, Ricardo.-
De: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] En nombre de Ken Rice Enviado el: lunes, 22 de octubre de 2012 16:50 Para: FreeSWITCH Users Help Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes Load Average is the number of threads/processes waiting in the kernel¹s run queue at sample time... Supposedly number of cores is the how much your load avg can get to before your boxes is exhausted but this isnt exactly true on modern hardware with highly multi-threaded software like FreeSWITCH...
Where did I get the 100+ number? From looking at top¹ or w¹ on a box to see what the current load avg is...
Sqlite is used all over FreeSWITCH from the coredb to mod_sofia and everything in between.... When you type show channels¹ from fs_cli it doesn¹t query the structures in RAM to show you that output, it pulls it from the sqlite db...
That being said, you can test quickly to see if sqlite is hammering the disk at some point by shifting freeswitch/db dir to a ram drive. Example code for linux such as Centos or Debian assuming freeswitch is installed to /usr/local/freeswitch
Shutdown freeswitch rm rf /usr/local/freeswitch/db/* mount t tmpfs tmpfs /usr/local/freeswitch/db Startup freeswitch
Once you have done that stick it back in rotation...
Note this will not affect anything but voicemail if you have voicemail running on that box and not using an external DB via mod_voicemail¹s DSN settings you¹ll lose the voicemail DB... (I would assume this is not the case since you are using this primarily as a Transcoder)
K
On 10/22/12 2:30 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: Hi Ken. Maybe that is the problem. I was sure that the ³load average² can¹t go more than #cores in your system. In my case I have 8 cores, so a ³load average² of 12 is high from my point of view. Where did you get the number 100 for this parameter? Anyway, what is running a sqlite in freeswitch. Does this have to do with the core.db file? Or with the cdr_sqlite module?. Thanks Ricardo.-
De: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] En nombre de Ken Rice Enviado el: lunes, 22 de octubre de 2012 16:18 Para: FreeSWITCH Users Help Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes Could it be just a normal sqlite running a vacuum?
Sqlite does this every so often or if its calling a sync....
Also freeswitch with a load avg of 12 for a short period of time isnt anything to really worry about. Due to the number of threads that¹s possible with FS (ie: 1 per call leg) its not unusual to see a box humming along and the load avg well above 100 in my experience... This is due to the number of threads waiting in the run queue when it was sampled... Now, that¹s not saying that this is not specifically indicative or a problem but something else to think about
On 10/22/12 1:52 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: This is the gdb from the proccess consuming more cpu : (gdb) thread 7 [Switching to thread 7 (Thread 31781)]#0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 (gdb) bt #0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 #1 0x00002aede19276de in unixSync (id=0x2aaaac726380, dataOnly=<value optimized out>) at ./src/os_unix.c:1229 #2 0x00002aede1951178 in syncJournal (pPager=0x1a1c200) at ./src/pager.c:2229 #3 0x00002aede1953b65 in sqlite3pager_sync (pPager=0x1a1c200, zMaster=0x0, nTrunc=0) at ./src/pager.c:3770 #4 0x00002aede193d07b in sqlite3BtreeSync (p=<value optimized out>, zMaster=0x0) at ./src/btree.c:6520 #5 0x00002aede1931f15 in sqlite3VdbeHalt (p=0x2aaaad17ac70) at ./src/vdbeaux.c:1062 #6 0x00002aede1962613 in sqlite3VdbeExec (p=0x2aaaad17ac70) at ./src/vdbe.c:2360 #7 0x00002aede193036c in sqlite3Step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:236 #8 sqlite3_step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:289 #9 0x00002aede1938978 in sqlite3_exec (db=0x1a1f070, zSql=0x2aede199e717 "COMMIT", xCallback=0, pArg=0x0, pzErrMsg=0x4178bed0) at ./src/legacy.c:78 #10 0x00002aede18ae0c1 in switch_core_db_exec (db=0x1a1f070, sql=0x2aede199e717 "COMMIT", callback=0, data=0x0, errmsg=0x4178bf38) at src/switch_core_db.c:93 #11 0x00002aede1897c18 in switch_cache_db_execute_sql_real (dbh=0x1a2dce0, sql=0x2aede199e717 "COMMIT", err=0x0) at src/switch_core_sqldb.c:445 #12 0x00002aede18985c3 in switch_cache_db_persistant_execute_trans (dbh=0x1a2dce0, sql=0x5f03bd0 "delete from calls where (caller_uuid='e64319c5-7f91-47a1-9561-42da291e499a' or callee_uuid='e64319c5-7f91-47a1-9561-42da291e499a') and hostname='transcoder.red.net <http://transcoder.red.net> <http://transcoder.red.net> <http://transcoder.red.net> ';\ndelete from calls where (call"..., retries=1) at src/switch_core_sqldb.c:787 #13 0x00002aede189bf1c in switch_core_sql_thread (thread=<value optimized out>, obj=<value optimized out>) at src/switch_core_sqldb.c:1026 #14 0x0000003c03c0673d in start_thread () from /lib64/libpthread.so.0 #15 0x0000003c030d3f6d in clone () from /lib64/libc.so.6
De: Ricardo Martinez [mailto:rmartinez@redvoiss.net] Enviado el: lunes, 22 de octubre de 2012 15:50 Para: 'FreeSWITCH Users Help' Asunto: RE: [Freeswitch-users] CPU Spikes every 20 minutes AS far as i can see there is no java running. This is the top at the moment of the spike : top - 15:48:29 up 17 days, 14:55, 3 users, load average: 14.07, 8.87, 6.48 Tasks: 158 total, 1 running, 157 sleeping, 0 stopped, 0 zombie Cpu(s): 12.6%us, 9.0%sy, 0.0%ni, 66.1%id, 2.7%wa, 0.3%hi, 9.2%si, 0.0%st Mem: 8171500k total, 8041556k used, 129944k free, 177688k buffers Swap: 10223608k total, 0k used, 10223608k free, 6951988k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31772 root 15 0 870m 481m 6880 S 212.4 6.0 9523:51 freeswitch 4210 root 19 0 36344 2940 1444 S 1.0 0.0 56:21.08 sngtc_server 29586 root 15 0 12740 1132 808 R 0.3 0.0 0:00.02 top 1 root 15 0 10348 704 588 S 0.0 0.0 0:01.13 init 2 root RT -5 0 0 0 S 0.0 0.0 0:00.41 migration/0 3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0 4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0 5 root RT -5 0 0 0 S 0.0 0.0 0:00.83 migration/1 6 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1 7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1 8 root RT -5 0 0 0 S 0.0 0.0 0:00.14 migration/2 9 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/2 10 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/2 11 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/3 12 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/3 13 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/3 14 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/4 15 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/4 16 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/4 17 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/5 18 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/5 19 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/5 20 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/6 Regards, Ricardo.- De: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] En nombre de Guillermo Ruiz Camauer Enviado el: lunes, 22 de octubre de 2012 15:38 Para: FreeSWITCH Users Help Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes Do you have anything like Java running that could be doing garbage collection?
What other processes are running on the machine? Does TOP show which process is causing the spike?
Guillermo
On Mon, Oct 22, 2012 at 3:29 PM, Ricardo Martinez <rmartinez@redvoiss.net> wrote: Hi Gabe. I'm mean between the spikes there is nothing different as regarding to the calls... the cps maintains the same.. there are no scripts running or crontabs running, the simultaneous calls are the same also. So, for some reason the "load average" start to grow (every 20 or 25 minutes) without any clear cause. I have debug with gdb the moment the server is with 12.5 of "load average" here : http://pastebin.freeswitch.org/20112
Can we detect a problem from here?
Regards, Ricardo.
-----Mensaje original----- De: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] En nombre de Gabriel Gunderson Enviado el: lunes, 22 de octubre de 2012 13:31 Para: FreeSWITCH Users Help Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes
On Mon, Oct 22, 2012 at 8:37 AM, Ricardo Martinez <rmartinez@redvoiss.net> wrote:
The problem is that I'm having weird spikes of "load average" cpu every 20 minutes (aprox), without any reason, reaching near of 12.5 of load average for this machine.
What have you done to verify that it happens 'without any reason'?
Gabe
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Ken http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org irc.freenode.net #freeswitch
Re: [Freeswitch-users] CPU Spikes every 20 minutes Ken. There is an average of 220 concurrent calls. Machine : DL380 G7 – 2 Quad Core – 8 Gig RAM. Centos 5.5 64bits Ricarod.- *De:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *En nombre de *Ken Rice *Enviado el:* lunes, 22 de octubre de 2012 17:47 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes How many concurrent calls are you doing on that box? On 10/22/12 3:33 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: Hi Ken. I mounted the db directory in RAM as you mention, but the “load average” still is high. You can see this here : Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 132G 8.6G 117G 7% / proc 0 0 0 - /proc sysfs 0 0 0 - /sys devpts 0 0 0 - /dev/pts /dev/cciss/c0d0p1 104M 20M 79M 21% /boot tmpfs 4.2G 0 4.2G 0% /dev/shm none 0 0 0 - /proc/sys/fs/binfmt_misc sunrpc 0 0 0 - /var/lib/nfs/rpc_pipefs *tmpfs 4.2G 1.3M 4.2G 1% /usr/local/freeswitch/db * This is a resume from uptime and total calls for my freeswitch : 17:20:07 up 17 days, Total Calls : 16:26, 3 users, load average: 4.77, 4.50, 4.66 , Total Calls : 209, Total sessions: 440 17:20:12 up 17 days, Total Calls : 16:27, 3 users, load average: 6.79, 4.92, 4.80 , Total Calls : 245, Total sessions: 510 17:20:17 up 17 days, Total Calls : 16:27, 3 users, load average: 6.25, 4.84, 4.77 , Total Calls : 235, Total sessions: 488 17:20:22 up 17 days, Total Calls : 16:27, 3 users, load average: 6.63, 4.94, 4.81 , Total Calls : 233, Total sessions: 484 17:20:27 up 17 days, Total Calls : 16:27, 3 users, load average: 6.42, 4.92, 4.80 , Total Calls : 226, Total sessions: 470 17:20:32 up 17 days, Total Calls : 16:27, 3 users, load average: 5.90, 4.84, 4.78 , Total Calls : 218, Total sessions: 454 17:20:37 up 17 days, Total Calls : 16:27, 3 users, load average: 5.43, 4.76, 4.75 , Total Calls : 198, Total sessions: 416 17:20:43 up 17 days, Total Calls : 16:27, 3 users, load average: 4.99, 4.68, 4.72 , Total Calls : 183, Total sessions: 384 17:20:48 up 17 days, Total Calls : 16:27, 3 users, load average: 4.59, 4.60, 4.70 , Total Calls : 172, Total sessions: 366 17:20:53 up 17 days, Total Calls : 16:27, 3 users, load average: 4.31, 4.54, 4.68 , Total Calls : 184, Total sessions: 388 17:20:58 up 17 days, Total Calls : 16:27, 3 users, load average: 3.96, 4.47, 4.65 , Total Calls : 207, Total sessions: 442 17:21:03 up 17 days, Total Calls : 16:27, 3 users, load average: 4.04, 4.48, 4.65 , Total Calls : 203, Total sessions: 436 17:21:08 up 17 days, Total Calls : 16:27, 3 users, load average: 3.72, 4.40, 4.63 , Total Calls : 198, Total sessions: 418 17:21:13 up 17 days, Total Calls : 16:28, 3 users, load average: 3.47, 4.32, 4.60 , Total Calls : 198, Total sessions: 422 17:21:18 up 17 days, Total Calls : 16:28, 3 users, load average: 4.23, 4.47, 4.65 , Total Calls : 194, Total sessions: 410 17:21:23 up 17 days, Total Calls : 16:28, 3 users, load average: 8.30, 5.31, 4.92 , Total Calls : 211, Total sessions: 450 17:21:28 up 17 days, Total Calls : 16:28, 3 users, load average: 8.11, 5.32, 4.92 , Total Calls : 251, Total sessions: 536 17:21:33 up 17 days, Total Calls : 16:28, 3 users, load average: 8.74, 5.49, 4.98 , Total Calls : 250, Total sessions: 530 17:21:38 up 17 days, Total Calls : 16:28, 3 users, load average: 9.48, 5.70, 5.05 , Total Calls : 236, Total sessions: 496 17:21:43 up 17 days, Total Calls : 16:28, 3 users, load average: 9.45, 5.76, 5.07 , Total Calls : 233, Total sessions: 492 17:21:48 up 17 days, Total Calls : 16:28, 3 users, load average: 9.57, 5.84, 5.10 , Total Calls : 235, Total sessions: 496 17:21:53 up 17 days, Total Calls : 16:28, 3 users, load average: 10.24, 6.04, 5.17 , Total Calls : 222, Total sessions: 466 17:21:58 up 17 days, Total Calls : 16:28, 3 users, load average: 10.46, 6.16, 5.21 , Total Calls : 215, Total sessions: 452 17:22:03 up 17 days, Total Calls : 16:28, 3 users, load average: 11.07, 6.36, 5.28 , Total Calls : 208, Total sessions: 430 17:22:08 up 17 days, Total Calls : 16:28, 3 users, load average: 11.22, 6.47, 5.32 , Total Calls : 202, Total sessions: 422 17:22:14 up 17 days, Total Calls : 16:29, 3 users, load average: 11.52, 6.61, 5.37 , Total Calls : 240, Total sessions: 500 17:22:19 up 17 days, Total Calls : 16:29, 3 users, load average: 11.96, 6.78, 5.44 , Total Calls : 239, Total sessions: 498 17:22:24 up 17 days, Total Calls : 16:29, 3 users, load average: 14.29, 7.35, 5.63 , Total Calls : 250, Total sessions: 516 17:22:29 up 17 days, Total Calls : 16:29, 3 users, load average: 13.78, 7.36, 5.64 , Total Calls : 236, Total sessions: 490 17:22:34 up 17 days, Total Calls : 16:29, 3 users, load average: 12.92, 7.29, 5.63 , Total Calls : 209, Total sessions: 436 17:22:39 up 17 days, Total Calls : 16:29, 3 users, load average: 11.97, 7.18, 5.60 , Total Calls : 180, Total sessions: 372 17:22:44 up 17 days, Total Calls : 16:29, 3 users, load average: 11.01, 7.06, 5.57 , Total Calls : 161, Total sessions: 342 17:22:49 up 17 days, Total Calls : 16:29, 3 users, load average: 11.57, 7.24, 5.64 , Total Calls : 175, Total sessions: 370 17:22:54 up 17 days, Total Calls : 16:29, 3 users, load average: 11.12, 7.22, 5.64 , Total Calls : 181, Total sessions: 382 17:22:59 up 17 days, Total Calls : 16:29, 3 users, load average: 11.19, 7.30, 5.67 , Total Calls : 178, Total sessions: 366 17:23:04 up 17 days, Total Calls : 16:29, 3 users, load average: 11.10, 7.35, 5.69 , Total Calls : 173, Total sessions: 358 17:23:09 up 17 days, Total Calls : 16:30, 3 users, load average: 11.57, 7.51, 5.76 , Total Calls : 182, Total sessions: 378 17:23:14 up 17 days, Total Calls : 16:30, 3 users, load average: 11.36, 7.53, 5.77 , Total Calls : 187, Total sessions: 386 17:23:19 up 17 days, Total Calls : 16:30, 3 users, load average: 12.21, 7.77, 5.86 , Total Calls : 190, Total sessions: 402 17:23:24 up 17 days, Total Calls : 16:30, 3 users, load average: 12.04, 7.81, 5.88 , Total Calls : 239, Total sessions: 496 17:23:29 up 17 days, Total Calls : 16:30, 3 users, load average: 12.27, 7.93, 5.93 , Total Calls : 257, Total sessions: 532 17:23:34 up 17 days, Total Calls : 16:30, 3 users, load average: 12.73, 8.09, 5.99 , Total Calls : 242, Total sessions: 506 17:23:39 up 17 days, Total Calls : 16:30, 3 users, load average: 12.67, 8.16, 6.03 , Total Calls : 233, Total sessions: 482 17:23:45 up 17 days, Total Calls : 16:30, 3 users, load average: 12.38, 8.17, 6.04 , Total Calls : 238, Total sessions: 492 17:23:50 up 17 days, Total Calls : 16:30, 3 users, load average: 12.67, 8.30, 6.10 , Total Calls : 236, Total sessions: 482 17:23:55 up 17 days, Total Calls : 16:30, 3 users, load average: 13.09, 8.46, 6.16 , Total Calls : 222, Total sessions: 458 17:24:00 up 17 days, Total Calls : 16:30, 3 users, load average: 12.37, 8.39, 6.15 , Total Calls : 221, Total sessions: 470 17:24:05 up 17 days, Total Calls : 16:30, 3 users, load average: 11.78, 8.33, 6.14 , Total Calls : 200, Total sessions: 422 17:24:10 up 17 days, Total Calls : 16:31, 3 users, load average: 13.64, 8.77, 6.30 , Total Calls : 214, Total sessions: 452 17:24:15 up 17 days, Total Calls : 16:31, 3 users, load average: 13.98, 8.93, 6.36 , Total Calls : 255, Total sessions: 536 17:24:20 up 17 days, Total Calls : 16:31, 3 users, load average: 14.23, 9.06, 6.42 , Total Calls : 252, Total sessions: 524 17:24:25 up 17 days, Total Calls : 16:31, 3 users, load average: 15.73, 9.46, 6.56 , Total Calls : 260, Total sessions: 536 17:24:30 up 17 days, Total Calls : 16:31, 3 users, load average: 15.03, 9.42, 6.56 , Total Calls : 246, Total sessions: 508 17:24:35 up 17 days, Total Calls : 16:31, 3 users, load average: 15.11, 9.52, 6.61 , Total Calls : 230, Total sessions: 478 17:24:40 up 17 days, Total Calls : 16:31, 3 users, load average: 14.94, 9.58, 6.64 , Total Calls : 194, Total sessions: 408 17:24:45 up 17 days, Total Calls : 16:31, 3 users, load average: 14.54, 9.59, 6.66 , Total Calls : 189, Total sessions: 396 17:24:50 up 17 days, Total Calls : 16:31, 3 users, load average: 13.94, 9.55, 6.66 , Total Calls : 202, Total sessions: 438 17:24:55 up 17 days, Total Calls : 16:31, 3 users, load average: 13.30, 9.49, 6.66 , Total Calls : 221, Total sessions: 470 17:25:00 up 17 days, Total Calls : 16:31, 3 users, load average: 12.88, 9.46, 6.67 , Total Calls : 220, Total sessions: 478 17:25:05 up 17 days, Total Calls : 16:31, 3 users, load average: 14.25, 9.80, 6.79 , Total Calls : 219, Total sessions: 480 17:25:10 up 17 days, Total Calls : 16:32, 3 users, load average: 13.67, 9.76, 6.79 , Total Calls : 222, Total sessions: 490 17:25:16 up 17 days, Total Calls : 16:32, 3 users, load average: 13.05, 9.69, 6.79 , Total Calls : 214, Total sessions: 470 17:25:21 up 17 days, Total Calls : 16:32, 3 users, load average: 13.53, 9.85, 6.85 , Total Calls : 210, Total sessions: 450 17:25:26 up 17 days, Total Calls : 16:32, 3 users, load average: 13.81, 9.97, 6.91 , Total Calls : 250, Total sessions: 524 17:25:31 up 17 days, Total Calls : 16:32, 3 users, load average: 12.70, 9.80, 6.87 , Total Calls : 258, Total sessions: 536 17:25:36 up 17 days, Total Calls : 16:32, 3 users, load average: 11.68, 9.64, 6.83 , Total Calls : 255, Total sessions: 528 17:25:41 up 17 days, Total Calls : 16:32, 3 users, load average: 10.83, 9.49, 6.80 , Total Calls : 246, Total sessions: 524 17:25:46 up 17 days, Total Calls : 16:32, 3 users, load average: 10.04, 9.35, 6.77 , Total Calls : 262, Total sessions: 564 17:25:51 up 17 days, Total Calls : 16:32, 3 users, load average: 9.32, 9.21, 6.74 , Total Calls : 238, Total sessions: 510 17:25:56 up 17 days, Total Calls : 16:32, 3 users, load average: 8.57, 9.06, 6.70 , Total Calls : 220, Total sessions: 476 17:26:01 up 17 days, Total Calls : 16:32, 3 users, load average: 7.88, 8.91, 6.67 , Total Calls : 228, Total sessions: 498 What else could be happening. Did I mention that I’m using two transcoding cards from Sangoma? Regards, Ricardo.- *De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Ken Rice *Enviado el:* lunes, 22 de octubre de 2012 16:50 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes Load Average is the number of threads/processes waiting in the kernel’s run queue at sample time... Supposedly number of cores is the how much your load avg can get to before your boxes is exhausted but this isnt exactly true on modern hardware with highly multi-threaded software like FreeSWITCH... Where did I get the 100+ number? From looking at ‘top’ or ‘w’ on a box to see what the current load avg is... Sqlite is used all over FreeSWITCH from the coredb to mod_sofia and everything in between.... When you type ‘show channels’ from fs_cli it doesn’t query the structures in RAM to show you that output, it pulls it from the sqlite db... That being said, you can test quickly to see if sqlite is hammering the disk at some point by shifting freeswitch/db dir to a ram drive. Example code for linux such as Centos or Debian assuming freeswitch is installed to /usr/local/freeswitch Shutdown freeswitch rm –rf /usr/local/freeswitch/db/* mount –t tmpfs tmpfs /usr/local/freeswitch/db Startup freeswitch Once you have done that stick it back in rotation... Note this will not affect anything but voicemail if you have voicemail running on that box and not using an external DB via mod_voicemail’s DSN settings you’ll lose the voicemail DB... (I would assume this is not the case since you are using this primarily as a Transcoder) K On 10/22/12 2:30 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: Hi Ken. Maybe that is the problem. I was sure that the “load average” can’t go more than #cores in your system. In my case I have 8 cores, so a “load average” of 12 is high from my point of view. Where did you get the number 100 for this parameter? Anyway, what is running a sqlite in freeswitch. Does this have to do with the core.db file? Or with the cdr_sqlite module?. Thanks Ricardo.- *De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Ken Rice *Enviado el:* lunes, 22 de octubre de 2012 16:18 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes Could it be just a normal sqlite running a vacuum? Sqlite does this every so often or if its calling a sync.... Also freeswitch with a load avg of 12 for a short period of time isnt anything to really worry about. Due to the number of threads that’s possible with FS (ie: 1 per call leg) its not unusual to see a box humming along and the load avg well above 100 in my experience... This is due to the number of threads waiting in the run queue when it was sampled... Now, that’s not saying that this is not specifically indicative or a problem but something else to think about On 10/22/12 1:52 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: This is the gdb from the proccess consuming more cpu : (gdb) thread 7 [Switching to thread 7 (Thread 31781)]#0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 (gdb) bt #0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 #1 0x00002aede19276de in unixSync (id=0x2aaaac726380, dataOnly=<value optimized out>) at ./src/os_unix.c:1229 #2 0x00002aede1951178 in syncJournal (pPager=0x1a1c200) at ./src/pager.c:2229 #3 0x00002aede1953b65 in sqlite3pager_sync (pPager=0x1a1c200, zMaster=0x0, nTrunc=0) at ./src/pager.c:3770 #4 0x00002aede193d07b in sqlite3BtreeSync (p=<value optimized out>, zMaster=0x0) at ./src/btree.c:6520 #5 0x00002aede1931f15 in sqlite3VdbeHalt (p=0x2aaaad17ac70) at ./src/vdbeaux.c:1062 #6 0x00002aede1962613 in sqlite3VdbeExec (p=0x2aaaad17ac70) at ./src/vdbe.c:2360 #7 0x00002aede193036c in sqlite3Step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:236 #8 sqlite3_step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:289 #9 0x00002aede1938978 in sqlite3_exec (db=0x1a1f070, zSql=0x2aede199e717 "COMMIT", xCallback=0, pArg=0x0, pzErrMsg=0x4178bed0) at ./src/legacy.c:78 #10 0x00002aede18ae0c1 in switch_core_db_exec (db=0x1a1f070, sql=0x2aede199e717 "COMMIT", callback=0, data=0x0, errmsg=0x4178bf38) at src/switch_core_db.c:93 #11 0x00002aede1897c18 in switch_cache_db_execute_sql_real (dbh=0x1a2dce0, sql=0x2aede199e717 "COMMIT", err=0x0) at src/switch_core_sqldb.c:445 #12 0x00002aede18985c3 in switch_cache_db_persistant_execute_trans (dbh=0x1a2dce0, sql=0x5f03bd0 "delete from calls where (caller_uuid='e64319c5-7f91-47a1-9561-42da291e499a' or callee_uuid='e64319c5-7f91-47a1-9561-42da291e499a') and hostname=' transcoder.red.net <http://transcoder.red.net> <http://transcoder.red.net> <http://transcoder.red.net> ';\ndelete from calls where (call"..., retries=1) at src/switch_core_sqldb.c:787 #13 0x00002aede189bf1c in switch_core_sql_thread (thread=<value optimized out>, obj=<value optimized out>) at src/switch_core_sqldb.c:1026 #14 0x0000003c03c0673d in start_thread () from /lib64/libpthread.so.0 #15 0x0000003c030d3f6d in clone () from /lib64/libc.so.6 *De:* Ricardo Martinez [mailto:rmartinez@redvoiss.net<rmartinez@redvoiss.net>] *Enviado el:* lunes, 22 de octubre de 2012 15:50 *Para:* 'FreeSWITCH Users Help' *Asunto:* RE: [Freeswitch-users] CPU Spikes every 20 minutes AS far as i can see there is no java running. This is the top at the moment of the spike : top - 15:48:29 up 17 days, 14:55, 3 users, load average: 14.07, 8.87, 6.48 Tasks: 158 total, 1 running, 157 sleeping, 0 stopped, 0 zombie Cpu(s): 12.6%us, 9.0%sy, 0.0%ni, 66.1%id, 2.7%wa, 0.3%hi, 9.2%si, 0.0%st Mem: 8171500k total, 8041556k used, 129944k free, 177688k buffers Swap: 10223608k total, 0k used, 10223608k free, 6951988k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31772 root 15 0 870m 481m 6880 S 212.4 6.0 9523:51 freeswitch 4210 root 19 0 36344 2940 1444 S 1.0 0.0 56:21.08 sngtc_server 29586 root 15 0 12740 1132 808 R 0.3 0.0 0:00.02 top 1 root 15 0 10348 704 588 S 0.0 0.0 0:01.13 init 2 root RT -5 0 0 0 S 0.0 0.0 0:00.41 migration/0 3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0 4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0 5 root RT -5 0 0 0 S 0.0 0.0 0:00.83 migration/1 6 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1 7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1 8 root RT -5 0 0 0 S 0.0 0.0 0:00.14 migration/2 9 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/2 10 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/2 11 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/3 12 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/3 13 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/3 14 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/4 15 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/4 16 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/4 17 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/5 18 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/5 19 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/5 20 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/6 Regards, Ricardo.- *De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Guillermo Ruiz Camauer *Enviado el:* lunes, 22 de octubre de 2012 15:38 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes Do you have anything like Java running that could be doing garbage collection? What other processes are running on the machine? Does TOP show which process is causing the spike? Guillermo On Mon, Oct 22, 2012 at 3:29 PM, Ricardo Martinez <rmartinez@redvoiss.net> wrote: Hi Gabe. I'm mean between the spikes there is nothing different as regarding to the calls... the cps maintains the same.. there are no scripts running or crontabs running, the simultaneous calls are the same also. So, for some reason the "load average" start to grow (every 20 or 25 minutes) without any clear cause. I have debug with gdb the moment the server is with 12.5 of "load average" here : http://pastebin.freeswitch.org/20112 Can we detect a problem from here? Regards, Ricardo. -----Mensaje original----- De: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] En nombre de Gabriel Gunderson Enviado el: lunes, 22 de octubre de 2012 13:31 Para: FreeSWITCH Users Help Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes On Mon, Oct 22, 2012 at 8:37 AM, Ricardo Martinez <rmartinez@redvoiss.net> wrote:
The problem is that I'm having weird spikes of "load average" cpu every 20 minutes (aprox), without any reason, reaching near of 12.5 of load average for this machine.
What have you done to verify that it happens 'without any reason'? Gabe _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Ken *http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org *irc.freenode.net #freeswitch
Keeping in mind that you are using a sangoma transcoder board, that means you are handling the equivalent of 440 concurrent calls worth of media for people not using sangoma transcoding hardware (yes theres 4 call legs in that scenario, 1 from endpoint A to FS, 1 from FS to D100, 1 From D100 back to FS, 1 from FS back out to the other end point). That doesn¹t sound like that big of a problem... You will have a fair bit of network traffic that way (the D100 shows up as another network interface as you are probably aware)... I would be looking at something like iotop or something else to see where that load is coming from Also, you have plenty of CPU free... So there is either a real problem here, or this is one of those things where the number of threads handling RTP and SIP are hanging around the run queue long enough to show up in the load average... On 10/22/12 3:53 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote:
Ken. There is an average of 220 concurrent calls. Machine : DL380 G7 2 Quad Core 8 Gig RAM. Centos 5.5 64bits Ricarod.-
De: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] En nombre de Ken Rice Enviado el: lunes, 22 de octubre de 2012 17:47 Para: FreeSWITCH Users Help Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes How many concurrent calls are you doing on that box?
On 10/22/12 3:33 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: Hi Ken. I mounted the db directory in RAM as you mention, but the ³load average² still is high. You can see this here : Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 132G 8.6G 117G 7% / proc 0 0 0 - /proc sysfs 0 0 0 - /sys devpts 0 0 0 - /dev/pts /dev/cciss/c0d0p1 104M 20M 79M 21% /boot tmpfs 4.2G 0 4.2G 0% /dev/shm none 0 0 0 - /proc/sys/fs/binfmt_misc sunrpc 0 0 0 - /var/lib/nfs/rpc_pipefs tmpfs 4.2G 1.3M 4.2G 1% /usr/local/freeswitch/db This is a resume from uptime and total calls for my freeswitch : 17:20:07 up 17 days, Total Calls : 16:26, 3 users, load average: 4.77, 4.50, 4.66 , Total Calls : 209, Total sessions: 440 17:20:12 up 17 days, Total Calls : 16:27, 3 users, load average: 6.79, 4.92, 4.80 , Total Calls : 245, Total sessions: 510 17:20:17 up 17 days, Total Calls : 16:27, 3 users, load average: 6.25, 4.84, 4.77 , Total Calls : 235, Total sessions: 488 17:20:22 up 17 days, Total Calls : 16:27, 3 users, load average: 6.63, 4.94, 4.81 , Total Calls : 233, Total sessions: 484 17:20:27 up 17 days, Total Calls : 16:27, 3 users, load average: 6.42, 4.92, 4.80 , Total Calls : 226, Total sessions: 470 17:20:32 up 17 days, Total Calls : 16:27, 3 users, load average: 5.90, 4.84, 4.78 , Total Calls : 218, Total sessions: 454 17:20:37 up 17 days, Total Calls : 16:27, 3 users, load average: 5.43, 4.76, 4.75 , Total Calls : 198, Total sessions: 416 17:20:43 up 17 days, Total Calls : 16:27, 3 users, load average: 4.99, 4.68, 4.72 , Total Calls : 183, Total sessions: 384 17:20:48 up 17 days, Total Calls : 16:27, 3 users, load average: 4.59, 4.60, 4.70 , Total Calls : 172, Total sessions: 366 17:20:53 up 17 days, Total Calls : 16:27, 3 users, load average: 4.31, 4.54, 4.68 , Total Calls : 184, Total sessions: 388 17:20:58 up 17 days, Total Calls : 16:27, 3 users, load average: 3.96, 4.47, 4.65 , Total Calls : 207, Total sessions: 442 17:21:03 up 17 days, Total Calls : 16:27, 3 users, load average: 4.04, 4.48, 4.65 , Total Calls : 203, Total sessions: 436 17:21:08 up 17 days, Total Calls : 16:27, 3 users, load average: 3.72, 4.40, 4.63 , Total Calls : 198, Total sessions: 418 17:21:13 up 17 days, Total Calls : 16:28, 3 users, load average: 3.47, 4.32, 4.60 , Total Calls : 198, Total sessions: 422 17:21:18 up 17 days, Total Calls : 16:28, 3 users, load average: 4.23, 4.47, 4.65 , Total Calls : 194, Total sessions: 410 17:21:23 up 17 days, Total Calls : 16:28, 3 users, load average: 8.30, 5.31, 4.92 , Total Calls : 211, Total sessions: 450 17:21:28 up 17 days, Total Calls : 16:28, 3 users, load average: 8.11, 5.32, 4.92 , Total Calls : 251, Total sessions: 536 17:21:33 up 17 days, Total Calls : 16:28, 3 users, load average: 8.74, 5.49, 4.98 , Total Calls : 250, Total sessions: 530 17:21:38 up 17 days, Total Calls : 16:28, 3 users, load average: 9.48, 5.70, 5.05 , Total Calls : 236, Total sessions: 496 17:21:43 up 17 days, Total Calls : 16:28, 3 users, load average: 9.45, 5.76, 5.07 , Total Calls : 233, Total sessions: 492 17:21:48 up 17 days, Total Calls : 16:28, 3 users, load average: 9.57, 5.84, 5.10 , Total Calls : 235, Total sessions: 496 17:21:53 up 17 days, Total Calls : 16:28, 3 users, load average: 10.24, 6.04, 5.17 , Total Calls : 222, Total sessions: 466 17:21:58 up 17 days, Total Calls : 16:28, 3 users, load average: 10.46, 6.16, 5.21 , Total Calls : 215, Total sessions: 452 17:22:03 up 17 days, Total Calls : 16:28, 3 users, load average: 11.07, 6.36, 5.28 , Total Calls : 208, Total sessions: 430 17:22:08 up 17 days, Total Calls : 16:28, 3 users, load average: 11.22, 6.47, 5.32 , Total Calls : 202, Total sessions: 422 17:22:14 up 17 days, Total Calls : 16:29, 3 users, load average: 11.52, 6.61, 5.37 , Total Calls : 240, Total sessions: 500 17:22:19 up 17 days, Total Calls : 16:29, 3 users, load average: 11.96, 6.78, 5.44 , Total Calls : 239, Total sessions: 498 17:22:24 up 17 days, Total Calls : 16:29, 3 users, load average: 14.29, 7.35, 5.63 , Total Calls : 250, Total sessions: 516 17:22:29 up 17 days, Total Calls : 16:29, 3 users, load average: 13.78, 7.36, 5.64 , Total Calls : 236, Total sessions: 490 17:22:34 up 17 days, Total Calls : 16:29, 3 users, load average: 12.92, 7.29, 5.63 , Total Calls : 209, Total sessions: 436 17:22:39 up 17 days, Total Calls : 16:29, 3 users, load average: 11.97, 7.18, 5.60 , Total Calls : 180, Total sessions: 372 17:22:44 up 17 days, Total Calls : 16:29, 3 users, load average: 11.01, 7.06, 5.57 , Total Calls : 161, Total sessions: 342 17:22:49 up 17 days, Total Calls : 16:29, 3 users, load average: 11.57, 7.24, 5.64 , Total Calls : 175, Total sessions: 370 17:22:54 up 17 days, Total Calls : 16:29, 3 users, load average: 11.12, 7.22, 5.64 , Total Calls : 181, Total sessions: 382 17:22:59 up 17 days, Total Calls : 16:29, 3 users, load average: 11.19, 7.30, 5.67 , Total Calls : 178, Total sessions: 366 17:23:04 up 17 days, Total Calls : 16:29, 3 users, load average: 11.10, 7.35, 5.69 , Total Calls : 173, Total sessions: 358 17:23:09 up 17 days, Total Calls : 16:30, 3 users, load average: 11.57, 7.51, 5.76 , Total Calls : 182, Total sessions: 378 17:23:14 up 17 days, Total Calls : 16:30, 3 users, load average: 11.36, 7.53, 5.77 , Total Calls : 187, Total sessions: 386 17:23:19 up 17 days, Total Calls : 16:30, 3 users, load average: 12.21, 7.77, 5.86 , Total Calls : 190, Total sessions: 402 17:23:24 up 17 days, Total Calls : 16:30, 3 users, load average: 12.04, 7.81, 5.88 , Total Calls : 239, Total sessions: 496 17:23:29 up 17 days, Total Calls : 16:30, 3 users, load average: 12.27, 7.93, 5.93 , Total Calls : 257, Total sessions: 532 17:23:34 up 17 days, Total Calls : 16:30, 3 users, load average: 12.73, 8.09, 5.99 , Total Calls : 242, Total sessions: 506 17:23:39 up 17 days, Total Calls : 16:30, 3 users, load average: 12.67, 8.16, 6.03 , Total Calls : 233, Total sessions: 482 17:23:45 up 17 days, Total Calls : 16:30, 3 users, load average: 12.38, 8.17, 6.04 , Total Calls : 238, Total sessions: 492 17:23:50 up 17 days, Total Calls : 16:30, 3 users, load average: 12.67, 8.30, 6.10 , Total Calls : 236, Total sessions: 482 17:23:55 up 17 days, Total Calls : 16:30, 3 users, load average: 13.09, 8.46, 6.16 , Total Calls : 222, Total sessions: 458 17:24:00 up 17 days, Total Calls : 16:30, 3 users, load average: 12.37, 8.39, 6.15 , Total Calls : 221, Total sessions: 470 17:24:05 up 17 days, Total Calls : 16:30, 3 users, load average: 11.78, 8.33, 6.14 , Total Calls : 200, Total sessions: 422 17:24:10 up 17 days, Total Calls : 16:31, 3 users, load average: 13.64, 8.77, 6.30 , Total Calls : 214, Total sessions: 452 17:24:15 up 17 days, Total Calls : 16:31, 3 users, load average: 13.98, 8.93, 6.36 , Total Calls : 255, Total sessions: 536 17:24:20 up 17 days, Total Calls : 16:31, 3 users, load average: 14.23, 9.06, 6.42 , Total Calls : 252, Total sessions: 524 17:24:25 up 17 days, Total Calls : 16:31, 3 users, load average: 15.73, 9.46, 6.56 , Total Calls : 260, Total sessions: 536 17:24:30 up 17 days, Total Calls : 16:31, 3 users, load average: 15.03, 9.42, 6.56 , Total Calls : 246, Total sessions: 508 17:24:35 up 17 days, Total Calls : 16:31, 3 users, load average: 15.11, 9.52, 6.61 , Total Calls : 230, Total sessions: 478 17:24:40 up 17 days, Total Calls : 16:31, 3 users, load average: 14.94, 9.58, 6.64 , Total Calls : 194, Total sessions: 408 17:24:45 up 17 days, Total Calls : 16:31, 3 users, load average: 14.54, 9.59, 6.66 , Total Calls : 189, Total sessions: 396 17:24:50 up 17 days, Total Calls : 16:31, 3 users, load average: 13.94, 9.55, 6.66 , Total Calls : 202, Total sessions: 438 17:24:55 up 17 days, Total Calls : 16:31, 3 users, load average: 13.30, 9.49, 6.66 , Total Calls : 221, Total sessions: 470 17:25:00 up 17 days, Total Calls : 16:31, 3 users, load average: 12.88, 9.46, 6.67 , Total Calls : 220, Total sessions: 478 17:25:05 up 17 days, Total Calls : 16:31, 3 users, load average: 14.25, 9.80, 6.79 , Total Calls : 219, Total sessions: 480 17:25:10 up 17 days, Total Calls : 16:32, 3 users, load average: 13.67, 9.76, 6.79 , Total Calls : 222, Total sessions: 490 17:25:16 up 17 days, Total Calls : 16:32, 3 users, load average: 13.05, 9.69, 6.79 , Total Calls : 214, Total sessions: 470 17:25:21 up 17 days, Total Calls : 16:32, 3 users, load average: 13.53, 9.85, 6.85 , Total Calls : 210, Total sessions: 450 17:25:26 up 17 days, Total Calls : 16:32, 3 users, load average: 13.81, 9.97, 6.91 , Total Calls : 250, Total sessions: 524 17:25:31 up 17 days, Total Calls : 16:32, 3 users, load average: 12.70, 9.80, 6.87 , Total Calls : 258, Total sessions: 536 17:25:36 up 17 days, Total Calls : 16:32, 3 users, load average: 11.68, 9.64, 6.83 , Total Calls : 255, Total sessions: 528 17:25:41 up 17 days, Total Calls : 16:32, 3 users, load average: 10.83, 9.49, 6.80 , Total Calls : 246, Total sessions: 524 17:25:46 up 17 days, Total Calls : 16:32, 3 users, load average: 10.04, 9.35, 6.77 , Total Calls : 262, Total sessions: 564 17:25:51 up 17 days, Total Calls : 16:32, 3 users, load average: 9.32, 9.21, 6.74 , Total Calls : 238, Total sessions: 510 17:25:56 up 17 days, Total Calls : 16:32, 3 users, load average: 8.57, 9.06, 6.70 , Total Calls : 220, Total sessions: 476 17:26:01 up 17 days, Total Calls : 16:32, 3 users, load average: 7.88, 8.91, 6.67 , Total Calls : 228, Total sessions: 498 What else could be happening. Did I mention that I¹m using two transcoding cards from Sangoma? Regards, Ricardo.-
De: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] En nombre de Ken Rice Enviado el: lunes, 22 de octubre de 2012 16:50 Para: FreeSWITCH Users Help Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes Load Average is the number of threads/processes waiting in the kernel¹s run queue at sample time... Supposedly number of cores is the how much your load avg can get to before your boxes is exhausted but this isnt exactly true on modern hardware with highly multi-threaded software like FreeSWITCH...
Where did I get the 100+ number? From looking at top¹ or w¹ on a box to see what the current load avg is...
Sqlite is used all over FreeSWITCH from the coredb to mod_sofia and everything in between.... When you type show channels¹ from fs_cli it doesn¹t query the structures in RAM to show you that output, it pulls it from the sqlite db...
That being said, you can test quickly to see if sqlite is hammering the disk at some point by shifting freeswitch/db dir to a ram drive. Example code for linux such as Centos or Debian assuming freeswitch is installed to /usr/local/freeswitch
Shutdown freeswitch rm rf /usr/local/freeswitch/db/* mount t tmpfs tmpfs /usr/local/freeswitch/db Startup freeswitch
Once you have done that stick it back in rotation...
Note this will not affect anything but voicemail if you have voicemail running on that box and not using an external DB via mod_voicemail¹s DSN settings you¹ll lose the voicemail DB... (I would assume this is not the case since you are using this primarily as a Transcoder)
K
On 10/22/12 2:30 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: Hi Ken. Maybe that is the problem. I was sure that the ³load average² can¹t go more than #cores in your system. In my case I have 8 cores, so a ³load average² of 12 is high from my point of view. Where did you get the number 100 for this parameter? Anyway, what is running a sqlite in freeswitch. Does this have to do with the core.db file? Or with the cdr_sqlite module?. Thanks Ricardo.-
De: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] En nombre de Ken Rice Enviado el: lunes, 22 de octubre de 2012 16:18 Para: FreeSWITCH Users Help Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes Could it be just a normal sqlite running a vacuum?
Sqlite does this every so often or if its calling a sync....
Also freeswitch with a load avg of 12 for a short period of time isnt anything to really worry about. Due to the number of threads that¹s possible with FS (ie: 1 per call leg) its not unusual to see a box humming along and the load avg well above 100 in my experience... This is due to the number of threads waiting in the run queue when it was sampled... Now, that¹s not saying that this is not specifically indicative or a problem but something else to think about
On 10/22/12 1:52 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: This is the gdb from the proccess consuming more cpu : (gdb) thread 7 [Switching to thread 7 (Thread 31781)]#0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 (gdb) bt #0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 #1 0x00002aede19276de in unixSync (id=0x2aaaac726380, dataOnly=<value optimized out>) at ./src/os_unix.c:1229 #2 0x00002aede1951178 in syncJournal (pPager=0x1a1c200) at ./src/pager.c:2229 #3 0x00002aede1953b65 in sqlite3pager_sync (pPager=0x1a1c200, zMaster=0x0, nTrunc=0) at ./src/pager.c:3770 #4 0x00002aede193d07b in sqlite3BtreeSync (p=<value optimized out>, zMaster=0x0) at ./src/btree.c:6520 #5 0x00002aede1931f15 in sqlite3VdbeHalt (p=0x2aaaad17ac70) at ./src/vdbeaux.c:1062 #6 0x00002aede1962613 in sqlite3VdbeExec (p=0x2aaaad17ac70) at ./src/vdbe.c:2360 #7 0x00002aede193036c in sqlite3Step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:236 #8 sqlite3_step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:289 #9 0x00002aede1938978 in sqlite3_exec (db=0x1a1f070, zSql=0x2aede199e717 "COMMIT", xCallback=0, pArg=0x0, pzErrMsg=0x4178bed0) at ./src/legacy.c:78 #10 0x00002aede18ae0c1 in switch_core_db_exec (db=0x1a1f070, sql=0x2aede199e717 "COMMIT", callback=0, data=0x0, errmsg=0x4178bf38) at src/switch_core_db.c:93 #11 0x00002aede1897c18 in switch_cache_db_execute_sql_real (dbh=0x1a2dce0, sql=0x2aede199e717 "COMMIT", err=0x0) at src/switch_core_sqldb.c:445 #12 0x00002aede18985c3 in switch_cache_db_persistant_execute_trans (dbh=0x1a2dce0, sql=0x5f03bd0 "delete from calls where (caller_uuid='e64319c5-7f91-47a1-9561-42da291e499a' or callee_uuid='e64319c5-7f91-47a1-9561-42da291e499a') and hostname='transcoder.red.net <http://transcoder.red.net> <http://transcoder.red.net> <http://transcoder.red.net> <http://transcoder.red.net> ';\ndelete from calls where (call"..., retries=1) at src/switch_core_sqldb.c:787 #13 0x00002aede189bf1c in switch_core_sql_thread (thread=<value optimized out>, obj=<value optimized out>) at src/switch_core_sqldb.c:1026 #14 0x0000003c03c0673d in start_thread () from /lib64/libpthread.so.0 #15 0x0000003c030d3f6d in clone () from /lib64/libc.so.6
De: Ricardo Martinez [mailto:rmartinez@redvoiss.net] Enviado el: lunes, 22 de octubre de 2012 15:50 Para: 'FreeSWITCH Users Help' Asunto: RE: [Freeswitch-users] CPU Spikes every 20 minutes AS far as i can see there is no java running. This is the top at the moment of the spike : top - 15:48:29 up 17 days, 14:55, 3 users, load average: 14.07, 8.87, 6.48 Tasks: 158 total, 1 running, 157 sleeping, 0 stopped, 0 zombie Cpu(s): 12.6%us, 9.0%sy, 0.0%ni, 66.1%id, 2.7%wa, 0.3%hi, 9.2%si, 0.0%st Mem: 8171500k total, 8041556k used, 129944k free, 177688k buffers Swap: 10223608k total, 0k used, 10223608k free, 6951988k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31772 root 15 0 870m 481m 6880 S 212.4 6.0 9523:51 freeswitch 4210 root 19 0 36344 2940 1444 S 1.0 0.0 56:21.08 sngtc_server 29586 root 15 0 12740 1132 808 R 0.3 0.0 0:00.02 top 1 root 15 0 10348 704 588 S 0.0 0.0 0:01.13 init 2 root RT -5 0 0 0 S 0.0 0.0 0:00.41 migration/0 3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0 4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0 5 root RT -5 0 0 0 S 0.0 0.0 0:00.83 migration/1 6 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1 7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1 8 root RT -5 0 0 0 S 0.0 0.0 0:00.14 migration/2 9 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/2 10 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/2 11 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/3 12 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/3 13 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/3 14 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/4 15 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/4 16 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/4 17 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/5 18 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/5 19 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/5 20 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/6 Regards, Ricardo.- De: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] En nombre de Guillermo Ruiz Camauer Enviado el: lunes, 22 de octubre de 2012 15:38 Para: FreeSWITCH Users Help Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes Do you have anything like Java running that could be doing garbage collection?
What other processes are running on the machine? Does TOP show which process is causing the spike?
Guillermo
On Mon, Oct 22, 2012 at 3:29 PM, Ricardo Martinez <rmartinez@redvoiss.net> wrote: Hi Gabe. I'm mean between the spikes there is nothing different as regarding to the calls... the cps maintains the same.. there are no scripts running or crontabs running, the simultaneous calls are the same also. So, for some reason the "load average" start to grow (every 20 or 25 minutes) without any clear cause. I have debug with gdb the moment the server is with 12.5 of "load average" here : http://pastebin.freeswitch.org/20112
Can we detect a problem from here?
Regards, Ricardo.
-----Mensaje original----- De: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] En nombre de Gabriel Gunderson Enviado el: lunes, 22 de octubre de 2012 13:31 Para: FreeSWITCH Users Help Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes
On Mon, Oct 22, 2012 at 8:37 AM, Ricardo Martinez <rmartinez@redvoiss.net> wrote:
The problem is that I'm having weird spikes of "load average" cpu every 20 minutes (aprox), without any reason, reaching near of 12.5 of load average for this machine.
What have you done to verify that it happens 'without any reason'?
Gabe
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Ken http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org irc.freenode.net #freeswitch
iotop doesn't exist yet in in 5.5 Try cent 5.7 or cent 6.3 or debian 6 if you want some more modern kernels to get iotop On Mon, Oct 22, 2012 at 4:59 PM, Ken Rice <krice@freeswitch.org> wrote:
Keeping in mind that you are using a sangoma transcoder board, that means you are handling the equivalent of 440 concurrent calls worth of media for people not using sangoma transcoding hardware (yes theres 4 call legs in that scenario, 1 from endpoint A to FS, 1 from FS to D100, 1 From D100 back to FS, 1 from FS back out to the other end point). That doesn’t sound like that big of a problem... You will have a fair bit of network traffic that way (the D100 shows up as another network interface as you are probably aware)...
I would be looking at something like iotop or something else to see where that load is coming from
Also, you have plenty of CPU free... So there is either a real problem here, or this is one of those things where the number of threads handling RTP and SIP are hanging around the run queue long enough to show up in the load average...
On 10/22/12 3:53 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote:
Ken. There is an average of 220 concurrent calls. Machine : DL380 G7 – 2 Quad Core – 8 Gig RAM. Centos 5.5 64bits
Ricarod.-
*De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Ken Rice *Enviado el:* lunes, 22 de octubre de 2012 17:47 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes
How many concurrent calls are you doing on that box?
On 10/22/12 3:33 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: Hi Ken. I mounted the db directory in RAM as you mention, but the “load average” still is high. You can see this here :
Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 132G 8.6G 117G 7% / proc 0 0 0 - /proc sysfs 0 0 0 - /sys devpts 0 0 0 - /dev/pts /dev/cciss/c0d0p1 104M 20M 79M 21% /boot tmpfs 4.2G 0 4.2G 0% /dev/shm none 0 0 0 - /proc/sys/fs/binfmt_misc sunrpc 0 0 0 - /var/lib/nfs/rpc_pipefs *tmpfs 4.2G 1.3M 4.2G 1% /usr/local/freeswitch/db * This is a resume from uptime and total calls for my freeswitch :
17:20:07 up 17 days, Total Calls : 16:26, 3 users, load average: 4.77, 4.50, 4.66 , Total Calls : 209, Total sessions: 440 17:20:12 up 17 days, Total Calls : 16:27, 3 users, load average: 6.79, 4.92, 4.80 , Total Calls : 245, Total sessions: 510 17:20:17 up 17 days, Total Calls : 16:27, 3 users, load average: 6.25, 4.84, 4.77 , Total Calls : 235, Total sessions: 488 17:20:22 up 17 days, Total Calls : 16:27, 3 users, load average: 6.63, 4.94, 4.81 , Total Calls : 233, Total sessions: 484 17:20:27 up 17 days, Total Calls : 16:27, 3 users, load average: 6.42, 4.92, 4.80 , Total Calls : 226, Total sessions: 470 17:20:32 up 17 days, Total Calls : 16:27, 3 users, load average: 5.90, 4.84, 4.78 , Total Calls : 218, Total sessions: 454 17:20:37 up 17 days, Total Calls : 16:27, 3 users, load average: 5.43, 4.76, 4.75 , Total Calls : 198, Total sessions: 416 17:20:43 up 17 days, Total Calls : 16:27, 3 users, load average: 4.99, 4.68, 4.72 , Total Calls : 183, Total sessions: 384 17:20:48 up 17 days, Total Calls : 16:27, 3 users, load average: 4.59, 4.60, 4.70 , Total Calls : 172, Total sessions: 366 17:20:53 up 17 days, Total Calls : 16:27, 3 users, load average: 4.31, 4.54, 4.68 , Total Calls : 184, Total sessions: 388 17:20:58 up 17 days, Total Calls : 16:27, 3 users, load average: 3.96, 4.47, 4.65 , Total Calls : 207, Total sessions: 442 17:21:03 up 17 days, Total Calls : 16:27, 3 users, load average: 4.04, 4.48, 4.65 , Total Calls : 203, Total sessions: 436 17:21:08 up 17 days, Total Calls : 16:27, 3 users, load average: 3.72, 4.40, 4.63 , Total Calls : 198, Total sessions: 418 17:21:13 up 17 days, Total Calls : 16:28, 3 users, load average: 3.47, 4.32, 4.60 , Total Calls : 198, Total sessions: 422 17:21:18 up 17 days, Total Calls : 16:28, 3 users, load average: 4.23, 4.47, 4.65 , Total Calls : 194, Total sessions: 410 17:21:23 up 17 days, Total Calls : 16:28, 3 users, load average: 8.30, 5.31, 4.92 , Total Calls : 211, Total sessions: 450 17:21:28 up 17 days, Total Calls : 16:28, 3 users, load average: 8.11, 5.32, 4.92 , Total Calls : 251, Total sessions: 536 17:21:33 up 17 days, Total Calls : 16:28, 3 users, load average: 8.74, 5.49, 4.98 , Total Calls : 250, Total sessions: 530 17:21:38 up 17 days, Total Calls : 16:28, 3 users, load average: 9.48, 5.70, 5.05 , Total Calls : 236, Total sessions: 496 17:21:43 up 17 days, Total Calls : 16:28, 3 users, load average: 9.45, 5.76, 5.07 , Total Calls : 233, Total sessions: 492 17:21:48 up 17 days, Total Calls : 16:28, 3 users, load average: 9.57, 5.84, 5.10 , Total Calls : 235, Total sessions: 496 17:21:53 up 17 days, Total Calls : 16:28, 3 users, load average: 10.24, 6.04, 5.17 , Total Calls : 222, Total sessions: 466 17:21:58 up 17 days, Total Calls : 16:28, 3 users, load average: 10.46, 6.16, 5.21 , Total Calls : 215, Total sessions: 452 17:22:03 up 17 days, Total Calls : 16:28, 3 users, load average: 11.07, 6.36, 5.28 , Total Calls : 208, Total sessions: 430 17:22:08 up 17 days, Total Calls : 16:28, 3 users, load average: 11.22, 6.47, 5.32 , Total Calls : 202, Total sessions: 422 17:22:14 up 17 days, Total Calls : 16:29, 3 users, load average: 11.52, 6.61, 5.37 , Total Calls : 240, Total sessions: 500 17:22:19 up 17 days, Total Calls : 16:29, 3 users, load average: 11.96, 6.78, 5.44 , Total Calls : 239, Total sessions: 498 17:22:24 up 17 days, Total Calls : 16:29, 3 users, load average: 14.29, 7.35, 5.63 , Total Calls : 250, Total sessions: 516 17:22:29 up 17 days, Total Calls : 16:29, 3 users, load average: 13.78, 7.36, 5.64 , Total Calls : 236, Total sessions: 490 17:22:34 up 17 days, Total Calls : 16:29, 3 users, load average: 12.92, 7.29, 5.63 , Total Calls : 209, Total sessions: 436 17:22:39 up 17 days, Total Calls : 16:29, 3 users, load average: 11.97, 7.18, 5.60 , Total Calls : 180, Total sessions: 372 17:22:44 up 17 days, Total Calls : 16:29, 3 users, load average: 11.01, 7.06, 5.57 , Total Calls : 161, Total sessions: 342 17:22:49 up 17 days, Total Calls : 16:29, 3 users, load average: 11.57, 7.24, 5.64 , Total Calls : 175, Total sessions: 370 17:22:54 up 17 days, Total Calls : 16:29, 3 users, load average: 11.12, 7.22, 5.64 , Total Calls : 181, Total sessions: 382 17:22:59 up 17 days, Total Calls : 16:29, 3 users, load average: 11.19, 7.30, 5.67 , Total Calls : 178, Total sessions: 366 17:23:04 up 17 days, Total Calls : 16:29, 3 users, load average: 11.10, 7.35, 5.69 , Total Calls : 173, Total sessions: 358 17:23:09 up 17 days, Total Calls : 16:30, 3 users, load average: 11.57, 7.51, 5.76 , Total Calls : 182, Total sessions: 378 17:23:14 up 17 days, Total Calls : 16:30, 3 users, load average: 11.36, 7.53, 5.77 , Total Calls : 187, Total sessions: 386 17:23:19 up 17 days, Total Calls : 16:30, 3 users, load average: 12.21, 7.77, 5.86 , Total Calls : 190, Total sessions: 402 17:23:24 up 17 days, Total Calls : 16:30, 3 users, load average: 12.04, 7.81, 5.88 , Total Calls : 239, Total sessions: 496 17:23:29 up 17 days, Total Calls : 16:30, 3 users, load average: 12.27, 7.93, 5.93 , Total Calls : 257, Total sessions: 532 17:23:34 up 17 days, Total Calls : 16:30, 3 users, load average: 12.73, 8.09, 5.99 , Total Calls : 242, Total sessions: 506 17:23:39 up 17 days, Total Calls : 16:30, 3 users, load average: 12.67, 8.16, 6.03 , Total Calls : 233, Total sessions: 482 17:23:45 up 17 days, Total Calls : 16:30, 3 users, load average: 12.38, 8.17, 6.04 , Total Calls : 238, Total sessions: 492 17:23:50 up 17 days, Total Calls : 16:30, 3 users, load average: 12.67, 8.30, 6.10 , Total Calls : 236, Total sessions: 482 17:23:55 up 17 days, Total Calls : 16:30, 3 users, load average: 13.09, 8.46, 6.16 , Total Calls : 222, Total sessions: 458 17:24:00 up 17 days, Total Calls : 16:30, 3 users, load average: 12.37, 8.39, 6.15 , Total Calls : 221, Total sessions: 470 17:24:05 up 17 days, Total Calls : 16:30, 3 users, load average: 11.78, 8.33, 6.14 , Total Calls : 200, Total sessions: 422 17:24:10 up 17 days, Total Calls : 16:31, 3 users, load average: 13.64, 8.77, 6.30 , Total Calls : 214, Total sessions: 452 17:24:15 up 17 days, Total Calls : 16:31, 3 users, load average: 13.98, 8.93, 6.36 , Total Calls : 255, Total sessions: 536 17:24:20 up 17 days, Total Calls : 16:31, 3 users, load average: 14.23, 9.06, 6.42 , Total Calls : 252, Total sessions: 524 17:24:25 up 17 days, Total Calls : 16:31, 3 users, load average: 15.73, 9.46, 6.56 , Total Calls : 260, Total sessions: 536 17:24:30 up 17 days, Total Calls : 16:31, 3 users, load average: 15.03, 9.42, 6.56 , Total Calls : 246, Total sessions: 508 17:24:35 up 17 days, Total Calls : 16:31, 3 users, load average: 15.11, 9.52, 6.61 , Total Calls : 230, Total sessions: 478 17:24:40 up 17 days, Total Calls : 16:31, 3 users, load average: 14.94, 9.58, 6.64 , Total Calls : 194, Total sessions: 408 17:24:45 up 17 days, Total Calls : 16:31, 3 users, load average: 14.54, 9.59, 6.66 , Total Calls : 189, Total sessions: 396 17:24:50 up 17 days, Total Calls : 16:31, 3 users, load average: 13.94, 9.55, 6.66 , Total Calls : 202, Total sessions: 438 17:24:55 up 17 days, Total Calls : 16:31, 3 users, load average: 13.30, 9.49, 6.66 , Total Calls : 221, Total sessions: 470 17:25:00 up 17 days, Total Calls : 16:31, 3 users, load average: 12.88, 9.46, 6.67 , Total Calls : 220, Total sessions: 478 17:25:05 up 17 days, Total Calls : 16:31, 3 users, load average: 14.25, 9.80, 6.79 , Total Calls : 219, Total sessions: 480 17:25:10 up 17 days, Total Calls : 16:32, 3 users, load average: 13.67, 9.76, 6.79 , Total Calls : 222, Total sessions: 490 17:25:16 up 17 days, Total Calls : 16:32, 3 users, load average: 13.05, 9.69, 6.79 , Total Calls : 214, Total sessions: 470 17:25:21 up 17 days, Total Calls : 16:32, 3 users, load average: 13.53, 9.85, 6.85 , Total Calls : 210, Total sessions: 450 17:25:26 up 17 days, Total Calls : 16:32, 3 users, load average: 13.81, 9.97, 6.91 , Total Calls : 250, Total sessions: 524 17:25:31 up 17 days, Total Calls : 16:32, 3 users, load average: 12.70, 9.80, 6.87 , Total Calls : 258, Total sessions: 536 17:25:36 up 17 days, Total Calls : 16:32, 3 users, load average: 11.68, 9.64, 6.83 , Total Calls : 255, Total sessions: 528 17:25:41 up 17 days, Total Calls : 16:32, 3 users, load average: 10.83, 9.49, 6.80 , Total Calls : 246, Total sessions: 524 17:25:46 up 17 days, Total Calls : 16:32, 3 users, load average: 10.04, 9.35, 6.77 , Total Calls : 262, Total sessions: 564 17:25:51 up 17 days, Total Calls : 16:32, 3 users, load average: 9.32, 9.21, 6.74 , Total Calls : 238, Total sessions: 510 17:25:56 up 17 days, Total Calls : 16:32, 3 users, load average: 8.57, 9.06, 6.70 , Total Calls : 220, Total sessions: 476 17:26:01 up 17 days, Total Calls : 16:32, 3 users, load average: 7.88, 8.91, 6.67 , Total Calls : 228, Total sessions: 498
What else could be happening. Did I mention that I’m using two transcoding cards from Sangoma?
Regards, Ricardo.-
*De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Ken Rice *Enviado el:* lunes, 22 de octubre de 2012 16:50 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes
Load Average is the number of threads/processes waiting in the kernel’s run queue at sample time... Supposedly number of cores is the how much your load avg can get to before your boxes is exhausted but this isnt exactly true on modern hardware with highly multi-threaded software like FreeSWITCH...
Where did I get the 100+ number? From looking at ‘top’ or ‘w’ on a box to see what the current load avg is...
Sqlite is used all over FreeSWITCH from the coredb to mod_sofia and everything in between.... When you type ‘show channels’ from fs_cli it doesn’t query the structures in RAM to show you that output, it pulls it from the sqlite db...
That being said, you can test quickly to see if sqlite is hammering the disk at some point by shifting freeswitch/db dir to a ram drive. Example code for linux such as Centos or Debian assuming freeswitch is installed to /usr/local/freeswitch
Shutdown freeswitch rm –rf /usr/local/freeswitch/db/* mount –t tmpfs tmpfs /usr/local/freeswitch/db Startup freeswitch
Once you have done that stick it back in rotation...
Note this will not affect anything but voicemail if you have voicemail running on that box and not using an external DB via mod_voicemail’s DSN settings you’ll lose the voicemail DB... (I would assume this is not the case since you are using this primarily as a Transcoder)
K
On 10/22/12 2:30 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: Hi Ken. Maybe that is the problem. I was sure that the “load average” can’t go more than #cores in your system. In my case I have 8 cores, so a “load average” of 12 is high from my point of view. Where did you get the number 100 for this parameter? Anyway, what is running a sqlite in freeswitch. Does this have to do with the core.db file? Or with the cdr_sqlite module?.
Thanks Ricardo.-
*De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Ken Rice *Enviado el:* lunes, 22 de octubre de 2012 16:18 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes
Could it be just a normal sqlite running a vacuum?
Sqlite does this every so often or if its calling a sync....
Also freeswitch with a load avg of 12 for a short period of time isnt anything to really worry about. Due to the number of threads that’s possible with FS (ie: 1 per call leg) its not unusual to see a box humming along and the load avg well above 100 in my experience... This is due to the number of threads waiting in the run queue when it was sampled... Now, that’s not saying that this is not specifically indicative or a problem but something else to think about
On 10/22/12 1:52 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: This is the gdb from the proccess consuming more cpu :
(gdb) thread 7 [Switching to thread 7 (Thread 31781)]#0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 (gdb) bt #0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 #1 0x00002aede19276de in unixSync (id=0x2aaaac726380, dataOnly=<value optimized out>) at ./src/os_unix.c:1229 #2 0x00002aede1951178 in syncJournal (pPager=0x1a1c200) at ./src/pager.c:2229 #3 0x00002aede1953b65 in sqlite3pager_sync (pPager=0x1a1c200, zMaster=0x0, nTrunc=0) at ./src/pager.c:3770 #4 0x00002aede193d07b in sqlite3BtreeSync (p=<value optimized out>, zMaster=0x0) at ./src/btree.c:6520 #5 0x00002aede1931f15 in sqlite3VdbeHalt (p=0x2aaaad17ac70) at ./src/vdbeaux.c:1062 #6 0x00002aede1962613 in sqlite3VdbeExec (p=0x2aaaad17ac70) at ./src/vdbe.c:2360 #7 0x00002aede193036c in sqlite3Step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:236 #8 sqlite3_step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:289 #9 0x00002aede1938978 in sqlite3_exec (db=0x1a1f070, zSql=0x2aede199e717 "COMMIT", xCallback=0, pArg=0x0, pzErrMsg=0x4178bed0) at ./src/legacy.c:78 #10 0x00002aede18ae0c1 in switch_core_db_exec (db=0x1a1f070, sql=0x2aede199e717 "COMMIT", callback=0, data=0x0, errmsg=0x4178bf38) at src/switch_core_db.c:93 #11 0x00002aede1897c18 in switch_cache_db_execute_sql_real (dbh=0x1a2dce0, sql=0x2aede199e717 "COMMIT", err=0x0) at src/switch_core_sqldb.c:445 #12 0x00002aede18985c3 in switch_cache_db_persistant_execute_trans (dbh=0x1a2dce0, sql=0x5f03bd0 "delete from calls where (caller_uuid='e64319c5-7f91-47a1-9561-42da291e499a' or callee_uuid='e64319c5-7f91-47a1-9561-42da291e499a') and hostname=' transcoder.red.net <http://transcoder.red.net> <http://transcoder.red.net> <http://transcoder.red.net> <http://transcoder.red.net> ';\ndelete from calls where (call"..., retries=1)
at src/switch_core_sqldb.c:787 #13 0x00002aede189bf1c in switch_core_sql_thread (thread=<value optimized out>, obj=<value optimized out>) at src/switch_core_sqldb.c:1026 #14 0x0000003c03c0673d in start_thread () from /lib64/libpthread.so.0 #15 0x0000003c030d3f6d in clone () from /lib64/libc.so.6
*De:* Ricardo Martinez [mailto:rmartinez@redvoiss.net<rmartinez@redvoiss.net>]
*Enviado el:* lunes, 22 de octubre de 2012 15:50 *Para:* 'FreeSWITCH Users Help' *Asunto:* RE: [Freeswitch-users] CPU Spikes every 20 minutes
AS far as i can see there is no java running. This is the top at the moment of the spike :
top - 15:48:29 up 17 days, 14:55, 3 users, load average: 14.07, 8.87, 6.48 Tasks: 158 total, 1 running, 157 sleeping, 0 stopped, 0 zombie Cpu(s): 12.6%us, 9.0%sy, 0.0%ni, 66.1%id, 2.7%wa, 0.3%hi, 9.2%si, 0.0%st Mem: 8171500k total, 8041556k used, 129944k free, 177688k buffers Swap: 10223608k total, 0k used, 10223608k free, 6951988k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31772 root 15 0 870m 481m 6880 S 212.4 6.0 9523:51 freeswitch 4210 root 19 0 36344 2940 1444 S 1.0 0.0 56:21.08 sngtc_server 29586 root 15 0 12740 1132 808 R 0.3 0.0 0:00.02 top 1 root 15 0 10348 704 588 S 0.0 0.0 0:01.13 init 2 root RT -5 0 0 0 S 0.0 0.0 0:00.41 migration/0 3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0 4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0 5 root RT -5 0 0 0 S 0.0 0.0 0:00.83 migration/1 6 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1 7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
8 root RT -5 0 0 0 S 0.0 0.0 0:00.14 migration/2 9 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/2 10 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/2 11 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/3 12 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/3 13 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/3 14 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/4 15 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/4 16 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/4 17 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/5 18 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/5 19 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/5 20 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/6
Regards, Ricardo.-
*De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Guillermo Ruiz Camauer *Enviado el:* lunes, 22 de octubre de 2012 15:38 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes
Do you have anything like Java running that could be doing garbage collection?
What other processes are running on the machine? Does TOP show which process is causing the spike?
Guillermo
On Mon, Oct 22, 2012 at 3:29 PM, Ricardo Martinez <rmartinez@redvoiss.net> wrote: Hi Gabe. I'm mean between the spikes there is nothing different as regarding to the calls... the cps maintains the same.. there are no scripts running or crontabs running, the simultaneous calls are the same also. So, for some reason the "load average" start to grow (every 20 or 25 minutes) without any clear cause. I have debug with gdb the moment the server is with 12.5 of "load average" here : http://pastebin.freeswitch.org/20112
Can we detect a problem from here?
Regards, Ricardo.
-----Mensaje original----- De: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] En nombre de Gabriel Gunderson Enviado el: lunes, 22 de octubre de 2012 13:31 Para: FreeSWITCH Users Help Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes
On Mon, Oct 22, 2012 at 8:37 AM, Ricardo Martinez <rmartinez@redvoiss.net> wrote:
The problem is that I'm having weird spikes of "load average" cpu every 20 minutes (aprox), without any reason, reaching near of 12.5 of load average for this machine.
What have you done to verify that it happens 'without any reason'?
Gabe
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Ken *http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org *irc.freenode.net #freeswitch
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
Is there an equivalent command in Centos 5.5? Ricardo.- *De:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *En nombre de *Anthony Minessale *Enviado el:* lunes, 22 de octubre de 2012 19:37 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes iotop doesn't exist yet in in 5.5 Try cent 5.7 or cent 6.3 or debian 6 if you want some more modern kernels to get iotop On Mon, Oct 22, 2012 at 4:59 PM, Ken Rice <krice@freeswitch.org> wrote: Keeping in mind that you are using a sangoma transcoder board, that means you are handling the equivalent of 440 concurrent calls worth of media for people not using sangoma transcoding hardware (yes theres 4 call legs in that scenario, 1 from endpoint A to FS, 1 from FS to D100, 1 From D100 back to FS, 1 from FS back out to the other end point). That doesn’t sound like that big of a problem... You will have a fair bit of network traffic that way (the D100 shows up as another network interface as you are probably aware)... I would be looking at something like iotop or something else to see where that load is coming from Also, you have plenty of CPU free... So there is either a real problem here, or this is one of those things where the number of threads handling RTP and SIP are hanging around the run queue long enough to show up in the load average... On 10/22/12 3:53 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: Ken. There is an average of 220 concurrent calls. Machine : DL380 G7 – 2 Quad Core – 8 Gig RAM. Centos 5.5 64bits Ricarod.- *De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Ken Rice *Enviado el:* lunes, 22 de octubre de 2012 17:47 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes How many concurrent calls are you doing on that box? On 10/22/12 3:33 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: Hi Ken. I mounted the db directory in RAM as you mention, but the “load average” still is high. You can see this here : Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 132G 8.6G 117G 7% / proc 0 0 0 - /proc sysfs 0 0 0 - /sys devpts 0 0 0 - /dev/pts /dev/cciss/c0d0p1 104M 20M 79M 21% /boot tmpfs 4.2G 0 4.2G 0% /dev/shm none 0 0 0 - /proc/sys/fs/binfmt_misc sunrpc 0 0 0 - /var/lib/nfs/rpc_pipefs *tmpfs 4.2G 1.3M 4.2G 1% /usr/local/freeswitch/db * This is a resume from uptime and total calls for my freeswitch : 17:20:07 up 17 days, Total Calls : 16:26, 3 users, load average: 4.77, 4.50, 4.66 , Total Calls : 209, Total sessions: 440 17:20:12 up 17 days, Total Calls : 16:27, 3 users, load average: 6.79, 4.92, 4.80 , Total Calls : 245, Total sessions: 510 17:20:17 up 17 days, Total Calls : 16:27, 3 users, load average: 6.25, 4.84, 4.77 , Total Calls : 235, Total sessions: 488 17:20:22 up 17 days, Total Calls : 16:27, 3 users, load average: 6.63, 4.94, 4.81 , Total Calls : 233, Total sessions: 484 17:20:27 up 17 days, Total Calls : 16:27, 3 users, load average: 6.42, 4.92, 4.80 , Total Calls : 226, Total sessions: 470 17:20:32 up 17 days, Total Calls : 16:27, 3 users, load average: 5.90, 4.84, 4.78 , Total Calls : 218, Total sessions: 454 17:20:37 up 17 days, Total Calls : 16:27, 3 users, load average: 5.43, 4.76, 4.75 , Total Calls : 198, Total sessions: 416 17:20:43 up 17 days, Total Calls : 16:27, 3 users, load average: 4.99, 4.68, 4.72 , Total Calls : 183, Total sessions: 384 17:20:48 up 17 days, Total Calls : 16:27, 3 users, load average: 4.59, 4.60, 4.70 , Total Calls : 172, Total sessions: 366 17:20:53 up 17 days, Total Calls : 16:27, 3 users, load average: 4.31, 4.54, 4.68 , Total Calls : 184, Total sessions: 388 17:20:58 up 17 days, Total Calls : 16:27, 3 users, load average: 3.96, 4.47, 4.65 , Total Calls : 207, Total sessions: 442 17:21:03 up 17 days, Total Calls : 16:27, 3 users, load average: 4.04, 4.48, 4.65 , Total Calls : 203, Total sessions: 436 17:21:08 up 17 days, Total Calls : 16:27, 3 users, load average: 3.72, 4.40, 4.63 , Total Calls : 198, Total sessions: 418 17:21:13 up 17 days, Total Calls : 16:28, 3 users, load average: 3.47, 4.32, 4.60 , Total Calls : 198, Total sessions: 422 17:21:18 up 17 days, Total Calls : 16:28, 3 users, load average: 4.23, 4.47, 4.65 , Total Calls : 194, Total sessions: 410 17:21:23 up 17 days, Total Calls : 16:28, 3 users, load average: 8.30, 5.31, 4.92 , Total Calls : 211, Total sessions: 450 17:21:28 up 17 days, Total Calls : 16:28, 3 users, load average: 8.11, 5.32, 4.92 , Total Calls : 251, Total sessions: 536 17:21:33 up 17 days, Total Calls : 16:28, 3 users, load average: 8.74, 5.49, 4.98 , Total Calls : 250, Total sessions: 530 17:21:38 up 17 days, Total Calls : 16:28, 3 users, load average: 9.48, 5.70, 5.05 , Total Calls : 236, Total sessions: 496 17:21:43 up 17 days, Total Calls : 16:28, 3 users, load average: 9.45, 5.76, 5.07 , Total Calls : 233, Total sessions: 492 17:21:48 up 17 days, Total Calls : 16:28, 3 users, load average: 9.57, 5.84, 5.10 , Total Calls : 235, Total sessions: 496 17:21:53 up 17 days, Total Calls : 16:28, 3 users, load average: 10.24, 6.04, 5.17 , Total Calls : 222, Total sessions: 466 17:21:58 up 17 days, Total Calls : 16:28, 3 users, load average: 10.46, 6.16, 5.21 , Total Calls : 215, Total sessions: 452 17:22:03 up 17 days, Total Calls : 16:28, 3 users, load average: 11.07, 6.36, 5.28 , Total Calls : 208, Total sessions: 430 17:22:08 up 17 days, Total Calls : 16:28, 3 users, load average: 11.22, 6.47, 5.32 , Total Calls : 202, Total sessions: 422 17:22:14 up 17 days, Total Calls : 16:29, 3 users, load average: 11.52, 6.61, 5.37 , Total Calls : 240, Total sessions: 500 17:22:19 up 17 days, Total Calls : 16:29, 3 users, load average: 11.96, 6.78, 5.44 , Total Calls : 239, Total sessions: 498 17:22:24 up 17 days, Total Calls : 16:29, 3 users, load average: 14.29, 7.35, 5.63 , Total Calls : 250, Total sessions: 516 17:22:29 up 17 days, Total Calls : 16:29, 3 users, load average: 13.78, 7.36, 5.64 , Total Calls : 236, Total sessions: 490 17:22:34 up 17 days, Total Calls : 16:29, 3 users, load average: 12.92, 7.29, 5.63 , Total Calls : 209, Total sessions: 436 17:22:39 up 17 days, Total Calls : 16:29, 3 users, load average: 11.97, 7.18, 5.60 , Total Calls : 180, Total sessions: 372 17:22:44 up 17 days, Total Calls : 16:29, 3 users, load average: 11.01, 7.06, 5.57 , Total Calls : 161, Total sessions: 342 17:22:49 up 17 days, Total Calls : 16:29, 3 users, load average: 11.57, 7.24, 5.64 , Total Calls : 175, Total sessions: 370 17:22:54 up 17 days, Total Calls : 16:29, 3 users, load average: 11.12, 7.22, 5.64 , Total Calls : 181, Total sessions: 382 17:22:59 up 17 days, Total Calls : 16:29, 3 users, load average: 11.19, 7.30, 5.67 , Total Calls : 178, Total sessions: 366 17:23:04 up 17 days, Total Calls : 16:29, 3 users, load average: 11.10, 7.35, 5.69 , Total Calls : 173, Total sessions: 358 17:23:09 up 17 days, Total Calls : 16:30, 3 users, load average: 11.57, 7.51, 5.76 , Total Calls : 182, Total sessions: 378 17:23:14 up 17 days, Total Calls : 16:30, 3 users, load average: 11.36, 7.53, 5.77 , Total Calls : 187, Total sessions: 386 17:23:19 up 17 days, Total Calls : 16:30, 3 users, load average: 12.21, 7.77, 5.86 , Total Calls : 190, Total sessions: 402 17:23:24 up 17 days, Total Calls : 16:30, 3 users, load average: 12.04, 7.81, 5.88 , Total Calls : 239, Total sessions: 496 17:23:29 up 17 days, Total Calls : 16:30, 3 users, load average: 12.27, 7.93, 5.93 , Total Calls : 257, Total sessions: 532 17:23:34 up 17 days, Total Calls : 16:30, 3 users, load average: 12.73, 8.09, 5.99 , Total Calls : 242, Total sessions: 506 17:23:39 up 17 days, Total Calls : 16:30, 3 users, load average: 12.67, 8.16, 6.03 , Total Calls : 233, Total sessions: 482 17:23:45 up 17 days, Total Calls : 16:30, 3 users, load average: 12.38, 8.17, 6.04 , Total Calls : 238, Total sessions: 492 17:23:50 up 17 days, Total Calls : 16:30, 3 users, load average: 12.67, 8.30, 6.10 , Total Calls : 236, Total sessions: 482 17:23:55 up 17 days, Total Calls : 16:30, 3 users, load average: 13.09, 8.46, 6.16 , Total Calls : 222, Total sessions: 458 17:24:00 up 17 days, Total Calls : 16:30, 3 users, load average: 12.37, 8.39, 6.15 , Total Calls : 221, Total sessions: 470 17:24:05 up 17 days, Total Calls : 16:30, 3 users, load average: 11.78, 8.33, 6.14 , Total Calls : 200, Total sessions: 422 17:24:10 up 17 days, Total Calls : 16:31, 3 users, load average: 13.64, 8.77, 6.30 , Total Calls : 214, Total sessions: 452 17:24:15 up 17 days, Total Calls : 16:31, 3 users, load average: 13.98, 8.93, 6.36 , Total Calls : 255, Total sessions: 536 17:24:20 up 17 days, Total Calls : 16:31, 3 users, load average: 14.23, 9.06, 6.42 , Total Calls : 252, Total sessions: 524 17:24:25 up 17 days, Total Calls : 16:31, 3 users, load average: 15.73, 9.46, 6.56 , Total Calls : 260, Total sessions: 536 17:24:30 up 17 days, Total Calls : 16:31, 3 users, load average: 15.03, 9.42, 6.56 , Total Calls : 246, Total sessions: 508 17:24:35 up 17 days, Total Calls : 16:31, 3 users, load average: 15.11, 9.52, 6.61 , Total Calls : 230, Total sessions: 478 17:24:40 up 17 days, Total Calls : 16:31, 3 users, load average: 14.94, 9.58, 6.64 , Total Calls : 194, Total sessions: 408 17:24:45 up 17 days, Total Calls : 16:31, 3 users, load average: 14.54, 9.59, 6.66 , Total Calls : 189, Total sessions: 396 17:24:50 up 17 days, Total Calls : 16:31, 3 users, load average: 13.94, 9.55, 6.66 , Total Calls : 202, Total sessions: 438 17:24:55 up 17 days, Total Calls : 16:31, 3 users, load average: 13.30, 9.49, 6.66 , Total Calls : 221, Total sessions: 470 17:25:00 up 17 days, Total Calls : 16:31, 3 users, load average: 12.88, 9.46, 6.67 , Total Calls : 220, Total sessions: 478 17:25:05 up 17 days, Total Calls : 16:31, 3 users, load average: 14.25, 9.80, 6.79 , Total Calls : 219, Total sessions: 480 17:25:10 up 17 days, Total Calls : 16:32, 3 users, load average: 13.67, 9.76, 6.79 , Total Calls : 222, Total sessions: 490 17:25:16 up 17 days, Total Calls : 16:32, 3 users, load average: 13.05, 9.69, 6.79 , Total Calls : 214, Total sessions: 470 17:25:21 up 17 days, Total Calls : 16:32, 3 users, load average: 13.53, 9.85, 6.85 , Total Calls : 210, Total sessions: 450 17:25:26 up 17 days, Total Calls : 16:32, 3 users, load average: 13.81, 9.97, 6.91 , Total Calls : 250, Total sessions: 524 17:25:31 up 17 days, Total Calls : 16:32, 3 users, load average: 12.70, 9.80, 6.87 , Total Calls : 258, Total sessions: 536 17:25:36 up 17 days, Total Calls : 16:32, 3 users, load average: 11.68, 9.64, 6.83 , Total Calls : 255, Total sessions: 528 17:25:41 up 17 days, Total Calls : 16:32, 3 users, load average: 10.83, 9.49, 6.80 , Total Calls : 246, Total sessions: 524 17:25:46 up 17 days, Total Calls : 16:32, 3 users, load average: 10.04, 9.35, 6.77 , Total Calls : 262, Total sessions: 564 17:25:51 up 17 days, Total Calls : 16:32, 3 users, load average: 9.32, 9.21, 6.74 , Total Calls : 238, Total sessions: 510 17:25:56 up 17 days, Total Calls : 16:32, 3 users, load average: 8.57, 9.06, 6.70 , Total Calls : 220, Total sessions: 476 17:26:01 up 17 days, Total Calls : 16:32, 3 users, load average: 7.88, 8.91, 6.67 , Total Calls : 228, Total sessions: 498 What else could be happening. Did I mention that I’m using two transcoding cards from Sangoma? Regards, Ricardo.- *De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Ken Rice *Enviado el:* lunes, 22 de octubre de 2012 16:50 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes Load Average is the number of threads/processes waiting in the kernel’s run queue at sample time... Supposedly number of cores is the how much your load avg can get to before your boxes is exhausted but this isnt exactly true on modern hardware with highly multi-threaded software like FreeSWITCH... Where did I get the 100+ number? From looking at ‘top’ or ‘w’ on a box to see what the current load avg is... Sqlite is used all over FreeSWITCH from the coredb to mod_sofia and everything in between.... When you type ‘show channels’ from fs_cli it doesn’t query the structures in RAM to show you that output, it pulls it from the sqlite db... That being said, you can test quickly to see if sqlite is hammering the disk at some point by shifting freeswitch/db dir to a ram drive. Example code for linux such as Centos or Debian assuming freeswitch is installed to /usr/local/freeswitch Shutdown freeswitch rm –rf /usr/local/freeswitch/db/* mount –t tmpfs tmpfs /usr/local/freeswitch/db Startup freeswitch Once you have done that stick it back in rotation... Note this will not affect anything but voicemail if you have voicemail running on that box and not using an external DB via mod_voicemail’s DSN settings you’ll lose the voicemail DB... (I would assume this is not the case since you are using this primarily as a Transcoder) K On 10/22/12 2:30 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: Hi Ken. Maybe that is the problem. I was sure that the “load average” can’t go more than #cores in your system. In my case I have 8 cores, so a “load average” of 12 is high from my point of view. Where did you get the number 100 for this parameter? Anyway, what is running a sqlite in freeswitch. Does this have to do with the core.db file? Or with the cdr_sqlite module?. Thanks Ricardo.- *De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Ken Rice *Enviado el:* lunes, 22 de octubre de 2012 16:18 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes Could it be just a normal sqlite running a vacuum? Sqlite does this every so often or if its calling a sync.... Also freeswitch with a load avg of 12 for a short period of time isnt anything to really worry about. Due to the number of threads that’s possible with FS (ie: 1 per call leg) its not unusual to see a box humming along and the load avg well above 100 in my experience... This is due to the number of threads waiting in the run queue when it was sampled... Now, that’s not saying that this is not specifically indicative or a problem but something else to think about On 10/22/12 1:52 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: This is the gdb from the proccess consuming more cpu : (gdb) thread 7 [Switching to thread 7 (Thread 31781)]#0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 (gdb) bt #0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 #1 0x00002aede19276de in unixSync (id=0x2aaaac726380, dataOnly=<value optimized out>) at ./src/os_unix.c:1229 #2 0x00002aede1951178 in syncJournal (pPager=0x1a1c200) at ./src/pager.c:2229 #3 0x00002aede1953b65 in sqlite3pager_sync (pPager=0x1a1c200, zMaster=0x0, nTrunc=0) at ./src/pager.c:3770 #4 0x00002aede193d07b in sqlite3BtreeSync (p=<value optimized out>, zMaster=0x0) at ./src/btree.c:6520 #5 0x00002aede1931f15 in sqlite3VdbeHalt (p=0x2aaaad17ac70) at ./src/vdbeaux.c:1062 #6 0x00002aede1962613 in sqlite3VdbeExec (p=0x2aaaad17ac70) at ./src/vdbe.c:2360 #7 0x00002aede193036c in sqlite3Step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:236 #8 sqlite3_step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:289 #9 0x00002aede1938978 in sqlite3_exec (db=0x1a1f070, zSql=0x2aede199e717 "COMMIT", xCallback=0, pArg=0x0, pzErrMsg=0x4178bed0) at ./src/legacy.c:78 #10 0x00002aede18ae0c1 in switch_core_db_exec (db=0x1a1f070, sql=0x2aede199e717 "COMMIT", callback=0, data=0x0, errmsg=0x4178bf38) at src/switch_core_db.c:93 #11 0x00002aede1897c18 in switch_cache_db_execute_sql_real (dbh=0x1a2dce0, sql=0x2aede199e717 "COMMIT", err=0x0) at src/switch_core_sqldb.c:445 #12 0x00002aede18985c3 in switch_cache_db_persistant_execute_trans (dbh=0x1a2dce0, sql=0x5f03bd0 "delete from calls where (caller_uuid='e64319c5-7f91-47a1-9561-42da291e499a' or callee_uuid='e64319c5-7f91-47a1-9561-42da291e499a') and hostname=' transcoder.red.net <http://transcoder.red.net> <http://transcoder.red.net> <http://transcoder.red.net> <http://transcoder.red.net> ';\ndelete from calls where (call"..., retries=1) at src/switch_core_sqldb.c:787 #13 0x00002aede189bf1c in switch_core_sql_thread (thread=<value optimized out>, obj=<value optimized out>) at src/switch_core_sqldb.c:1026 #14 0x0000003c03c0673d in start_thread () from /lib64/libpthread.so.0 #15 0x0000003c030d3f6d in clone () from /lib64/libc.so.6 *De:* Ricardo Martinez [mailto:rmartinez@redvoiss.net<rmartinez@redvoiss.net>] *Enviado el:* lunes, 22 de octubre de 2012 15:50 *Para:* 'FreeSWITCH Users Help' *Asunto:* RE: [Freeswitch-users] CPU Spikes every 20 minutes AS far as i can see there is no java running. This is the top at the moment of the spike : top - 15:48:29 up 17 days, 14:55, 3 users, load average: 14.07, 8.87, 6.48 Tasks: 158 total, 1 running, 157 sleeping, 0 stopped, 0 zombie Cpu(s): 12.6%us, 9.0%sy, 0.0%ni, 66.1%id, 2.7%wa, 0.3%hi, 9.2%si, 0.0%st Mem: 8171500k total, 8041556k used, 129944k free, 177688k buffers Swap: 10223608k total, 0k used, 10223608k free, 6951988k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31772 root 15 0 870m 481m 6880 S 212.4 6.0 9523:51 freeswitch 4210 root 19 0 36344 2940 1444 S 1.0 0.0 56:21.08 sngtc_server 29586 root 15 0 12740 1132 808 R 0.3 0.0 0:00.02 top 1 root 15 0 10348 704 588 S 0.0 0.0 0:01.13 init 2 root RT -5 0 0 0 S 0.0 0.0 0:00.41 migration/0 3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0 4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0 5 root RT -5 0 0 0 S 0.0 0.0 0:00.83 migration/1 6 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1 7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1 8 root RT -5 0 0 0 S 0.0 0.0 0:00.14 migration/2 9 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/2 10 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/2 11 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/3 12 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/3 13 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/3 14 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/4 15 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/4 16 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/4 17 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/5 18 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/5 19 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/5 20 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/6 Regards, Ricardo.- *De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Guillermo Ruiz Camauer *Enviado el:* lunes, 22 de octubre de 2012 15:38 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes Do you have anything like Java running that could be doing garbage collection? What other processes are running on the machine? Does TOP show which process is causing the spike? Guillermo On Mon, Oct 22, 2012 at 3:29 PM, Ricardo Martinez <rmartinez@redvoiss.net> wrote: Hi Gabe. I'm mean between the spikes there is nothing different as regarding to the calls... the cps maintains the same.. there are no scripts running or crontabs running, the simultaneous calls are the same also. So, for some reason the "load average" start to grow (every 20 or 25 minutes) without any clear cause. I have debug with gdb the moment the server is with 12.5 of "load average" here : http://pastebin.freeswitch.org/20112 Can we detect a problem from here? Regards, Ricardo. -----Mensaje original----- De: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] En nombre de Gabriel Gunderson Enviado el: lunes, 22 de octubre de 2012 13:31 Para: FreeSWITCH Users Help Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes On Mon, Oct 22, 2012 at 8:37 AM, Ricardo Martinez <rmartinez@redvoiss.net> wrote:
The problem is that I'm having weird spikes of "load average" cpu every 20 minutes (aprox), without any reason, reaching near of 12.5 of load average for this machine.
What have you done to verify that it happens 'without any reason'? Gabe _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Ken *http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org *irc.freenode.net #freeswitch _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
No, If you have these kinds of problems my only advice is what I already said in my last post about newer OS. You can try top -H -p <current pid of FS process> watch for any high cpu usage in a particular thread, gcore it and post a bt. On Tue, Oct 23, 2012 at 8:03 AM, Ricardo Martinez <rmartinez@redvoiss.net>wrote:
Is there an equivalent command in Centos 5.5?
Ricardo.-
*De:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *En nombre de *Anthony Minessale *Enviado el:* lunes, 22 de octubre de 2012 19:37
*Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes
iotop doesn't exist yet in in 5.5
Try cent 5.7
or cent 6.3 or debian 6 if you want some more modern kernels to get iotop
On Mon, Oct 22, 2012 at 4:59 PM, Ken Rice <krice@freeswitch.org> wrote:
Keeping in mind that you are using a sangoma transcoder board, that means you are handling the equivalent of 440 concurrent calls worth of media for people not using sangoma transcoding hardware (yes theres 4 call legs in that scenario, 1 from endpoint A to FS, 1 from FS to D100, 1 From D100 back to FS, 1 from FS back out to the other end point). That doesn’t sound like that big of a problem... You will have a fair bit of network traffic that way (the D100 shows up as another network interface as you are probably aware)...
I would be looking at something like iotop or something else to see where that load is coming from
Also, you have plenty of CPU free... So there is either a real problem here, or this is one of those things where the number of threads handling RTP and SIP are hanging around the run queue long enough to show up in the load average...
On 10/22/12 3:53 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote:
Ken. There is an average of 220 concurrent calls. Machine : DL380 G7 – 2 Quad Core – 8 Gig RAM. Centos 5.5 64bits
Ricarod.-
*De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Ken Rice *Enviado el:* lunes, 22 de octubre de 2012 17:47 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes
How many concurrent calls are you doing on that box?
On 10/22/12 3:33 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: Hi Ken. I mounted the db directory in RAM as you mention, but the “load average” still is high. You can see this here :
Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 132G 8.6G 117G 7% / proc 0 0 0 - /proc sysfs 0 0 0 - /sys devpts 0 0 0 - /dev/pts /dev/cciss/c0d0p1 104M 20M 79M 21% /boot tmpfs 4.2G 0 4.2G 0% /dev/shm none 0 0 0 - /proc/sys/fs/binfmt_misc sunrpc 0 0 0 - /var/lib/nfs/rpc_pipefs *tmpfs 4.2G 1.3M 4.2G 1% /usr/local/freeswitch/db * This is a resume from uptime and total calls for my freeswitch :
17:20:07 up 17 days, Total Calls : 16:26, 3 users, load average: 4.77, 4.50, 4.66 , Total Calls : 209, Total sessions: 440 17:20:12 up 17 days, Total Calls : 16:27, 3 users, load average: 6.79, 4.92, 4.80 , Total Calls : 245, Total sessions: 510 17:20:17 up 17 days, Total Calls : 16:27, 3 users, load average: 6.25, 4.84, 4.77 , Total Calls : 235, Total sessions: 488 17:20:22 up 17 days, Total Calls : 16:27, 3 users, load average: 6.63, 4.94, 4.81 , Total Calls : 233, Total sessions: 484 17:20:27 up 17 days, Total Calls : 16:27, 3 users, load average: 6.42, 4.92, 4.80 , Total Calls : 226, Total sessions: 470 17:20:32 up 17 days, Total Calls : 16:27, 3 users, load average: 5.90, 4.84, 4.78 , Total Calls : 218, Total sessions: 454 17:20:37 up 17 days, Total Calls : 16:27, 3 users, load average: 5.43, 4.76, 4.75 , Total Calls : 198, Total sessions: 416 17:20:43 up 17 days, Total Calls : 16:27, 3 users, load average: 4.99, 4.68, 4.72 , Total Calls : 183, Total sessions: 384 17:20:48 up 17 days, Total Calls : 16:27, 3 users, load average: 4.59, 4.60, 4.70 , Total Calls : 172, Total sessions: 366 17:20:53 up 17 days, Total Calls : 16:27, 3 users, load average: 4.31, 4.54, 4.68 , Total Calls : 184, Total sessions: 388 17:20:58 up 17 days, Total Calls : 16:27, 3 users, load average: 3.96, 4.47, 4.65 , Total Calls : 207, Total sessions: 442 17:21:03 up 17 days, Total Calls : 16:27, 3 users, load average: 4.04, 4.48, 4.65 , Total Calls : 203, Total sessions: 436 17:21:08 up 17 days, Total Calls : 16:27, 3 users, load average: 3.72, 4.40, 4.63 , Total Calls : 198, Total sessions: 418 17:21:13 up 17 days, Total Calls : 16:28, 3 users, load average: 3.47, 4.32, 4.60 , Total Calls : 198, Total sessions: 422 17:21:18 up 17 days, Total Calls : 16:28, 3 users, load average: 4.23, 4.47, 4.65 , Total Calls : 194, Total sessions: 410 17:21:23 up 17 days, Total Calls : 16:28, 3 users, load average: 8.30, 5.31, 4.92 , Total Calls : 211, Total sessions: 450 17:21:28 up 17 days, Total Calls : 16:28, 3 users, load average: 8.11, 5.32, 4.92 , Total Calls : 251, Total sessions: 536 17:21:33 up 17 days, Total Calls : 16:28, 3 users, load average: 8.74, 5.49, 4.98 , Total Calls : 250, Total sessions: 530 17:21:38 up 17 days, Total Calls : 16:28, 3 users, load average: 9.48, 5.70, 5.05 , Total Calls : 236, Total sessions: 496 17:21:43 up 17 days, Total Calls : 16:28, 3 users, load average: 9.45, 5.76, 5.07 , Total Calls : 233, Total sessions: 492 17:21:48 up 17 days, Total Calls : 16:28, 3 users, load average: 9.57, 5.84, 5.10 , Total Calls : 235, Total sessions: 496 17:21:53 up 17 days, Total Calls : 16:28, 3 users, load average: 10.24, 6.04, 5.17 , Total Calls : 222, Total sessions: 466 17:21:58 up 17 days, Total Calls : 16:28, 3 users, load average: 10.46, 6.16, 5.21 , Total Calls : 215, Total sessions: 452 17:22:03 up 17 days, Total Calls : 16:28, 3 users, load average: 11.07, 6.36, 5.28 , Total Calls : 208, Total sessions: 430 17:22:08 up 17 days, Total Calls : 16:28, 3 users, load average: 11.22, 6.47, 5.32 , Total Calls : 202, Total sessions: 422 17:22:14 up 17 days, Total Calls : 16:29, 3 users, load average: 11.52, 6.61, 5.37 , Total Calls : 240, Total sessions: 500 17:22:19 up 17 days, Total Calls : 16:29, 3 users, load average: 11.96, 6.78, 5.44 , Total Calls : 239, Total sessions: 498 17:22:24 up 17 days, Total Calls : 16:29, 3 users, load average: 14.29, 7.35, 5.63 , Total Calls : 250, Total sessions: 516 17:22:29 up 17 days, Total Calls : 16:29, 3 users, load average: 13.78, 7.36, 5.64 , Total Calls : 236, Total sessions: 490 17:22:34 up 17 days, Total Calls : 16:29, 3 users, load average: 12.92, 7.29, 5.63 , Total Calls : 209, Total sessions: 436 17:22:39 up 17 days, Total Calls : 16:29, 3 users, load average: 11.97, 7.18, 5.60 , Total Calls : 180, Total sessions: 372 17:22:44 up 17 days, Total Calls : 16:29, 3 users, load average: 11.01, 7.06, 5.57 , Total Calls : 161, Total sessions: 342 17:22:49 up 17 days, Total Calls : 16:29, 3 users, load average: 11.57, 7.24, 5.64 , Total Calls : 175, Total sessions: 370 17:22:54 up 17 days, Total Calls : 16:29, 3 users, load average: 11.12, 7.22, 5.64 , Total Calls : 181, Total sessions: 382 17:22:59 up 17 days, Total Calls : 16:29, 3 users, load average: 11.19, 7.30, 5.67 , Total Calls : 178, Total sessions: 366 17:23:04 up 17 days, Total Calls : 16:29, 3 users, load average: 11.10, 7.35, 5.69 , Total Calls : 173, Total sessions: 358 17:23:09 up 17 days, Total Calls : 16:30, 3 users, load average: 11.57, 7.51, 5.76 , Total Calls : 182, Total sessions: 378 17:23:14 up 17 days, Total Calls : 16:30, 3 users, load average: 11.36, 7.53, 5.77 , Total Calls : 187, Total sessions: 386 17:23:19 up 17 days, Total Calls : 16:30, 3 users, load average: 12.21, 7.77, 5.86 , Total Calls : 190, Total sessions: 402 17:23:24 up 17 days, Total Calls : 16:30, 3 users, load average: 12.04, 7.81, 5.88 , Total Calls : 239, Total sessions: 496 17:23:29 up 17 days, Total Calls : 16:30, 3 users, load average: 12.27, 7.93, 5.93 , Total Calls : 257, Total sessions: 532 17:23:34 up 17 days, Total Calls : 16:30, 3 users, load average: 12.73, 8.09, 5.99 , Total Calls : 242, Total sessions: 506 17:23:39 up 17 days, Total Calls : 16:30, 3 users, load average: 12.67, 8.16, 6.03 , Total Calls : 233, Total sessions: 482 17:23:45 up 17 days, Total Calls : 16:30, 3 users, load average: 12.38, 8.17, 6.04 , Total Calls : 238, Total sessions: 492 17:23:50 up 17 days, Total Calls : 16:30, 3 users, load average: 12.67, 8.30, 6.10 , Total Calls : 236, Total sessions: 482 17:23:55 up 17 days, Total Calls : 16:30, 3 users, load average: 13.09, 8.46, 6.16 , Total Calls : 222, Total sessions: 458 17:24:00 up 17 days, Total Calls : 16:30, 3 users, load average: 12.37, 8.39, 6.15 , Total Calls : 221, Total sessions: 470 17:24:05 up 17 days, Total Calls : 16:30, 3 users, load average: 11.78, 8.33, 6.14 , Total Calls : 200, Total sessions: 422 17:24:10 up 17 days, Total Calls : 16:31, 3 users, load average: 13.64, 8.77, 6.30 , Total Calls : 214, Total sessions: 452 17:24:15 up 17 days, Total Calls : 16:31, 3 users, load average: 13.98, 8.93, 6.36 , Total Calls : 255, Total sessions: 536 17:24:20 up 17 days, Total Calls : 16:31, 3 users, load average: 14.23, 9.06, 6.42 , Total Calls : 252, Total sessions: 524 17:24:25 up 17 days, Total Calls : 16:31, 3 users, load average: 15.73, 9.46, 6.56 , Total Calls : 260, Total sessions: 536 17:24:30 up 17 days, Total Calls : 16:31, 3 users, load average: 15.03, 9.42, 6.56 , Total Calls : 246, Total sessions: 508 17:24:35 up 17 days, Total Calls : 16:31, 3 users, load average: 15.11, 9.52, 6.61 , Total Calls : 230, Total sessions: 478 17:24:40 up 17 days, Total Calls : 16:31, 3 users, load average: 14.94, 9.58, 6.64 , Total Calls : 194, Total sessions: 408 17:24:45 up 17 days, Total Calls : 16:31, 3 users, load average: 14.54, 9.59, 6.66 , Total Calls : 189, Total sessions: 396 17:24:50 up 17 days, Total Calls : 16:31, 3 users, load average: 13.94, 9.55, 6.66 , Total Calls : 202, Total sessions: 438 17:24:55 up 17 days, Total Calls : 16:31, 3 users, load average: 13.30, 9.49, 6.66 , Total Calls : 221, Total sessions: 470 17:25:00 up 17 days, Total Calls : 16:31, 3 users, load average: 12.88, 9.46, 6.67 , Total Calls : 220, Total sessions: 478 17:25:05 up 17 days, Total Calls : 16:31, 3 users, load average: 14.25, 9.80, 6.79 , Total Calls : 219, Total sessions: 480 17:25:10 up 17 days, Total Calls : 16:32, 3 users, load average: 13.67, 9.76, 6.79 , Total Calls : 222, Total sessions: 490 17:25:16 up 17 days, Total Calls : 16:32, 3 users, load average: 13.05, 9.69, 6.79 , Total Calls : 214, Total sessions: 470 17:25:21 up 17 days, Total Calls : 16:32, 3 users, load average: 13.53, 9.85, 6.85 , Total Calls : 210, Total sessions: 450 17:25:26 up 17 days, Total Calls : 16:32, 3 users, load average: 13.81, 9.97, 6.91 , Total Calls : 250, Total sessions: 524 17:25:31 up 17 days, Total Calls : 16:32, 3 users, load average: 12.70, 9.80, 6.87 , Total Calls : 258, Total sessions: 536 17:25:36 up 17 days, Total Calls : 16:32, 3 users, load average: 11.68, 9.64, 6.83 , Total Calls : 255, Total sessions: 528 17:25:41 up 17 days, Total Calls : 16:32, 3 users, load average: 10.83, 9.49, 6.80 , Total Calls : 246, Total sessions: 524 17:25:46 up 17 days, Total Calls : 16:32, 3 users, load average: 10.04, 9.35, 6.77 , Total Calls : 262, Total sessions: 564 17:25:51 up 17 days, Total Calls : 16:32, 3 users, load average: 9.32, 9.21, 6.74 , Total Calls : 238, Total sessions: 510 17:25:56 up 17 days, Total Calls : 16:32, 3 users, load average: 8.57, 9.06, 6.70 , Total Calls : 220, Total sessions: 476 17:26:01 up 17 days, Total Calls : 16:32, 3 users, load average: 7.88, 8.91, 6.67 , Total Calls : 228, Total sessions: 498
What else could be happening. Did I mention that I’m using two transcoding cards from Sangoma?
Regards, Ricardo.-
*De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Ken Rice *Enviado el:* lunes, 22 de octubre de 2012 16:50 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes
Load Average is the number of threads/processes waiting in the kernel’s run queue at sample time... Supposedly number of cores is the how much your load avg can get to before your boxes is exhausted but this isnt exactly true on modern hardware with highly multi-threaded software like FreeSWITCH...
Where did I get the 100+ number? From looking at ‘top’ or ‘w’ on a box to see what the current load avg is...
Sqlite is used all over FreeSWITCH from the coredb to mod_sofia and everything in between.... When you type ‘show channels’ from fs_cli it doesn’t query the structures in RAM to show you that output, it pulls it from the sqlite db...
That being said, you can test quickly to see if sqlite is hammering the disk at some point by shifting freeswitch/db dir to a ram drive. Example code for linux such as Centos or Debian assuming freeswitch is installed to /usr/local/freeswitch
Shutdown freeswitch rm –rf /usr/local/freeswitch/db/* mount –t tmpfs tmpfs /usr/local/freeswitch/db Startup freeswitch
Once you have done that stick it back in rotation...
Note this will not affect anything but voicemail if you have voicemail running on that box and not using an external DB via mod_voicemail’s DSN settings you’ll lose the voicemail DB... (I would assume this is not the case since you are using this primarily as a Transcoder)
K
On 10/22/12 2:30 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: Hi Ken. Maybe that is the problem. I was sure that the “load average” can’t go more than #cores in your system. In my case I have 8 cores, so a “load average” of 12 is high from my point of view. Where did you get the number 100 for this parameter? Anyway, what is running a sqlite in freeswitch. Does this have to do with the core.db file? Or with the cdr_sqlite module?.
Thanks Ricardo.-
*De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Ken Rice *Enviado el:* lunes, 22 de octubre de 2012 16:18 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes
Could it be just a normal sqlite running a vacuum?
Sqlite does this every so often or if its calling a sync....
Also freeswitch with a load avg of 12 for a short period of time isnt anything to really worry about. Due to the number of threads that’s possible with FS (ie: 1 per call leg) its not unusual to see a box humming along and the load avg well above 100 in my experience... This is due to the number of threads waiting in the run queue when it was sampled... Now, that’s not saying that this is not specifically indicative or a problem but something else to think about
On 10/22/12 1:52 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: This is the gdb from the proccess consuming more cpu :
(gdb) thread 7 [Switching to thread 7 (Thread 31781)]#0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 (gdb) bt #0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 #1 0x00002aede19276de in unixSync (id=0x2aaaac726380, dataOnly=<value optimized out>) at ./src/os_unix.c:1229 #2 0x00002aede1951178 in syncJournal (pPager=0x1a1c200) at ./src/pager.c:2229 #3 0x00002aede1953b65 in sqlite3pager_sync (pPager=0x1a1c200, zMaster=0x0, nTrunc=0) at ./src/pager.c:3770 #4 0x00002aede193d07b in sqlite3BtreeSync (p=<value optimized out>, zMaster=0x0) at ./src/btree.c:6520 #5 0x00002aede1931f15 in sqlite3VdbeHalt (p=0x2aaaad17ac70) at ./src/vdbeaux.c:1062 #6 0x00002aede1962613 in sqlite3VdbeExec (p=0x2aaaad17ac70) at ./src/vdbe.c:2360 #7 0x00002aede193036c in sqlite3Step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:236 #8 sqlite3_step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:289 #9 0x00002aede1938978 in sqlite3_exec (db=0x1a1f070, zSql=0x2aede199e717 "COMMIT", xCallback=0, pArg=0x0, pzErrMsg=0x4178bed0) at ./src/legacy.c:78 #10 0x00002aede18ae0c1 in switch_core_db_exec (db=0x1a1f070, sql=0x2aede199e717 "COMMIT", callback=0, data=0x0, errmsg=0x4178bf38) at src/switch_core_db.c:93 #11 0x00002aede1897c18 in switch_cache_db_execute_sql_real (dbh=0x1a2dce0, sql=0x2aede199e717 "COMMIT", err=0x0) at src/switch_core_sqldb.c:445 #12 0x00002aede18985c3 in switch_cache_db_persistant_execute_trans (dbh=0x1a2dce0,
sql=0x5f03bd0 "delete from calls where (caller_uuid='e64319c5-7f91-47a1-9561-42da291e499a' or callee_uuid='e64319c5-7f91-47a1-9561-42da291e499a') and hostname=' transcoder.red.net <http://transcoder.red.net> <http://transcoder.red.net> <http://transcoder.red.net> <http://transcoder.red.net> ';\ndelete from calls where (call"..., retries=1)
at src/switch_core_sqldb.c:787 #13 0x00002aede189bf1c in switch_core_sql_thread (thread=<value optimized out>, obj=<value optimized out>) at src/switch_core_sqldb.c:1026 #14 0x0000003c03c0673d in start_thread () from /lib64/libpthread.so.0 #15 0x0000003c030d3f6d in clone () from /lib64/libc.so.6
*De:* Ricardo Martinez [mailto:rmartinez@redvoiss.net<rmartinez@redvoiss.net>]
*Enviado el:* lunes, 22 de octubre de 2012 15:50 *Para:* 'FreeSWITCH Users Help' *Asunto:* RE: [Freeswitch-users] CPU Spikes every 20 minutes
AS far as i can see there is no java running. This is the top at the moment of the spike :
top - 15:48:29 up 17 days, 14:55, 3 users, load average: 14.07, 8.87, 6.48 Tasks: 158 total, 1 running, 157 sleeping, 0 stopped, 0 zombie Cpu(s): 12.6%us, 9.0%sy, 0.0%ni, 66.1%id, 2.7%wa, 0.3%hi, 9.2%si, 0.0%st Mem: 8171500k total, 8041556k used, 129944k free, 177688k buffers Swap: 10223608k total, 0k used, 10223608k free, 6951988k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31772 root 15 0 870m 481m 6880 S 212.4 6.0 9523:51 freeswitch 4210 root 19 0 36344 2940 1444 S 1.0 0.0 56:21.08 sngtc_server 29586 root 15 0 12740 1132 808 R 0.3 0.0 0:00.02 top 1 root 15 0 10348 704 588 S 0.0 0.0 0:01.13 init 2 root RT -5 0 0 0 S 0.0 0.0 0:00.41 migration/0 3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0 4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0 5 root RT -5 0 0 0 S 0.0 0.0 0:00.83 migration/1 6 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1 7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
8 root RT -5 0 0 0 S 0.0 0.0 0:00.14 migration/2 9 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/2 10 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/2 11 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/3 12 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/3 13 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/3 14 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/4 15 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/4 16 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/4 17 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/5 18 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/5 19 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/5 20 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/6
Regards, Ricardo.-
*De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Guillermo Ruiz Camauer *Enviado el:* lunes, 22 de octubre de 2012 15:38 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes
Do you have anything like Java running that could be doing garbage collection?
What other processes are running on the machine? Does TOP show which process is causing the spike?
Guillermo
On Mon, Oct 22, 2012 at 3:29 PM, Ricardo Martinez <rmartinez@redvoiss.net> wrote: Hi Gabe. I'm mean between the spikes there is nothing different as regarding to the calls... the cps maintains the same.. there are no scripts running or crontabs running, the simultaneous calls are the same also. So, for some reason the "load average" start to grow (every 20 or 25 minutes) without any clear cause. I have debug with gdb the moment the server is with 12.5 of "load average" here : http://pastebin.freeswitch.org/20112
Can we detect a problem from here?
Regards, Ricardo.
-----Mensaje original----- De: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] En nombre de Gabriel Gunderson Enviado el: lunes, 22 de octubre de 2012 13:31 Para: FreeSWITCH Users Help Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes
On Mon, Oct 22, 2012 at 8:37 AM, Ricardo Martinez <rmartinez@redvoiss.net> wrote:
The problem is that I'm having weird spikes of "load average" cpu every 20 minutes (aprox), without any reason, reaching near of 12.5 of load average for this machine.
What have you done to verify that it happens 'without any reason'?
Gabe
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Ken *http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org *irc.freenode.net #freeswitch
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
I used the dstat commad and posted the results in another answer. I already do the gcore and bt this is the output : *http://pastebin.freeswitch.org/20112* The thread 7 seems to get al the cpu load. Regards, Ricardo.- *De:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *En nombre de *Anthony Minessale *Enviado el:* martes, 23 de octubre de 2012 19:38 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes No, If you have these kinds of problems my only advice is what I already said in my last post about newer OS. You can try top -H -p <current pid of FS process> watch for any high cpu usage in a particular thread, gcore it and post a bt. On Tue, Oct 23, 2012 at 8:03 AM, Ricardo Martinez <rmartinez@redvoiss.net> wrote: Is there an equivalent command in Centos 5.5? Ricardo.- *De:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *En nombre de *Anthony Minessale *Enviado el:* lunes, 22 de octubre de 2012 19:37 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes iotop doesn't exist yet in in 5.5 Try cent 5.7 or cent 6.3 or debian 6 if you want some more modern kernels to get iotop On Mon, Oct 22, 2012 at 4:59 PM, Ken Rice <krice@freeswitch.org> wrote: Keeping in mind that you are using a sangoma transcoder board, that means you are handling the equivalent of 440 concurrent calls worth of media for people not using sangoma transcoding hardware (yes theres 4 call legs in that scenario, 1 from endpoint A to FS, 1 from FS to D100, 1 From D100 back to FS, 1 from FS back out to the other end point). That doesn’t sound like that big of a problem... You will have a fair bit of network traffic that way (the D100 shows up as another network interface as you are probably aware)... I would be looking at something like iotop or something else to see where that load is coming from Also, you have plenty of CPU free... So there is either a real problem here, or this is one of those things where the number of threads handling RTP and SIP are hanging around the run queue long enough to show up in the load average... On 10/22/12 3:53 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: Ken. There is an average of 220 concurrent calls. Machine : DL380 G7 – 2 Quad Core – 8 Gig RAM. Centos 5.5 64bits Ricarod.- *De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Ken Rice *Enviado el:* lunes, 22 de octubre de 2012 17:47 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes How many concurrent calls are you doing on that box? On 10/22/12 3:33 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: Hi Ken. I mounted the db directory in RAM as you mention, but the “load average” still is high. You can see this here : Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 132G 8.6G 117G 7% / proc 0 0 0 - /proc sysfs 0 0 0 - /sys devpts 0 0 0 - /dev/pts /dev/cciss/c0d0p1 104M 20M 79M 21% /boot tmpfs 4.2G 0 4.2G 0% /dev/shm none 0 0 0 - /proc/sys/fs/binfmt_misc sunrpc 0 0 0 - /var/lib/nfs/rpc_pipefs *tmpfs 4.2G 1.3M 4.2G 1% /usr/local/freeswitch/db * This is a resume from uptime and total calls for my freeswitch : 17:20:07 up 17 days, Total Calls : 16:26, 3 users, load average: 4.77, 4.50, 4.66 , Total Calls : 209, Total sessions: 440 17:20:12 up 17 days, Total Calls : 16:27, 3 users, load average: 6.79, 4.92, 4.80 , Total Calls : 245, Total sessions: 510 17:20:17 up 17 days, Total Calls : 16:27, 3 users, load average: 6.25, 4.84, 4.77 , Total Calls : 235, Total sessions: 488 17:20:22 up 17 days, Total Calls : 16:27, 3 users, load average: 6.63, 4.94, 4.81 , Total Calls : 233, Total sessions: 484 17:20:27 up 17 days, Total Calls : 16:27, 3 users, load average: 6.42, 4.92, 4.80 , Total Calls : 226, Total sessions: 470 17:20:32 up 17 days, Total Calls : 16:27, 3 users, load average: 5.90, 4.84, 4.78 , Total Calls : 218, Total sessions: 454 17:20:37 up 17 days, Total Calls : 16:27, 3 users, load average: 5.43, 4.76, 4.75 , Total Calls : 198, Total sessions: 416 17:20:43 up 17 days, Total Calls : 16:27, 3 users, load average: 4.99, 4.68, 4.72 , Total Calls : 183, Total sessions: 384 17:20:48 up 17 days, Total Calls : 16:27, 3 users, load average: 4.59, 4.60, 4.70 , Total Calls : 172, Total sessions: 366 17:20:53 up 17 days, Total Calls : 16:27, 3 users, load average: 4.31, 4.54, 4.68 , Total Calls : 184, Total sessions: 388 17:20:58 up 17 days, Total Calls : 16:27, 3 users, load average: 3.96, 4.47, 4.65 , Total Calls : 207, Total sessions: 442 17:21:03 up 17 days, Total Calls : 16:27, 3 users, load average: 4.04, 4.48, 4.65 , Total Calls : 203, Total sessions: 436 17:21:08 up 17 days, Total Calls : 16:27, 3 users, load average: 3.72, 4.40, 4.63 , Total Calls : 198, Total sessions: 418 17:21:13 up 17 days, Total Calls : 16:28, 3 users, load average: 3.47, 4.32, 4.60 , Total Calls : 198, Total sessions: 422 17:21:18 up 17 days, Total Calls : 16:28, 3 users, load average: 4.23, 4.47, 4.65 , Total Calls : 194, Total sessions: 410 17:21:23 up 17 days, Total Calls : 16:28, 3 users, load average: 8.30, 5.31, 4.92 , Total Calls : 211, Total sessions: 450 17:21:28 up 17 days, Total Calls : 16:28, 3 users, load average: 8.11, 5.32, 4.92 , Total Calls : 251, Total sessions: 536 17:21:33 up 17 days, Total Calls : 16:28, 3 users, load average: 8.74, 5.49, 4.98 , Total Calls : 250, Total sessions: 530 17:21:38 up 17 days, Total Calls : 16:28, 3 users, load average: 9.48, 5.70, 5.05 , Total Calls : 236, Total sessions: 496 17:21:43 up 17 days, Total Calls : 16:28, 3 users, load average: 9.45, 5.76, 5.07 , Total Calls : 233, Total sessions: 492 17:21:48 up 17 days, Total Calls : 16:28, 3 users, load average: 9.57, 5.84, 5.10 , Total Calls : 235, Total sessions: 496 17:21:53 up 17 days, Total Calls : 16:28, 3 users, load average: 10.24, 6.04, 5.17 , Total Calls : 222, Total sessions: 466 17:21:58 up 17 days, Total Calls : 16:28, 3 users, load average: 10.46, 6.16, 5.21 , Total Calls : 215, Total sessions: 452 17:22:03 up 17 days, Total Calls : 16:28, 3 users, load average: 11.07, 6.36, 5.28 , Total Calls : 208, Total sessions: 430 17:22:08 up 17 days, Total Calls : 16:28, 3 users, load average: 11.22, 6.47, 5.32 , Total Calls : 202, Total sessions: 422 17:22:14 up 17 days, Total Calls : 16:29, 3 users, load average: 11.52, 6.61, 5.37 , Total Calls : 240, Total sessions: 500 17:22:19 up 17 days, Total Calls : 16:29, 3 users, load average: 11.96, 6.78, 5.44 , Total Calls : 239, Total sessions: 498 17:22:24 up 17 days, Total Calls : 16:29, 3 users, load average: 14.29, 7.35, 5.63 , Total Calls : 250, Total sessions: 516 17:22:29 up 17 days, Total Calls : 16:29, 3 users, load average: 13.78, 7.36, 5.64 , Total Calls : 236, Total sessions: 490 17:22:34 up 17 days, Total Calls : 16:29, 3 users, load average: 12.92, 7.29, 5.63 , Total Calls : 209, Total sessions: 436 17:22:39 up 17 days, Total Calls : 16:29, 3 users, load average: 11.97, 7.18, 5.60 , Total Calls : 180, Total sessions: 372 17:22:44 up 17 days, Total Calls : 16:29, 3 users, load average: 11.01, 7.06, 5.57 , Total Calls : 161, Total sessions: 342 17:22:49 up 17 days, Total Calls : 16:29, 3 users, load average: 11.57, 7.24, 5.64 , Total Calls : 175, Total sessions: 370 17:22:54 up 17 days, Total Calls : 16:29, 3 users, load average: 11.12, 7.22, 5.64 , Total Calls : 181, Total sessions: 382 17:22:59 up 17 days, Total Calls : 16:29, 3 users, load average: 11.19, 7.30, 5.67 , Total Calls : 178, Total sessions: 366 17:23:04 up 17 days, Total Calls : 16:29, 3 users, load average: 11.10, 7.35, 5.69 , Total Calls : 173, Total sessions: 358 17:23:09 up 17 days, Total Calls : 16:30, 3 users, load average: 11.57, 7.51, 5.76 , Total Calls : 182, Total sessions: 378 17:23:14 up 17 days, Total Calls : 16:30, 3 users, load average: 11.36, 7.53, 5.77 , Total Calls : 187, Total sessions: 386 17:23:19 up 17 days, Total Calls : 16:30, 3 users, load average: 12.21, 7.77, 5.86 , Total Calls : 190, Total sessions: 402 17:23:24 up 17 days, Total Calls : 16:30, 3 users, load average: 12.04, 7.81, 5.88 , Total Calls : 239, Total sessions: 496 17:23:29 up 17 days, Total Calls : 16:30, 3 users, load average: 12.27, 7.93, 5.93 , Total Calls : 257, Total sessions: 532 17:23:34 up 17 days, Total Calls : 16:30, 3 users, load average: 12.73, 8.09, 5.99 , Total Calls : 242, Total sessions: 506 17:23:39 up 17 days, Total Calls : 16:30, 3 users, load average: 12.67, 8.16, 6.03 , Total Calls : 233, Total sessions: 482 17:23:45 up 17 days, Total Calls : 16:30, 3 users, load average: 12.38, 8.17, 6.04 , Total Calls : 238, Total sessions: 492 17:23:50 up 17 days, Total Calls : 16:30, 3 users, load average: 12.67, 8.30, 6.10 , Total Calls : 236, Total sessions: 482 17:23:55 up 17 days, Total Calls : 16:30, 3 users, load average: 13.09, 8.46, 6.16 , Total Calls : 222, Total sessions: 458 17:24:00 up 17 days, Total Calls : 16:30, 3 users, load average: 12.37, 8.39, 6.15 , Total Calls : 221, Total sessions: 470 17:24:05 up 17 days, Total Calls : 16:30, 3 users, load average: 11.78, 8.33, 6.14 , Total Calls : 200, Total sessions: 422 17:24:10 up 17 days, Total Calls : 16:31, 3 users, load average: 13.64, 8.77, 6.30 , Total Calls : 214, Total sessions: 452 17:24:15 up 17 days, Total Calls : 16:31, 3 users, load average: 13.98, 8.93, 6.36 , Total Calls : 255, Total sessions: 536 17:24:20 up 17 days, Total Calls : 16:31, 3 users, load average: 14.23, 9.06, 6.42 , Total Calls : 252, Total sessions: 524 17:24:25 up 17 days, Total Calls : 16:31, 3 users, load average: 15.73, 9.46, 6.56 , Total Calls : 260, Total sessions: 536 17:24:30 up 17 days, Total Calls : 16:31, 3 users, load average: 15.03, 9.42, 6.56 , Total Calls : 246, Total sessions: 508 17:24:35 up 17 days, Total Calls : 16:31, 3 users, load average: 15.11, 9.52, 6.61 , Total Calls : 230, Total sessions: 478 17:24:40 up 17 days, Total Calls : 16:31, 3 users, load average: 14.94, 9.58, 6.64 , Total Calls : 194, Total sessions: 408 17:24:45 up 17 days, Total Calls : 16:31, 3 users, load average: 14.54, 9.59, 6.66 , Total Calls : 189, Total sessions: 396 17:24:50 up 17 days, Total Calls : 16:31, 3 users, load average: 13.94, 9.55, 6.66 , Total Calls : 202, Total sessions: 438 17:24:55 up 17 days, Total Calls : 16:31, 3 users, load average: 13.30, 9.49, 6.66 , Total Calls : 221, Total sessions: 470 17:25:00 up 17 days, Total Calls : 16:31, 3 users, load average: 12.88, 9.46, 6.67 , Total Calls : 220, Total sessions: 478 17:25:05 up 17 days, Total Calls : 16:31, 3 users, load average: 14.25, 9.80, 6.79 , Total Calls : 219, Total sessions: 480 17:25:10 up 17 days, Total Calls : 16:32, 3 users, load average: 13.67, 9.76, 6.79 , Total Calls : 222, Total sessions: 490 17:25:16 up 17 days, Total Calls : 16:32, 3 users, load average: 13.05, 9.69, 6.79 , Total Calls : 214, Total sessions: 470 17:25:21 up 17 days, Total Calls : 16:32, 3 users, load average: 13.53, 9.85, 6.85 , Total Calls : 210, Total sessions: 450 17:25:26 up 17 days, Total Calls : 16:32, 3 users, load average: 13.81, 9.97, 6.91 , Total Calls : 250, Total sessions: 524 17:25:31 up 17 days, Total Calls : 16:32, 3 users, load average: 12.70, 9.80, 6.87 , Total Calls : 258, Total sessions: 536 17:25:36 up 17 days, Total Calls : 16:32, 3 users, load average: 11.68, 9.64, 6.83 , Total Calls : 255, Total sessions: 528 17:25:41 up 17 days, Total Calls : 16:32, 3 users, load average: 10.83, 9.49, 6.80 , Total Calls : 246, Total sessions: 524 17:25:46 up 17 days, Total Calls : 16:32, 3 users, load average: 10.04, 9.35, 6.77 , Total Calls : 262, Total sessions: 564 17:25:51 up 17 days, Total Calls : 16:32, 3 users, load average: 9.32, 9.21, 6.74 , Total Calls : 238, Total sessions: 510 17:25:56 up 17 days, Total Calls : 16:32, 3 users, load average: 8.57, 9.06, 6.70 , Total Calls : 220, Total sessions: 476 17:26:01 up 17 days, Total Calls : 16:32, 3 users, load average: 7.88, 8.91, 6.67 , Total Calls : 228, Total sessions: 498 What else could be happening. Did I mention that I’m using two transcoding cards from Sangoma? Regards, Ricardo.- *De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Ken Rice *Enviado el:* lunes, 22 de octubre de 2012 16:50 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes Load Average is the number of threads/processes waiting in the kernel’s run queue at sample time... Supposedly number of cores is the how much your load avg can get to before your boxes is exhausted but this isnt exactly true on modern hardware with highly multi-threaded software like FreeSWITCH... Where did I get the 100+ number? From looking at ‘top’ or ‘w’ on a box to see what the current load avg is... Sqlite is used all over FreeSWITCH from the coredb to mod_sofia and everything in between.... When you type ‘show channels’ from fs_cli it doesn’t query the structures in RAM to show you that output, it pulls it from the sqlite db... That being said, you can test quickly to see if sqlite is hammering the disk at some point by shifting freeswitch/db dir to a ram drive. Example code for linux such as Centos or Debian assuming freeswitch is installed to /usr/local/freeswitch Shutdown freeswitch rm –rf /usr/local/freeswitch/db/* mount –t tmpfs tmpfs /usr/local/freeswitch/db Startup freeswitch Once you have done that stick it back in rotation... Note this will not affect anything but voicemail if you have voicemail running on that box and not using an external DB via mod_voicemail’s DSN settings you’ll lose the voicemail DB... (I would assume this is not the case since you are using this primarily as a Transcoder) K On 10/22/12 2:30 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: Hi Ken. Maybe that is the problem. I was sure that the “load average” can’t go more than #cores in your system. In my case I have 8 cores, so a “load average” of 12 is high from my point of view. Where did you get the number 100 for this parameter? Anyway, what is running a sqlite in freeswitch. Does this have to do with the core.db file? Or with the cdr_sqlite module?. Thanks Ricardo.- *De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Ken Rice *Enviado el:* lunes, 22 de octubre de 2012 16:18 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes Could it be just a normal sqlite running a vacuum? Sqlite does this every so often or if its calling a sync.... Also freeswitch with a load avg of 12 for a short period of time isnt anything to really worry about. Due to the number of threads that’s possible with FS (ie: 1 per call leg) its not unusual to see a box humming along and the load avg well above 100 in my experience... This is due to the number of threads waiting in the run queue when it was sampled... Now, that’s not saying that this is not specifically indicative or a problem but something else to think about On 10/22/12 1:52 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: This is the gdb from the proccess consuming more cpu : (gdb) thread 7 [Switching to thread 7 (Thread 31781)]#0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 (gdb) bt #0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 #1 0x00002aede19276de in unixSync (id=0x2aaaac726380, dataOnly=<value optimized out>) at ./src/os_unix.c:1229 #2 0x00002aede1951178 in syncJournal (pPager=0x1a1c200) at ./src/pager.c:2229 #3 0x00002aede1953b65 in sqlite3pager_sync (pPager=0x1a1c200, zMaster=0x0, nTrunc=0) at ./src/pager.c:3770 #4 0x00002aede193d07b in sqlite3BtreeSync (p=<value optimized out>, zMaster=0x0) at ./src/btree.c:6520 #5 0x00002aede1931f15 in sqlite3VdbeHalt (p=0x2aaaad17ac70) at ./src/vdbeaux.c:1062 #6 0x00002aede1962613 in sqlite3VdbeExec (p=0x2aaaad17ac70) at ./src/vdbe.c:2360 #7 0x00002aede193036c in sqlite3Step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:236 #8 sqlite3_step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:289 #9 0x00002aede1938978 in sqlite3_exec (db=0x1a1f070, zSql=0x2aede199e717 "COMMIT", xCallback=0, pArg=0x0, pzErrMsg=0x4178bed0) at ./src/legacy.c:78 #10 0x00002aede18ae0c1 in switch_core_db_exec (db=0x1a1f070, sql=0x2aede199e717 "COMMIT", callback=0, data=0x0, errmsg=0x4178bf38) at src/switch_core_db.c:93 #11 0x00002aede1897c18 in switch_cache_db_execute_sql_real (dbh=0x1a2dce0, sql=0x2aede199e717 "COMMIT", err=0x0) at src/switch_core_sqldb.c:445 #12 0x00002aede18985c3 in switch_cache_db_persistant_execute_trans (dbh=0x1a2dce0, sql=0x5f03bd0 "delete from calls where (caller_uuid='e64319c5-7f91-47a1-9561-42da291e499a' or callee_uuid='e64319c5-7f91-47a1-9561-42da291e499a') and hostname=' transcoder.red.net <http://transcoder.red.net> <http://transcoder.red.net> <http://transcoder.red.net> <http://transcoder.red.net> ';\ndelete from calls where (call"..., retries=1) at src/switch_core_sqldb.c:787 #13 0x00002aede189bf1c in switch_core_sql_thread (thread=<value optimized out>, obj=<value optimized out>) at src/switch_core_sqldb.c:1026 #14 0x0000003c03c0673d in start_thread () from /lib64/libpthread.so.0 #15 0x0000003c030d3f6d in clone () from /lib64/libc.so.6 *De:* Ricardo Martinez [mailto:rmartinez@redvoiss.net<rmartinez@redvoiss.net>] *Enviado el:* lunes, 22 de octubre de 2012 15:50 *Para:* 'FreeSWITCH Users Help' *Asunto:* RE: [Freeswitch-users] CPU Spikes every 20 minutes AS far as i can see there is no java running. This is the top at the moment of the spike : top - 15:48:29 up 17 days, 14:55, 3 users, load average: 14.07, 8.87, 6.48 Tasks: 158 total, 1 running, 157 sleeping, 0 stopped, 0 zombie Cpu(s): 12.6%us, 9.0%sy, 0.0%ni, 66.1%id, 2.7%wa, 0.3%hi, 9.2%si, 0.0%st Mem: 8171500k total, 8041556k used, 129944k free, 177688k buffers Swap: 10223608k total, 0k used, 10223608k free, 6951988k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31772 root 15 0 870m 481m 6880 S 212.4 6.0 9523:51 freeswitch 4210 root 19 0 36344 2940 1444 S 1.0 0.0 56:21.08 sngtc_server 29586 root 15 0 12740 1132 808 R 0.3 0.0 0:00.02 top 1 root 15 0 10348 704 588 S 0.0 0.0 0:01.13 init 2 root RT -5 0 0 0 S 0.0 0.0 0:00.41 migration/0 3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0 4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0 5 root RT -5 0 0 0 S 0.0 0.0 0:00.83 migration/1 6 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1 7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1 8 root RT -5 0 0 0 S 0.0 0.0 0:00.14 migration/2 9 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/2 10 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/2 11 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/3 12 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/3 13 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/3 14 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/4 15 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/4 16 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/4 17 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/5 18 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/5 19 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/5 20 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/6 Regards, Ricardo.- *De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Guillermo Ruiz Camauer *Enviado el:* lunes, 22 de octubre de 2012 15:38 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes Do you have anything like Java running that could be doing garbage collection? What other processes are running on the machine? Does TOP show which process is causing the spike? Guillermo On Mon, Oct 22, 2012 at 3:29 PM, Ricardo Martinez <rmartinez@redvoiss.net> wrote: Hi Gabe. I'm mean between the spikes there is nothing different as regarding to the calls... the cps maintains the same.. there are no scripts running or crontabs running, the simultaneous calls are the same also. So, for some reason the "load average" start to grow (every 20 or 25 minutes) without any clear cause. I have debug with gdb the moment the server is with 12.5 of "load average" here : http://pastebin.freeswitch.org/20112 Can we detect a problem from here? Regards, Ricardo. -----Mensaje original----- De: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] En nombre de Gabriel Gunderson Enviado el: lunes, 22 de octubre de 2012 13:31 Para: FreeSWITCH Users Help Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes On Mon, Oct 22, 2012 at 8:37 AM, Ricardo Martinez <rmartinez@redvoiss.net> wrote:
The problem is that I'm having weird spikes of "load average" cpu every 20 minutes (aprox), without any reason, reaching near of 12.5 of load average for this machine.
What have you done to verify that it happens 'without any reason'? Gabe _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Ken *http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org *irc.freenode.net #freeswitch _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900 _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
Thread 7 is the sql thread. again it points to the sql. If you are only concerned by the load average and are not actually experiencing any noticeable problems you should not be concerned. If you have an actual problem, try a newer OS. Sangoma cards have been known to consume more resources on older CentOS due to incompatibility with modern processors. On Wed, Oct 24, 2012 at 8:02 AM, Ricardo Martinez <rmartinez@redvoiss.net>wrote:
I used the dstat commad and posted the results in another answer.
I already do the gcore and bt this is the output : *http://pastebin.freeswitch.org/20112*
The thread 7 seems to get al the cpu load.
Regards,
Ricardo.-
*De:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *En nombre de *Anthony Minessale *Enviado el:* martes, 23 de octubre de 2012 19:38
*Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes
No,
If you have these kinds of problems my only advice is what I already said in my last post about newer OS.
You can try top -H -p <current pid of FS process>
watch for any high cpu usage in a particular thread, gcore it and post a bt.
On Tue, Oct 23, 2012 at 8:03 AM, Ricardo Martinez <rmartinez@redvoiss.net> wrote:
Is there an equivalent command in Centos 5.5?
Ricardo.-
*De:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *En nombre de *Anthony Minessale *Enviado el:* lunes, 22 de octubre de 2012 19:37
*Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes
iotop doesn't exist yet in in 5.5
Try cent 5.7
or cent 6.3 or debian 6 if you want some more modern kernels to get iotop
On Mon, Oct 22, 2012 at 4:59 PM, Ken Rice <krice@freeswitch.org> wrote:
Keeping in mind that you are using a sangoma transcoder board, that means you are handling the equivalent of 440 concurrent calls worth of media for people not using sangoma transcoding hardware (yes theres 4 call legs in that scenario, 1 from endpoint A to FS, 1 from FS to D100, 1 From D100 back to FS, 1 from FS back out to the other end point). That doesn’t sound like that big of a problem... You will have a fair bit of network traffic that way (the D100 shows up as another network interface as you are probably aware)...
I would be looking at something like iotop or something else to see where that load is coming from
Also, you have plenty of CPU free... So there is either a real problem here, or this is one of those things where the number of threads handling RTP and SIP are hanging around the run queue long enough to show up in the load average...
On 10/22/12 3:53 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote:
Ken. There is an average of 220 concurrent calls. Machine : DL380 G7 – 2 Quad Core – 8 Gig RAM. Centos 5.5 64bits
Ricarod.-
*De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Ken Rice *Enviado el:* lunes, 22 de octubre de 2012 17:47 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes
How many concurrent calls are you doing on that box?
On 10/22/12 3:33 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: Hi Ken. I mounted the db directory in RAM as you mention, but the “load average” still is high. You can see this here :
Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 132G 8.6G 117G 7% / proc 0 0 0 - /proc sysfs 0 0 0 - /sys devpts 0 0 0 - /dev/pts /dev/cciss/c0d0p1 104M 20M 79M 21% /boot tmpfs 4.2G 0 4.2G 0% /dev/shm none 0 0 0 - /proc/sys/fs/binfmt_misc sunrpc 0 0 0 - /var/lib/nfs/rpc_pipefs *tmpfs 4.2G 1.3M 4.2G 1% /usr/local/freeswitch/db * This is a resume from uptime and total calls for my freeswitch :
17:20:07 up 17 days, Total Calls : 16:26, 3 users, load average: 4.77, 4.50, 4.66 , Total Calls : 209, Total sessions: 440 17:20:12 up 17 days, Total Calls : 16:27, 3 users, load average: 6.79, 4.92, 4.80 , Total Calls : 245, Total sessions: 510 17:20:17 up 17 days, Total Calls : 16:27, 3 users, load average: 6.25, 4.84, 4.77 , Total Calls : 235, Total sessions: 488 17:20:22 up 17 days, Total Calls : 16:27, 3 users, load average: 6.63, 4.94, 4.81 , Total Calls : 233, Total sessions: 484 17:20:27 up 17 days, Total Calls : 16:27, 3 users, load average: 6.42, 4.92, 4.80 , Total Calls : 226, Total sessions: 470 17:20:32 up 17 days, Total Calls : 16:27, 3 users, load average: 5.90, 4.84, 4.78 , Total Calls : 218, Total sessions: 454 17:20:37 up 17 days, Total Calls : 16:27, 3 users, load average: 5.43, 4.76, 4.75 , Total Calls : 198, Total sessions: 416 17:20:43 up 17 days, Total Calls : 16:27, 3 users, load average: 4.99, 4.68, 4.72 , Total Calls : 183, Total sessions: 384 17:20:48 up 17 days, Total Calls : 16:27, 3 users, load average: 4.59, 4.60, 4.70 , Total Calls : 172, Total sessions: 366 17:20:53 up 17 days, Total Calls : 16:27, 3 users, load average: 4.31, 4.54, 4.68 , Total Calls : 184, Total sessions: 388 17:20:58 up 17 days, Total Calls : 16:27, 3 users, load average: 3.96, 4.47, 4.65 , Total Calls : 207, Total sessions: 442 17:21:03 up 17 days, Total Calls : 16:27, 3 users, load average: 4.04, 4.48, 4.65 , Total Calls : 203, Total sessions: 436 17:21:08 up 17 days, Total Calls : 16:27, 3 users, load average: 3.72, 4.40, 4.63 , Total Calls : 198, Total sessions: 418 17:21:13 up 17 days, Total Calls : 16:28, 3 users, load average: 3.47, 4.32, 4.60 , Total Calls : 198, Total sessions: 422 17:21:18 up 17 days, Total Calls : 16:28, 3 users, load average: 4.23, 4.47, 4.65 , Total Calls : 194, Total sessions: 410 17:21:23 up 17 days, Total Calls : 16:28, 3 users, load average: 8.30, 5.31, 4.92 , Total Calls : 211, Total sessions: 450 17:21:28 up 17 days, Total Calls : 16:28, 3 users, load average: 8.11, 5.32, 4.92 , Total Calls : 251, Total sessions: 536 17:21:33 up 17 days, Total Calls : 16:28, 3 users, load average: 8.74, 5.49, 4.98 , Total Calls : 250, Total sessions: 530 17:21:38 up 17 days, Total Calls : 16:28, 3 users, load average: 9.48, 5.70, 5.05 , Total Calls : 236, Total sessions: 496 17:21:43 up 17 days, Total Calls : 16:28, 3 users, load average: 9.45, 5.76, 5.07 , Total Calls : 233, Total sessions: 492 17:21:48 up 17 days, Total Calls : 16:28, 3 users, load average: 9.57, 5.84, 5.10 , Total Calls : 235, Total sessions: 496 17:21:53 up 17 days, Total Calls : 16:28, 3 users, load average: 10.24, 6.04, 5.17 , Total Calls : 222, Total sessions: 466 17:21:58 up 17 days, Total Calls : 16:28, 3 users, load average: 10.46, 6.16, 5.21 , Total Calls : 215, Total sessions: 452 17:22:03 up 17 days, Total Calls : 16:28, 3 users, load average: 11.07, 6.36, 5.28 , Total Calls : 208, Total sessions: 430 17:22:08 up 17 days, Total Calls : 16:28, 3 users, load average: 11.22, 6.47, 5.32 , Total Calls : 202, Total sessions: 422 17:22:14 up 17 days, Total Calls : 16:29, 3 users, load average: 11.52, 6.61, 5.37 , Total Calls : 240, Total sessions: 500 17:22:19 up 17 days, Total Calls : 16:29, 3 users, load average: 11.96, 6.78, 5.44 , Total Calls : 239, Total sessions: 498 17:22:24 up 17 days, Total Calls : 16:29, 3 users, load average: 14.29, 7.35, 5.63 , Total Calls : 250, Total sessions: 516 17:22:29 up 17 days, Total Calls : 16:29, 3 users, load average: 13.78, 7.36, 5.64 , Total Calls : 236, Total sessions: 490 17:22:34 up 17 days, Total Calls : 16:29, 3 users, load average: 12.92, 7.29, 5.63 , Total Calls : 209, Total sessions: 436 17:22:39 up 17 days, Total Calls : 16:29, 3 users, load average: 11.97, 7.18, 5.60 , Total Calls : 180, Total sessions: 372 17:22:44 up 17 days, Total Calls : 16:29, 3 users, load average: 11.01, 7.06, 5.57 , Total Calls : 161, Total sessions: 342 17:22:49 up 17 days, Total Calls : 16:29, 3 users, load average: 11.57, 7.24, 5.64 , Total Calls : 175, Total sessions: 370 17:22:54 up 17 days, Total Calls : 16:29, 3 users, load average: 11.12, 7.22, 5.64 , Total Calls : 181, Total sessions: 382 17:22:59 up 17 days, Total Calls : 16:29, 3 users, load average: 11.19, 7.30, 5.67 , Total Calls : 178, Total sessions: 366 17:23:04 up 17 days, Total Calls : 16:29, 3 users, load average: 11.10, 7.35, 5.69 , Total Calls : 173, Total sessions: 358 17:23:09 up 17 days, Total Calls : 16:30, 3 users, load average: 11.57, 7.51, 5.76 , Total Calls : 182, Total sessions: 378 17:23:14 up 17 days, Total Calls : 16:30, 3 users, load average: 11.36, 7.53, 5.77 , Total Calls : 187, Total sessions: 386 17:23:19 up 17 days, Total Calls : 16:30, 3 users, load average: 12.21, 7.77, 5.86 , Total Calls : 190, Total sessions: 402 17:23:24 up 17 days, Total Calls : 16:30, 3 users, load average: 12.04, 7.81, 5.88 , Total Calls : 239, Total sessions: 496 17:23:29 up 17 days, Total Calls : 16:30, 3 users, load average: 12.27, 7.93, 5.93 , Total Calls : 257, Total sessions: 532 17:23:34 up 17 days, Total Calls : 16:30, 3 users, load average: 12.73, 8.09, 5.99 , Total Calls : 242, Total sessions: 506 17:23:39 up 17 days, Total Calls : 16:30, 3 users, load average: 12.67, 8.16, 6.03 , Total Calls : 233, Total sessions: 482 17:23:45 up 17 days, Total Calls : 16:30, 3 users, load average: 12.38, 8.17, 6.04 , Total Calls : 238, Total sessions: 492 17:23:50 up 17 days, Total Calls : 16:30, 3 users, load average: 12.67, 8.30, 6.10 , Total Calls : 236, Total sessions: 482 17:23:55 up 17 days, Total Calls : 16:30, 3 users, load average: 13.09, 8.46, 6.16 , Total Calls : 222, Total sessions: 458 17:24:00 up 17 days, Total Calls : 16:30, 3 users, load average: 12.37, 8.39, 6.15 , Total Calls : 221, Total sessions: 470 17:24:05 up 17 days, Total Calls : 16:30, 3 users, load average: 11.78, 8.33, 6.14 , Total Calls : 200, Total sessions: 422 17:24:10 up 17 days, Total Calls : 16:31, 3 users, load average: 13.64, 8.77, 6.30 , Total Calls : 214, Total sessions: 452 17:24:15 up 17 days, Total Calls : 16:31, 3 users, load average: 13.98, 8.93, 6.36 , Total Calls : 255, Total sessions: 536 17:24:20 up 17 days, Total Calls : 16:31, 3 users, load average: 14.23, 9.06, 6.42 , Total Calls : 252, Total sessions: 524 17:24:25 up 17 days, Total Calls : 16:31, 3 users, load average: 15.73, 9.46, 6.56 , Total Calls : 260, Total sessions: 536 17:24:30 up 17 days, Total Calls : 16:31, 3 users, load average: 15.03, 9.42, 6.56 , Total Calls : 246, Total sessions: 508 17:24:35 up 17 days, Total Calls : 16:31, 3 users, load average: 15.11, 9.52, 6.61 , Total Calls : 230, Total sessions: 478 17:24:40 up 17 days, Total Calls : 16:31, 3 users, load average: 14.94, 9.58, 6.64 , Total Calls : 194, Total sessions: 408 17:24:45 up 17 days, Total Calls : 16:31, 3 users, load average: 14.54, 9.59, 6.66 , Total Calls : 189, Total sessions: 396 17:24:50 up 17 days, Total Calls : 16:31, 3 users, load average: 13.94, 9.55, 6.66 , Total Calls : 202, Total sessions: 438 17:24:55 up 17 days, Total Calls : 16:31, 3 users, load average: 13.30, 9.49, 6.66 , Total Calls : 221, Total sessions: 470 17:25:00 up 17 days, Total Calls : 16:31, 3 users, load average: 12.88, 9.46, 6.67 , Total Calls : 220, Total sessions: 478 17:25:05 up 17 days, Total Calls : 16:31, 3 users, load average: 14.25, 9.80, 6.79 , Total Calls : 219, Total sessions: 480 17:25:10 up 17 days, Total Calls : 16:32, 3 users, load average: 13.67, 9.76, 6.79 , Total Calls : 222, Total sessions: 490 17:25:16 up 17 days, Total Calls : 16:32, 3 users, load average: 13.05, 9.69, 6.79 , Total Calls : 214, Total sessions: 470 17:25:21 up 17 days, Total Calls : 16:32, 3 users, load average: 13.53, 9.85, 6.85 , Total Calls : 210, Total sessions: 450 17:25:26 up 17 days, Total Calls : 16:32, 3 users, load average: 13.81, 9.97, 6.91 , Total Calls : 250, Total sessions: 524 17:25:31 up 17 days, Total Calls : 16:32, 3 users, load average: 12.70, 9.80, 6.87 , Total Calls : 258, Total sessions: 536 17:25:36 up 17 days, Total Calls : 16:32, 3 users, load average: 11.68, 9.64, 6.83 , Total Calls : 255, Total sessions: 528 17:25:41 up 17 days, Total Calls : 16:32, 3 users, load average: 10.83, 9.49, 6.80 , Total Calls : 246, Total sessions: 524 17:25:46 up 17 days, Total Calls : 16:32, 3 users, load average: 10.04, 9.35, 6.77 , Total Calls : 262, Total sessions: 564 17:25:51 up 17 days, Total Calls : 16:32, 3 users, load average: 9.32, 9.21, 6.74 , Total Calls : 238, Total sessions: 510 17:25:56 up 17 days, Total Calls : 16:32, 3 users, load average: 8.57, 9.06, 6.70 , Total Calls : 220, Total sessions: 476 17:26:01 up 17 days, Total Calls : 16:32, 3 users, load average: 7.88, 8.91, 6.67 , Total Calls : 228, Total sessions: 498
What else could be happening. Did I mention that I’m using two transcoding cards from Sangoma?
Regards, Ricardo.-
*De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Ken Rice *Enviado el:* lunes, 22 de octubre de 2012 16:50 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes
Load Average is the number of threads/processes waiting in the kernel’s run queue at sample time... Supposedly number of cores is the how much your load avg can get to before your boxes is exhausted but this isnt exactly true on modern hardware with highly multi-threaded software like FreeSWITCH...
Where did I get the 100+ number? From looking at ‘top’ or ‘w’ on a box to see what the current load avg is...
Sqlite is used all over FreeSWITCH from the coredb to mod_sofia and everything in between.... When you type ‘show channels’ from fs_cli it doesn’t query the structures in RAM to show you that output, it pulls it from the sqlite db...
That being said, you can test quickly to see if sqlite is hammering the disk at some point by shifting freeswitch/db dir to a ram drive. Example code for linux such as Centos or Debian assuming freeswitch is installed to /usr/local/freeswitch
Shutdown freeswitch rm –rf /usr/local/freeswitch/db/* mount –t tmpfs tmpfs /usr/local/freeswitch/db Startup freeswitch
Once you have done that stick it back in rotation...
Note this will not affect anything but voicemail if you have voicemail running on that box and not using an external DB via mod_voicemail’s DSN settings you’ll lose the voicemail DB... (I would assume this is not the case since you are using this primarily as a Transcoder)
K
On 10/22/12 2:30 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: Hi Ken. Maybe that is the problem. I was sure that the “load average” can’t go more than #cores in your system. In my case I have 8 cores, so a “load average” of 12 is high from my point of view. Where did you get the number 100 for this parameter? Anyway, what is running a sqlite in freeswitch. Does this have to do with the core.db file? Or with the cdr_sqlite module?.
Thanks Ricardo.-
*De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Ken Rice *Enviado el:* lunes, 22 de octubre de 2012 16:18 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes
Could it be just a normal sqlite running a vacuum?
Sqlite does this every so often or if its calling a sync....
Also freeswitch with a load avg of 12 for a short period of time isnt anything to really worry about. Due to the number of threads that’s possible with FS (ie: 1 per call leg) its not unusual to see a box humming along and the load avg well above 100 in my experience... This is due to the number of threads waiting in the run queue when it was sampled... Now, that’s not saying that this is not specifically indicative or a problem but something else to think about
On 10/22/12 1:52 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: This is the gdb from the proccess consuming more cpu :
(gdb) thread 7 [Switching to thread 7 (Thread 31781)]#0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 (gdb) bt #0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 #1 0x00002aede19276de in unixSync (id=0x2aaaac726380, dataOnly=<value optimized out>) at ./src/os_unix.c:1229 #2 0x00002aede1951178 in syncJournal (pPager=0x1a1c200) at ./src/pager.c:2229 #3 0x00002aede1953b65 in sqlite3pager_sync (pPager=0x1a1c200, zMaster=0x0, nTrunc=0) at ./src/pager.c:3770 #4 0x00002aede193d07b in sqlite3BtreeSync (p=<value optimized out>, zMaster=0x0) at ./src/btree.c:6520 #5 0x00002aede1931f15 in sqlite3VdbeHalt (p=0x2aaaad17ac70) at ./src/vdbeaux.c:1062 #6 0x00002aede1962613 in sqlite3VdbeExec (p=0x2aaaad17ac70) at ./src/vdbe.c:2360 #7 0x00002aede193036c in sqlite3Step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:236 #8 sqlite3_step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:289 #9 0x00002aede1938978 in sqlite3_exec (db=0x1a1f070, zSql=0x2aede199e717 "COMMIT", xCallback=0, pArg=0x0, pzErrMsg=0x4178bed0) at ./src/legacy.c:78 #10 0x00002aede18ae0c1 in switch_core_db_exec (db=0x1a1f070, sql=0x2aede199e717 "COMMIT", callback=0, data=0x0, errmsg=0x4178bf38) at src/switch_core_db.c:93 #11 0x00002aede1897c18 in switch_cache_db_execute_sql_real (dbh=0x1a2dce0, sql=0x2aede199e717 "COMMIT", err=0x0) at src/switch_core_sqldb.c:445 #12 0x00002aede18985c3 in switch_cache_db_persistant_execute_trans (dbh=0x1a2dce0,
sql=0x5f03bd0 "delete from calls where (caller_uuid='e64319c5-7f91-47a1-9561-42da291e499a' or callee_uuid='e64319c5-7f91-47a1-9561-42da291e499a') and hostname=' transcoder.red.net <http://transcoder.red.net> <http://transcoder.red.net> <http://transcoder.red.net> <http://transcoder.red.net> ';\ndelete from calls where (call"..., retries=1)
at src/switch_core_sqldb.c:787 #13 0x00002aede189bf1c in switch_core_sql_thread (thread=<value optimized out>, obj=<value optimized out>) at src/switch_core_sqldb.c:1026 #14 0x0000003c03c0673d in start_thread () from /lib64/libpthread.so.0 #15 0x0000003c030d3f6d in clone () from /lib64/libc.so.6
*De:* Ricardo Martinez [mailto:rmartinez@redvoiss.net<rmartinez@redvoiss.net>]
*Enviado el:* lunes, 22 de octubre de 2012 15:50 *Para:* 'FreeSWITCH Users Help' *Asunto:* RE: [Freeswitch-users] CPU Spikes every 20 minutes
AS far as i can see there is no java running. This is the top at the moment of the spike :
top - 15:48:29 up 17 days, 14:55, 3 users, load average: 14.07, 8.87, 6.48 Tasks: 158 total, 1 running, 157 sleeping, 0 stopped, 0 zombie Cpu(s): 12.6%us, 9.0%sy, 0.0%ni, 66.1%id, 2.7%wa, 0.3%hi, 9.2%si, 0.0%st Mem: 8171500k total, 8041556k used, 129944k free, 177688k buffers Swap: 10223608k total, 0k used, 10223608k free, 6951988k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31772 root 15 0 870m 481m 6880 S 212.4 6.0 9523:51 freeswitch 4210 root 19 0 36344 2940 1444 S 1.0 0.0 56:21.08 sngtc_server 29586 root 15 0 12740 1132 808 R 0.3 0.0 0:00.02 top 1 root 15 0 10348 704 588 S 0.0 0.0 0:01.13 init 2 root RT -5 0 0 0 S 0.0 0.0 0:00.41 migration/0 3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0 4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0 5 root RT -5 0 0 0 S 0.0 0.0 0:00.83 migration/1 6 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1 7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
8 root RT -5 0 0 0 S 0.0 0.0 0:00.14 migration/2 9 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/2 10 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/2 11 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/3 12 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/3 13 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/3 14 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/4 15 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/4 16 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/4 17 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/5 18 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/5 19 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/5 20 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/6
Regards, Ricardo.-
*De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Guillermo Ruiz Camauer *Enviado el:* lunes, 22 de octubre de 2012 15:38 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes
Do you have anything like Java running that could be doing garbage collection?
What other processes are running on the machine? Does TOP show which process is causing the spike?
Guillermo
On Mon, Oct 22, 2012 at 3:29 PM, Ricardo Martinez <rmartinez@redvoiss.net> wrote: Hi Gabe. I'm mean between the spikes there is nothing different as regarding to the calls... the cps maintains the same.. there are no scripts running or crontabs running, the simultaneous calls are the same also. So, for some reason the "load average" start to grow (every 20 or 25 minutes) without any clear cause. I have debug with gdb the moment the server is with 12.5 of "load average" here : http://pastebin.freeswitch.org/20112
Can we detect a problem from here?
Regards, Ricardo.
-----Mensaje original----- De: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] En nombre de Gabriel Gunderson Enviado el: lunes, 22 de octubre de 2012 13:31 Para: FreeSWITCH Users Help Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes
On Mon, Oct 22, 2012 at 8:37 AM, Ricardo Martinez <rmartinez@redvoiss.net> wrote:
The problem is that I'm having weird spikes of "load average" cpu every 20 minutes (aprox), without any reason, reaching near of 12.5 of load average for this machine.
What have you done to verify that it happens 'without any reason'?
Gabe
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Ken *http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org *irc.freenode.net #freeswitch
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
Re: [Freeswitch-users] CPU Spikes every 20 minutes Hello Ken. In fact i have two transcoding cards for this machine. I was able to gather performance statistics with “dstat”. This is what I have for a certain period of time (every 5 second), including the moment it reach high “load average” (14 or 15). I included the interrupts for eth4 and eth5 (83 and 91) which are the PCIe Sangoma Transcoding Cards. I noticed certain correlation between the “net received” and the interrupts for the eth4, could this be the cause of the high load average ??? If so?, are the RTP packets the cause of this?. What do you think?! Regards, Ricardo.- [root@transcoder rmartinez]# more /proc/interrupts 83: 35 11112478 1027490479 2378114054 0 1484697027 620462655 1028635757 PCI-MSI eth4 91: 36 1124521815 1276819784 451886364 0 1090122523 248909954 549224594 PCI-MSI eth5 ----total-cpu-usage---- -net/total- -dsk/total- ---load-avg--- ---procs--- ---system-- -interrupts usr sys idl wai hiq siq| recv send| read writ| 1m 5m 15m |run blk new| int csw | 83 91 3 2 90 1 0 2| 0 0 |6751B 717k| 0.7 3.7 5.4| 0 0 4| 17k 31k|1371 258 11 8 72 0 0 8| 11M 8045k| 0 74k| 1.4 3.8 5.4| 0 0 11| 46k 81k| 10k 7150 11 8 72 0 0 8| 11M 7933k| 0 9830B| 1.3 3.7 5.4| 2 0 12| 46k 96k| 11k 7216 13 9 71 0 0 8| 11M 8051k| 0 13k| 1.2 3.6 5.3| 18 0 37| 46k 121k| 10k 7195 13 9 68 0 0 8| 11M 8072k| 0 9830B| 1.1 3.6 5.3| 3 0 21| 46k 163k| 11k 7211 14 9 67 0 0 9| 12M 8506k| 0 9830B| 1 3.5 5.3| 0 0 14| 49k 131k| 11k 7215 13 10 67 0 0 10| 12M 9065k| 0 9830B| 0.9 3.5 5.2| 0 0 9| 50k 116k| 11k 7298 13 10 67 0 0 10| 13M 9396k| 0 78k| 1.6 3.6 5.3| 0 0 12| 51k 104k| 11k 7330 16 10 64 0 0 9| 13M 9391k| 0 202k| 2.3 3.7 5.3| 2 0 17| 51k 106k| 11k 7391 22 9 60 0 0 9| 13M 9246k| 0 54k| 3 3.8 5.3| 2 0 11| 50k 102k| 11k 7278 20 9 61 0 0 9| 12M 9026k| 0 18k| 3.1 3.8 5.3| 4 0 13| 49k 98k| 11k 7220 12 9 71 0 0 9| 12M 8576k| 0 32k| 3.2 3.8 5.3| 7 0 13| 47k 94k| 10k 7257 12 8 72 0 0 7| 11M 8233k| 0 9830B| 3.2 3.8 5.3| 9 0 30| 46k 86k| 10k 7087 11 8 73 0 0 8| 11M 7968k| 0 9830B| 5.8 4.3 5.5| 5 0 17| 45k 85k| 11k 7121 12 8 71 0 0 8| 11M 8234k| 0 219k| 5.3 4.3 5.4| 3 0 16| 47k 102k| 11k 7223 12 8 71 0 0 8| 11M 8221k| 0 653k| 4.9 4.2 5.4| 0 0 15| 47k 99k| 11k 7186 12 8 72 0 0 8| 11M 7854k| 0 45k| 4.5 4.1 5.4| 0 0 18| 46k 108k| 11k 7277 12 8 70 0 0 9| 11M 7965k| 0 9830B| 4.2 4 5.3| 1 0 22| 46k 115k| 11k 7157 12 8 71 0 0 8| 11M 7881k| 0 9830B| 4.2 4 5.3| 4 0 11| 46k 110k| 11k 7171 12 8 72 0 0 8| 11M 8080k| 0 9830B| 3.8 4 5.3| 9 0 16| 47k 102k| 11k 7174 11 8 72 0 0 8| 11M 8143k| 0 9830B| 3.5 3.9 5.3| 7 0 11| 46k 107k| 10k 7242 12 8 72 0 0 8| 11M 8297k| 0 112k| 3.2 3.9 5.3| 4 0 12| 47k 103k| 11k 7093 12 9 71 0 0 8| 12M 8535k| 0 9830B| 3 3.8 5.2| 0 0 17| 47k 102k| 11k 7223 12 9 70 0 0 8| 12M 8718k| 0 13k| 2.7 3.7 5.2| 0 0 19| 48k 105k| 11k 7238 12 9 70 0 0 9| 12M 8692k| 0 9830B| 3.2 3.8 5.2| 3 0 16| 48k 96k| 11k 7213 12 8 71 0 0 8| 12M 8458k| 0 14k| 3.2 3.8 5.2| 1 0 14| 48k 85k| 12k 7163 12 9 70 0 0 9| 12M 8751k| 0 13k| 3 3.7 5.2| 2 0 12| 49k 101k| 11k 7092 14 10 67 0 0 9| 12M 8861k| 0 13k| 2.7 3.7 5.2| 8 0 42| 49k 119k| 12k 7024 14 10 65 0 0 9| 12M 8657k| 0 42k| 2.5 3.6 5.1| 7 0 20| 50k 168k| 12k 7280 16 11 63 0 0 11| 13M 9386k| 0 9830B| 3.7 3.9 5.2| 4 0 12| 53k 139k| 13k 7500 14 10 65 0 0 10| 13M 9228k| 0 9830B| 3.8 3.9 5.2| 4 0 9| 52k 117k| 13k 7423 13 10 67 0 0 10| 13M 8937k| 0 9830B| 3.5 3.8 5.2| 5 0 16| 50k 109k| 12k 7441 14 10 67 0 0 10| 13M 9262k| 0 9830B| 4 3.9 5.2| 2 0 13| 51k 113k| 12k 7414 13 9 68 0 0 9| 12M 8906k| 0 9830B| 3.7 3.8 5.2| 0 0 12| 50k 107k| 12k 7459 13 9 68 0 0 9| 12M 8992k| 0 9830B| 3.4 3.8 5.1| 4 0 10| 50k 101k| 12k 7376 12 9 70 0 0 9| 12M 8637k| 0 18k| 3.1 3.7 5.1| 0 0 14| 49k 89k| 12k 7169 13 9 70 0 0 8| 12M 8452k| 0 9830B| 2.9 3.6 5.1| 2 0 34| 48k 92k| 11k 7172 12 9 70 0 0 9| 12M 8644k| 0 9830B| 2.7 3.6 5| 0 0 12| 49k 93k| 11k 7091 12 9 69 0 0 9| 12M 8658k| 0 13k| 2.7 3.6 5| 0 0 15| 49k 105k| 12k 7242 12 9 69 0 0 9| 12M 8912k| 0 13k| 2.9 3.6 5| 0 0 11| 50k 103k| 11k 7320 12 9 69 0 0 9| 12M 8722k| 0 77k| 3 3.6 5| 0 0 11| 49k 103k| 11k 7313 13 9 68 0 0 9| 12M 8786k| 0 21k| 3 3.6 5| 0 0 16| 50k 101k| 11k 7435 12 9 70 0 0 9| 12M 8388k| 0 66k| 2.7 3.5 5| 5 0 10| 48k 103k| 11k 7399 12 9 69 0 0 10| 12M 8622k| 0 9830B| 2.9 3.6 5| 6 0 15| 49k 96k| 11k 7301 12 9 69 0 0 9| 12M 8625k| 0 9830B| 3 3.6 5| 6 0 16| 49k 102k| 11k 7256 12 9 69 0 0 9| 12M 8847k| 0 9830B| 3.4 3.6 5| 2 0 12| 50k 100k| 11k 7332 13 9 68 0 0 9| 12M 9005k| 0 9830B| 3.7 3.7 5| 2 0 20| 50k 103k| 11k 7125 13 9 68 0 0 9| 12M 8844k| 0 9830B| 4.5 3.9 5.1| 1 0 23| 50k 108k| 11k 7332 13 10 68 0 0 9| 13M 9000k| 0 9830B| 4.4 3.9 5| 4 0 13| 51k 93k| 12k 7233 13 10 68 0 0 9| 13M 9368k| 0 11k| 6.2 4.2 5.2| 2 0 8| 52k 87k| 12k 7365 13 9 69 0 0 9| 13M 9259k| 0 9830B| 6.2 4.3 5.2| 0 0 10| 51k 92k| 12k 7395 15 10 66 0 0 9| 12M 8785k| 0 13k| 6.1 4.3 5.2| 5 0 45| 49k 135k| 11k 7516 14 10 65 0 0 10| 12M 8661k| 0 70k| 7.1 4.5 5.2| 11 0 17| 50k 157k| 12k 7518 15 11 63 0 0 11| 13M 9044k| 0 14k| 7.7 4.7 5.3| 3 0 14| 52k 136k| 12k 7517 14 11 64 0 0 10| 13M 9218k| 0 9830B| 7.9 4.8 5.3| 0 0 13| 52k 125k| 12k 7445 14 10 64 0 0 11| 14M 9925k| 0 9830B| 7.9 4.8 5.3| 0 0 11| 54k 106k| 12k 7415 13 10 66 0 0 10| 13M 9696k| 0 67k| 7.7 4.8 5.3| 0 0 11| 53k 100k| 12k 7352 13 10 67 0 0 10| 13M 9255k| 0 9830B| 8.1 5 5.4| 0 0 11| 52k 98k| 12k 7373 12 8 71 0 0 9| 12M 8433k| 0 9830B| 7.7 4.9 5.4| 0 0 12| 48k 96k| 11k 7195 12 8 72 0 0 8| 12M 8120k| 0 9830B| 7.7 5 5.4| 0 0 19| 47k 92k| 11k 7241 11 8 72 0 0 8| 11M 8053k| 0 9830B| 7.7 5 5.4| 0 0 14| 47k 88k| 11k 7179 11 8 72 0 0 8| 11M 8074k| 0 9830B| 8.1 5.2 5.4| 1 0 11| 47k 90k| 11k 7220 12 8 71 0 0 8| 12M 8218k| 0 9830B| 8.3 5.2 5.5| 1 0 11| 47k 93k| 11k 7065 12 9 72 0 0 8| 12M 8235k| 0 13k| 9 5.4 5.5| 2 0 17| 48k 95k| 11k 7278 12 9 71 0 0 8| 11M 8213k| 0 70k| 8.9 5.5 5.5| 8 0 16| 47k 100k| 11k 7129 12 9 70 0 0 8| 11M 8211k| 0 9830B| 9.4 5.6 5.6| 2 0 13| 47k 102k| 11k 7142 12 9 71 0 0 8| 12M 8464k| 0 9830B| 10 5.9 5.7| 3 0 13| 48k 97k| 11k 7082 12 8 71 0 0 8| 12M 8402k| 0 9830B| 11 6 5.7| 1 0 9| 48k 96k| 11k 7135 11 8 71 0 0 9| 12M 8357k| 0 9830B| 9.8 5.9 5.7| 0 0 10| 47k 96k| 11k 7073 12 8 71 0 0 9| 12M 8393k| 0 11k| 9.9 6 5.7| 1 0 12| 47k 92k| 11k 6885 11 8 71 0 0 9| 11M 8348k| 0 64k| 10 6.2 5.8| 6 0 18| 47k 100k| 11k 6980 11 8 72 0 0 8| 11M 8290k| 0 9830B| 14 7 6| 0 0 12| 47k 96k| 11k 7000 13 9 69 0 0 9| 12M 8683k| 0 9830B| 13 6.9 6| 0 0 28| 49k 89k| 11k 7083 13 9 68 0 0 9| 13M 9172k| 0 9830B| 13 7 6.1| 0 0 14| 51k 96k| 12k 7357 13 10 68 0 0 9| 12M 8999k| 0 9830B| 13 7.1 6.1| 0 0 11| 50k 102k| 12k 7422 14 10 67 0 0 9| 12M 8817k| 0 138k| 13 7.2 6.1| 0 0 36| 50k 122k| 12k 7240 14 10 66 0 0 9| 12M 8564k| 0 110k| 13 7.4 6.2| 3 0 22| 50k 164k| 12k 7302 14 10 66 0 0 9| 12M 8591k| 0 139k| 13 7.3 6.2| 2 0 11| 50k 139k| 12k 7416 14 10 67 0 0 9| 13M 8776k| 0 9830B| 12 7.4 6.2| 0 0 10| 50k 111k| 12k 7325 12 9 69 0 0 9| 12M 8674k| 0 9830B| 12 7.4 6.2| 0 0 14| 49k 99k| 12k 7354 13 9 69 0 0 9| 12M 8688k| 0 9830B| 12 7.4 6.2| 2 0 13| 49k 99k| 11k 7268 12 9 70 0 0 9| 12M 8450k| 0 14k| 12 7.4 6.2| 0 0 13| 48k 95k| 11k 7238 12 8 72 0 0 8| 11M 8189k| 0 9830B| 11 7.4 6.2| 2 0 14| 47k 90k| 11k 7291 11 8 74 0 0 8| 11M 7815k| 0 9830B| 13 7.8 6.3| 0 0 16| 46k 85k| 11k 7260 10 8 74 0 0 8| 11M 7947k| 0 14k| 13 7.9 6.4| 3 0 11| 46k 76k| 11k 6884 10 7 75 0 0 7| 11M 7654k| 0 13k| 15 8.4 6.6| 2 0 13| 44k 79k| 11k 6749 9 7 76 0 0 7| 10M 7458k| 0 9830B| 14 8.3 6.6| 0 0 10| 44k 87k| 10k 6682 10 7 75 0 0 7| 10M 7410k| 0 53k| 14 8.4 6.6| 0 0 13| 44k 92k| 10k 6710 10 8 76 0 0 7| 10M 7315k| 0 48k| 13 8.3 6.6| 0 0 10| 43k 89k|9904 6805 10 7 76 0 0 7| 10M 7131k| 0 9830B| 13 8.3 6.6| 0 0 14| 43k 90k|9589 6967 10 7 75 0 0 7| 10M 7073k| 0 13k| 12 8.3 6.6| 6 0 18| 42k 99k|9492 7072 11 8 74 0 0 7| 10M 7395k| 0 58k| 12 8.2 6.6| 6 0 14| 44k 103k|9674 7188 10 8 74 0 0 7| 11M 7694k| 0 9830B| 11 8.2 6.6| 2 0 10| 45k 88k|9831 7138 10 7 75 0 0 7| 11M 7677k| 0 9830B| 12 8.4 6.6| 0 0 10| 45k 82k|9837 7177 10 7 76 0 0 7| 10M 7219k| 0 9830B| 12 8.4 6.7| 0 0 16| 43k 87k|9660 6971 9 7 77 0 0 6| 10M 7071k| 0 9830B| 11 8.3 6.6| 0 0 10| 42k 86k|9757 6957 10 7 77 0 0 7| 10M 7153k| 0 13k| 11 8.4 6.7| 1 0 22| 43k 78k|9739 7148 10 7 76 0 0 7| 10M 7380k| 0 13k| 13 8.7 6.8| 0 0 9| 44k 80k| 10k 7219 10 7 76 0 0 7| 10M 7406k| 0 13k| 12 8.6 6.8| 1 0 11| 44k 83k| 10k 7157 12 8 73 0 0 7| 10M 7137k| 0 13k| 11 8.4 6.7| 9 0 45| 43k 131k| 10k 7094 11 8 72 0 0 8| 10M 7207k| 0 9830B| 10 8.3 6.7| 1 0 10| 44k 152k| 11k 7140 12 9 71 0 0 8| 11M 7470k| 0 27k| 9.3 8.2 6.7| 2 0 13| 46k 127k| 11k 7211 12 9 70 0 0 9| 12M 8019k| 0 9830B| 9 8.2 6.7| 1 0 6| 47k 101k| 11k 7214 12 9 71 0 0 9| 12M 8341k| 0 9830B| 8.3 8.1 6.6| 3 0 11| 47k 84k| 11k 7219 11 7 74 0 0 8| 11M 7868k| 0 9830B| 7.6 7.9 6.6| 1 0 8| 46k 87k| 11k 7131 10 7 75 0 0 7| 11M 7473k| 0 18k| 7.2 7.8 6.6| 0 0 9| 44k 83k| 10k 7121 9 7 77 0 0 7| 10M 7199k| 0 9830B| 6.6 7.7 6.5| 0 0 12| 43k 77k| 10k 7158 8 6 79 0 0 7|9926k 6836k| 0 9830B| 6.1 7.6 6.5| 1 0 8| 41k 77k|9540 7188 8 6 80 0 0 6|9677k 6700k| 0 9830B| 5.6 7.4 6.5| 0 0 9| 41k 66k|9341 7022 7 5 82 0 0 6|9288k 6355k| 0 14k| 5.2 7.3 6.4| 0 0 8| 39k 62k|9211 6843 7 5 82 0 0 6|8843k 6095k| 0 9830B| 4.8 7.2 6.4| 0 0 8| 38k 66k|8828 6758 7 5 82 0 0 6|8557k 5993k| 0 13k| 4.4 7.1 6.3| 1 0 12| 38k 70k|8588 6722 7 5 83 0 0 5|8174k 5676k| 0 16k| 4.1 7 6.3| 0 0 11| 36k 78k|8144 6618 7 5 82 0 0 5|8325k 5828k| 0 9830B| 3.7 6.9 6.3| 1 0 11| 37k 78k|8364 6675 8 6 80 0 0 6|8738k 6220k| 0 9830B| 3.4 6.7 6.2| 0 0 14| 39k 82k|8506 6689 9 6 79 0 0 6|9161k 6565k| 0 9830B| 3.2 6.6 6.2| 0 0 9| 40k 84k|8770 6633 8 6 79 0 0 6|9315k 6616k| 0 9830B| 2.9 6.5 6.2| 0 0 9| 40k 77k|8844 6679 8 6 78 0 0 6|9506k 6819k| 0 9830B| 2.7 6.4 6.2| 2 0 10| 41k 74k|8743 6843 9 7 78 0 0 6|9481k 6765k| 0 9830B| 2.5 6.3 6.1| 2 0 18| 41k 85k|8911 6804 8 6 79 0 0 6|9397k 6663k| 0 13k| 2.3 6.2 6.1| 0 0 9| 41k 82k|8989 6845 9 7 77 0 0 6|9732k 6765k| 0 9830B| 2.2 6.1 6.1| 3 0 18| 41k 72k|9168 6916 8 7 78 0 0 7|9871k 6700k| 0 9830B| 2.2 6.1 6| 0 0 7| 41k 76k|9155 6935 9 6 78 0 0 6|9772k 6711k| 0 9830B| 2 6 6| 0 0 10| 41k 75k|9085 6744 10 7 76 0 0 7|9361k 6501k| 0 13k| 1.9 5.9 6| 1 0 39| 40k 115k|9030 6723 11 8 73 0 0 7|9831k 6844k| 0 9830B| 1.7 5.8 5.9| 0 0 12| 42k 144k|9639 6716 11 8 74 0 0 8| 10M 6913k| 0 18k| 1.8 5.7 5.9| 0 0 6| 43k 109k| 10k 6945 11 8 74 0 0 8| 11M 7346k| 0 38k| 1.6 5.6 5.9| 2 0 9| 44k 87k| 10k 7047 10 7 76 0 0 7| 10M 7147k| 0 82k| 1.5 5.5 5.9| 1 0 12| 43k 81k|9820 6942 9 7 77 0 0 7| 9.9M 6957k| 0 11k| 1.4 5.4 5.8| 0 0 9| 42k 79k|9475 6768 9 7 78 0 0 6|9602k 6597k| 0 9830B| 1.2 5.3 5.8| 0 0 11| 40k 80k|8916 6728 8 6 80 0 0 6|9115k 6146k| 0 9830B| 1.2 5.3 5.8| 1 0 9| 38k 73k|8590 6662 7 5 83 0 0 5|8426k 5678k| 0 9830B| 1.1 5.2 5.7| 0 0 9| 37k 68k|8257 6586 6 4 84 0 0 5|7898k 5440k| 0 9830B| 1 5.1 5.7| 0 0 5| 35k 60k|7739 6405 7 5 84 0 0 5|7634k 5303k| 0 94k| 1.2 5 5.7| 0 0 11| 35k 67k|7725 6449 6 5 84 0 0 5|7765k 5485k| 0 21k| 1.1 5 5.7| 0 0 8| 35k 70k|7948 6350 7 5 83 0 0 5|7803k 5452k| 0 112k| 1.1 4.9 5.6| 0 0 12| 35k 75k|7938 6248 8 5 82 0 0 5|8024k 5693k| 0 107k| 1.2 4.8 5.6| 1 0 18| 36k 84k|7882 6416 7 5 82 0 0 5|8300k 5918k| 0 14k| 1.1 4.8 5.6| 3 0 11| 37k 81k|8191 6540 8 6 80 0 0 6|8396k 5990k| 0 9830B| 1 4.7 5.5| 2 0 16| 37k 86k|8253 6200 8 6 80 0 0 6|8752k 6305k| 0 13k| 0.9 4.6 5.5| 4 0 10| 39k 89k|8395 6314 *De:* freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-u sers-bounces@lists.freeswitch.org] *En nombre de *Ken Rice *Enviado el:* lunes, 22 de octubre de 2012 19:00 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes Keeping in mind that you are using a sangoma transcoder board, that means you are handling the equivalent of 440 concurrent calls worth of media for people not using sangoma transcoding hardware (yes theres 4 call legs in that scenario, 1 from endpoint A to FS, 1 from FS to D100, 1 From D100 back to FS, 1 from FS back out to the other end point). That doesn’t sound like that big of a problem... You will have a fair bit of network traffic that way (the D100 shows up as another network interface as you are probably aware)... I would be looking at something like iotop or something else to see where that load is coming from Also, you have plenty of CPU free... So there is either a real problem here, or this is one of those things where the number of threads handling RTP and SIP are hanging around the run queue long enough to show up in the load average... On 10/22/12 3:53 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: Ken. There is an average of 220 concurrent calls. Machine : DL380 G7 – 2 Quad Core – 8 Gig RAM. Centos 5.5 64bits Ricarod.- *De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Ken Rice *Enviado el:* lunes, 22 de octubre de 2012 17:47 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes How many concurrent calls are you doing on that box? On 10/22/12 3:33 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: Hi Ken. I mounted the db directory in RAM as you mention, but the “load average” still is high. You can see this here : Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 132G 8.6G 117G 7% / proc 0 0 0 - /proc sysfs 0 0 0 - /sys devpts 0 0 0 - /dev/pts /dev/cciss/c0d0p1 104M 20M 79M 21% /boot tmpfs 4.2G 0 4.2G 0% /dev/shm none 0 0 0 - /proc/sys/fs/binfmt_misc sunrpc 0 0 0 - /var/lib/nfs/rpc_pipefs *tmpfs 4.2G 1.3M 4.2G 1% /usr/local/freeswitch/db * This is a resume from uptime and total calls for my freeswitch : 17:20:07 up 17 days, Total Calls : 16:26, 3 users, load average: 4.77, 4.50, 4.66 , Total Calls : 209, Total sessions: 440 17:20:12 up 17 days, Total Calls : 16:27, 3 users, load average: 6.79, 4.92, 4.80 , Total Calls : 245, Total sessions: 510 17:20:17 up 17 days, Total Calls : 16:27, 3 users, load average: 6.25, 4.84, 4.77 , Total Calls : 235, Total sessions: 488 17:20:22 up 17 days, Total Calls : 16:27, 3 users, load average: 6.63, 4.94, 4.81 , Total Calls : 233, Total sessions: 484 17:20:27 up 17 days, Total Calls : 16:27, 3 users, load average: 6.42, 4.92, 4.80 , Total Calls : 226, Total sessions: 470 17:20:32 up 17 days, Total Calls : 16:27, 3 users, load average: 5.90, 4.84, 4.78 , Total Calls : 218, Total sessions: 454 17:20:37 up 17 days, Total Calls : 16:27, 3 users, load average: 5.43, 4.76, 4.75 , Total Calls : 198, Total sessions: 416 17:20:43 up 17 days, Total Calls : 16:27, 3 users, load average: 4.99, 4.68, 4.72 , Total Calls : 183, Total sessions: 384 17:20:48 up 17 days, Total Calls : 16:27, 3 users, load average: 4.59, 4.60, 4.70 , Total Calls : 172, Total sessions: 366 17:20:53 up 17 days, Total Calls : 16:27, 3 users, load average: 4.31, 4.54, 4.68 , Total Calls : 184, Total sessions: 388 17:20:58 up 17 days, Total Calls : 16:27, 3 users, load average: 3.96, 4.47, 4.65 , Total Calls : 207, Total sessions: 442 17:21:03 up 17 days, Total Calls : 16:27, 3 users, load average: 4.04, 4.48, 4.65 , Total Calls : 203, Total sessions: 436 17:21:08 up 17 days, Total Calls : 16:27, 3 users, load average: 3.72, 4.40, 4.63 , Total Calls : 198, Total sessions: 418 17:21:13 up 17 days, Total Calls : 16:28, 3 users, load average: 3.47, 4.32, 4.60 , Total Calls : 198, Total sessions: 422 17:21:18 up 17 days, Total Calls : 16:28, 3 users, load average: 4.23, 4.47, 4.65 , Total Calls : 194, Total sessions: 410 17:21:23 up 17 days, Total Calls : 16:28, 3 users, load average: 8.30, 5.31, 4.92 , Total Calls : 211, Total sessions: 450 17:21:28 up 17 days, Total Calls : 16:28, 3 users, load average: 8.11, 5.32, 4.92 , Total Calls : 251, Total sessions: 536 17:21:33 up 17 days, Total Calls : 16:28, 3 users, load average: 8.74, 5.49, 4.98 , Total Calls : 250, Total sessions: 530 17:21:38 up 17 days, Total Calls : 16:28, 3 users, load average: 9.48, 5.70, 5.05 , Total Calls : 236, Total sessions: 496 17:21:43 up 17 days, Total Calls : 16:28, 3 users, load average: 9.45, 5.76, 5.07 , Total Calls : 233, Total sessions: 492 17:21:48 up 17 days, Total Calls : 16:28, 3 users, load average: 9.57, 5.84, 5.10 , Total Calls : 235, Total sessions: 496 17:21:53 up 17 days, Total Calls : 16:28, 3 users, load average: 10.24, 6.04, 5.17 , Total Calls : 222, Total sessions: 466 17:21:58 up 17 days, Total Calls : 16:28, 3 users, load average: 10.46, 6.16, 5.21 , Total Calls : 215, Total sessions: 452 17:22:03 up 17 days, Total Calls : 16:28, 3 users, load average: 11.07, 6.36, 5.28 , Total Calls : 208, Total sessions: 430 17:22:08 up 17 days, Total Calls : 16:28, 3 users, load average: 11.22, 6.47, 5.32 , Total Calls : 202, Total sessions: 422 17:22:14 up 17 days, Total Calls : 16:29, 3 users, load average: 11.52, 6.61, 5.37 , Total Calls : 240, Total sessions: 500 17:22:19 up 17 days, Total Calls : 16:29, 3 users, load average: 11.96, 6.78, 5.44 , Total Calls : 239, Total sessions: 498 17:22:24 up 17 days, Total Calls : 16:29, 3 users, load average: 14.29, 7.35, 5.63 , Total Calls : 250, Total sessions: 516 17:22:29 up 17 days, Total Calls : 16:29, 3 users, load average: 13.78, 7.36, 5.64 , Total Calls : 236, Total sessions: 490 17:22:34 up 17 days, Total Calls : 16:29, 3 users, load average: 12.92, 7.29, 5.63 , Total Calls : 209, Total sessions: 436 17:22:39 up 17 days, Total Calls : 16:29, 3 users, load average: 11.97, 7.18, 5.60 , Total Calls : 180, Total sessions: 372 17:22:44 up 17 days, Total Calls : 16:29, 3 users, load average: 11.01, 7.06, 5.57 , Total Calls : 161, Total sessions: 342 17:22:49 up 17 days, Total Calls : 16:29, 3 users, load average: 11.57, 7.24, 5.64 , Total Calls : 175, Total sessions: 370 17:22:54 up 17 days, Total Calls : 16:29, 3 users, load average: 11.12, 7.22, 5.64 , Total Calls : 181, Total sessions: 382 17:22:59 up 17 days, Total Calls : 16:29, 3 users, load average: 11.19, 7.30, 5.67 , Total Calls : 178, Total sessions: 366 17:23:04 up 17 days, Total Calls : 16:29, 3 users, load average: 11.10, 7.35, 5.69 , Total Calls : 173, Total sessions: 358 17:23:09 up 17 days, Total Calls : 16:30, 3 users, load average: 11.57, 7.51, 5.76 , Total Calls : 182, Total sessions: 378 17:23:14 up 17 days, Total Calls : 16:30, 3 users, load average: 11.36, 7.53, 5.77 , Total Calls : 187, Total sessions: 386 17:23:19 up 17 days, Total Calls : 16:30, 3 users, load average: 12.21, 7.77, 5.86 , Total Calls : 190, Total sessions: 402 17:23:24 up 17 days, Total Calls : 16:30, 3 users, load average: 12.04, 7.81, 5.88 , Total Calls : 239, Total sessions: 496 17:23:29 up 17 days, Total Calls : 16:30, 3 users, load average: 12.27, 7.93, 5.93 , Total Calls : 257, Total sessions: 532 17:23:34 up 17 days, Total Calls : 16:30, 3 users, load average: 12.73, 8.09, 5.99 , Total Calls : 242, Total sessions: 506 17:23:39 up 17 days, Total Calls : 16:30, 3 users, load average: 12.67, 8.16, 6.03 , Total Calls : 233, Total sessions: 482 17:23:45 up 17 days, Total Calls : 16:30, 3 users, load average: 12.38, 8.17, 6.04 , Total Calls : 238, Total sessions: 492 17:23:50 up 17 days, Total Calls : 16:30, 3 users, load average: 12.67, 8.30, 6.10 , Total Calls : 236, Total sessions: 482 17:23:55 up 17 days, Total Calls : 16:30, 3 users, load average: 13.09, 8.46, 6.16 , Total Calls : 222, Total sessions: 458 17:24:00 up 17 days, Total Calls : 16:30, 3 users, load average: 12.37, 8.39, 6.15 , Total Calls : 221, Total sessions: 470 17:24:05 up 17 days, Total Calls : 16:30, 3 users, load average: 11.78, 8.33, 6.14 , Total Calls : 200, Total sessions: 422 17:24:10 up 17 days, Total Calls : 16:31, 3 users, load average: 13.64, 8.77, 6.30 , Total Calls : 214, Total sessions: 452 17:24:15 up 17 days, Total Calls : 16:31, 3 users, load average: 13.98, 8.93, 6.36 , Total Calls : 255, Total sessions: 536 17:24:20 up 17 days, Total Calls : 16:31, 3 users, load average: 14.23, 9.06, 6.42 , Total Calls : 252, Total sessions: 524 17:24:25 up 17 days, Total Calls : 16:31, 3 users, load average: 15.73, 9.46, 6.56 , Total Calls : 260, Total sessions: 536 17:24:30 up 17 days, Total Calls : 16:31, 3 users, load average: 15.03, 9.42, 6.56 , Total Calls : 246, Total sessions: 508 17:24:35 up 17 days, Total Calls : 16:31, 3 users, load average: 15.11, 9.52, 6.61 , Total Calls : 230, Total sessions: 478 17:24:40 up 17 days, Total Calls : 16:31, 3 users, load average: 14.94, 9.58, 6.64 , Total Calls : 194, Total sessions: 408 17:24:45 up 17 days, Total Calls : 16:31, 3 users, load average: 14.54, 9.59, 6.66 , Total Calls : 189, Total sessions: 396 17:24:50 up 17 days, Total Calls : 16:31, 3 users, load average: 13.94, 9.55, 6.66 , Total Calls : 202, Total sessions: 438 17:24:55 up 17 days, Total Calls : 16:31, 3 users, load average: 13.30, 9.49, 6.66 , Total Calls : 221, Total sessions: 470 17:25:00 up 17 days, Total Calls : 16:31, 3 users, load average: 12.88, 9.46, 6.67 , Total Calls : 220, Total sessions: 478 17:25:05 up 17 days, Total Calls : 16:31, 3 users, load average: 14.25, 9.80, 6.79 , Total Calls : 219, Total sessions: 480 17:25:10 up 17 days, Total Calls : 16:32, 3 users, load average: 13.67, 9.76, 6.79 , Total Calls : 222, Total sessions: 490 17:25:16 up 17 days, Total Calls : 16:32, 3 users, load average: 13.05, 9.69, 6.79 , Total Calls : 214, Total sessions: 470 17:25:21 up 17 days, Total Calls : 16:32, 3 users, load average: 13.53, 9.85, 6.85 , Total Calls : 210, Total sessions: 450 17:25:26 up 17 days, Total Calls : 16:32, 3 users, load average: 13.81, 9.97, 6.91 , Total Calls : 250, Total sessions: 524 17:25:31 up 17 days, Total Calls : 16:32, 3 users, load average: 12.70, 9.80, 6.87 , Total Calls : 258, Total sessions: 536 17:25:36 up 17 days, Total Calls : 16:32, 3 users, load average: 11.68, 9.64, 6.83 , Total Calls : 255, Total sessions: 528 17:25:41 up 17 days, Total Calls : 16:32, 3 users, load average: 10.83, 9.49, 6.80 , Total Calls : 246, Total sessions: 524 17:25:46 up 17 days, Total Calls : 16:32, 3 users, load average: 10.04, 9.35, 6.77 , Total Calls : 262, Total sessions: 564 17:25:51 up 17 days, Total Calls : 16:32, 3 users, load average: 9.32, 9.21, 6.74 , Total Calls : 238, Total sessions: 510 17:25:56 up 17 days, Total Calls : 16:32, 3 users, load average: 8.57, 9.06, 6.70 , Total Calls : 220, Total sessions: 476 17:26:01 up 17 days, Total Calls : 16:32, 3 users, load average: 7.88, 8.91, 6.67 , Total Calls : 228, Total sessions: 498 What else could be happening. Did I mention that I’m using two transcoding cards from Sangoma? Regards, Ricardo.- *De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Ken Rice *Enviado el:* lunes, 22 de octubre de 2012 16:50 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes Load Average is the number of threads/processes waiting in the kernel’s run queue at sample time... Supposedly number of cores is the how much your load avg can get to before your boxes is exhausted but this isnt exactly true on modern hardware with highly multi-threaded software like FreeSWITCH... Where did I get the 100+ number? From looking at ‘top’ or ‘w’ on a box to see what the current load avg is... Sqlite is used all over FreeSWITCH from the coredb to mod_sofia and everything in between.... When you type ‘show channels’ from fs_cli it doesn’t query the structures in RAM to show you that output, it pulls it from the sqlite db... That being said, you can test quickly to see if sqlite is hammering the disk at some point by shifting freeswitch/db dir to a ram drive. Example code for linux such as Centos or Debian assuming freeswitch is installed to /usr/local/freeswitch Shutdown freeswitch rm –rf /usr/local/freeswitch/db/* mount –t tmpfs tmpfs /usr/local/freeswitch/db Startup freeswitch Once you have done that stick it back in rotation... Note this will not affect anything but voicemail if you have voicemail running on that box and not using an external DB via mod_voicemail’s DSN settings you’ll lose the voicemail DB... (I would assume this is not the case since you are using this primarily as a Transcoder) K On 10/22/12 2:30 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: Hi Ken. Maybe that is the problem. I was sure that the “load average” can’t go more than #cores in your system. In my case I have 8 cores, so a “load average” of 12 is high from my point of view. Where did you get the number 100 for this parameter? Anyway, what is running a sqlite in freeswitch. Does this have to do with the core.db file? Or with the cdr_sqlite module?. Thanks Ricardo.- *De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Ken Rice *Enviado el:* lunes, 22 de octubre de 2012 16:18 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes Could it be just a normal sqlite running a vacuum? Sqlite does this every so often or if its calling a sync.... Also freeswitch with a load avg of 12 for a short period of time isnt anything to really worry about. Due to the number of threads that’s possible with FS (ie: 1 per call leg) its not unusual to see a box humming along and the load avg well above 100 in my experience... This is due to the number of threads waiting in the run queue when it was sampled... Now, that’s not saying that this is not specifically indicative or a problem but something else to think about On 10/22/12 1:52 PM, "Ricardo Martinez" <rmartinez@redvoiss.net> wrote: This is the gdb from the proccess consuming more cpu : (gdb) thread 7 [Switching to thread 7 (Thread 31781)]#0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 (gdb) bt #0 0x0000003c03c0e047 in fsync () from /lib64/libpthread.so.0 #1 0x00002aede19276de in unixSync (id=0x2aaaac726380, dataOnly=<value optimized out>) at ./src/os_unix.c:1229 #2 0x00002aede1951178 in syncJournal (pPager=0x1a1c200) at ./src/pager.c:2229 #3 0x00002aede1953b65 in sqlite3pager_sync (pPager=0x1a1c200, zMaster=0x0, nTrunc=0) at ./src/pager.c:3770 #4 0x00002aede193d07b in sqlite3BtreeSync (p=<value optimized out>, zMaster=0x0) at ./src/btree.c:6520 #5 0x00002aede1931f15 in sqlite3VdbeHalt (p=0x2aaaad17ac70) at ./src/vdbeaux.c:1062 #6 0x00002aede1962613 in sqlite3VdbeExec (p=0x2aaaad17ac70) at ./src/vdbe.c:2360 #7 0x00002aede193036c in sqlite3Step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:236 #8 sqlite3_step (pStmt=0x2aaaad17ac70) at ./src/vdbeapi.c:289 #9 0x00002aede1938978 in sqlite3_exec (db=0x1a1f070, zSql=0x2aede199e717 "COMMIT", xCallback=0, pArg=0x0, pzErrMsg=0x4178bed0) at ./src/legacy.c:78 #10 0x00002aede18ae0c1 in switch_core_db_exec (db=0x1a1f070, sql=0x2aede199e717 "COMMIT", callback=0, data=0x0, errmsg=0x4178bf38) at src/switch_core_db.c:93 #11 0x00002aede1897c18 in switch_cache_db_execute_sql_real (dbh=0x1a2dce0, sql=0x2aede199e717 "COMMIT", err=0x0) at src/switch_core_sqldb.c:445 #12 0x00002aede18985c3 in switch_cache_db_persistant_execute_trans (dbh=0x1a2dce0, sql=0x5f03bd0 "delete from calls where (caller_uuid='e64319c5-7f91-47a1-9561-42da291e499a' or callee_uuid='e64319c5-7f91-47a1-9561-42da291e499a') and hostname=' transcoder.red.net <http://transcoder.red.net> <http://transcoder.red.net> <http://transcoder.red.net> <http://transcoder.red.net> ';\ndelete from calls where (call"..., retries=1) at src/switch_core_sqldb.c:787 #13 0x00002aede189bf1c in switch_core_sql_thread (thread=<value optimized out>, obj=<value optimized out>) at src/switch_core_sqldb.c:1026 #14 0x0000003c03c0673d in start_thread () from /lib64/libpthread.so.0 #15 0x0000003c030d3f6d in clone () from /lib64/libc.so.6 *De:* Ricardo Martinez [mailto:rmartinez@redvoiss.net<rmartinez@redvoiss.net>] *Enviado el:* lunes, 22 de octubre de 2012 15:50 *Para:* 'FreeSWITCH Users Help' *Asunto:* RE: [Freeswitch-users] CPU Spikes every 20 minutes AS far as i can see there is no java running. This is the top at the moment of the spike : top - 15:48:29 up 17 days, 14:55, 3 users, load average: 14.07, 8.87, 6.48 Tasks: 158 total, 1 running, 157 sleeping, 0 stopped, 0 zombie Cpu(s): 12.6%us, 9.0%sy, 0.0%ni, 66.1%id, 2.7%wa, 0.3%hi, 9.2%si, 0.0%st Mem: 8171500k total, 8041556k used, 129944k free, 177688k buffers Swap: 10223608k total, 0k used, 10223608k free, 6951988k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31772 root 15 0 870m 481m 6880 S 212.4 6.0 9523:51 freeswitch 4210 root 19 0 36344 2940 1444 S 1.0 0.0 56:21.08 sngtc_server 29586 root 15 0 12740 1132 808 R 0.3 0.0 0:00.02 top 1 root 15 0 10348 704 588 S 0.0 0.0 0:01.13 init 2 root RT -5 0 0 0 S 0.0 0.0 0:00.41 migration/0 3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0 4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0 5 root RT -5 0 0 0 S 0.0 0.0 0:00.83 migration/1 6 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1 7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1 8 root RT -5 0 0 0 S 0.0 0.0 0:00.14 migration/2 9 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/2 10 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/2 11 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/3 12 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/3 13 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/3 14 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/4 15 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/4 16 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/4 17 root RT -5 0 0 0 S 0.0 0.0 0:00.06 migration/5 18 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/5 19 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/5 20 root RT -5 0 0 0 S 0.0 0.0 0:00.07 migration/6 Regards, Ricardo.- *De:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] *En nombre de *Guillermo Ruiz Camauer *Enviado el:* lunes, 22 de octubre de 2012 15:38 *Para:* FreeSWITCH Users Help *Asunto:* Re: [Freeswitch-users] CPU Spikes every 20 minutes Do you have anything like Java running that could be doing garbage collection? What other processes are running on the machine? Does TOP show which process is causing the spike? Guillermo On Mon, Oct 22, 2012 at 3:29 PM, Ricardo Martinez <rmartinez@redvoiss.net> wrote: Hi Gabe. I'm mean between the spikes there is nothing different as regarding to the calls... the cps maintains the same.. there are no scripts running or crontabs running, the simultaneous calls are the same also. So, for some reason the "load average" start to grow (every 20 or 25 minutes) without any clear cause. I have debug with gdb the moment the server is with 12.5 of "load average" here : http://pastebin.freeswitch.org/20112 Can we detect a problem from here? Regards, Ricardo. -----Mensaje original----- De: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org<freeswitch-users-bounces@lists.freeswitch.org>] En nombre de Gabriel Gunderson Enviado el: lunes, 22 de octubre de 2012 13:31 Para: FreeSWITCH Users Help Asunto: Re: [Freeswitch-users] CPU Spikes every 20 minutes On Mon, Oct 22, 2012 at 8:37 AM, Ricardo Martinez <rmartinez@redvoiss.net> wrote:
The problem is that I'm having weird spikes of "load average" cpu every 20 minutes (aprox), without any reason, reaching near of 12.5 of load average for this machine.
What have you done to verify that it happens 'without any reason'? Gabe _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Ken *http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org *irc.freenode.net #freeswitch
participants (5)
-
Anthony Minessale
-
Gabriel Gunderson
-
Guillermo Ruiz Camauer
-
Ken Rice
-
Ricardo Martinez