Hi, We're currently experiencing a weird issue with one of our freeswitch servers where the CPU just goes berserk and stays at 100%. There are no active calls at that moment, but FS is still using 100% CPU. I was unable to get a debug using gdb, since freeswitch does not crash. We had this issue on 1.6.13 and then I upgraded to 1.6.15 this morning, but the problem still persists. There is currently only one active application running on this box which is using LUA. Maybe that is causing the issue? We checked the code but there are no weird loops etc. Perhaps we are missing something? Any ideas how to trace this? Regards, Grant
Hi, You can use the following commands: #get a core: gcore -o /home/gcore-$(date +%s) $(cat /var/run/freeswitch/freeswitch.pid) #check thread usage: top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1 #memory mapping: cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps you can get some info from fs: fs_cli -x "sofia global siptrace on" fs_cli -x "fsctl debug_level 10" fs_cli -x "sofia loglevel all 9" fs_cli -x "fsctl debug_sql " fs_cli -x "sofia global debug presence" Put everything into a jira so we can take a look. Regards, António On 03/10/2017 11:08 AM, Grant Bagdasarian wrote:
Hi,
We’re currently experiencing a weird issue with one of our freeswitch servers where the CPU just goes berserk and stays at 100%.
There are no active calls at that moment, but FS is still using 100% CPU.
I was unable to get a debug using gdb, since freeswitch does not crash.
We had this issue on 1.6.13 and then I upgraded to 1.6.15 this morning, but the problem still persists.
There is currently only one active application running on this box which is using LUA. Maybe that is causing the issue?
We checked the code but there are no weird loops etc. Perhaps we are missing something?
Any ideas how to trace this?
Regards,
Grant
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Saludos / Regards / Cumprimentos, António silva
Make sure you lua script has all while loops including session:ready() as a condition to continue. Its a common mistake that leads to loops spinning out of control on hungup channels. On Fri, Mar 10, 2017 at 6:11 AM, Antonio Silva <asilva@wirelessmundi.com> wrote:
Hi,
You can use the following commands:
#get a core: gcore -o /home/gcore-$(date +%s) $(cat /var/run/freeswitch/freeswitch.pid)
#check thread usage: top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1
#memory mapping: cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps
you can get some info from fs:
fs_cli -x "sofia global siptrace on" fs_cli -x "fsctl debug_level 10" fs_cli -x "sofia loglevel all 9" fs_cli -x "fsctl debug_sql " fs_cli -x "sofia global debug presence"
Put everything into a jira so we can take a look.
Regards, António
On 03/10/2017 11:08 AM, Grant Bagdasarian wrote:
Hi,
We’re currently experiencing a weird issue with one of our freeswitch servers where the CPU just goes berserk and stays at 100%.
There are no active calls at that moment, but FS is still using 100% CPU.
I was unable to get a debug using gdb, since freeswitch does not crash.
We had this issue on 1.6.13 and then I upgraded to 1.6.15 this morning, but the problem still persists.
There is currently only one active application running on this box which is using LUA. Maybe that is causing the issue?
We checked the code but there are no weird loops etc. Perhaps we are missing something?
Any ideas how to trace this?
Regards,
Grant
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.orghttp://www.freeswitchsolutions.com
Official FreeSWITCH Siteshttp://www.freeswitch.orghttp://confluence.freeswitch.orghttp://www.cluecon....
FreeSWITCH-users mailing listFreeSWITCH-users@lists.freeswitch.orghttp://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-usershttp://www.frees...
--
Saludos / Regards / Cumprimentos, António silva
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
Hi Antony and Antonio, Thank you for the tips. We haven’t had freeswitch going berserk anymore (not sure as to why it did perform without issue over the weekend), but will continue to monitor it and run the commands to trace the issue. Regards, Grant From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Anthony Minessale Sent: zaterdag 11 maart 2017 00:07 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] 100% CPU usage Make sure you lua script has all while loops including session:ready() as a condition to continue. Its a common mistake that leads to loops spinning out of control on hungup channels. On Fri, Mar 10, 2017 at 6:11 AM, Antonio Silva <asilva@wirelessmundi.com<mailto:asilva@wirelessmundi.com>> wrote: Hi, You can use the following commands: #get a core: gcore -o /home/gcore-$(date +%s) $(cat /var/run/freeswitch/freeswitch.pid) #check thread usage: top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1 #memory mapping: cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps you can get some info from fs: fs_cli -x "sofia global siptrace on" fs_cli -x "fsctl debug_level 10" fs_cli -x "sofia loglevel all 9" fs_cli -x "fsctl debug_sql " fs_cli -x "sofia global debug presence" Put everything into a jira so we can take a look. Regards, António On 03/10/2017 11:08 AM, Grant Bagdasarian wrote: Hi, We’re currently experiencing a weird issue with one of our freeswitch servers where the CPU just goes berserk and stays at 100%. There are no active calls at that moment, but FS is still using 100% CPU. I was unable to get a debug using gdb, since freeswitch does not crash. We had this issue on 1.6.13 and then I upgraded to 1.6.15 this morning, but the problem still persists. There is currently only one active application running on this box which is using LUA. Maybe that is causing the issue? We checked the code but there are no weird loops etc. Perhaps we are missing something? Any ideas how to trace this? Regards, Grant _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Saludos / Regards / Cumprimentos, António silva _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net<http://irc.freenode.net> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
Attached are the logs for: #check thread usage: top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1 #memory mapping: cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps We’re unable to run gcore since freeswitch hogs all available cpu’s at 100%. I tried changing the nice settings real-time but that didn’t work either. At one point we had a huge spike in inbound traffic towards our freeswitch (valid traffic). The traffic is handled by a lua application. What we saw was calls hanging in freeswitch while they’ve already been physically disconnected. After the huge spike freeswitch behaved normally for a couple of minutes and then suddenly started eating all the available cpu cores at 100%. We have some db queries which are fired in the lua application, so these may cause the application to hang, but we haven’t reproduced it yet. I’ll setup a test lab with the same configuration and fire sipp at it to see if I can reproduce the issue. However, I probably won’t be able to run gcore for debugging. Any other suggestions? Also we saw a lot of open udp ports and some had a huge read buffer size which did not decrease. [cid:image001.png@01D2A238.14965330] From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Grant Bagdasarian Sent: maandag 13 maart 2017 09:04 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] 100% CPU usage Hi Antony and Antonio, Thank you for the tips. We haven’t had freeswitch going berserk anymore (not sure as to why it did perform without issue over the weekend), but will continue to monitor it and run the commands to trace the issue. Regards, Grant From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Anthony Minessale Sent: zaterdag 11 maart 2017 00:07 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage Make sure you lua script has all while loops including session:ready() as a condition to continue. Its a common mistake that leads to loops spinning out of control on hungup channels. On Fri, Mar 10, 2017 at 6:11 AM, Antonio Silva <asilva@wirelessmundi.com<mailto:asilva@wirelessmundi.com>> wrote: Hi, You can use the following commands: #get a core: gcore -o /home/gcore-$(date +%s) $(cat /var/run/freeswitch/freeswitch.pid) #check thread usage: top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1 #memory mapping: cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps you can get some info from fs: fs_cli -x "sofia global siptrace on" fs_cli -x "fsctl debug_level 10" fs_cli -x "sofia loglevel all 9" fs_cli -x "fsctl debug_sql " fs_cli -x "sofia global debug presence" Put everything into a jira so we can take a look. Regards, António On 03/10/2017 11:08 AM, Grant Bagdasarian wrote: Hi, We’re currently experiencing a weird issue with one of our freeswitch servers where the CPU just goes berserk and stays at 100%. There are no active calls at that moment, but FS is still using 100% CPU. I was unable to get a debug using gdb, since freeswitch does not crash. We had this issue on 1.6.13 and then I upgraded to 1.6.15 this morning, but the problem still persists. There is currently only one active application running on this box which is using LUA. Maybe that is causing the issue? We checked the code but there are no weird loops etc. Perhaps we are missing something? Any ideas how to trace this? Regards, Grant _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Saludos / Regards / Cumprimentos, António silva _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net<http://irc.freenode.net> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
I already told you why a week ago. Your script most likely has an endless loop in it that does not check the condition session:ready() However, Feel free to contact consulting@freeswitch.org and we can formally diagnose it for a fee. On Tue, Mar 21, 2017 at 6:42 AM Grant Bagdasarian <gb@cm.nl> wrote:
Attached are the logs for:
#check thread usage:
top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1
#memory mapping:
cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps
We’re unable to run gcore since freeswitch hogs all available cpu’s at 100%.
I tried changing the nice settings real-time but that didn’t work either.
At one point we had a huge spike in inbound traffic towards our freeswitch (valid traffic).
The traffic is handled by a lua application.
What we saw was calls hanging in freeswitch while they’ve already been physically disconnected.
After the huge spike freeswitch behaved normally for a couple of minutes and then suddenly started eating all the available cpu cores at 100%.
We have some db queries which are fired in the lua application, so these may cause the application to hang, but we haven’t reproduced it yet.
I’ll setup a test lab with the same configuration and fire sipp at it to see if I can reproduce the issue.
However, I probably won’t be able to run gcore for debugging. Any other suggestions?
Also we saw a lot of open udp ports and some had a huge read buffer size which did not decrease.
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Grant Bagdasarian *Sent:* maandag 13 maart 2017 09:04
*To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] 100% CPU usage
Hi Antony and Antonio,
Thank you for the tips.
We haven’t had freeswitch going berserk anymore (not sure as to why it did perform without issue over the weekend), but will continue to monitor it and run the commands to trace the issue.
Regards,
Grant
*From:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org <freeswitch-users-bounces@lists.freeswitch.org>] *On Behalf Of *Anthony Minessale *Sent:* zaterdag 11 maart 2017 00:07 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] 100% CPU usage
Make sure you lua script has all while loops including session:ready() as a condition to continue. Its a common mistake that leads to loops spinning out of control on hungup channels.
On Fri, Mar 10, 2017 at 6:11 AM, Antonio Silva <asilva@wirelessmundi.com> wrote:
Hi,
You can use the following commands:
#get a core:
gcore -o /home/gcore-$(date +%s) $(cat /var/run/freeswitch/freeswitch.pid)
#check thread usage:
top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1
#memory mapping:
cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps
you can get some info from fs:
fs_cli -x "sofia global siptrace on"
fs_cli -x "fsctl debug_level 10"
fs_cli -x "sofia loglevel all 9"
fs_cli -x "fsctl debug_sql "
fs_cli -x "sofia global debug presence"
Put everything into a jira so we can take a look.
Regards, António
On 03/10/2017 11:08 AM, Grant Bagdasarian wrote:
Hi,
We’re currently experiencing a weird issue with one of our freeswitch servers where the CPU just goes berserk and stays at 100%.
There are no active calls at that moment, but FS is still using 100% CPU.
I was unable to get a debug using gdb, since freeswitch does not crash.
We had this issue on 1.6.13 and then I upgraded to 1.6.15 this morning, but the problem still persists.
There is currently only one active application running on this box which is using LUA. Maybe that is causing the issue?
We checked the code but there are no weird loops etc. Perhaps we are missing something?
Any ideas how to trace this?
Regards,
Grant
_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com
Official FreeSWITCH Sites
http://confluence.freeswitch.org
FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
--
Saludos / Regards / Cumprimentos,
António silva
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
--
Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH
☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call
☎ sip:888@conference.freeswitch.org ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s
https://www.youtube.com/watch?v=NLaDpGQuZDA _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
Anthony, we triple checked the script and its dependencies. There are no loops at all in the script. The imported scripts are also written by us and they too do not contain any loops. It’s just basic call handling with some playback and play_and_get_digits calls, and some pcall calls to query a database. From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Anthony Minessale Sent: dinsdag 21 maart 2017 16:02 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] 100% CPU usage I already told you why a week ago. Your script most likely has an endless loop in it that does not check the condition session:ready() However, Feel free to contact consulting@freeswitch.org<mailto:consulting@freeswitch.org> and we can formally diagnose it for a fee. On Tue, Mar 21, 2017 at 6:42 AM Grant Bagdasarian <gb@cm.nl<mailto:gb@cm.nl>> wrote: Attached are the logs for: #check thread usage: top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1 #memory mapping: cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps We’re unable to run gcore since freeswitch hogs all available cpu’s at 100%. I tried changing the nice settings real-time but that didn’t work either. At one point we had a huge spike in inbound traffic towards our freeswitch (valid traffic). The traffic is handled by a lua application. What we saw was calls hanging in freeswitch while they’ve already been physically disconnected. After the huge spike freeswitch behaved normally for a couple of minutes and then suddenly started eating all the available cpu cores at 100%. We have some db queries which are fired in the lua application, so these may cause the application to hang, but we haven’t reproduced it yet. I’ll setup a test lab with the same configuration and fire sipp at it to see if I can reproduce the issue. However, I probably won’t be able to run gcore for debugging. Any other suggestions? Also we saw a lot of open udp ports and some had a huge read buffer size which did not decrease. [cid:image001.png@01D2A238.14965330] From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Grant Bagdasarian Sent: maandag 13 maart 2017 09:04 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage Hi Antony and Antonio, Thank you for the tips. We haven’t had freeswitch going berserk anymore (not sure as to why it did perform without issue over the weekend), but will continue to monitor it and run the commands to trace the issue. Regards, Grant From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Anthony Minessale Sent: zaterdag 11 maart 2017 00:07 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage Make sure you lua script has all while loops including session:ready() as a condition to continue. Its a common mistake that leads to loops spinning out of control on hungup channels. On Fri, Mar 10, 2017 at 6:11 AM, Antonio Silva <asilva@wirelessmundi.com<mailto:asilva@wirelessmundi.com>> wrote: Hi, You can use the following commands: #get a core: gcore -o /home/gcore-$(date +%s) $(cat /var/run/freeswitch/freeswitch.pid) #check thread usage: top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1 #memory mapping: cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps you can get some info from fs: fs_cli -x "sofia global siptrace on" fs_cli -x "fsctl debug_level 10" fs_cli -x "sofia loglevel all 9" fs_cli -x "fsctl debug_sql " fs_cli -x "sofia global debug presence" Put everything into a jira so we can take a look. Regards, António On 03/10/2017 11:08 AM, Grant Bagdasarian wrote: Hi, We’re currently experiencing a weird issue with one of our freeswitch servers where the CPU just goes berserk and stays at 100%. There are no active calls at that moment, but FS is still using 100% CPU. I was unable to get a debug using gdb, since freeswitch does not crash. We had this issue on 1.6.13 and then I upgraded to 1.6.15 this morning, but the problem still persists. There is currently only one active application running on this box which is using LUA. Maybe that is causing the issue? We checked the code but there are no weird loops etc. Perhaps we are missing something? Any ideas how to trace this? Regards, Grant _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Saludos / Regards / Cumprimentos, António silva _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net<http://irc.freenode.net> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net<http://irc.freenode.net> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
Are you using ‘if (session:ready()) then’ in your scripts before you execute? Are you using virtual or physical hardware and what spec? From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Grant Bagdasarian Sent: 22 March 2017 07:31 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] 100% CPU usage Anthony, we triple checked the script and its dependencies. There are no loops at all in the script. The imported scripts are also written by us and they too do not contain any loops. It’s just basic call handling with some playback and play_and_get_digits calls, and some pcall calls to query a database. From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Anthony Minessale Sent: dinsdag 21 maart 2017 16:02 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage I already told you why a week ago. Your script most likely has an endless loop in it that does not check the condition session:ready() However, Feel free to contact consulting@freeswitch.org<mailto:consulting@freeswitch.org> and we can formally diagnose it for a fee. On Tue, Mar 21, 2017 at 6:42 AM Grant Bagdasarian <gb@cm.nl<mailto:gb@cm.nl>> wrote: Attached are the logs for: #check thread usage: top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1 #memory mapping: cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps We’re unable to run gcore since freeswitch hogs all available cpu’s at 100%. I tried changing the nice settings real-time but that didn’t work either. At one point we had a huge spike in inbound traffic towards our freeswitch (valid traffic). The traffic is handled by a lua application. What we saw was calls hanging in freeswitch while they’ve already been physically disconnected. After the huge spike freeswitch behaved normally for a couple of minutes and then suddenly started eating all the available cpu cores at 100%. We have some db queries which are fired in the lua application, so these may cause the application to hang, but we haven’t reproduced it yet. I’ll setup a test lab with the same configuration and fire sipp at it to see if I can reproduce the issue. However, I probably won’t be able to run gcore for debugging. Any other suggestions? Also we saw a lot of open udp ports and some had a huge read buffer size which did not decrease. [cid:image001.png@01D2A238.14965330] From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Grant Bagdasarian Sent: maandag 13 maart 2017 09:04 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage Hi Antony and Antonio, Thank you for the tips. We haven’t had freeswitch going berserk anymore (not sure as to why it did perform without issue over the weekend), but will continue to monitor it and run the commands to trace the issue. Regards, Grant From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Anthony Minessale Sent: zaterdag 11 maart 2017 00:07 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage Make sure you lua script has all while loops including session:ready() as a condition to continue. Its a common mistake that leads to loops spinning out of control on hungup channels. On Fri, Mar 10, 2017 at 6:11 AM, Antonio Silva <asilva@wirelessmundi.com<mailto:asilva@wirelessmundi.com>> wrote: Hi, You can use the following commands: #get a core: gcore -o /home/gcore-$(date +%s) $(cat /var/run/freeswitch/freeswitch.pid) #check thread usage: top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1 #memory mapping: cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps you can get some info from fs: fs_cli -x "sofia global siptrace on" fs_cli -x "fsctl debug_level 10" fs_cli -x "sofia loglevel all 9" fs_cli -x "fsctl debug_sql " fs_cli -x "sofia global debug presence" Put everything into a jira so we can take a look. Regards, António On 03/10/2017 11:08 AM, Grant Bagdasarian wrote: Hi, We’re currently experiencing a weird issue with one of our freeswitch servers where the CPU just goes berserk and stays at 100%. There are no active calls at that moment, but FS is still using 100% CPU. I was unable to get a debug using gdb, since freeswitch does not crash. We had this issue on 1.6.13 and then I upgraded to 1.6.15 this morning, but the problem still persists. There is currently only one active application running on this box which is using LUA. Maybe that is causing the issue? We checked the code but there are no weird loops etc. Perhaps we are missing something? Any ideas how to trace this? Regards, Grant _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Saludos / Regards / Cumprimentos, António silva _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net<http://irc.freenode.net> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net<http://irc.freenode.net> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ [http://www.itec-support.co.uk/wp-content/uploads/2016/07/email_logo.jpg] Shaun Stokes - Infrastructure Analyst T : 01453 700713 E : shaun.stokes@itec-support.co.uk W : www.itec-support.co.uk Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001 CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment. ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________
We had some issues using pcall functions like io.popen when trying to execute external commands, that block mod_lua, i end up replacing that with os.execute. Recently i also had some issue with cpu hitting 100%,for what i could compile from debugging was something related with malloc calls. after compiling fs with tcmalloc the issue disappear, you could try this approach as well. You can check jira: https://freeswitch.org/jira/browse/FS-9957 and also in the malling: "CPU sys load issue" for more info. Regards, António On 03/22/2017 09:52 AM, Shaun Stokes wrote:
Are you using ‘if (session:ready()) then’ in your scripts before you execute?
Are you using virtual or physical hardware and what spec?
*From:*freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Grant Bagdasarian *Sent:* 22 March 2017 07:31 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] 100% CPU usage
Anthony, we triple checked the script and its dependencies. There are no loops at all in the script. The imported scripts are also written by us and they too do not contain any loops.
It’s just basic call handling with some playback and play_and_get_digits calls, and some pcall calls to query a database.
*From:*freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Anthony Minessale *Sent:* dinsdag 21 maart 2017 16:02 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org>> *Subject:* Re: [Freeswitch-users] 100% CPU usage
I already told you why a week ago. Your script most likely has an endless loop in it that does not check the condition session:ready()
However, Feel free to contact consulting@freeswitch.org <mailto:consulting@freeswitch.org> and we can formally diagnose it for a fee.
On Tue, Mar 21, 2017 at 6:42 AM Grant Bagdasarian <gb@cm.nl <mailto:gb@cm.nl>> wrote:
Attached are the logs for:
#check thread usage:
top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1
#memory mapping:
cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps
We’re unable to run gcore since freeswitch hogs all available cpu’s at 100%.
I tried changing the nice settings real-time but that didn’t work either.
At one point we had a huge spike in inbound traffic towards our freeswitch (valid traffic).
The traffic is handled by a lua application.
What we saw was calls hanging in freeswitch while they’ve already been physically disconnected.
After the huge spike freeswitch behaved normally for a couple of minutes and then suddenly started eating all the available cpu cores at 100%.
We have some db queries which are fired in the lua application, so these may cause the application to hang, but we haven’t reproduced it yet.
I’ll setup a test lab with the same configuration and fire sipp at it to see if I can reproduce the issue.
However, I probably won’t be able to run gcore for debugging. Any other suggestions?
Also we saw a lot of open udp ports and some had a huge read buffer size which did not decrease.
*From:*freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>] *On Behalf Of *Grant Bagdasarian *Sent:* maandag 13 maart 2017 09:04
*To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org>> *Subject:* Re: [Freeswitch-users] 100% CPU usage
Hi Antony and Antonio,
Thank you for the tips.
We haven’t had freeswitch going berserk anymore (not sure as to why it did perform without issue over the weekend), but will continue to monitor it and run the commands to trace the issue.
Regards,
Grant
*From:*freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Anthony Minessale *Sent:* zaterdag 11 maart 2017 00:07 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org>> *Subject:* Re: [Freeswitch-users] 100% CPU usage
Make sure you lua script has all while loops including session:ready() as a condition to continue. Its a common mistake that leads to loops spinning out of control on hungup channels.
On Fri, Mar 10, 2017 at 6:11 AM, Antonio Silva <asilva@wirelessmundi.com <mailto:asilva@wirelessmundi.com>> wrote:
Hi,
You can use the following commands:
#get a core:
gcore -o /home/gcore-$(date +%s) $(cat /var/run/freeswitch/freeswitch.pid)
#check thread usage:
top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1
#memory mapping:
cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps
you can get some info from fs:
fs_cli -x "sofia global siptrace on"
fs_cli -x "fsctl debug_level 10"
fs_cli -x "sofia loglevel all 9"
fs_cli -x "fsctl debug_sql "
fs_cli -x "sofia global debug presence"
Put everything into a jira so we can take a look.
Regards, António
On 03/10/2017 11:08 AM, Grant Bagdasarian wrote:
Hi,
We’re currently experiencing a weird issue with one of our freeswitch servers where the CPU just goes berserk and stays at 100%.
There are no active calls at that moment, but FS is still using 100% CPU.
I was unable to get a debug using gdb, since freeswitch does not crash.
We had this issue on 1.6.13 and then I upgraded to 1.6.15 this morning, but the problem still persists.
There is currently only one active application running on this box which is using LUA. Maybe that is causing the issue?
We checked the code but there are no weird loops etc. Perhaps we are missing something?
Any ideas how to trace this?
Regards,
Grant
_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org <mailto:consulting@freeswitch.org>
http://www.freeswitchsolutions.com
Official FreeSWITCH Sites
http://confluence.freeswitch.org
FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org>
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
--
Saludos / Regards / Cumprimentos,
António silva
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
--
Anthony Minessale II ♬@anthmfs ♬@FreeSWITCH ♬
☞http://freeswitch.org/ ☞http://cluecon.com/ ☞http://twitter.com/FreeSWITCH
☞irc.freenode.net <http://irc.freenode.net> #freeswitch ☞_http://freeswitch.org/g+_
ClueCon Weekly Development Call
☎sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> ☎+19193869900
https://www.youtube.com/watch?v=9XXgW34t40s
https://www.youtube.com/watch?v=NLaDpGQuZDA
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
--
Anthony Minessale II ♬@anthmfs ♬@FreeSWITCH ♬
☞http://freeswitch.org/ ☞http://cluecon.com/ ☞http://twitter.com/FreeSWITCH
☞irc.freenode.net <http://irc.freenode.net> #freeswitch ☞_http://freeswitch.org/g+_
ClueCon Weekly Development Call
☎sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> ☎+19193869900
https://www.youtube.com/watch?v=9XXgW34t40s
https://www.youtube.com/watch?v=NLaDpGQuZDA
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________
Shaun Stokes - Infrastructure Analyst
T : 01453 700713 E : shaun.stokes@itec-support.co.uk W : www.itec-support.co.uk
Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001
CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment.
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Saludos / Regards / Cumprimentos, António silva
@Shuan, yes we are using the session:ready() method before we execute. We are running all of our servers virtually and never had any issues up until now. This same application ran on older versions of freeswitch without issues, even during and after high load. @Antonio, thanks for the tips! I’ll have a look and try some stuff. From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Antonio Silva Sent: woensdag 22 maart 2017 10:09 To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] 100% CPU usage We had some issues using pcall functions like io.popen when trying to execute external commands, that block mod_lua, i end up replacing that with os.execute. Recently i also had some issue with cpu hitting 100%,for what i could compile from debugging was something related with malloc calls. after compiling fs with tcmalloc the issue disappear, you could try this approach as well. You can check jira: https://freeswitch.org/jira/browse/FS-9957 and also in the malling: "CPU sys load issue" for more info. Regards, António On 03/22/2017 09:52 AM, Shaun Stokes wrote: Are you using ‘if (session:ready()) then’ in your scripts before you execute? Are you using virtual or physical hardware and what spec? From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Grant Bagdasarian Sent: 22 March 2017 07:31 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org><mailto:freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] 100% CPU usage Anthony, we triple checked the script and its dependencies. There are no loops at all in the script. The imported scripts are also written by us and they too do not contain any loops. It’s just basic call handling with some playback and play_and_get_digits calls, and some pcall calls to query a database. From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Anthony Minessale Sent: dinsdag 21 maart 2017 16:02 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage I already told you why a week ago. Your script most likely has an endless loop in it that does not check the condition session:ready() However, Feel free to contact consulting@freeswitch.org<mailto:consulting@freeswitch.org> and we can formally diagnose it for a fee. On Tue, Mar 21, 2017 at 6:42 AM Grant Bagdasarian <gb@cm.nl<mailto:gb@cm.nl>> wrote: Attached are the logs for: #check thread usage: top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1 #memory mapping: cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps We’re unable to run gcore since freeswitch hogs all available cpu’s at 100%. I tried changing the nice settings real-time but that didn’t work either. At one point we had a huge spike in inbound traffic towards our freeswitch (valid traffic). The traffic is handled by a lua application. What we saw was calls hanging in freeswitch while they’ve already been physically disconnected. After the huge spike freeswitch behaved normally for a couple of minutes and then suddenly started eating all the available cpu cores at 100%. We have some db queries which are fired in the lua application, so these may cause the application to hang, but we haven’t reproduced it yet. I’ll setup a test lab with the same configuration and fire sipp at it to see if I can reproduce the issue. However, I probably won’t be able to run gcore for debugging. Any other suggestions? Also we saw a lot of open udp ports and some had a huge read buffer size which did not decrease. [cid:image001.png@01D2A238.14965330] From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Grant Bagdasarian Sent: maandag 13 maart 2017 09:04 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage Hi Antony and Antonio, Thank you for the tips. We haven’t had freeswitch going berserk anymore (not sure as to why it did perform without issue over the weekend), but will continue to monitor it and run the commands to trace the issue. Regards, Grant From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Anthony Minessale Sent: zaterdag 11 maart 2017 00:07 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage Make sure you lua script has all while loops including session:ready() as a condition to continue. Its a common mistake that leads to loops spinning out of control on hungup channels. On Fri, Mar 10, 2017 at 6:11 AM, Antonio Silva <asilva@wirelessmundi.com<mailto:asilva@wirelessmundi.com>> wrote: Hi, You can use the following commands: #get a core: gcore -o /home/gcore-$(date +%s) $(cat /var/run/freeswitch/freeswitch.pid) #check thread usage: top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1 #memory mapping: cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps you can get some info from fs: fs_cli -x "sofia global siptrace on" fs_cli -x "fsctl debug_level 10" fs_cli -x "sofia loglevel all 9" fs_cli -x "fsctl debug_sql " fs_cli -x "sofia global debug presence" Put everything into a jira so we can take a look. Regards, António On 03/10/2017 11:08 AM, Grant Bagdasarian wrote: Hi, We’re currently experiencing a weird issue with one of our freeswitch servers where the CPU just goes berserk and stays at 100%. There are no active calls at that moment, but FS is still using 100% CPU. I was unable to get a debug using gdb, since freeswitch does not crash. We had this issue on 1.6.13 and then I upgraded to 1.6.15 this morning, but the problem still persists. There is currently only one active application running on this box which is using LUA. Maybe that is causing the issue? We checked the code but there are no weird loops etc. Perhaps we are missing something? Any ideas how to trace this? Regards, Grant _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Saludos / Regards / Cumprimentos, António silva _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net<http://irc.freenode.net> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net<http://irc.freenode.net> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ [http://www.itec-support.co.uk/wp-content/uploads/2016/07/email_logo.jpg] Shaun Stokes - Infrastructure Analyst T : 01453 700713 E : shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk> W : www.itec-support.co.uk<http://www.itec-support.co.uk> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001 CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment. ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Saludos / Regards / Cumprimentos, António silva
It’s probably a long shot, but if you’re able to replicate the problem it might be worth trying to replicate this on physical hardware. That way you eliminate your virtual environment from the problem, there are a number of factors which exist in virtual environments depending on your hyper-visor which can contribute to performance issues of which we’ve experienced in the past. It would also be worth trying to isolate what the script is doing when the high CPU begins, if you’ve got a test environment you can use to replicate the issue then you could try tweaking the script so you can eliminate certain actions from the list of causes, there’s maybe a FreeSWITCH module which is being called which triggers this. From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Grant Bagdasarian Sent: 22 March 2017 11:20 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] 100% CPU usage @Shuan, yes we are using the session:ready() method before we execute. We are running all of our servers virtually and never had any issues up until now. This same application ran on older versions of freeswitch without issues, even during and after high load. @Antonio, thanks for the tips! I’ll have a look and try some stuff. From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Antonio Silva Sent: woensdag 22 maart 2017 10:09 To: freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] 100% CPU usage We had some issues using pcall functions like io.popen when trying to execute external commands, that block mod_lua, i end up replacing that with os.execute. Recently i also had some issue with cpu hitting 100%,for what i could compile from debugging was something related with malloc calls. after compiling fs with tcmalloc the issue disappear, you could try this approach as well. You can check jira: https://freeswitch.org/jira/browse/FS-9957 and also in the malling: "CPU sys load issue" for more info. Regards, António On 03/22/2017 09:52 AM, Shaun Stokes wrote: Are you using ‘if (session:ready()) then’ in your scripts before you execute? Are you using virtual or physical hardware and what spec? From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Grant Bagdasarian Sent: 22 March 2017 07:31 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org><mailto:freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] 100% CPU usage Anthony, we triple checked the script and its dependencies. There are no loops at all in the script. The imported scripts are also written by us and they too do not contain any loops. It’s just basic call handling with some playback and play_and_get_digits calls, and some pcall calls to query a database. From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Anthony Minessale Sent: dinsdag 21 maart 2017 16:02 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage I already told you why a week ago. Your script most likely has an endless loop in it that does not check the condition session:ready() However, Feel free to contact consulting@freeswitch.org<mailto:consulting@freeswitch.org> and we can formally diagnose it for a fee. On Tue, Mar 21, 2017 at 6:42 AM Grant Bagdasarian <gb@cm.nl<mailto:gb@cm.nl>> wrote: Attached are the logs for: #check thread usage: top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1 #memory mapping: cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps We’re unable to run gcore since freeswitch hogs all available cpu’s at 100%. I tried changing the nice settings real-time but that didn’t work either. At one point we had a huge spike in inbound traffic towards our freeswitch (valid traffic). The traffic is handled by a lua application. What we saw was calls hanging in freeswitch while they’ve already been physically disconnected. After the huge spike freeswitch behaved normally for a couple of minutes and then suddenly started eating all the available cpu cores at 100%. We have some db queries which are fired in the lua application, so these may cause the application to hang, but we haven’t reproduced it yet. I’ll setup a test lab with the same configuration and fire sipp at it to see if I can reproduce the issue. However, I probably won’t be able to run gcore for debugging. Any other suggestions? Also we saw a lot of open udp ports and some had a huge read buffer size which did not decrease. [cid:image001.png@01D2A238.14965330] From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Grant Bagdasarian Sent: maandag 13 maart 2017 09:04 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage Hi Antony and Antonio, Thank you for the tips. We haven’t had freeswitch going berserk anymore (not sure as to why it did perform without issue over the weekend), but will continue to monitor it and run the commands to trace the issue. Regards, Grant From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Anthony Minessale Sent: zaterdag 11 maart 2017 00:07 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage Make sure you lua script has all while loops including session:ready() as a condition to continue. Its a common mistake that leads to loops spinning out of control on hungup channels. On Fri, Mar 10, 2017 at 6:11 AM, Antonio Silva <asilva@wirelessmundi.com<mailto:asilva@wirelessmundi.com>> wrote: Hi, You can use the following commands: #get a core: gcore -o /home/gcore-$(date +%s) $(cat /var/run/freeswitch/freeswitch.pid) #check thread usage: top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1 #memory mapping: cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps you can get some info from fs: fs_cli -x "sofia global siptrace on" fs_cli -x "fsctl debug_level 10" fs_cli -x "sofia loglevel all 9" fs_cli -x "fsctl debug_sql " fs_cli -x "sofia global debug presence" Put everything into a jira so we can take a look. Regards, António On 03/10/2017 11:08 AM, Grant Bagdasarian wrote: Hi, We’re currently experiencing a weird issue with one of our freeswitch servers where the CPU just goes berserk and stays at 100%. There are no active calls at that moment, but FS is still using 100% CPU. I was unable to get a debug using gdb, since freeswitch does not crash. We had this issue on 1.6.13 and then I upgraded to 1.6.15 this morning, but the problem still persists. There is currently only one active application running on this box which is using LUA. Maybe that is causing the issue? We checked the code but there are no weird loops etc. Perhaps we are missing something? Any ideas how to trace this? Regards, Grant _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Saludos / Regards / Cumprimentos, António silva _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net<http://irc.freenode.net> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net<http://irc.freenode.net> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ [http://www.itec-support.co.uk/wp-content/uploads/2016/07/email_logo.jpg] Shaun Stokes - Infrastructure Analyst T : 01453 700713 E : shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk> W : www.itec-support.co.uk<http://www.itec-support.co.uk> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001 CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment. ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Saludos / Regards / Cumprimentos, António silva ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ [http://www.itec-support.co.uk/wp-content/uploads/2016/07/email_logo.jpg] Shaun Stokes - Infrastructure Analyst T : 01453 700713 E : shaun.stokes@itec-support.co.uk W : www.itec-support.co.uk Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001 CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment. ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________
top -H -p `pidof freeswitch` Note the pid using 100% cpu gcore `pidof freeswitch` Attach it to a jira. If you think its a bug in FS you should report it to Jira not to this list. https://freeswitch.org/jira On Wed, Mar 22, 2017 at 10:14 AM Shaun Stokes < shaun.stokes@itec-support.co.uk> wrote:
It’s probably a long shot, but if you’re able to replicate the problem it might be worth trying to replicate this on physical hardware. That way you eliminate your virtual environment from the problem, there are a number of factors which exist in virtual environments depending on your hyper-visor which can contribute to performance issues of which we’ve experienced in the past.
It would also be worth trying to isolate what the script is doing when the high CPU begins, if you’ve got a test environment you can use to replicate the issue then you could try tweaking the script so you can eliminate certain actions from the list of causes, there’s maybe a FreeSWITCH module which is being called which triggers this.
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Grant Bagdasarian *Sent:* 22 March 2017 11:20
*To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] 100% CPU usage
@Shuan, yes we are using the session:ready() method before we execute.
We are running all of our servers virtually and never had any issues up until now.
This same application ran on older versions of freeswitch without issues, even during and after high load.
@Antonio, thanks for the tips! I’ll have a look and try some stuff.
*From:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org <freeswitch-users-bounces@lists.freeswitch.org>] *On Behalf Of *Antonio Silva *Sent:* woensdag 22 maart 2017 10:09 *To:* freeswitch-users@lists.freeswitch.org *Subject:* Re: [Freeswitch-users] 100% CPU usage
We had some issues using pcall functions like io.popen when trying to execute external commands, that block mod_lua, i end up replacing that with os.execute.
Recently i also had some issue with cpu hitting 100%,for what i could compile from debugging was something related with malloc calls. after compiling fs with tcmalloc the issue disappear, you could try this approach as well.
You can check jira: https://freeswitch.org/jira/browse/FS-9957 and also in the malling: "CPU sys load issue" for more info.
Regards, António
On 03/22/2017 09:52 AM, Shaun Stokes wrote:
Are you using ‘if (session:ready()) then’ in your scripts before you execute?
Are you using virtual or physical hardware and what spec?
*From:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org <freeswitch-users-bounces@lists.freeswitch.org>] *On Behalf Of *Grant Bagdasarian *Sent:* 22 March 2017 07:31 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] 100% CPU usage
Anthony, we triple checked the script and its dependencies. There are no loops at all in the script. The imported scripts are also written by us and they too do not contain any loops.
It’s just basic call handling with some playback and play_and_get_digits calls, and some pcall calls to query a database.
*From:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org <freeswitch-users-bounces@lists.freeswitch.org>] *On Behalf Of *Anthony Minessale *Sent:* dinsdag 21 maart 2017 16:02 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] 100% CPU usage
I already told you why a week ago. Your script most likely has an endless loop in it that does not check the condition session:ready()
However, Feel free to contact consulting@freeswitch.org and we can formally diagnose it for a fee.
On Tue, Mar 21, 2017 at 6:42 AM Grant Bagdasarian <gb@cm.nl> wrote:
Attached are the logs for:
#check thread usage:
top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1
#memory mapping:
cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps
We’re unable to run gcore since freeswitch hogs all available cpu’s at 100%.
I tried changing the nice settings real-time but that didn’t work either.
At one point we had a huge spike in inbound traffic towards our freeswitch (valid traffic).
The traffic is handled by a lua application.
What we saw was calls hanging in freeswitch while they’ve already been physically disconnected.
After the huge spike freeswitch behaved normally for a couple of minutes and then suddenly started eating all the available cpu cores at 100%.
We have some db queries which are fired in the lua application, so these may cause the application to hang, but we haven’t reproduced it yet.
I’ll setup a test lab with the same configuration and fire sipp at it to see if I can reproduce the issue.
However, I probably won’t be able to run gcore for debugging. Any other suggestions?
Also we saw a lot of open udp ports and some had a huge read buffer size which did not decrease.
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Grant Bagdasarian *Sent:* maandag 13 maart 2017 09:04
*To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] 100% CPU usage
Hi Antony and Antonio,
Thank you for the tips.
We haven’t had freeswitch going berserk anymore (not sure as to why it did perform without issue over the weekend), but will continue to monitor it and run the commands to trace the issue.
Regards,
Grant
*From:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org <freeswitch-users-bounces@lists.freeswitch.org>] *On Behalf Of *Anthony Minessale *Sent:* zaterdag 11 maart 2017 00:07 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] 100% CPU usage
Make sure you lua script has all while loops including session:ready() as a condition to continue. Its a common mistake that leads to loops spinning out of control on hungup channels.
On Fri, Mar 10, 2017 at 6:11 AM, Antonio Silva <asilva@wirelessmundi.com> wrote:
Hi,
You can use the following commands:
#get a core:
gcore -o /home/gcore-$(date +%s) $(cat /var/run/freeswitch/freeswitch.pid)
#check thread usage:
top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1
#memory mapping:
cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps
you can get some info from fs:
fs_cli -x "sofia global siptrace on"
fs_cli -x "fsctl debug_level 10"
fs_cli -x "sofia loglevel all 9"
fs_cli -x "fsctl debug_sql "
fs_cli -x "sofia global debug presence"
Put everything into a jira so we can take a look.
Regards, António
On 03/10/2017 11:08 AM, Grant Bagdasarian wrote:
Hi,
We’re currently experiencing a weird issue with one of our freeswitch servers where the CPU just goes berserk and stays at 100%.
There are no active calls at that moment, but FS is still using 100% CPU.
I was unable to get a debug using gdb, since freeswitch does not crash.
We had this issue on 1.6.13 and then I upgraded to 1.6.15 this morning, but the problem still persists.
There is currently only one active application running on this box which is using LUA. Maybe that is causing the issue?
We checked the code but there are no weird loops etc. Perhaps we are missing something?
Any ideas how to trace this?
Regards,
Grant
_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com
Official FreeSWITCH Sites
http://confluence.freeswitch.org
FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
--
Saludos / Regards / Cumprimentos,
António silva
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
--
Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH
☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call
☎ sip:888@conference.freeswitch.org ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s
https://www.youtube.com/watch?v=NLaDpGQuZDA
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
--
Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH
☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call
☎ sip:888@conference.freeswitch.org ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s
https://www.youtube.com/watch?v=NLaDpGQuZDA
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________
Shaun Stokes - Infrastructure Analyst
T :
01453 700713
E :
shaun.stokes@itec-support.co.uk
W :
www.itec-support.co.uk
Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001
CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment.
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________
_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com
Official FreeSWITCH Sites
http://confluence.freeswitch.org
FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
--
Saludos / Regards / Cumprimentos,
António silva
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ Shaun Stokes - Infrastructure Analyst T : 01453 700713 E : shaun.stokes@itec-support.co.uk W : www.itec-support.co.uk Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001
CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment.
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
Hi All, I’m having a very similar issue however it’s not staying at 100% CPU on physical hardware. I’ve created a JIRA but wanted to see if anyone else had any feedback on possible causes? I’m using freeswitch 1.6.15 (previously 1.6.13). What happens is that FS will jump to 100% CPU and stay there for 1-2 seconds then drop back down but this is then causing massive jitter/call quality issues for about 10-15 seconds after the spike. The strange thing that I’ve noticed is that even with 0 active calls, FS is using 15-30% cpu constantly with only 40-60 registered endpoints. I’m watching the logs in DEBUG mode and not seeing anything there that’s indicative of any issue, no loops that I can see, and it seems to be happening when there’s even just 1 or 2 active calls on the system. I’m wondering if there’s any other suggestions on where to look because this is getting very frustrating for me and my users. Thanks much for any pointers! Cj B
On Mar 22, 2017, at 8:13 AM, Anthony Minessale <anthony.minessale@gmail.com> wrote:
top -H -p `pidof freeswitch` Note the pid using 100% cpu
gcore `pidof freeswitch`
Attach it to a jira. If you think its a bug in FS you should report it to Jira not to this list.
https://freeswitch.org/jira <https://freeswitch.org/jira>
On Wed, Mar 22, 2017 at 10:14 AM Shaun Stokes <shaun.stokes@itec-support.co.uk <mailto:shaun.stokes@itec-support.co.uk>> wrote: It’s probably a long shot, but if you’re able to replicate the problem it might be worth trying to replicate this on physical hardware. That way you eliminate your virtual environment from the problem, there are a number of factors which exist in virtual environments depending on your hyper-visor which can contribute to performance issues of which we’ve experienced in the past.
It would also be worth trying to isolate what the script is doing when the high CPU begins, if you’ve got a test environment you can use to replicate the issue then you could try tweaking the script so you can eliminate certain actions from the list of causes, there’s maybe a FreeSWITCH module which is being called which triggers this.
From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Grant Bagdasarian Sent: 22 March 2017 11:20
To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage
@Shuan, yes we are using the session:ready() method before we execute.
We are running all of our servers virtually and never had any issues up until now.
This same application ran on older versions of freeswitch without issues, even during and after high load.
@Antonio, thanks for the tips! I’ll have a look and try some stuff.
From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Antonio Silva Sent: woensdag 22 maart 2017 10:09 To: freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] 100% CPU usage
We had some issues using pcall functions like io.popen when trying to execute external commands, that block mod_lua, i end up replacing that with os.execute.
Recently i also had some issue with cpu hitting 100%,for what i could compile from debugging was something related with malloc calls. after compiling fs with tcmalloc the issue disappear, you could try this approach as well.
You can check jira: https://freeswitch.org/jira/browse/FS-9957 <https://freeswitch.org/jira/browse/FS-9957> and also in the malling: "CPU sys load issue" for more info.
Regards, António
On 03/22/2017 09:52 AM, Shaun Stokes wrote:
Are you using ‘if (session:ready()) then’ in your scripts before you execute?
Are you using virtual or physical hardware and what spec?
From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Grant Bagdasarian Sent: 22 March 2017 07:31 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> <mailto:freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] 100% CPU usage
Anthony, we triple checked the script and its dependencies. There are no loops at all in the script. The imported scripts are also written by us and they too do not contain any loops.
It’s just basic call handling with some playback and play_and_get_digits calls, and some pcall calls to query a database.
From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Anthony Minessale Sent: dinsdag 21 maart 2017 16:02 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage
I already told you why a week ago. Your script most likely has an endless loop in it that does not check the condition session:ready()
However, Feel free to contact consulting@freeswitch.org <mailto:consulting@freeswitch.org> and we can formally diagnose it for a fee.
On Tue, Mar 21, 2017 at 6:42 AM Grant Bagdasarian <gb@cm.nl <mailto:gb@cm.nl>> wrote:
Attached are the logs for:
#check thread usage: top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1
#memory mapping: cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps
We’re unable to run gcore since freeswitch hogs all available cpu’s at 100%.
I tried changing the nice settings real-time but that didn’t work either.
At one point we had a huge spike in inbound traffic towards our freeswitch (valid traffic).
The traffic is handled by a lua application.
What we saw was calls hanging in freeswitch while they’ve already been physically disconnected.
After the huge spike freeswitch behaved normally for a couple of minutes and then suddenly started eating all the available cpu cores at 100%.
We have some db queries which are fired in the lua application, so these may cause the application to hang, but we haven’t reproduced it yet.
I’ll setup a test lab with the same configuration and fire sipp at it to see if I can reproduce the issue.
However, I probably won’t be able to run gcore for debugging. Any other suggestions?
Also we saw a lot of open udp ports and some had a huge read buffer size which did not decrease.
From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Grant Bagdasarian Sent: maandag 13 maart 2017 09:04
To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage
Hi Antony and Antonio,
Thank you for the tips.
We haven’t had freeswitch going berserk anymore (not sure as to why it did perform without issue over the weekend), but will continue to monitor it and run the commands to trace the issue.
Regards,
Grant
From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Anthony Minessale Sent: zaterdag 11 maart 2017 00:07 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage
Make sure you lua script has all while loops including session:ready() as a condition to continue. Its a common mistake that leads to loops spinning out of control on hungup channels.
On Fri, Mar 10, 2017 at 6:11 AM, Antonio Silva <asilva@wirelessmundi.com <mailto:asilva@wirelessmundi.com>> wrote:
Hi,
You can use the following commands:
#get a core: gcore -o /home/gcore-$(date +%s) $(cat /var/run/freeswitch/freeswitch.pid)
#check thread usage: top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1
#memory mapping: cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps
you can get some info from fs:
fs_cli -x "sofia global siptrace on" fs_cli -x "fsctl debug_level 10" fs_cli -x "sofia loglevel all 9" fs_cli -x "fsctl debug_sql " fs_cli -x "sofia global debug presence"
Put everything into a jira so we can take a look.
Regards, António
On 03/10/2017 11:08 AM, Grant Bagdasarian wrote:
Hi,
We’re currently experiencing a weird issue with one of our freeswitch servers where the CPU just goes berserk and stays at 100%.
There are no active calls at that moment, but FS is still using 100% CPU.
I was unable to get a debug using gdb, since freeswitch does not crash.
We had this issue on 1.6.13 and then I upgraded to 1.6.15 this morning, but the problem still persists.
There is currently only one active application running on this box which is using LUA. Maybe that is causing the issue?
We checked the code but there are no weird loops etc. Perhaps we are missing something?
Any ideas how to trace this?
Regards,
Grant
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://confluence.freeswitch.org <http://confluence.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org <http://www.freeswitch.org/>
--
Saludos / Regards / Cumprimentos, António silva
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://confluence.freeswitch.org <http://confluence.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org <http://www.freeswitch.org/>
--
Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ <http://freeswitch.org/> ☞ http://cluecon.com/ <http://cluecon.com/> ☞ http://twitter.com/FreeSWITCH <http://twitter.com/FreeSWITCH> ☞ irc.freenode.net <http://irc.freenode.net/> #freeswitch ☞ http://freeswitch.org/g+ <http://freeswitch.org/g+> ClueCon Weekly Development Call
☎ sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s <https://www.youtube.com/watch?v=9XXgW34t40s> https://www.youtube.com/watch?v=NLaDpGQuZDA <https://www.youtube.com/watch?v=NLaDpGQuZDA> _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://confluence.freeswitch.org <http://confluence.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org <http://www.freeswitch.org/> --
Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ <http://freeswitch.org/> ☞ http://cluecon.com/ <http://cluecon.com/> ☞ http://twitter.com/FreeSWITCH <http://twitter.com/FreeSWITCH> ☞ irc.freenode.net <http://irc.freenode.net/> #freeswitch ☞ http://freeswitch.org/g+ <http://freeswitch.org/g+> ClueCon Weekly Development Call
☎ sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s <https://www.youtube.com/watch?v=9XXgW34t40s> https://www.youtube.com/watch?v=NLaDpGQuZDA <https://www.youtube.com/watch?v=NLaDpGQuZDA>
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________
Shaun Stokes - Infrastructure Analyst
T :
01453 700713
E :
shaun.stokes@itec-support.co.uk <mailto:shaun.stokes@itec-support.co.uk> W :
www.itec-support.co.uk <http://www.itec-support.co.uk/> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001
CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment.
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://confluence.freeswitch.org <http://confluence.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org <http://www.freeswitch.org/>
--
Saludos / Regards / Cumprimentos, António silva
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________
Shaun Stokes - Infrastructure Analyst <> T : <> 01453 700713 <> E : <> shaun.stokes@itec-support.co.uk <> W : <> www.itec-support.co.uk <> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF <> Company No. 06908001 <>
CONFIDENTIALITY NOTICE <> This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. <> The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment. <> ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://confluence.freeswitch.org <http://confluence.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org <http://www.freeswitch.org/> -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ <http://freeswitch.org/> ☞ http://cluecon.com/ <http://cluecon.com/> ☞ http://twitter.com/FreeSWITCH <http://twitter.com/FreeSWITCH> ☞ irc.freenode.net <http://irc.freenode.net/> #freeswitch ☞ http://freeswitch.org/g+ <http://freeswitch.org/g+>
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s <https://www.youtube.com/watch?v=9XXgW34t40s> https://www.youtube.com/watch?v=NLaDpGQuZDA <https://www.youtube.com/watch?v=NLaDpGQuZDA> _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Try to avoid doing things like pointing out how frustrated you are when seeking help here. It makes those trying to help you feel like you are being entitled. The suggestion on what to do was clearly outlined in previous posts in this thread including the exact commands to run when filing a jira. On Wed, Mar 29, 2017 at 12:20 PM, Cj B <blackc2004@gmail.com> wrote:
Hi All,
I’m having a very similar issue however it’s not staying at 100% CPU on physical hardware. I’ve created a JIRA but wanted to see if anyone else had any feedback on possible causes? I’m using freeswitch 1.6.15 (previously 1.6.13). What happens is that FS will jump to 100% CPU and stay there for 1-2 seconds then drop back down but this is then causing massive jitter/call quality issues for about 10-15 seconds after the spike. The strange thing that I’ve noticed is that even with 0 active calls, FS is using 15-30% cpu constantly with only 40-60 registered endpoints.
I’m watching the logs in DEBUG mode and not seeing anything there that’s indicative of any issue, no loops that I can see, and it seems to be happening when there’s even just 1 or 2 active calls on the system.
I’m wondering if there’s any other suggestions on where to look because this is getting very frustrating for me and my users.
Thanks much for any pointers! Cj B
On Mar 22, 2017, at 8:13 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
top -H -p `pidof freeswitch` Note the pid using 100% cpu
gcore `pidof freeswitch`
Attach it to a jira. If you think its a bug in FS you should report it to Jira not to this list.
On Wed, Mar 22, 2017 at 10:14 AM Shaun Stokes < shaun.stokes@itec-support.co.uk> wrote:
It’s probably a long shot, but if you’re able to replicate the problem it might be worth trying to replicate this on physical hardware. That way you eliminate your virtual environment from the problem, there are a number of factors which exist in virtual environments depending on your hyper-visor which can contribute to performance issues of which we’ve experienced in the past.
It would also be worth trying to isolate what the script is doing when the high CPU begins, if you’ve got a test environment you can use to replicate the issue then you could try tweaking the script so you can eliminate certain actions from the list of causes, there’s maybe a FreeSWITCH module which is being called which triggers this.
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Grant Bagdasarian *Sent:* 22 March 2017 11:20
*To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] 100% CPU usage
@Shuan, yes we are using the session:ready() method before we execute.
We are running all of our servers virtually and never had any issues up until now.
This same application ran on older versions of freeswitch without issues, even during and after high load.
@Antonio, thanks for the tips! I’ll have a look and try some stuff.
*From:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org <freeswitch-users-bounces@lists.freeswitch.org>] *On Behalf Of *Antonio Silva *Sent:* woensdag 22 maart 2017 10:09 *To:* freeswitch-users@lists.freeswitch.org *Subject:* Re: [Freeswitch-users] 100% CPU usage
We had some issues using pcall functions like io.popen when trying to execute external commands, that block mod_lua, i end up replacing that with os.execute.
Recently i also had some issue with cpu hitting 100%,for what i could compile from debugging was something related with malloc calls. after compiling fs with tcmalloc the issue disappear, you could try this approach as well.
You can check jira: https://freeswitch.org/jira/browse/FS-9957 and also in the malling: "CPU sys load issue" for more info.
Regards, António
On 03/22/2017 09:52 AM, Shaun Stokes wrote:
Are you using ‘if (session:ready()) then’ in your scripts before you execute?
Are you using virtual or physical hardware and what spec?
*From:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org <freeswitch-users-bounces@lists.freeswitch.org>] *On Behalf Of *Grant Bagdasarian *Sent:* 22 March 2017 07:31 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] 100% CPU usage
Anthony, we triple checked the script and its dependencies. There are no loops at all in the script. The imported scripts are also written by us and they too do not contain any loops.
It’s just basic call handling with some playback and play_and_get_digits calls, and some pcall calls to query a database.
*From:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org <freeswitch-users-bounces@lists.freeswitch.org>] *On Behalf Of *Anthony Minessale *Sent:* dinsdag 21 maart 2017 16:02 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] 100% CPU usage
I already told you why a week ago. Your script most likely has an endless loop in it that does not check the condition session:ready()
However, Feel free to contact consulting@freeswitch.org and we can formally diagnose it for a fee.
On Tue, Mar 21, 2017 at 6:42 AM Grant Bagdasarian <gb@cm.nl> wrote:
Attached are the logs for:
#check thread usage:
top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1
#memory mapping:
cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps
We’re unable to run gcore since freeswitch hogs all available cpu’s at 100%.
I tried changing the nice settings real-time but that didn’t work either.
At one point we had a huge spike in inbound traffic towards our freeswitch (valid traffic).
The traffic is handled by a lua application.
What we saw was calls hanging in freeswitch while they’ve already been physically disconnected.
After the huge spike freeswitch behaved normally for a couple of minutes and then suddenly started eating all the available cpu cores at 100%.
We have some db queries which are fired in the lua application, so these may cause the application to hang, but we haven’t reproduced it yet.
I’ll setup a test lab with the same configuration and fire sipp at it to see if I can reproduce the issue.
However, I probably won’t be able to run gcore for debugging. Any other suggestions?
Also we saw a lot of open udp ports and some had a huge read buffer size which did not decrease.
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Grant Bagdasarian *Sent:* maandag 13 maart 2017 09:04
*To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] 100% CPU usage
Hi Antony and Antonio,
Thank you for the tips.
We haven’t had freeswitch going berserk anymore (not sure as to why it did perform without issue over the weekend), but will continue to monitor it and run the commands to trace the issue.
Regards,
Grant
*From:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org <freeswitch-users-bounces@lists.freeswitch.org>] *On Behalf Of *Anthony Minessale *Sent:* zaterdag 11 maart 2017 00:07 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] 100% CPU usage
Make sure you lua script has all while loops including session:ready() as a condition to continue. Its a common mistake that leads to loops spinning out of control on hungup channels.
On Fri, Mar 10, 2017 at 6:11 AM, Antonio Silva <asilva@wirelessmundi.com> wrote:
Hi,
You can use the following commands:
#get a core:
gcore -o /home/gcore-$(date +%s) $(cat /var/run/freeswitch/freeswitch.pid)
#check thread usage:
top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1
#memory mapping:
cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps
you can get some info from fs:
fs_cli -x "sofia global siptrace on"
fs_cli -x "fsctl debug_level 10"
fs_cli -x "sofia loglevel all 9"
fs_cli -x "fsctl debug_sql "
fs_cli -x "sofia global debug presence"
Put everything into a jira so we can take a look.
Regards, António
On 03/10/2017 11:08 AM, Grant Bagdasarian wrote:
Hi,
We’re currently experiencing a weird issue with one of our freeswitch servers where the CPU just goes berserk and stays at 100%.
There are no active calls at that moment, but FS is still using 100% CPU.
I was unable to get a debug using gdb, since freeswitch does not crash.
We had this issue on 1.6.13 and then I upgraded to 1.6.15 this morning, but the problem still persists.
There is currently only one active application running on this box which is using LUA. Maybe that is causing the issue?
We checked the code but there are no weird loops etc. Perhaps we are missing something?
Any ideas how to trace this?
Regards,
Grant
_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com
Official FreeSWITCH Sites
http://confluence.freeswitch.org
FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
--
Saludos / Regards / Cumprimentos,
António silva
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
--
Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH
☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call
☎ sip:888@conference.freeswitch.org ☎ +19193869900 <(919)%20386-9900>
https://www.youtube.com/watch?v=9XXgW34t40s
https://www.youtube.com/watch?v=NLaDpGQuZDA
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
--
Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH
☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call
☎ sip:888@conference.freeswitch.org ☎ +19193869900 <(919)%20386-9900>
https://www.youtube.com/watch?v=9XXgW34t40s
https://www.youtube.com/watch?v=NLaDpGQuZDA
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________
Shaun Stokes - Infrastructure Analyst
T :
01453 700713
E :
shaun.stokes@itec-support.co.uk
W :
www.itec-support.co.uk
Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001
CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment.
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________
_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com
Official FreeSWITCH Sites
http://confluence.freeswitch.org
FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
--
Saludos / Regards / Cumprimentos,
António silva
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ Shaun Stokes - Infrastructure Analyst T : 01453 700713 E : shaun.stokes@itec-support.co.uk W : www.itec-support.co.uk Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001
CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment.
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900 <(919)%20386-9900>
https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
Anthony, I apologize I’m appearing “entitled”. I was simply asking for any other pointers. I’ve followed all the suggestions here when submitting the jira and have done so. I was just asking if there were any other tips/hints about what to look at Thanks for your response. Cj B
On Mar 29, 2017, at 10:28 AM, Anthony Minessale <anthony.minessale@gmail.com> wrote:
Try to avoid doing things like pointing out how frustrated you are when seeking help here. It makes those trying to help you feel like you are being entitled.
The suggestion on what to do was clearly outlined in previous posts in this thread including the exact commands to run when filing a jira.
On Wed, Mar 29, 2017 at 12:20 PM, Cj B <blackc2004@gmail.com <mailto:blackc2004@gmail.com>> wrote: Hi All,
I’m having a very similar issue however it’s not staying at 100% CPU on physical hardware. I’ve created a JIRA but wanted to see if anyone else had any feedback on possible causes? I’m using freeswitch 1.6.15 (previously 1.6.13). What happens is that FS will jump to 100% CPU and stay there for 1-2 seconds then drop back down but this is then causing massive jitter/call quality issues for about 10-15 seconds after the spike. The strange thing that I’ve noticed is that even with 0 active calls, FS is using 15-30% cpu constantly with only 40-60 registered endpoints.
I’m watching the logs in DEBUG mode and not seeing anything there that’s indicative of any issue, no loops that I can see, and it seems to be happening when there’s even just 1 or 2 active calls on the system.
I’m wondering if there’s any other suggestions on where to look because this is getting very frustrating for me and my users.
Thanks much for any pointers! Cj B
On Mar 22, 2017, at 8:13 AM, Anthony Minessale <anthony.minessale@gmail.com <mailto:anthony.minessale@gmail.com>> wrote:
top -H -p `pidof freeswitch` Note the pid using 100% cpu
gcore `pidof freeswitch`
Attach it to a jira. If you think its a bug in FS you should report it to Jira not to this list.
https://freeswitch.org/jira <https://freeswitch.org/jira>
On Wed, Mar 22, 2017 at 10:14 AM Shaun Stokes <shaun.stokes@itec-support.co.uk <mailto:shaun.stokes@itec-support.co.uk>> wrote: It’s probably a long shot, but if you’re able to replicate the problem it might be worth trying to replicate this on physical hardware. That way you eliminate your virtual environment from the problem, there are a number of factors which exist in virtual environments depending on your hyper-visor which can contribute to performance issues of which we’ve experienced in the past.
It would also be worth trying to isolate what the script is doing when the high CPU begins, if you’ve got a test environment you can use to replicate the issue then you could try tweaking the script so you can eliminate certain actions from the list of causes, there’s maybe a FreeSWITCH module which is being called which triggers this.
From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Grant Bagdasarian Sent: 22 March 2017 11:20
To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage
@Shuan, yes we are using the session:ready() method before we execute.
We are running all of our servers virtually and never had any issues up until now.
This same application ran on older versions of freeswitch without issues, even during and after high load.
@Antonio, thanks for the tips! I’ll have a look and try some stuff.
From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Antonio Silva Sent: woensdag 22 maart 2017 10:09 To: freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] 100% CPU usage
We had some issues using pcall functions like io.popen when trying to execute external commands, that block mod_lua, i end up replacing that with os.execute.
Recently i also had some issue with cpu hitting 100%,for what i could compile from debugging was something related with malloc calls. after compiling fs with tcmalloc the issue disappear, you could try this approach as well.
You can check jira: https://freeswitch.org/jira/browse/FS-9957 <https://freeswitch.org/jira/browse/FS-9957> and also in the malling: "CPU sys load issue" for more info.
Regards, António
On 03/22/2017 09:52 AM, Shaun Stokes wrote:
Are you using ‘if (session:ready()) then’ in your scripts before you execute?
Are you using virtual or physical hardware and what spec?
From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Grant Bagdasarian Sent: 22 March 2017 07:31 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> <mailto:freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] 100% CPU usage
Anthony, we triple checked the script and its dependencies. There are no loops at all in the script. The imported scripts are also written by us and they too do not contain any loops.
It’s just basic call handling with some playback and play_and_get_digits calls, and some pcall calls to query a database.
From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Anthony Minessale Sent: dinsdag 21 maart 2017 16:02 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage
I already told you why a week ago. Your script most likely has an endless loop in it that does not check the condition session:ready()
However, Feel free to contact consulting@freeswitch.org <mailto:consulting@freeswitch.org> and we can formally diagnose it for a fee.
On Tue, Mar 21, 2017 at 6:42 AM Grant Bagdasarian <gb@cm.nl <mailto:gb@cm.nl>> wrote:
Attached are the logs for:
#check thread usage: top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1
#memory mapping: cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps
We’re unable to run gcore since freeswitch hogs all available cpu’s at 100%.
I tried changing the nice settings real-time but that didn’t work either.
At one point we had a huge spike in inbound traffic towards our freeswitch (valid traffic).
The traffic is handled by a lua application.
What we saw was calls hanging in freeswitch while they’ve already been physically disconnected.
After the huge spike freeswitch behaved normally for a couple of minutes and then suddenly started eating all the available cpu cores at 100%.
We have some db queries which are fired in the lua application, so these may cause the application to hang, but we haven’t reproduced it yet.
I’ll setup a test lab with the same configuration and fire sipp at it to see if I can reproduce the issue.
However, I probably won’t be able to run gcore for debugging. Any other suggestions?
Also we saw a lot of open udp ports and some had a huge read buffer size which did not decrease.
From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Grant Bagdasarian Sent: maandag 13 maart 2017 09:04
To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage
Hi Antony and Antonio,
Thank you for the tips.
We haven’t had freeswitch going berserk anymore (not sure as to why it did perform without issue over the weekend), but will continue to monitor it and run the commands to trace the issue.
Regards,
Grant
From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Anthony Minessale Sent: zaterdag 11 maart 2017 00:07 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage
Make sure you lua script has all while loops including session:ready() as a condition to continue. Its a common mistake that leads to loops spinning out of control on hungup channels.
On Fri, Mar 10, 2017 at 6:11 AM, Antonio Silva <asilva@wirelessmundi.com <mailto:asilva@wirelessmundi.com>> wrote:
Hi,
You can use the following commands:
#get a core: gcore -o /home/gcore-$(date +%s) $(cat /var/run/freeswitch/freeswitch.pid)
#check thread usage: top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1
#memory mapping: cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps
you can get some info from fs:
fs_cli -x "sofia global siptrace on" fs_cli -x "fsctl debug_level 10" fs_cli -x "sofia loglevel all 9" fs_cli -x "fsctl debug_sql " fs_cli -x "sofia global debug presence"
Put everything into a jira so we can take a look.
Regards, António
On 03/10/2017 11:08 AM, Grant Bagdasarian wrote:
Hi,
We’re currently experiencing a weird issue with one of our freeswitch servers where the CPU just goes berserk and stays at 100%.
There are no active calls at that moment, but FS is still using 100% CPU.
I was unable to get a debug using gdb, since freeswitch does not crash.
We had this issue on 1.6.13 and then I upgraded to 1.6.15 this morning, but the problem still persists.
There is currently only one active application running on this box which is using LUA. Maybe that is causing the issue?
We checked the code but there are no weird loops etc. Perhaps we are missing something?
Any ideas how to trace this?
Regards,
Grant
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://confluence.freeswitch.org <http://confluence.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org <http://www.freeswitch.org/>
--
Saludos / Regards / Cumprimentos, António silva
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://confluence.freeswitch.org <http://confluence.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org <http://www.freeswitch.org/>
--
Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ <http://freeswitch.org/> ☞ http://cluecon.com/ <http://cluecon.com/> ☞ http://twitter.com/FreeSWITCH <http://twitter.com/FreeSWITCH> ☞ irc.freenode.net <http://irc.freenode.net/> #freeswitch ☞ http://freeswitch.org/g+ <http://freeswitch.org/g+> ClueCon Weekly Development Call
☎ sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900 <tel:(919)%20386-9900>
https://www.youtube.com/watch?v=9XXgW34t40s <https://www.youtube.com/watch?v=9XXgW34t40s> https://www.youtube.com/watch?v=NLaDpGQuZDA <https://www.youtube.com/watch?v=NLaDpGQuZDA> _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://confluence.freeswitch.org <http://confluence.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org <http://www.freeswitch.org/> --
Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ <http://freeswitch.org/> ☞ http://cluecon.com/ <http://cluecon.com/> ☞ http://twitter.com/FreeSWITCH <http://twitter.com/FreeSWITCH> ☞ irc.freenode.net <http://irc.freenode.net/> #freeswitch ☞ http://freeswitch.org/g+ <http://freeswitch.org/g+> ClueCon Weekly Development Call
☎ sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900 <tel:(919)%20386-9900>
https://www.youtube.com/watch?v=9XXgW34t40s <https://www.youtube.com/watch?v=9XXgW34t40s> https://www.youtube.com/watch?v=NLaDpGQuZDA <https://www.youtube.com/watch?v=NLaDpGQuZDA>
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________
Shaun Stokes - Infrastructure Analyst
T :
01453 700713
E :
shaun.stokes@itec-support.co.uk <mailto:shaun.stokes@itec-support.co.uk> W :
www.itec-support.co.uk <http://www.itec-support.co.uk/> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001
CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment.
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://confluence.freeswitch.org <http://confluence.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org <http://www.freeswitch.org/>
--
Saludos / Regards / Cumprimentos, António silva
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________
Shaun Stokes - Infrastructure Analyst <> T : <> 01453 700713 <> E : <> shaun.stokes@itec-support.co.uk <> W : <> www.itec-support.co.uk <> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF <> Company No. 06908001 <>
CONFIDENTIALITY NOTICE <> This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. <> The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment. <> ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://confluence.freeswitch.org <http://confluence.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org <http://www.freeswitch.org/> -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ <http://freeswitch.org/> ☞ http://cluecon.com/ <http://cluecon.com/> ☞ http://twitter.com/FreeSWITCH <http://twitter.com/FreeSWITCH> ☞ irc.freenode.net <http://irc.freenode.net/> #freeswitch ☞ http://freeswitch.org/g+ <http://freeswitch.org/g+>
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900 <tel:(919)%20386-9900>
https://www.youtube.com/watch?v=9XXgW34t40s <https://www.youtube.com/watch?v=9XXgW34t40s> https://www.youtube.com/watch?v=NLaDpGQuZDA <https://www.youtube.com/watch?v=NLaDpGQuZDA> _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://confluence.freeswitch.org <http://confluence.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org <http://www.freeswitch.org/>
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://confluence.freeswitch.org <http://confluence.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org <http://www.freeswitch.org/>
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ <http://freeswitch.org/> ☞ http://cluecon.com/ <http://cluecon.com/> ☞ http://twitter.com/FreeSWITCH <http://twitter.com/FreeSWITCH> ☞ irc.freenode.net <http://irc.freenode.net/> #freeswitch ☞ http://freeswitch.org/g+ <http://freeswitch.org/g+>
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s <https://www.youtube.com/watch?v=9XXgW34t40s> https://www.youtube.com/watch?v=NLaDpGQuZDA <https://www.youtube.com/watch?v=NLaDpGQuZDA> _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Cj, I’m going to do some stress testing today using sipp. Once I have results and a possible solution, I’ll post it here. From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Cj B Sent: woensdag 29 maart 2017 19:31 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] 100% CPU usage Anthony, I apologize I’m appearing “entitled”. I was simply asking for any other pointers. I’ve followed all the suggestions here when submitting the jira and have done so. I was just asking if there were any other tips/hints about what to look at Thanks for your response. Cj B On Mar 29, 2017, at 10:28 AM, Anthony Minessale <anthony.minessale@gmail.com<mailto:anthony.minessale@gmail.com>> wrote: Try to avoid doing things like pointing out how frustrated you are when seeking help here. It makes those trying to help you feel like you are being entitled. The suggestion on what to do was clearly outlined in previous posts in this thread including the exact commands to run when filing a jira. On Wed, Mar 29, 2017 at 12:20 PM, Cj B <blackc2004@gmail.com<mailto:blackc2004@gmail.com>> wrote: Hi All, I’m having a very similar issue however it’s not staying at 100% CPU on physical hardware. I’ve created a JIRA but wanted to see if anyone else had any feedback on possible causes? I’m using freeswitch 1.6.15 (previously 1.6.13). What happens is that FS will jump to 100% CPU and stay there for 1-2 seconds then drop back down but this is then causing massive jitter/call quality issues for about 10-15 seconds after the spike. The strange thing that I’ve noticed is that even with 0 active calls, FS is using 15-30% cpu constantly with only 40-60 registered endpoints. I’m watching the logs in DEBUG mode and not seeing anything there that’s indicative of any issue, no loops that I can see, and it seems to be happening when there’s even just 1 or 2 active calls on the system. I’m wondering if there’s any other suggestions on where to look because this is getting very frustrating for me and my users. Thanks much for any pointers! Cj B On Mar 22, 2017, at 8:13 AM, Anthony Minessale <anthony.minessale@gmail.com<mailto:anthony.minessale@gmail.com>> wrote: top -H -p `pidof freeswitch` Note the pid using 100% cpu gcore `pidof freeswitch` Attach it to a jira. If you think its a bug in FS you should report it to Jira not to this list. https://freeswitch.org/jira On Wed, Mar 22, 2017 at 10:14 AM Shaun Stokes <shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>> wrote: It’s probably a long shot, but if you’re able to replicate the problem it might be worth trying to replicate this on physical hardware. That way you eliminate your virtual environment from the problem, there are a number of factors which exist in virtual environments depending on your hyper-visor which can contribute to performance issues of which we’ve experienced in the past. It would also be worth trying to isolate what the script is doing when the high CPU begins, if you’ve got a test environment you can use to replicate the issue then you could try tweaking the script so you can eliminate certain actions from the list of causes, there’s maybe a FreeSWITCH module which is being called which triggers this. From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Grant Bagdasarian Sent: 22 March 2017 11:20 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage @Shuan, yes we are using the session:ready() method before we execute. We are running all of our servers virtually and never had any issues up until now. This same application ran on older versions of freeswitch without issues, even during and after high load. @Antonio, thanks for the tips! I’ll have a look and try some stuff. From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Antonio Silva Sent: woensdag 22 maart 2017 10:09 To: freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] 100% CPU usage We had some issues using pcall functions like io.popen when trying to execute external commands, that block mod_lua, i end up replacing that with os.execute. Recently i also had some issue with cpu hitting 100%,for what i could compile from debugging was something related with malloc calls. after compiling fs with tcmalloc the issue disappear, you could try this approach as well. You can check jira: https://freeswitch.org/jira/browse/FS-9957 and also in the malling: "CPU sys load issue" for more info. Regards, António On 03/22/2017 09:52 AM, Shaun Stokes wrote: Are you using ‘if (session:ready()) then’ in your scripts before you execute? Are you using virtual or physical hardware and what spec? From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Grant Bagdasarian Sent: 22 March 2017 07:31 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org><mailto:freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] 100% CPU usage Anthony, we triple checked the script and its dependencies. There are no loops at all in the script. The imported scripts are also written by us and they too do not contain any loops. It’s just basic call handling with some playback and play_and_get_digits calls, and some pcall calls to query a database. From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Anthony Minessale Sent: dinsdag 21 maart 2017 16:02 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage I already told you why a week ago. Your script most likely has an endless loop in it that does not check the condition session:ready() However, Feel free to contact consulting@freeswitch.org<mailto:consulting@freeswitch.org> and we can formally diagnose it for a fee. On Tue, Mar 21, 2017 at 6:42 AM Grant Bagdasarian <gb@cm.nl<mailto:gb@cm.nl>> wrote: Attached are the logs for: #check thread usage: top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1 #memory mapping: cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps We’re unable to run gcore since freeswitch hogs all available cpu’s at 100%. I tried changing the nice settings real-time but that didn’t work either. At one point we had a huge spike in inbound traffic towards our freeswitch (valid traffic). The traffic is handled by a lua application. What we saw was calls hanging in freeswitch while they’ve already been physically disconnected. After the huge spike freeswitch behaved normally for a couple of minutes and then suddenly started eating all the available cpu cores at 100%. We have some db queries which are fired in the lua application, so these may cause the application to hang, but we haven’t reproduced it yet. I’ll setup a test lab with the same configuration and fire sipp at it to see if I can reproduce the issue. However, I probably won’t be able to run gcore for debugging. Any other suggestions? Also we saw a lot of open udp ports and some had a huge read buffer size which did not decrease. From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Grant Bagdasarian Sent: maandag 13 maart 2017 09:04 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage Hi Antony and Antonio, Thank you for the tips. We haven’t had freeswitch going berserk anymore (not sure as to why it did perform without issue over the weekend), but will continue to monitor it and run the commands to trace the issue. Regards, Grant From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Anthony Minessale Sent: zaterdag 11 maart 2017 00:07 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage Make sure you lua script has all while loops including session:ready() as a condition to continue. Its a common mistake that leads to loops spinning out of control on hungup channels. On Fri, Mar 10, 2017 at 6:11 AM, Antonio Silva <asilva@wirelessmundi.com<mailto:asilva@wirelessmundi.com>> wrote: Hi, You can use the following commands: #get a core: gcore -o /home/gcore-$(date +%s) $(cat /var/run/freeswitch/freeswitch.pid) #check thread usage: top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1 #memory mapping: cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps you can get some info from fs: fs_cli -x "sofia global siptrace on" fs_cli -x "fsctl debug_level 10" fs_cli -x "sofia loglevel all 9" fs_cli -x "fsctl debug_sql " fs_cli -x "sofia global debug presence" Put everything into a jira so we can take a look. Regards, António On 03/10/2017 11:08 AM, Grant Bagdasarian wrote: Hi, We’re currently experiencing a weird issue with one of our freeswitch servers where the CPU just goes berserk and stays at 100%. There are no active calls at that moment, but FS is still using 100% CPU. I was unable to get a debug using gdb, since freeswitch does not crash. We had this issue on 1.6.13 and then I upgraded to 1.6.15 this morning, but the problem still persists. There is currently only one active application running on this box which is using LUA. Maybe that is causing the issue? We checked the code but there are no weird loops etc. Perhaps we are missing something? Any ideas how to trace this? Regards, Grant _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> -- Saludos / Regards / Cumprimentos, António silva _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net<http://irc.freenode.net/> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900<tel:(919)%20386-9900> https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net<http://irc.freenode.net/> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900<tel:(919)%20386-9900> https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ [http://www.itec-support.co.uk/wp-content/uploads/2016/07/email_logo.jpg] Shaun Stokes - Infrastructure Analyst T : 01453 700713 E : shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk> W : www.itec-support.co.uk<http://www.itec-support.co.uk/> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001 CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment. ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> -- Saludos / Regards / Cumprimentos, António silva ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ [http://www.itec-support.co.uk/wp-content/uploads/2016/07/email_logo.jpg] Shaun Stokes - Infrastructure Analyst T : 01453 700713 E : shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk> W : www.itec-support.co.uk<http://www.itec-support.co.uk> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001 CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment. ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net<http://irc.freenode.net/> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900<tel:(919)%20386-9900> https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net<http://irc.freenode.net/> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
The strange thing that I’ve noticed is that even with 0 active calls, FS is using 15-30% cpu constantly with only 40-60 registered endpoints.
How are you handling the registrations? Those endpoints will send REGISTER packets frequently which will individually get processed. So if they're calling an API though mod_xml_curl that'll be some extra load without active calls. mod_sofia will be updating the registration database which is sqlite by default and can be moved to a ramdisk or to ODBC. You might also find clients are registering unusually frequently. On 29 March 2017 at 18:20, Cj B <blackc2004@gmail.com> wrote:
Hi All,
I’m having a very similar issue however it’s not staying at 100% CPU on physical hardware. I’ve created a JIRA but wanted to see if anyone else had any feedback on possible causes? I’m using freeswitch 1.6.15 (previously 1.6.13). What happens is that FS will jump to 100% CPU and stay there for 1-2 seconds then drop back down but this is then causing massive jitter/call quality issues for about 10-15 seconds after the spike. The strange thing that I’ve noticed is that even with 0 active calls, FS is using 15-30% cpu constantly with only 40-60 registered endpoints.
I’m watching the logs in DEBUG mode and not seeing anything there that’s indicative of any issue, no loops that I can see, and it seems to be happening when there’s even just 1 or 2 active calls on the system.
I’m wondering if there’s any other suggestions on where to look because this is getting very frustrating for me and my users.
Thanks much for any pointers! Cj B
On Mar 22, 2017, at 8:13 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
top -H -p `pidof freeswitch` Note the pid using 100% cpu
gcore `pidof freeswitch`
Attach it to a jira. If you think its a bug in FS you should report it to Jira not to this list.
On Wed, Mar 22, 2017 at 10:14 AM Shaun Stokes < shaun.stokes@itec-support.co.uk> wrote:
It’s probably a long shot, but if you’re able to replicate the problem it might be worth trying to replicate this on physical hardware. That way you eliminate your virtual environment from the problem, there are a number of factors which exist in virtual environments depending on your hyper-visor which can contribute to performance issues of which we’ve experienced in the past.
It would also be worth trying to isolate what the script is doing when the high CPU begins, if you’ve got a test environment you can use to replicate the issue then you could try tweaking the script so you can eliminate certain actions from the list of causes, there’s maybe a FreeSWITCH module which is being called which triggers this.
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Grant Bagdasarian *Sent:* 22 March 2017 11:20
*To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] 100% CPU usage
@Shuan, yes we are using the session:ready() method before we execute.
We are running all of our servers virtually and never had any issues up until now.
This same application ran on older versions of freeswitch without issues, even during and after high load.
@Antonio, thanks for the tips! I’ll have a look and try some stuff.
*From:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org <freeswitch-users-bounces@lists.freeswitch.org>] *On Behalf Of *Antonio Silva *Sent:* woensdag 22 maart 2017 10:09 *To:* freeswitch-users@lists.freeswitch.org *Subject:* Re: [Freeswitch-users] 100% CPU usage
We had some issues using pcall functions like io.popen when trying to execute external commands, that block mod_lua, i end up replacing that with os.execute.
Recently i also had some issue with cpu hitting 100%,for what i could compile from debugging was something related with malloc calls. after compiling fs with tcmalloc the issue disappear, you could try this approach as well.
You can check jira: https://freeswitch.org/jira/browse/FS-9957 and also in the malling: "CPU sys load issue" for more info.
Regards, António
On 03/22/2017 09:52 AM, Shaun Stokes wrote:
Are you using ‘if (session:ready()) then’ in your scripts before you execute?
Are you using virtual or physical hardware and what spec?
*From:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org <freeswitch-users-bounces@lists.freeswitch.org>] *On Behalf Of *Grant Bagdasarian *Sent:* 22 March 2017 07:31 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] 100% CPU usage
Anthony, we triple checked the script and its dependencies. There are no loops at all in the script. The imported scripts are also written by us and they too do not contain any loops.
It’s just basic call handling with some playback and play_and_get_digits calls, and some pcall calls to query a database.
*From:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org <freeswitch-users-bounces@lists.freeswitch.org>] *On Behalf Of *Anthony Minessale *Sent:* dinsdag 21 maart 2017 16:02 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] 100% CPU usage
I already told you why a week ago. Your script most likely has an endless loop in it that does not check the condition session:ready()
However, Feel free to contact consulting@freeswitch.org and we can formally diagnose it for a fee.
On Tue, Mar 21, 2017 at 6:42 AM Grant Bagdasarian <gb@cm.nl> wrote:
Attached are the logs for:
#check thread usage:
top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1
#memory mapping:
cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps
We’re unable to run gcore since freeswitch hogs all available cpu’s at 100%.
I tried changing the nice settings real-time but that didn’t work either.
At one point we had a huge spike in inbound traffic towards our freeswitch (valid traffic).
The traffic is handled by a lua application.
What we saw was calls hanging in freeswitch while they’ve already been physically disconnected.
After the huge spike freeswitch behaved normally for a couple of minutes and then suddenly started eating all the available cpu cores at 100%.
We have some db queries which are fired in the lua application, so these may cause the application to hang, but we haven’t reproduced it yet.
I’ll setup a test lab with the same configuration and fire sipp at it to see if I can reproduce the issue.
However, I probably won’t be able to run gcore for debugging. Any other suggestions?
Also we saw a lot of open udp ports and some had a huge read buffer size which did not decrease.
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Grant Bagdasarian *Sent:* maandag 13 maart 2017 09:04
*To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] 100% CPU usage
Hi Antony and Antonio,
Thank you for the tips.
We haven’t had freeswitch going berserk anymore (not sure as to why it did perform without issue over the weekend), but will continue to monitor it and run the commands to trace the issue.
Regards,
Grant
*From:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org <freeswitch-users-bounces@lists.freeswitch.org>] *On Behalf Of *Anthony Minessale *Sent:* zaterdag 11 maart 2017 00:07 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] 100% CPU usage
Make sure you lua script has all while loops including session:ready() as a condition to continue. Its a common mistake that leads to loops spinning out of control on hungup channels.
On Fri, Mar 10, 2017 at 6:11 AM, Antonio Silva <asilva@wirelessmundi.com> wrote:
Hi,
You can use the following commands:
#get a core:
gcore -o /home/gcore-$(date +%s) $(cat /var/run/freeswitch/freeswitch.pid)
#check thread usage:
top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1
#memory mapping:
cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps
you can get some info from fs:
fs_cli -x "sofia global siptrace on"
fs_cli -x "fsctl debug_level 10"
fs_cli -x "sofia loglevel all 9"
fs_cli -x "fsctl debug_sql "
fs_cli -x "sofia global debug presence"
Put everything into a jira so we can take a look.
Regards, António
On 03/10/2017 11:08 AM, Grant Bagdasarian wrote:
Hi,
We’re currently experiencing a weird issue with one of our freeswitch servers where the CPU just goes berserk and stays at 100%.
There are no active calls at that moment, but FS is still using 100% CPU.
I was unable to get a debug using gdb, since freeswitch does not crash.
We had this issue on 1.6.13 and then I upgraded to 1.6.15 this morning, but the problem still persists.
There is currently only one active application running on this box which is using LUA. Maybe that is causing the issue?
We checked the code but there are no weird loops etc. Perhaps we are missing something?
Any ideas how to trace this?
Regards,
Grant
_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com
Official FreeSWITCH Sites
http://confluence.freeswitch.org
FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
--
Saludos / Regards / Cumprimentos,
António silva
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
--
Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH
☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call
☎ sip:888@conference.freeswitch.org ☎ +19193869900 <(919)%20386-9900>
https://www.youtube.com/watch?v=9XXgW34t40s
https://www.youtube.com/watch?v=NLaDpGQuZDA
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
--
Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH
☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call
☎ sip:888@conference.freeswitch.org ☎ +19193869900 <(919)%20386-9900>
https://www.youtube.com/watch?v=9XXgW34t40s
https://www.youtube.com/watch?v=NLaDpGQuZDA
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________
Shaun Stokes - Infrastructure Analyst
T :
01453 700713
E :
shaun.stokes@itec-support.co.uk
W :
www.itec-support.co.uk
Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001
CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment.
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________
_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com
Official FreeSWITCH Sites
http://confluence.freeswitch.org
FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
--
Saludos / Regards / Cumprimentos,
António silva
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ Shaun Stokes - Infrastructure Analyst T : 01453 700713 E : shaun.stokes@itec-support.co.uk W : www.itec-support.co.uk Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001
CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment.
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900 <(919)%20386-9900>
https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Cj B, What hyper-visor are you using? Also, what spec (AMD or Intel host, number of cores, RAM, 15k SAS or SSD storage, local or network storage etc)? This sounds very similar to problems we had on virtual hardware but we were closer to 400 registrations before we ran into problems, on physical hardware of the same spec we managed to get up to 800 registered extensions before we ran into limitations on the SIP profile (different symptoms). We’re now running roughly 1200 registered extensions across 2 SIP profiles on the same physical hardware average CPU under load roughly 25%. Shaun From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Cj B Sent: 29 March 2017 18:21 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] 100% CPU usage Hi All, I’m having a very similar issue however it’s not staying at 100% CPU on physical hardware. I’ve created a JIRA but wanted to see if anyone else had any feedback on possible causes? I’m using freeswitch 1.6.15 (previously 1.6.13). What happens is that FS will jump to 100% CPU and stay there for 1-2 seconds then drop back down but this is then causing massive jitter/call quality issues for about 10-15 seconds after the spike. The strange thing that I’ve noticed is that even with 0 active calls, FS is using 15-30% cpu constantly with only 40-60 registered endpoints. I’m watching the logs in DEBUG mode and not seeing anything there that’s indicative of any issue, no loops that I can see, and it seems to be happening when there’s even just 1 or 2 active calls on the system. I’m wondering if there’s any other suggestions on where to look because this is getting very frustrating for me and my users. Thanks much for any pointers! Cj B On Mar 22, 2017, at 8:13 AM, Anthony Minessale <anthony.minessale@gmail.com<mailto:anthony.minessale@gmail.com>> wrote: top -H -p `pidof freeswitch` Note the pid using 100% cpu gcore `pidof freeswitch` Attach it to a jira. If you think its a bug in FS you should report it to Jira not to this list. https://freeswitch.org/jira On Wed, Mar 22, 2017 at 10:14 AM Shaun Stokes <shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>> wrote: It’s probably a long shot, but if you’re able to replicate the problem it might be worth trying to replicate this on physical hardware. That way you eliminate your virtual environment from the problem, there are a number of factors which exist in virtual environments depending on your hyper-visor which can contribute to performance issues of which we’ve experienced in the past. It would also be worth trying to isolate what the script is doing when the high CPU begins, if you’ve got a test environment you can use to replicate the issue then you could try tweaking the script so you can eliminate certain actions from the list of causes, there’s maybe a FreeSWITCH module which is being called which triggers this. From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Grant Bagdasarian Sent: 22 March 2017 11:20 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage @Shuan, yes we are using the session:ready() method before we execute. We are running all of our servers virtually and never had any issues up until now. This same application ran on older versions of freeswitch without issues, even during and after high load. @Antonio, thanks for the tips! I’ll have a look and try some stuff. From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Antonio Silva Sent: woensdag 22 maart 2017 10:09 To: freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] 100% CPU usage We had some issues using pcall functions like io.popen when trying to execute external commands, that block mod_lua, i end up replacing that with os.execute. Recently i also had some issue with cpu hitting 100%,for what i could compile from debugging was something related with malloc calls. after compiling fs with tcmalloc the issue disappear, you could try this approach as well. You can check jira: https://freeswitch.org/jira/browse/FS-9957 and also in the malling: "CPU sys load issue" for more info. Regards, António On 03/22/2017 09:52 AM, Shaun Stokes wrote: Are you using ‘if (session:ready()) then’ in your scripts before you execute? Are you using virtual or physical hardware and what spec? From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Grant Bagdasarian Sent: 22 March 2017 07:31 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org><mailto:freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] 100% CPU usage Anthony, we triple checked the script and its dependencies. There are no loops at all in the script. The imported scripts are also written by us and they too do not contain any loops. It’s just basic call handling with some playback and play_and_get_digits calls, and some pcall calls to query a database. From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Anthony Minessale Sent: dinsdag 21 maart 2017 16:02 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage I already told you why a week ago. Your script most likely has an endless loop in it that does not check the condition session:ready() However, Feel free to contact consulting@freeswitch.org<mailto:consulting@freeswitch.org> and we can formally diagnose it for a fee. On Tue, Mar 21, 2017 at 6:42 AM Grant Bagdasarian <gb@cm.nl<mailto:gb@cm.nl>> wrote: Attached are the logs for: #check thread usage: top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1 #memory mapping: cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps We’re unable to run gcore since freeswitch hogs all available cpu’s at 100%. I tried changing the nice settings real-time but that didn’t work either. At one point we had a huge spike in inbound traffic towards our freeswitch (valid traffic). The traffic is handled by a lua application. What we saw was calls hanging in freeswitch while they’ve already been physically disconnected. After the huge spike freeswitch behaved normally for a couple of minutes and then suddenly started eating all the available cpu cores at 100%. We have some db queries which are fired in the lua application, so these may cause the application to hang, but we haven’t reproduced it yet. I’ll setup a test lab with the same configuration and fire sipp at it to see if I can reproduce the issue. However, I probably won’t be able to run gcore for debugging. Any other suggestions? Also we saw a lot of open udp ports and some had a huge read buffer size which did not decrease. From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Grant Bagdasarian Sent: maandag 13 maart 2017 09:04 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage Hi Antony and Antonio, Thank you for the tips. We haven’t had freeswitch going berserk anymore (not sure as to why it did perform without issue over the weekend), but will continue to monitor it and run the commands to trace the issue. Regards, Grant From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Anthony Minessale Sent: zaterdag 11 maart 2017 00:07 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage Make sure you lua script has all while loops including session:ready() as a condition to continue. Its a common mistake that leads to loops spinning out of control on hungup channels. On Fri, Mar 10, 2017 at 6:11 AM, Antonio Silva <asilva@wirelessmundi.com<mailto:asilva@wirelessmundi.com>> wrote: Hi, You can use the following commands: #get a core: gcore -o /home/gcore-$(date +%s) $(cat /var/run/freeswitch/freeswitch.pid) #check thread usage: top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1 #memory mapping: cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps you can get some info from fs: fs_cli -x "sofia global siptrace on" fs_cli -x "fsctl debug_level 10" fs_cli -x "sofia loglevel all 9" fs_cli -x "fsctl debug_sql " fs_cli -x "sofia global debug presence" Put everything into a jira so we can take a look. Regards, António On 03/10/2017 11:08 AM, Grant Bagdasarian wrote: Hi, We’re currently experiencing a weird issue with one of our freeswitch servers where the CPU just goes berserk and stays at 100%. There are no active calls at that moment, but FS is still using 100% CPU. I was unable to get a debug using gdb, since freeswitch does not crash. We had this issue on 1.6.13 and then I upgraded to 1.6.15 this morning, but the problem still persists. There is currently only one active application running on this box which is using LUA. Maybe that is causing the issue? We checked the code but there are no weird loops etc. Perhaps we are missing something? Any ideas how to trace this? Regards, Grant _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> -- Saludos / Regards / Cumprimentos, António silva _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net<http://irc.freenode.net/> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net<http://irc.freenode.net/> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ [http://www.itec-support.co.uk/wp-content/uploads/2016/07/email_logo.jpg] Shaun Stokes - Infrastructure Analyst T : 01453 700713 E : shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk> W : www.itec-support.co.uk<http://www.itec-support.co.uk/> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001 CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment. ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> -- Saludos / Regards / Cumprimentos, António silva ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ [http://www.itec-support.co.uk/wp-content/uploads/2016/07/email_logo.jpg] Shaun Stokes - Infrastructure Analyst T : 01453 700713 E : shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk> W : www.itec-support.co.uk<http://www.itec-support.co.uk> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001 CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment. ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net<http://irc.freenode.net/> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ [http://www.itec-support.co.uk/wp-content/uploads/2016/07/email_logo.jpg] Shaun Stokes - Infrastructure Analyst T : 01453 700713 E : shaun.stokes@itec-support.co.uk W : www.itec-support.co.uk Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001 CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment. ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________
Also, a few questions that might help us solve this. What is the registration expiry on the devices (i.e. how frequently are they trying to re-register)? If you have 60 devices which are re-registering every 30 seconds (120 registrations per minute) this would be comparable to over 2000 devices re-registering every 20 minutes. What are you SIP profile forced registry expiry settings (sip-force-expires, sip-expires-max-deviation)? Are you using presence (manage-presence)? From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Shaun Stokes Sent: 31 March 2017 08:17 To: 'FreeSWITCH Users Help' <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] 100% CPU usage Cj B, What hyper-visor are you using? Also, what spec (AMD or Intel host, number of cores, RAM, 15k SAS or SSD storage, local or network storage etc)? This sounds very similar to problems we had on virtual hardware but we were closer to 400 registrations before we ran into problems, on physical hardware of the same spec we managed to get up to 800 registered extensions before we ran into limitations on the SIP profile (different symptoms). We’re now running roughly 1200 registered extensions across 2 SIP profiles on the same physical hardware average CPU under load roughly 25%. Shaun From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Cj B Sent: 29 March 2017 18:21 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage Hi All, I’m having a very similar issue however it’s not staying at 100% CPU on physical hardware. I’ve created a JIRA but wanted to see if anyone else had any feedback on possible causes? I’m using freeswitch 1.6.15 (previously 1.6.13). What happens is that FS will jump to 100% CPU and stay there for 1-2 seconds then drop back down but this is then causing massive jitter/call quality issues for about 10-15 seconds after the spike. The strange thing that I’ve noticed is that even with 0 active calls, FS is using 15-30% cpu constantly with only 40-60 registered endpoints. I’m watching the logs in DEBUG mode and not seeing anything there that’s indicative of any issue, no loops that I can see, and it seems to be happening when there’s even just 1 or 2 active calls on the system. I’m wondering if there’s any other suggestions on where to look because this is getting very frustrating for me and my users. Thanks much for any pointers! Cj B On Mar 22, 2017, at 8:13 AM, Anthony Minessale <anthony.minessale@gmail.com<mailto:anthony.minessale@gmail.com>> wrote: top -H -p `pidof freeswitch` Note the pid using 100% cpu gcore `pidof freeswitch` Attach it to a jira. If you think its a bug in FS you should report it to Jira not to this list. https://freeswitch.org/jira On Wed, Mar 22, 2017 at 10:14 AM Shaun Stokes <shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>> wrote: It’s probably a long shot, but if you’re able to replicate the problem it might be worth trying to replicate this on physical hardware. That way you eliminate your virtual environment from the problem, there are a number of factors which exist in virtual environments depending on your hyper-visor which can contribute to performance issues of which we’ve experienced in the past. It would also be worth trying to isolate what the script is doing when the high CPU begins, if you’ve got a test environment you can use to replicate the issue then you could try tweaking the script so you can eliminate certain actions from the list of causes, there’s maybe a FreeSWITCH module which is being called which triggers this. From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Grant Bagdasarian Sent: 22 March 2017 11:20 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage @Shuan, yes we are using the session:ready() method before we execute. We are running all of our servers virtually and never had any issues up until now. This same application ran on older versions of freeswitch without issues, even during and after high load. @Antonio, thanks for the tips! I’ll have a look and try some stuff. From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Antonio Silva Sent: woensdag 22 maart 2017 10:09 To: freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] 100% CPU usage We had some issues using pcall functions like io.popen when trying to execute external commands, that block mod_lua, i end up replacing that with os.execute. Recently i also had some issue with cpu hitting 100%,for what i could compile from debugging was something related with malloc calls. after compiling fs with tcmalloc the issue disappear, you could try this approach as well. You can check jira: https://freeswitch.org/jira/browse/FS-9957 and also in the malling: "CPU sys load issue" for more info. Regards, António On 03/22/2017 09:52 AM, Shaun Stokes wrote: Are you using ‘if (session:ready()) then’ in your scripts before you execute? Are you using virtual or physical hardware and what spec? From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Grant Bagdasarian Sent: 22 March 2017 07:31 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org><mailto:freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] 100% CPU usage Anthony, we triple checked the script and its dependencies. There are no loops at all in the script. The imported scripts are also written by us and they too do not contain any loops. It’s just basic call handling with some playback and play_and_get_digits calls, and some pcall calls to query a database. From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Anthony Minessale Sent: dinsdag 21 maart 2017 16:02 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage I already told you why a week ago. Your script most likely has an endless loop in it that does not check the condition session:ready() However, Feel free to contact consulting@freeswitch.org<mailto:consulting@freeswitch.org> and we can formally diagnose it for a fee. On Tue, Mar 21, 2017 at 6:42 AM Grant Bagdasarian <gb@cm.nl<mailto:gb@cm.nl>> wrote: Attached are the logs for: #check thread usage: top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1 #memory mapping: cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps We’re unable to run gcore since freeswitch hogs all available cpu’s at 100%. I tried changing the nice settings real-time but that didn’t work either. At one point we had a huge spike in inbound traffic towards our freeswitch (valid traffic). The traffic is handled by a lua application. What we saw was calls hanging in freeswitch while they’ve already been physically disconnected. After the huge spike freeswitch behaved normally for a couple of minutes and then suddenly started eating all the available cpu cores at 100%. We have some db queries which are fired in the lua application, so these may cause the application to hang, but we haven’t reproduced it yet. I’ll setup a test lab with the same configuration and fire sipp at it to see if I can reproduce the issue. However, I probably won’t be able to run gcore for debugging. Any other suggestions? Also we saw a lot of open udp ports and some had a huge read buffer size which did not decrease. From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Grant Bagdasarian Sent: maandag 13 maart 2017 09:04 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage Hi Antony and Antonio, Thank you for the tips. We haven’t had freeswitch going berserk anymore (not sure as to why it did perform without issue over the weekend), but will continue to monitor it and run the commands to trace the issue. Regards, Grant From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Anthony Minessale Sent: zaterdag 11 maart 2017 00:07 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage Make sure you lua script has all while loops including session:ready() as a condition to continue. Its a common mistake that leads to loops spinning out of control on hungup channels. On Fri, Mar 10, 2017 at 6:11 AM, Antonio Silva <asilva@wirelessmundi.com<mailto:asilva@wirelessmundi.com>> wrote: Hi, You can use the following commands: #get a core: gcore -o /home/gcore-$(date +%s) $(cat /var/run/freeswitch/freeswitch.pid) #check thread usage: top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1 #memory mapping: cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps you can get some info from fs: fs_cli -x "sofia global siptrace on" fs_cli -x "fsctl debug_level 10" fs_cli -x "sofia loglevel all 9" fs_cli -x "fsctl debug_sql " fs_cli -x "sofia global debug presence" Put everything into a jira so we can take a look. Regards, António On 03/10/2017 11:08 AM, Grant Bagdasarian wrote: Hi, We’re currently experiencing a weird issue with one of our freeswitch servers where the CPU just goes berserk and stays at 100%. There are no active calls at that moment, but FS is still using 100% CPU. I was unable to get a debug using gdb, since freeswitch does not crash. We had this issue on 1.6.13 and then I upgraded to 1.6.15 this morning, but the problem still persists. There is currently only one active application running on this box which is using LUA. Maybe that is causing the issue? We checked the code but there are no weird loops etc. Perhaps we are missing something? Any ideas how to trace this? Regards, Grant _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> -- Saludos / Regards / Cumprimentos, António silva _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net<http://irc.freenode.net/> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net<http://irc.freenode.net/> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ [http://www.itec-support.co.uk/wp-content/uploads/2016/07/email_logo.jpg] Shaun Stokes - Infrastructure Analyst T : 01453 700713 E : shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk> W : www.itec-support.co.uk<http://www.itec-support.co.uk/> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001 CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment. ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> -- Saludos / Regards / Cumprimentos, António silva ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ [http://www.itec-support.co.uk/wp-content/uploads/2016/07/email_logo.jpg] Shaun Stokes - Infrastructure Analyst T : 01453 700713 E : shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk> W : www.itec-support.co.uk<http://www.itec-support.co.uk> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001 CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment. ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net<http://irc.freenode.net/> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ [http://www.itec-support.co.uk/wp-content/uploads/2016/07/email_logo.jpg] Shaun Stokes - Infrastructure Analyst T : 01453 700713 E : shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk> W : www.itec-support.co.uk<http://www.itec-support.co.uk> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001 CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment. ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ [http://www.itec-support.co.uk/wp-content/uploads/2016/07/email_logo.jpg] - T : E : W : www.itec-support.co.uk<http://www.itec-support.co.uk> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001 CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment. [http://www.itec-support.co.uk/wp-content/uploads/2016/07/email_logo.jpg] Shaun Stokes - Infrastructure Analyst T : 01453 700713 E : shaun.stokes@itec-support.co.uk W : www.itec-support.co.uk Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001 CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment. ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________
Hi All, Thanks for you detailed replies and suggestions… I spent yesterday setting up a test box and doing some testing. Just for information, my production boxes are Intel(R) Xeon(R) CPU E5430 @ 2.66GHz and Intel(R) Xeon(R) CPU L5520 @ 2.27GHz with 16gb RAM on raw hardware, 15ksas drives, local storage and core.db is in memory. My expiry is set to 360seconds and I do have manage-presence. I do not appear to have sip-force-expires set that I see anywhere. Here’s some interesting things I found. Test box was a digitalocean $5 droplet, 1 CPU, 512RAM. Configured it exactly as my production boxes, FS 1.6.15, all the same scripts, same number of Destinations, same number of domains, etc. I registered 10 phones to it and had 6 active calls, including 1 video call with no problems at all with expected CPU usage for such a low power box (around 50% CPU) With this information I then started looking at other things on my production boxes. I did a service freeswitch restart and when freeswitch came back up it was complaining that 5060 and 8021 were still in use. I did a netstat -npl | grep 8021 and noticed there was an “sh” process still active on both those ports. I killed the process and FS came back up. However I noticed that my RAM usage hadn’t decreased at all. I would expect that upon restating FS, it would free up RAM, am I mistaken here? Perhaps whatever that Sh process was is a cleanup? I then rebooted one of my servers and memory dropped and noticed that CPU usage had also dropped to about 1-2% when idle. So I rebooted the other servers as well and CPU usage dropped. It’s now been about 13 hours and I see CPU starting to climb again and my RAM is 100% used again. I will continue to monitor today and send any updates. Thanks again for everyone’s suggestions! Cj B
On Mar 31, 2017, at 12:34 AM, Shaun Stokes <shaun.stokes@itec-support.co.uk> wrote:
Also, a few questions that might help us solve this.
What is the registration expiry on the devices (i.e. how frequently are they trying to re-register)? If you have 60 devices which are re-registering every 30 seconds (120 registrations per minute) this would be comparable to over 2000 devices re-registering every 20 minutes.
What are you SIP profile forced registry expiry settings (sip-force-expires, sip-expires-max-deviation)?
Are you using presence (manage-presence)?
From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Shaun Stokes Sent: 31 March 2017 08:17 To: 'FreeSWITCH Users Help' <freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage
Cj B,
What hyper-visor are you using? Also, what spec (AMD or Intel host, number of cores, RAM, 15k SAS or SSD storage, local or network storage etc)?
This sounds very similar to problems we had on virtual hardware but we were closer to 400 registrations before we ran into problems, on physical hardware of the same spec we managed to get up to 800 registered extensions before we ran into limitations on the SIP profile (different symptoms). We’re now running roughly 1200 registered extensions across 2 SIP profiles on the same physical hardware average CPU under load roughly 25%.
Shaun
From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Cj B Sent: 29 March 2017 18:21 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage
Hi All,
I’m having a very similar issue however it’s not staying at 100% CPU on physical hardware. I’ve created a JIRA but wanted to see if anyone else had any feedback on possible causes? I’m using freeswitch 1.6.15 (previously 1.6.13). What happens is that FS will jump to 100% CPU and stay there for 1-2 seconds then drop back down but this is then causing massive jitter/call quality issues for about 10-15 seconds after the spike. The strange thing that I’ve noticed is that even with 0 active calls, FS is using 15-30% cpu constantly with only 40-60 registered endpoints.
I’m watching the logs in DEBUG mode and not seeing anything there that’s indicative of any issue, no loops that I can see, and it seems to be happening when there’s even just 1 or 2 active calls on the system.
I’m wondering if there’s any other suggestions on where to look because this is getting very frustrating for me and my users.
Thanks much for any pointers! Cj B
On Mar 22, 2017, at 8:13 AM, Anthony Minessale <anthony.minessale@gmail.com <mailto:anthony.minessale@gmail.com>> wrote:
top -H -p `pidof freeswitch` Note the pid using 100% cpu
gcore `pidof freeswitch`
Attach it to a jira. If you think its a bug in FS you should report it to Jira not to this list.
https://freeswitch.org/jira <https://freeswitch.org/jira>
On Wed, Mar 22, 2017 at 10:14 AM Shaun Stokes <shaun.stokes@itec-support.co.uk <mailto:shaun.stokes@itec-support.co.uk>> wrote: It’s probably a long shot, but if you’re able to replicate the problem it might be worth trying to replicate this on physical hardware. That way you eliminate your virtual environment from the problem, there are a number of factors which exist in virtual environments depending on your hyper-visor which can contribute to performance issues of which we’ve experienced in the past.
It would also be worth trying to isolate what the script is doing when the high CPU begins, if you’ve got a test environment you can use to replicate the issue then you could try tweaking the script so you can eliminate certain actions from the list of causes, there’s maybe a FreeSWITCH module which is being called which triggers this.
From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Grant Bagdasarian Sent: 22 March 2017 11:20
To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage
@Shuan, yes we are using the session:ready() method before we execute. We are running all of our servers virtually and never had any issues up until now. This same application ran on older versions of freeswitch without issues, even during and after high load.
@Antonio, thanks for the tips! I’ll have a look and try some stuff.
From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Antonio Silva Sent: woensdag 22 maart 2017 10:09 To: freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] 100% CPU usage
We had some issues using pcall functions like io.popen when trying to execute external commands, that block mod_lua, i end up replacing that with os.execute.
Recently i also had some issue with cpu hitting 100%,for what i could compile from debugging was something related with malloc calls. after compiling fs with tcmalloc the issue disappear, you could try this approach as well.
You can check jira: https://freeswitch.org/jira/browse/FS-9957 <https://freeswitch.org/jira/browse/FS-9957> and also in the malling: "CPU sys load issue" for more info.
Regards, António
On 03/22/2017 09:52 AM, Shaun Stokes wrote: Are you using ‘if (session:ready()) then’ in your scripts before you execute?
Are you using virtual or physical hardware and what spec?
From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>]On Behalf Of Grant Bagdasarian Sent: 22 March 2017 07:31 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> <mailto:freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] 100% CPU usage
Anthony, we triple checked the script and its dependencies. There are no loops at all in the script. The imported scripts are also written by us and they too do not contain any loops. It’s just basic call handling with some playback and play_and_get_digits calls, and some pcall calls to query a database.
From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>]On Behalf Of Anthony Minessale Sent: dinsdag 21 maart 2017 16:02 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage
I already told you why a week ago. Your script most likely has an endless loop in it that does not check the condition session:ready()
However, Feel free to contact consulting@freeswitch.org <mailto:consulting@freeswitch.org> and we can formally diagnose it for a fee.
On Tue, Mar 21, 2017 at 6:42 AM Grant Bagdasarian <gb@cm.nl <mailto:gb@cm.nl>> wrote: Attached are the logs for: #check thread usage: top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1
#memory mapping: cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps
We’re unable to run gcore since freeswitch hogs all available cpu’s at 100%. I tried changing the nice settings real-time but that didn’t work either.
At one point we had a huge spike in inbound traffic towards our freeswitch (valid traffic). The traffic is handled by a lua application. What we saw was calls hanging in freeswitch while they’ve already been physically disconnected. After the huge spike freeswitch behaved normally for a couple of minutes and then suddenly started eating all the available cpu cores at 100%.
We have some db queries which are fired in the lua application, so these may cause the application to hang, but we haven’t reproduced it yet.
I’ll setup a test lab with the same configuration and fire sipp at it to see if I can reproduce the issue. However, I probably won’t be able to run gcore for debugging. Any other suggestions?
Also we saw a lot of open udp ports and some had a huge read buffer size which did not decrease.
From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>]On Behalf Of Grant Bagdasarian Sent: maandag 13 maart 2017 09:04
To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage
Hi Antony and Antonio,
Thank you for the tips. We haven’t had freeswitch going berserk anymore (not sure as to why it did perform without issue over the weekend), but will continue to monitor it and run the commands to trace the issue.
Regards,
Grant
From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>]On Behalf Of Anthony Minessale Sent: zaterdag 11 maart 2017 00:07 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage
Make sure you lua script has all while loops including session:ready() as a condition to continue. Its a common mistake that leads to loops spinning out of control on hungup channels.
On Fri, Mar 10, 2017 at 6:11 AM, Antonio Silva <asilva@wirelessmundi.com <mailto:asilva@wirelessmundi.com>> wrote: Hi,
You can use the following commands:
#get a core: gcore -o /home/gcore-$(date +%s) $(cat /var/run/freeswitch/freeswitch.pid)
#check thread usage: top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1
#memory mapping: cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps
you can get some info from fs:
fs_cli -x "sofia global siptrace on" fs_cli -x "fsctl debug_level 10" fs_cli -x "sofia loglevel all 9" fs_cli -x "fsctl debug_sql " fs_cli -x "sofia global debug presence"
Put everything into a jira so we can take a look.
Regards, António
On 03/10/2017 11:08 AM, Grant Bagdasarian wrote: Hi,
We’re currently experiencing a weird issue with one of our freeswitch servers where the CPU just goes berserk and stays at 100%. There are no active calls at that moment, but FS is still using 100% CPU.
I was unable to get a debug using gdb, since freeswitch does not crash.
We had this issue on 1.6.13 and then I upgraded to 1.6.15 this morning, but the problem still persists. There is currently only one active application running on this box which is using LUA. Maybe that is causing the issue? We checked the code but there are no weird loops etc. Perhaps we are missing something?
Any ideas how to trace this?
Regards,
Grant
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://confluence.freeswitch.org <http://confluence.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org <http://www.freeswitch.org/>
--
Saludos / Regards / Cumprimentos, António silva
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://confluence.freeswitch.org <http://confluence.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org <http://www.freeswitch.org/>
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ <http://freeswitch.org/> ☞ http://cluecon.com/ <http://cluecon.com/> ☞ http://twitter.com/FreeSWITCH <http://twitter.com/FreeSWITCH> ☞ irc.freenode.net <http://irc.freenode.net/> #freeswitch ☞ http://freeswitch.org/g+ <http://freeswitch.org/g+> ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s <https://www.youtube.com/watch?v=9XXgW34t40s> https://www.youtube.com/watch?v=NLaDpGQuZDA <https://www.youtube.com/watch?v=NLaDpGQuZDA> _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://confluence.freeswitch.org <http://confluence.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org <http://www.freeswitch.org/> -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ <http://freeswitch.org/> ☞ http://cluecon.com/ <http://cluecon.com/> ☞ http://twitter.com/FreeSWITCH <http://twitter.com/FreeSWITCH> ☞ irc.freenode.net <http://irc.freenode.net/> #freeswitch ☞ http://freeswitch.org/g+ <http://freeswitch.org/g+> ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s <https://www.youtube.com/watch?v=9XXgW34t40s> https://www.youtube.com/watch?v=NLaDpGQuZDA <https://www.youtube.com/watch?v=NLaDpGQuZDA>
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________
Shaun Stokes - Infrastructure Analyst T : 01453 700713 E : shaun.stokes@itec-support.co.uk <mailto:shaun.stokes@itec-support.co.uk> W : www.itec-support.co.uk <http://www.itec-support.co.uk/> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001
CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment.
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://confluence.freeswitch.org <http://confluence.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org <http://www.freeswitch.org/>
--
Saludos / Regards / Cumprimentos, António silva
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________
Shaun Stokes - Infrastructure Analyst T : 01453 700713 E : shaun.stokes@itec-support.co.uk <mailto:shaun.stokes@itec-support.co.uk> W : www.itec-support.co.uk <http://www.itec-support.co.uk/> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001
CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment.
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://confluence.freeswitch.org <http://confluence.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org <http://www.freeswitch.org/> -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ <http://freeswitch.org/> ☞ http://cluecon.com/ <http://cluecon.com/> ☞ http://twitter.com/FreeSWITCH <http://twitter.com/FreeSWITCH> ☞ irc.freenode.net <http://irc.freenode.net/> #freeswitch ☞ http://freeswitch.org/g+ <http://freeswitch.org/g+> ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s <https://www.youtube.com/watch?v=9XXgW34t40s> https://www.youtube.com/watch?v=NLaDpGQuZDA <https://www.youtube.com/watch?v=NLaDpGQuZDA> _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://confluence.freeswitch.org <http://confluence.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org <http://www.freeswitch.org/>
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________
Shaun Stokes - Infrastructure Analyst T : 01453 700713 E : shaun.stokes@itec-support.co.uk <mailto:shaun.stokes@itec-support.co.uk> W : www.itec-support.co.uk <http://www.itec-support.co.uk/> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001
CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment.
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________
- T : E : W : www.itec-support.co.uk <http://www.itec-support.co.uk/> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001
CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment.
Shaun Stokes - Infrastructure Analyst <> T : <> 01453 700713 <> E : <> <>shaun.stokes@itec-support.co.uk <mailto:shaun.stokes@itec-support.co.uk> W : <> <>www.itec-support.co.uk <http://www.itec-support.co.uk/> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF <> Company No. 06908001 <>
CONFIDENTIALITY NOTICE <> This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. <> The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment. <> ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://confluence.freeswitch.org <http://confluence.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org <http://www.freeswitch.org/>
Just a small update. We did some testing today, and it looks like calls to the database are causing the call inside freeswitch to hang. We traced it back to the call to dbh:query where the call just hangs. It doesn’t even have to be high load. Generating only a total of 10 calls with 10 CPS causes 1 or 2 channels to hang. What is interesting is when I do a uuid_kill freeswitch cli reports it has been killed, but the show channels command still shows the channel there. Another uuid_kill returns “Error no such channel”. The channel id still shows up in the uuid_kill list. When we kill the sql session in mysql itself, that’s when freeswitch starts to go berserk and the CPU spikes to 100%. You can either kill it manually or wait for the mysql server timeout. Either way, 100% cpu usage. We put the session:ready() check everywhere possible, without success. We also rewrote some parts of the db access, connecting and releasing in the connection pool, but with no luck. From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Cj B Sent: vrijdag 31 maart 2017 17:04 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] 100% CPU usage Hi All, Thanks for you detailed replies and suggestions… I spent yesterday setting up a test box and doing some testing. Just for information, my production boxes are Intel(R) Xeon(R) CPU E5430 @ 2.66GHz and Intel(R) Xeon(R) CPU L5520 @ 2.27GHz with 16gb RAM on raw hardware, 15ksas drives, local storage and core.db is in memory. My expiry is set to 360seconds and I do have manage-presence. I do not appear to have sip-force-expires set that I see anywhere. Here’s some interesting things I found. * Test box was a digitalocean $5 droplet, 1 CPU, 512RAM. Configured it exactly as my production boxes, FS 1.6.15, all the same scripts, same number of Destinations, same number of domains, etc. I registered 10 phones to it and had 6 active calls, including 1 video call with no problems at all with expected CPU usage for such a low power box (around 50% CPU) * With this information I then started looking at other things on my production boxes. I did a service freeswitch restart and when freeswitch came back up it was complaining that 5060 and 8021 were still in use. I did a netstat -npl | grep 8021 and noticed there was an “sh” process still active on both those ports. I killed the process and FS came back up. However I noticed that my RAM usage hadn’t decreased at all. I would expect that upon restating FS, it would free up RAM, am I mistaken here? Perhaps whatever that Sh process was is a cleanup? * I then rebooted one of my servers and memory dropped and noticed that CPU usage had also dropped to about 1-2% when idle. So I rebooted the other servers as well and CPU usage dropped. It’s now been about 13 hours and I see CPU starting to climb again and my RAM is 100% used again. I will continue to monitor today and send any updates. Thanks again for everyone’s suggestions! Cj B On Mar 31, 2017, at 12:34 AM, Shaun Stokes <shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>> wrote: Also, a few questions that might help us solve this. What is the registration expiry on the devices (i.e. how frequently are they trying to re-register)? If you have 60 devices which are re-registering every 30 seconds (120 registrations per minute) this would be comparable to over 2000 devices re-registering every 20 minutes. What are you SIP profile forced registry expiry settings (sip-force-expires, sip-expires-max-deviation)? Are you using presence (manage-presence)? From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Shaun Stokes Sent: 31 March 2017 08:17 To: 'FreeSWITCH Users Help' <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage Cj B, What hyper-visor are you using? Also, what spec (AMD or Intel host, number of cores, RAM, 15k SAS or SSD storage, local or network storage etc)? This sounds very similar to problems we had on virtual hardware but we were closer to 400 registrations before we ran into problems, on physical hardware of the same spec we managed to get up to 800 registered extensions before we ran into limitations on the SIP profile (different symptoms). We’re now running roughly 1200 registered extensions across 2 SIP profiles on the same physical hardware average CPU under load roughly 25%. Shaun From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Cj B Sent: 29 March 2017 18:21 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage Hi All, I’m having a very similar issue however it’s not staying at 100% CPU on physical hardware. I’ve created a JIRA but wanted to see if anyone else had any feedback on possible causes? I’m using freeswitch 1.6.15 (previously 1.6.13). What happens is that FS will jump to 100% CPU and stay there for 1-2 seconds then drop back down but this is then causing massive jitter/call quality issues for about 10-15 seconds after the spike. The strange thing that I’ve noticed is that even with 0 active calls, FS is using 15-30% cpu constantly with only 40-60 registered endpoints. I’m watching the logs in DEBUG mode and not seeing anything there that’s indicative of any issue, no loops that I can see, and it seems to be happening when there’s even just 1 or 2 active calls on the system. I’m wondering if there’s any other suggestions on where to look because this is getting very frustrating for me and my users. Thanks much for any pointers! Cj B On Mar 22, 2017, at 8:13 AM, Anthony Minessale <anthony.minessale@gmail.com<mailto:anthony.minessale@gmail.com>> wrote: top -H -p `pidof freeswitch` Note the pid using 100% cpu gcore `pidof freeswitch` Attach it to a jira. If you think its a bug in FS you should report it to Jira not to this list. https://freeswitch.org/jira On Wed, Mar 22, 2017 at 10:14 AM Shaun Stokes <shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>> wrote: It’s probably a long shot, but if you’re able to replicate the problem it might be worth trying to replicate this on physical hardware. That way you eliminate your virtual environment from the problem, there are a number of factors which exist in virtual environments depending on your hyper-visor which can contribute to performance issues of which we’ve experienced in the past. It would also be worth trying to isolate what the script is doing when the high CPU begins, if you’ve got a test environment you can use to replicate the issue then you could try tweaking the script so you can eliminate certain actions from the list of causes, there’s maybe a FreeSWITCH module which is being called which triggers this. From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Grant Bagdasarian Sent: 22 March 2017 11:20 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage @Shuan, yes we are using the session:ready() method before we execute. We are running all of our servers virtually and never had any issues up until now. This same application ran on older versions of freeswitch without issues, even during and after high load. @Antonio, thanks for the tips! I’ll have a look and try some stuff. From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Antonio Silva Sent: woensdag 22 maart 2017 10:09 To: freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] 100% CPU usage We had some issues using pcall functions like io.popen when trying to execute external commands, that block mod_lua, i end up replacing that with os.execute. Recently i also had some issue with cpu hitting 100%,for what i could compile from debugging was something related with malloc calls. after compiling fs with tcmalloc the issue disappear, you could try this approach as well. You can check jira: https://freeswitch.org/jira/browse/FS-9957 and also in the malling: "CPU sys load issue" for more info. Regards, António On 03/22/2017 09:52 AM, Shaun Stokes wrote: Are you using ‘if (session:ready()) then’ in your scripts before you execute? Are you using virtual or physical hardware and what spec? From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org]On Behalf Of Grant Bagdasarian Sent: 22 March 2017 07:31 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org><mailto:freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] 100% CPU usage Anthony, we triple checked the script and its dependencies. There are no loops at all in the script. The imported scripts are also written by us and they too do not contain any loops. It’s just basic call handling with some playback and play_and_get_digits calls, and some pcall calls to query a database. From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org]On Behalf Of Anthony Minessale Sent: dinsdag 21 maart 2017 16:02 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage I already told you why a week ago. Your script most likely has an endless loop in it that does not check the condition session:ready() However, Feel free to contact consulting@freeswitch.org<mailto:consulting@freeswitch.org> and we can formally diagnose it for a fee. On Tue, Mar 21, 2017 at 6:42 AM Grant Bagdasarian <gb@cm.nl<mailto:gb@cm.nl>> wrote: Attached are the logs for: #check thread usage: top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1 #memory mapping: cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps We’re unable to run gcore since freeswitch hogs all available cpu’s at 100%. I tried changing the nice settings real-time but that didn’t work either. At one point we had a huge spike in inbound traffic towards our freeswitch (valid traffic). The traffic is handled by a lua application. What we saw was calls hanging in freeswitch while they’ve already been physically disconnected. After the huge spike freeswitch behaved normally for a couple of minutes and then suddenly started eating all the available cpu cores at 100%. We have some db queries which are fired in the lua application, so these may cause the application to hang, but we haven’t reproduced it yet. I’ll setup a test lab with the same configuration and fire sipp at it to see if I can reproduce the issue. However, I probably won’t be able to run gcore for debugging. Any other suggestions? Also we saw a lot of open udp ports and some had a huge read buffer size which did not decrease. From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>]On Behalf Of Grant Bagdasarian Sent: maandag 13 maart 2017 09:04 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage Hi Antony and Antonio, Thank you for the tips. We haven’t had freeswitch going berserk anymore (not sure as to why it did perform without issue over the weekend), but will continue to monitor it and run the commands to trace the issue. Regards, Grant From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org]On Behalf Of Anthony Minessale Sent: zaterdag 11 maart 2017 00:07 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage Make sure you lua script has all while loops including session:ready() as a condition to continue. Its a common mistake that leads to loops spinning out of control on hungup channels. On Fri, Mar 10, 2017 at 6:11 AM, Antonio Silva <asilva@wirelessmundi.com<mailto:asilva@wirelessmundi.com>> wrote: Hi, You can use the following commands: #get a core: gcore -o /home/gcore-$(date +%s) $(cat /var/run/freeswitch/freeswitch.pid) #check thread usage: top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1 #memory mapping: cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps you can get some info from fs: fs_cli -x "sofia global siptrace on" fs_cli -x "fsctl debug_level 10" fs_cli -x "sofia loglevel all 9" fs_cli -x "fsctl debug_sql " fs_cli -x "sofia global debug presence" Put everything into a jira so we can take a look. Regards, António On 03/10/2017 11:08 AM, Grant Bagdasarian wrote: Hi, We’re currently experiencing a weird issue with one of our freeswitch servers where the CPU just goes berserk and stays at 100%. There are no active calls at that moment, but FS is still using 100% CPU. I was unable to get a debug using gdb, since freeswitch does not crash. We had this issue on 1.6.13 and then I upgraded to 1.6.15 this morning, but the problem still persists. There is currently only one active application running on this box which is using LUA. Maybe that is causing the issue? We checked the code but there are no weird loops etc. Perhaps we are missing something? Any ideas how to trace this? Regards, Grant _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> -- Saludos / Regards / Cumprimentos, António silva _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net<http://irc.freenode.net/> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net<http://irc.freenode.net/> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ [http://www.itec-support.co.uk/wp-content/uploads/2016/07/email_logo.jpg] Shaun Stokes - Infrastructure Analyst T : 01453 700713 E : shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk> W : www.itec-support.co.uk<http://www.itec-support.co.uk/> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001 CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment. ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> -- Saludos / Regards / Cumprimentos, António silva ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ [http://www.itec-support.co.uk/wp-content/uploads/2016/07/email_logo.jpg] Shaun Stokes - Infrastructure Analyst T : 01453 700713 E : shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk> W : www.itec-support.co.uk<http://www.itec-support.co.uk/> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001 CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment. ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net<http://irc.freenode.net/> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ [http://www.itec-support.co.uk/wp-content/uploads/2016/07/email_logo.jpg] Shaun Stokes - Infrastructure Analyst T : 01453 700713 E : shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk> W : www.itec-support.co.uk<http://www.itec-support.co.uk/> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001 CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment. ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ [http://www.itec-support.co.uk/wp-content/uploads/2016/07/email_logo.jpg] - T : E : W : www.itec-support.co.uk<http://www.itec-support.co.uk/> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001 CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment. [http://www.itec-support.co.uk/wp-content/uploads/2016/07/email_logo.jpg] Shaun Stokes - Infrastructure Analyst T : 01453 700713 E : shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk> W : www.itec-support.co.uk<http://www.itec-support.co.uk/> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001 CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment. ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/>
So when you uuid_kill it and it doesn't end, that means that the thread is blocking on something and it can't end until that blocking call returns. So if you try again to uuid_kill it it will not work because you have already tagged the session to end. If you kill the mysql connection and it runs out of control yet you say you have 0 loops in the program that do not respect session:ready() it seems like you do have something in an endless loop still. Do you see anything specific in the log when you do this killing of the db? or get another capture. However, you have drawn me into answering you on this list for some reason. You should be updating the JIRA. On Mon, Apr 3, 2017 at 9:38 AM, Grant Bagdasarian <gb@cm.nl> wrote:
Just a small update.
We did some testing today, and it looks like calls to the database are causing the call inside freeswitch to hang.
We traced it back to the call to dbh:query where the call just hangs. It doesn’t even have to be high load. Generating only a total of 10 calls with 10 CPS causes 1 or 2 channels to hang.
What is interesting is when I do a uuid_kill freeswitch cli reports it has been killed, but the show channels command still shows the channel there. Another uuid_kill returns “Error no such channel”. The channel id still shows up in the uuid_kill list.
When we kill the sql session in mysql itself, that’s when freeswitch starts to go berserk and the CPU spikes to 100%. You can either kill it manually or wait for the mysql server timeout. Either way, 100% cpu usage.
We put the session:ready() check everywhere possible, without success. We also rewrote some parts of the db access, connecting and releasing in the connection pool, but with no luck.
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Cj B *Sent:* vrijdag 31 maart 2017 17:04
*To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] 100% CPU usage
Hi All,
Thanks for you detailed replies and suggestions… I spent yesterday setting up a test box and doing some testing.
Just for information, my production boxes are Intel(R) Xeon(R) CPU E5430 @ 2.66GHz and Intel(R) Xeon(R) CPU L5520 @ 2.27GHz with 16gb RAM on raw hardware, 15ksas drives, local storage and core.db is in memory. My expiry is set to 360seconds and I do have manage-presence. I do not appear to have sip-force-expires set that I see anywhere.
Here’s some interesting things I found.
- Test box was a digitalocean $5 droplet, 1 CPU, 512RAM. Configured it exactly as my production boxes, FS 1.6.15, all the same scripts, same number of Destinations, same number of domains, etc. I registered 10 phones to it and had 6 active calls, including 1 video call with no problems at all with expected CPU usage for such a low power box (around 50% CPU) - With this information I then started looking at other things on my production boxes. I did a service freeswitch restart and when freeswitch came back up it was complaining that 5060 and 8021 were still in use. I did a netstat -npl | grep 8021 and noticed there was an “sh” process still active on both those ports. I killed the process and FS came back up. However I noticed that my RAM usage hadn’t decreased at all. I would expect that upon restating FS, it would free up RAM, am I mistaken here? Perhaps whatever that Sh process was is a cleanup? - I then rebooted one of my servers and memory dropped and noticed that CPU usage had also dropped to about 1-2% when idle. So I rebooted the other servers as well and CPU usage dropped.
It’s now been about 13 hours and I see CPU starting to climb again and my RAM is 100% used again.
I will continue to monitor today and send any updates. Thanks again for everyone’s suggestions!
Cj B
On Mar 31, 2017, at 12:34 AM, Shaun Stokes <shaun.stokes@itec-support.co. uk> wrote:
Also, a few questions that might help us solve this.
What is the registration expiry on the devices (i.e. how frequently are they trying to re-register)? If you have 60 devices which are re-registering every 30 seconds (120 registrations per minute) this would be comparable to over 2000 devices re-registering every 20 minutes.
What are you SIP profile forced registry expiry settings (sip-force-expires, sip-expires-max-deviation)?
Are you using presence (manage-presence)?
*From:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org <freeswitch-users-bounces@lists.freeswitch.org>] *On Behalf Of *Shaun Stokes *Sent:* 31 March 2017 08:17 *To:* 'FreeSWITCH Users Help' <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] 100% CPU usage
Cj B,
What hyper-visor are you using? Also, what spec (AMD or Intel host, number of cores, RAM, 15k SAS or SSD storage, local or network storage etc)?
This sounds very similar to problems we had on virtual hardware but we were closer to 400 registrations before we ran into problems, on physical hardware of the same spec we managed to get up to 800 registered extensions before we ran into limitations on the SIP profile (different symptoms). We’re now running roughly 1200 registered extensions across 2 SIP profiles on the same physical hardware average CPU under load roughly 25%.
Shaun
*From:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org <freeswitch-users-bounces@lists.freeswitch.org>] *On Behalf Of *Cj B *Sent:* 29 March 2017 18:21 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] 100% CPU usage
Hi All,
I’m having a very similar issue however it’s not staying at 100% CPU on physical hardware. I’ve created a JIRA but wanted to see if anyone else had any feedback on possible causes? I’m using freeswitch 1.6.15 (previously 1.6.13). What happens is that FS will jump to 100% CPU and stay there for 1-2 seconds then drop back down but this is then causing massive jitter/call quality issues for about 10-15 seconds after the spike. The strange thing that I’ve noticed is that even with 0 active calls, FS is using 15-30% cpu constantly with only 40-60 registered endpoints.
I’m watching the logs in DEBUG mode and not seeing anything there that’s indicative of any issue, no loops that I can see, and it seems to be happening when there’s even just 1 or 2 active calls on the system.
I’m wondering if there’s any other suggestions on where to look because this is getting very frustrating for me and my users.
Thanks much for any pointers!
Cj B
On Mar 22, 2017, at 8:13 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
top -H -p `pidof freeswitch`
Note the pid using 100% cpu
gcore `pidof freeswitch`
Attach it to a jira. If you think its a bug in FS you should report it to Jira not to this list.
On Wed, Mar 22, 2017 at 10:14 AM Shaun Stokes < shaun.stokes@itec-support.co.uk> wrote:
It’s probably a long shot, but if you’re able to replicate the problem it might be worth trying to replicate this on physical hardware. That way you eliminate your virtual environment from the problem, there are a number of factors which exist in virtual environments depending on your hyper-visor which can contribute to performance issues of which we’ve experienced in the past.
It would also be worth trying to isolate what the script is doing when the high CPU begins, if you’ve got a test environment you can use to replicate the issue then you could try tweaking the script so you can eliminate certain actions from the list of causes, there’s maybe a FreeSWITCH module which is being called which triggers this.
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of* Grant Bagdasarian *Sent:* 22 March 2017 11:20
*To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] 100% CPU usage
@Shuan, yes we are using the session:ready() method before we execute.
We are running all of our servers virtually and never had any issues up until now.
This same application ran on older versions of freeswitch without issues, even during and after high load.
@Antonio, thanks for the tips! I’ll have a look and try some stuff.
*From:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org <freeswitch-users-bounces@lists.freeswitch.org>] *On Behalf Of* Antonio Silva *Sent:* woensdag 22 maart 2017 10:09 *To:* freeswitch-users@lists.freeswitch.org *Subject:* Re: [Freeswitch-users] 100% CPU usage
We had some issues using pcall functions like io.popen when trying to execute external commands, that block mod_lua, i end up replacing that with os.execute.
Recently i also had some issue with cpu hitting 100%,for what i could compile from debugging was something related with malloc calls. after compiling fs with tcmalloc the issue disappear, you could try this approach as well.
You can check jira: https://freeswitch.org/jira/browse/FS-9957 and also in the malling: "CPU sys load issue" for more info.
Regards, António
On 03/22/2017 09:52 AM, Shaun Stokes wrote:
Are you using ‘if (session:ready()) then’ in your scripts before you execute?
Are you using virtual or physical hardware and what spec?
*From:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org <freeswitch-users-bounces@lists.freeswitch.org>]*On Behalf Of* Grant Bagdasarian *Sent:* 22 March 2017 07:31 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] 100% CPU usage
Anthony, we triple checked the script and its dependencies. There are no loops at all in the script. The imported scripts are also written by us and they too do not contain any loops.
It’s just basic call handling with some playback and play_and_get_digits calls, and some pcall calls to query a database.
*From:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org <freeswitch-users-bounces@lists.freeswitch.org>]*On Behalf Of* Anthony Minessale *Sent:* dinsdag 21 maart 2017 16:02 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] 100% CPU usage
I already told you why a week ago. Your script most likely has an endless loop in it that does not check the condition session:ready()
However, Feel free to contact consulting@freeswitch.org and we can formally diagnose it for a fee.
On Tue, Mar 21, 2017 at 6:42 AM Grant Bagdasarian <gb@cm.nl> wrote:
Attached are the logs for:
#check thread usage:
top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1
#memory mapping:
cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps
We’re unable to run gcore since freeswitch hogs all available cpu’s at 100%.
I tried changing the nice settings real-time but that didn’t work either.
At one point we had a huge spike in inbound traffic towards our freeswitch (valid traffic).
The traffic is handled by a lua application.
What we saw was calls hanging in freeswitch while they’ve already been physically disconnected.
After the huge spike freeswitch behaved normally for a couple of minutes and then suddenly started eating all the available cpu cores at 100%.
We have some db queries which are fired in the lua application, so these may cause the application to hang, but we haven’t reproduced it yet.
I’ll setup a test lab with the same configuration and fire sipp at it to see if I can reproduce the issue.
However, I probably won’t be able to run gcore for debugging. Any other suggestions?
Also we saw a lot of open udp ports and some had a huge read buffer size which did not decrease.
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org]*On Behalf Of* Grant Bagdasarian *Sent:* maandag 13 maart 2017 09:04
*To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] 100% CPU usage
Hi Antony and Antonio,
Thank you for the tips.
We haven’t had freeswitch going berserk anymore (not sure as to why it did perform without issue over the weekend), but will continue to monitor it and run the commands to trace the issue.
Regards,
Grant
*From:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org <freeswitch-users-bounces@lists.freeswitch.org>]*On Behalf Of* Anthony Minessale *Sent:* zaterdag 11 maart 2017 00:07 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] 100% CPU usage
Make sure you lua script has all while loops including session:ready() as a condition to continue. Its a common mistake that leads to loops spinning out of control on hungup channels.
On Fri, Mar 10, 2017 at 6:11 AM, Antonio Silva <asilva@wirelessmundi.com> wrote:
Hi,
You can use the following commands:
#get a core:
gcore -o /home/gcore-$(date +%s) $(cat /var/run/freeswitch/freeswitch.pid)
#check thread usage:
top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1
#memory mapping:
cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps
you can get some info from fs:
fs_cli -x "sofia global siptrace on"
fs_cli -x "fsctl debug_level 10"
fs_cli -x "sofia loglevel all 9"
fs_cli -x "fsctl debug_sql "
fs_cli -x "sofia global debug presence"
Put everything into a jira so we can take a look.
Regards, António
On 03/10/2017 11:08 AM, Grant Bagdasarian wrote:
Hi,
We’re currently experiencing a weird issue with one of our freeswitch servers where the CPU just goes berserk and stays at 100%.
There are no active calls at that moment, but FS is still using 100% CPU.
I was unable to get a debug using gdb, since freeswitch does not crash.
We had this issue on 1.6.13 and then I upgraded to 1.6.15 this morning, but the problem still persists.
There is currently only one active application running on this box which is using LUA. Maybe that is causing the issue?
We checked the code but there are no weird loops etc. Perhaps we are missing something?
Any ideas how to trace this?
Regards,
Grant
_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com
Official FreeSWITCH Sites
http://confluence.freeswitch.org
FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
--
Saludos / Regards / Cumprimentos,
António silva
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
--
Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http:// twitter.com/FreeSWITCH
☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call
☎ sip:888@conference.freeswitch.org ☎ +19193869900 <(919)%20386-9900>
https://www.youtube.com/watch?v=9XXgW34t40s
https://www.youtube.com/watch?v=NLaDpGQuZDA
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
--
Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http:// twitter.com/FreeSWITCH
☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call
☎ sip:888@conference.freeswitch.org ☎ +19193869900 <(919)%20386-9900>
https://www.youtube.com/watch?v=9XXgW34t40s
https://www.youtube.com/watch?v=NLaDpGQuZDA
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________
Shaun Stokes - Infrastructure Analyst
T :
01453 700713
E :
shaun.stokes@itec-support.co.uk
W :
www.itec-support.co.uk
Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001
CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment.
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________
_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com
Official FreeSWITCH Sites
http://confluence.freeswitch.org
FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
--
Saludos / Regards / Cumprimentos,
António silva
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________
Shaun Stokes - Infrastructure Analyst
T :
01453 700713
E :
shaun.stokes@itec-support.co.uk
W :
www.itec-support.co.uk
Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001
CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment.
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
--
Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http:// twitter.com/FreeSWITCH
☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call
☎ sip:888@conference.freeswitch.org ☎ +19193869900 <(919)%20386-9900>
https://www.youtube.com/watch?v=9XXgW34t40s
https://www.youtube.com/watch?v=NLaDpGQuZDA
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________
Shaun Stokes - Infrastructure Analyst
T :
01453 700713
E :
shaun.stokes@itec-support.co.uk
W :
www.itec-support.co.uk
Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001
CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment.
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________
-
T :
E :
W :
www.itec-support.co.uk
Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001
CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment.
Shaun Stokes - Infrastructure Analyst
T :
01453 700713
E :
shaun.stokes@itec-support.co.uk
W :
www.itec-support.co.uk
Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001
CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment.
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
It sounds like you may have narrowed it down to a DB query, if you aren’t already then I’d suggest logging the output of your query and the results. You can use something similar to the this to see at which point it fails (or loops): freeswitch.consoleLog("DEBUG", "text here "..variable_here.." \n"); I’d also suggest checking your SQL logs if you haven’t already to check there isn’t anything out of the ordinary when the problem occurs. If you replace the DB query(s) with static variables for testing purposes does this resolve your issue? From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Grant Bagdasarian Sent: 03 April 2017 14:38 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] 100% CPU usage Just a small update. We did some testing today, and it looks like calls to the database are causing the call inside freeswitch to hang. We traced it back to the call to dbh:query where the call just hangs. It doesn’t even have to be high load. Generating only a total of 10 calls with 10 CPS causes 1 or 2 channels to hang. What is interesting is when I do a uuid_kill freeswitch cli reports it has been killed, but the show channels command still shows the channel there. Another uuid_kill returns “Error no such channel”. The channel id still shows up in the uuid_kill list. When we kill the sql session in mysql itself, that’s when freeswitch starts to go berserk and the CPU spikes to 100%. You can either kill it manually or wait for the mysql server timeout. Either way, 100% cpu usage. We put the session:ready() check everywhere possible, without success. We also rewrote some parts of the db access, connecting and releasing in the connection pool, but with no luck. From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Cj B Sent: vrijdag 31 maart 2017 17:04 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage Hi All, Thanks for you detailed replies and suggestions… I spent yesterday setting up a test box and doing some testing. Just for information, my production boxes are Intel(R) Xeon(R) CPU E5430 @ 2.66GHz and Intel(R) Xeon(R) CPU L5520 @ 2.27GHz with 16gb RAM on raw hardware, 15ksas drives, local storage and core.db is in memory. My expiry is set to 360seconds and I do have manage-presence. I do not appear to have sip-force-expires set that I see anywhere. Here’s some interesting things I found. * Test box was a digitalocean $5 droplet, 1 CPU, 512RAM. Configured it exactly as my production boxes, FS 1.6.15, all the same scripts, same number of Destinations, same number of domains, etc. I registered 10 phones to it and had 6 active calls, including 1 video call with no problems at all with expected CPU usage for such a low power box (around 50% CPU) * With this information I then started looking at other things on my production boxes. I did a service freeswitch restart and when freeswitch came back up it was complaining that 5060 and 8021 were still in use. I did a netstat -npl | grep 8021 and noticed there was an “sh” process still active on both those ports. I killed the process and FS came back up. However I noticed that my RAM usage hadn’t decreased at all. I would expect that upon restating FS, it would free up RAM, am I mistaken here? Perhaps whatever that Sh process was is a cleanup? * I then rebooted one of my servers and memory dropped and noticed that CPU usage had also dropped to about 1-2% when idle. So I rebooted the other servers as well and CPU usage dropped. It’s now been about 13 hours and I see CPU starting to climb again and my RAM is 100% used again. I will continue to monitor today and send any updates. Thanks again for everyone’s suggestions! Cj B On Mar 31, 2017, at 12:34 AM, Shaun Stokes <shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>> wrote: Also, a few questions that might help us solve this. What is the registration expiry on the devices (i.e. how frequently are they trying to re-register)? If you have 60 devices which are re-registering every 30 seconds (120 registrations per minute) this would be comparable to over 2000 devices re-registering every 20 minutes. What are you SIP profile forced registry expiry settings (sip-force-expires, sip-expires-max-deviation)? Are you using presence (manage-presence)? From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Shaun Stokes Sent: 31 March 2017 08:17 To: 'FreeSWITCH Users Help' <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage Cj B, What hyper-visor are you using? Also, what spec (AMD or Intel host, number of cores, RAM, 15k SAS or SSD storage, local or network storage etc)? This sounds very similar to problems we had on virtual hardware but we were closer to 400 registrations before we ran into problems, on physical hardware of the same spec we managed to get up to 800 registered extensions before we ran into limitations on the SIP profile (different symptoms). We’re now running roughly 1200 registered extensions across 2 SIP profiles on the same physical hardware average CPU under load roughly 25%. Shaun From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Cj B Sent: 29 March 2017 18:21 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage Hi All, I’m having a very similar issue however it’s not staying at 100% CPU on physical hardware. I’ve created a JIRA but wanted to see if anyone else had any feedback on possible causes? I’m using freeswitch 1.6.15 (previously 1.6.13). What happens is that FS will jump to 100% CPU and stay there for 1-2 seconds then drop back down but this is then causing massive jitter/call quality issues for about 10-15 seconds after the spike. The strange thing that I’ve noticed is that even with 0 active calls, FS is using 15-30% cpu constantly with only 40-60 registered endpoints. I’m watching the logs in DEBUG mode and not seeing anything there that’s indicative of any issue, no loops that I can see, and it seems to be happening when there’s even just 1 or 2 active calls on the system. I’m wondering if there’s any other suggestions on where to look because this is getting very frustrating for me and my users. Thanks much for any pointers! Cj B On Mar 22, 2017, at 8:13 AM, Anthony Minessale <anthony.minessale@gmail.com<mailto:anthony.minessale@gmail.com>> wrote: top -H -p `pidof freeswitch` Note the pid using 100% cpu gcore `pidof freeswitch` Attach it to a jira. If you think its a bug in FS you should report it to Jira not to this list. https://freeswitch.org/jira On Wed, Mar 22, 2017 at 10:14 AM Shaun Stokes <shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>> wrote: It’s probably a long shot, but if you’re able to replicate the problem it might be worth trying to replicate this on physical hardware. That way you eliminate your virtual environment from the problem, there are a number of factors which exist in virtual environments depending on your hyper-visor which can contribute to performance issues of which we’ve experienced in the past. It would also be worth trying to isolate what the script is doing when the high CPU begins, if you’ve got a test environment you can use to replicate the issue then you could try tweaking the script so you can eliminate certain actions from the list of causes, there’s maybe a FreeSWITCH module which is being called which triggers this. From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Grant Bagdasarian Sent: 22 March 2017 11:20 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage @Shuan, yes we are using the session:ready() method before we execute. We are running all of our servers virtually and never had any issues up until now. This same application ran on older versions of freeswitch without issues, even during and after high load. @Antonio, thanks for the tips! I’ll have a look and try some stuff. From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Antonio Silva Sent: woensdag 22 maart 2017 10:09 To: freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] 100% CPU usage We had some issues using pcall functions like io.popen when trying to execute external commands, that block mod_lua, i end up replacing that with os.execute. Recently i also had some issue with cpu hitting 100%,for what i could compile from debugging was something related with malloc calls. after compiling fs with tcmalloc the issue disappear, you could try this approach as well. You can check jira: https://freeswitch.org/jira/browse/FS-9957 and also in the malling: "CPU sys load issue" for more info. Regards, António On 03/22/2017 09:52 AM, Shaun Stokes wrote: Are you using ‘if (session:ready()) then’ in your scripts before you execute? Are you using virtual or physical hardware and what spec? From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org]On Behalf Of Grant Bagdasarian Sent: 22 March 2017 07:31 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org><mailto:freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] 100% CPU usage Anthony, we triple checked the script and its dependencies. There are no loops at all in the script. The imported scripts are also written by us and they too do not contain any loops. It’s just basic call handling with some playback and play_and_get_digits calls, and some pcall calls to query a database. From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org]On Behalf Of Anthony Minessale Sent: dinsdag 21 maart 2017 16:02 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage I already told you why a week ago. Your script most likely has an endless loop in it that does not check the condition session:ready() However, Feel free to contact consulting@freeswitch.org<mailto:consulting@freeswitch.org> and we can formally diagnose it for a fee. On Tue, Mar 21, 2017 at 6:42 AM Grant Bagdasarian <gb@cm.nl<mailto:gb@cm.nl>> wrote: Attached are the logs for: #check thread usage: top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1 #memory mapping: cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps We’re unable to run gcore since freeswitch hogs all available cpu’s at 100%. I tried changing the nice settings real-time but that didn’t work either. At one point we had a huge spike in inbound traffic towards our freeswitch (valid traffic). The traffic is handled by a lua application. What we saw was calls hanging in freeswitch while they’ve already been physically disconnected. After the huge spike freeswitch behaved normally for a couple of minutes and then suddenly started eating all the available cpu cores at 100%. We have some db queries which are fired in the lua application, so these may cause the application to hang, but we haven’t reproduced it yet. I’ll setup a test lab with the same configuration and fire sipp at it to see if I can reproduce the issue. However, I probably won’t be able to run gcore for debugging. Any other suggestions? Also we saw a lot of open udp ports and some had a huge read buffer size which did not decrease. From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>]On Behalf Of Grant Bagdasarian Sent: maandag 13 maart 2017 09:04 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage Hi Antony and Antonio, Thank you for the tips. We haven’t had freeswitch going berserk anymore (not sure as to why it did perform without issue over the weekend), but will continue to monitor it and run the commands to trace the issue. Regards, Grant From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org]On Behalf Of Anthony Minessale Sent: zaterdag 11 maart 2017 00:07 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] 100% CPU usage Make sure you lua script has all while loops including session:ready() as a condition to continue. Its a common mistake that leads to loops spinning out of control on hungup channels. On Fri, Mar 10, 2017 at 6:11 AM, Antonio Silva <asilva@wirelessmundi.com<mailto:asilva@wirelessmundi.com>> wrote: Hi, You can use the following commands: #get a core: gcore -o /home/gcore-$(date +%s) $(cat /var/run/freeswitch/freeswitch.pid) #check thread usage: top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1 #memory mapping: cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps you can get some info from fs: fs_cli -x "sofia global siptrace on" fs_cli -x "fsctl debug_level 10" fs_cli -x "sofia loglevel all 9" fs_cli -x "fsctl debug_sql " fs_cli -x "sofia global debug presence" Put everything into a jira so we can take a look. Regards, António On 03/10/2017 11:08 AM, Grant Bagdasarian wrote: Hi, We’re currently experiencing a weird issue with one of our freeswitch servers where the CPU just goes berserk and stays at 100%. There are no active calls at that moment, but FS is still using 100% CPU. I was unable to get a debug using gdb, since freeswitch does not crash. We had this issue on 1.6.13 and then I upgraded to 1.6.15 this morning, but the problem still persists. There is currently only one active application running on this box which is using LUA. Maybe that is causing the issue? We checked the code but there are no weird loops etc. Perhaps we are missing something? Any ideas how to trace this? Regards, Grant _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> -- Saludos / Regards / Cumprimentos, António silva _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net<http://irc.freenode.net/> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net<http://irc.freenode.net/> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ [http://www.itec-support.co.uk/wp-content/uploads/2016/07/email_logo.jpg] Shaun Stokes - Infrastructure Analyst T : 01453 700713 E : shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk> W : www.itec-support.co.uk<http://www.itec-support.co.uk/> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001 CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment. ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> -- Saludos / Regards / Cumprimentos, António silva ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ [http://www.itec-support.co.uk/wp-content/uploads/2016/07/email_logo.jpg] Shaun Stokes - Infrastructure Analyst T : 01453 700713 E : shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk> W : www.itec-support.co.uk<http://www.itec-support.co.uk/> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001 CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment. ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net<http://irc.freenode.net/> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ [http://www.itec-support.co.uk/wp-content/uploads/2016/07/email_logo.jpg] Shaun Stokes - Infrastructure Analyst T : 01453 700713 E : shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk> W : www.itec-support.co.uk<http://www.itec-support.co.uk/> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001 CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment. ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ [http://www.itec-support.co.uk/wp-content/uploads/2016/07/email_logo.jpg] - T : E : W : www.itec-support.co.uk<http://www.itec-support.co.uk/> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001 CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment. [http://www.itec-support.co.uk/wp-content/uploads/2016/07/email_logo.jpg] Shaun Stokes - Infrastructure Analyst T : 01453 700713 E : shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk> W : www.itec-support.co.uk<http://www.itec-support.co.uk/> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001 CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment. ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ [http://www.itec-support.co.uk/wp-content/uploads/2016/07/email_logo.jpg] Shaun Stokes - Infrastructure Analyst T : 01453 700713 E : shaun.stokes@itec-support.co.uk W : www.itec-support.co.uk Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF Company No. 06908001 CONFIDENTIALITY NOTICE This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately. The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment. ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________
participants (6)
-
Anthony Minessale
-
Antonio Silva
-
Cj B
-
Grant Bagdasarian
-
Shaun Stokes
-
Steven Ayre