[Freeswitch-users] 100% CPU usage

[Anthony Minessale]

I already told you why a week ago.  Your script most likely has an endless
loop in it that does not check the condition session:ready()

However, Feel free to contact consulting at freeswitch.org and we can formally
diagnose it for a fee.




On Tue, Mar 21, 2017 at 6:42 AM Grant Bagdasarian <gb at cm.nl> wrote:

> Attached are the logs for:
>
> #check thread usage:
>
> top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1
>
>
>
>
>
> #memory mapping:
>
> cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps
>
>
>
> We’re unable to run gcore since freeswitch hogs all available cpu’s at
> 100%.
>
> I tried changing the nice settings real-time but that didn’t work either.
>
>
>
> At one point we had a huge spike in inbound traffic towards our freeswitch
> (valid traffic).
>
> The traffic is handled by a lua application.
>
> What we saw was calls hanging in freeswitch while they’ve already been
> physically disconnected.
>
> After the huge spike freeswitch behaved normally for a couple of minutes
> and then suddenly started eating all the available cpu cores at 100%.
>
>
>
> We have some db queries which are fired in the lua application, so these
> may cause the application to hang, but we haven’t reproduced it yet.
>
>
>
> I’ll setup a test lab with the same configuration and fire sipp at it to
> see if I can reproduce the issue.
>
> However, I probably won’t be able to run gcore for debugging. Any other
> suggestions?
>
>
>
> Also we saw a lot of open udp ports and some had a huge read buffer size
> which did not decrease.
>
>
>
>
>
> *From:* freeswitch-users-bounces at lists.freeswitch.org [mailto:
> freeswitch-users-bounces at lists.freeswitch.org] *On Behalf Of *Grant
> Bagdasarian
> *Sent:* maandag 13 maart 2017 09:04
>
>
> *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
> *Subject:* Re: [Freeswitch-users] 100% CPU usage
>
>
>
> Hi Antony and Antonio,
>
>
>
> Thank you for the tips.
>
> We haven’t had freeswitch going berserk anymore (not sure as to why it did
> perform without issue over the weekend), but will continue to monitor it
> and run the commands to trace the issue.
>
>
>
> Regards,
>
>
>
> Grant
>
>
>
> *From:* freeswitch-users-bounces at lists.freeswitch.org [
> mailto:freeswitch-users-bounces at lists.freeswitch.org
> <freeswitch-users-bounces at lists.freeswitch.org>] *On Behalf Of *Anthony
> Minessale
> *Sent:* zaterdag 11 maart 2017 00:07
> *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
> *Subject:* Re: [Freeswitch-users] 100% CPU usage
>
>
>
> Make sure you lua script has all while loops including session:ready() as
> a condition to continue.  Its a common mistake that leads to loops spinning
> out of control on hungup channels.
>
>
>
>
>
> On Fri, Mar 10, 2017 at 6:11 AM, Antonio Silva <asilva at wirelessmundi.com>
> wrote:
>
> Hi,
>
> You can use the following commands:
>
> #get a core:
>
> gcore -o /home/gcore-$(date +%s) $(cat /var/run/freeswitch/freeswitch.pid)
>
>
>
>
>
> #check thread usage:
>
> top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1
>
>
>
>
>
> #memory mapping:
>
> cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps
>
>
>
> you can get some info from fs:
>
> fs_cli -x "sofia global siptrace on"
>
> fs_cli -x "fsctl debug_level 10"
>
> fs_cli -x "sofia loglevel all 9"
>
> fs_cli -x "fsctl debug_sql "
>
> fs_cli -x "sofia global debug presence"
>
>
>
> Put everything into a jira so we can take a look.
>
>
> Regards,
> António
>
>
>
> On 03/10/2017 11:08 AM, Grant Bagdasarian wrote:
>
> Hi,
>
>
>
> We’re currently experiencing a weird issue with one of our freeswitch
> servers where the CPU just goes berserk and stays at 100%.
>
> There are no active calls at that moment, but FS is still using 100% CPU.
>
>
>
> I was unable to get a debug using gdb, since freeswitch does not crash.
>
>
>
> We had this issue on 1.6.13 and then I upgraded to 1.6.15 this morning,
> but the problem still persists.
>
> There is currently only one active application running on this box which
> is using LUA. Maybe that is causing the issue?
>
> We checked the code but there are no weird loops etc. Perhaps we are
> missing something?
>
>
>
> Any ideas how to trace this?
>
>
>
> Regards,
>
>
>
> Grant
>
>
>
>
>
> _________________________________________________________________________
>
> Professional FreeSWITCH Consulting Services:
>
> consulting at freeswitch.org
>
> http://www.freeswitchsolutions.com
>
>
>
> Official FreeSWITCH Sites
>
> http://www.freeswitch.org
>
> http://confluence.freeswitch.org
>
> http://www.cluecon.com
>
>
>
> FreeSWITCH-users mailing list
>
> FreeSWITCH-users at lists.freeswitch.org
>
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>
> http://www.freeswitch.org
>
>
>
> --
>
>
>
> Saludos / Regards / Cumprimentos,
>
> António silva
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
>
>
>
> --
>
> Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬
>
>
>
> ☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞
> http://twitter.com/FreeSWITCH
>
> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
> <http://freeswitch.org/g+>*
>
> ClueCon Weekly Development Call
>
> ☎ sip:888 at conference.freeswitch.org  ☎ +19193869900
>
>
>
> https://www.youtube.com/watch?v=9XXgW34t40s
>
> https://www.youtube.com/watch?v=NLaDpGQuZDA
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org

-- 
Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬

☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞
http://twitter.com/FreeSWITCH
☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
<http://freeswitch.org/g+>*

ClueCon Weekly Development Call
☎ sip:888 at conference.freeswitch.org  ☎ +19193869900

https://www.youtube.com/watch?v=9XXgW34t40s
https://www.youtube.com/watch?v=NLaDpGQuZDA
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20170321/b755ef55/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 41068 bytes
Desc: not available
Url : http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20170321/b755ef55/attachment-0001.png