[Freeswitch-users] 100% CPU usage

[Grant Bagdasarian]

Attached are the logs for:

#check thread usage:

top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1





#memory mapping:

cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps

We’re unable to run gcore since freeswitch hogs all available cpu’s at 100%.
I tried changing the nice settings real-time but that didn’t work either.

At one point we had a huge spike in inbound traffic towards our freeswitch (valid traffic).
The traffic is handled by a lua application.
What we saw was calls hanging in freeswitch while they’ve already been physically disconnected.
After the huge spike freeswitch behaved normally for a couple of minutes and then suddenly started eating all the available cpu cores at 100%.

We have some db queries which are fired in the lua application, so these may cause the application to hang, but we haven’t reproduced it yet.

I’ll setup a test lab with the same configuration and fire sipp at it to see if I can reproduce the issue.
However, I probably won’t be able to run gcore for debugging. Any other suggestions?

Also we saw a lot of open udp ports and some had a huge read buffer size which did not decrease.

[cid:image001.png at 01D2A238.14965330]

From: freeswitch-users-bounces at lists.freeswitch.org [mailto:freeswitch-users-bounces at lists.freeswitch.org] On Behalf Of Grant Bagdasarian
Sent: maandag 13 maart 2017 09:04
To: FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
Subject: Re: [Freeswitch-users] 100% CPU usage

Hi Antony and Antonio,

Thank you for the tips.
We haven’t had freeswitch going berserk anymore (not sure as to why it did perform without issue over the weekend), but will continue to monitor it and run the commands to trace the issue.

Regards,

Grant

From: freeswitch-users-bounces at lists.freeswitch.org<mailto:freeswitch-users-bounces at lists.freeswitch.org> [mailto:freeswitch-users-bounces at lists.freeswitch.org] On Behalf Of Anthony Minessale
Sent: zaterdag 11 maart 2017 00:07
To: FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org<mailto:freeswitch-users at lists.freeswitch.org>>
Subject: Re: [Freeswitch-users] 100% CPU usage

Make sure you lua script has all while loops including session:ready() as a condition to continue.  Its a common mistake that leads to loops spinning out of control on hungup channels.


On Fri, Mar 10, 2017 at 6:11 AM, Antonio Silva <asilva at wirelessmundi.com<mailto:asilva at wirelessmundi.com>> wrote:
Hi,

You can use the following commands:

#get a core:

gcore -o /home/gcore-$(date +%s) $(cat /var/run/freeswitch/freeswitch.pid)





#check thread usage:

top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1





#memory mapping:

cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps


you can get some info from fs:

fs_cli -x "sofia global siptrace on"

fs_cli -x "fsctl debug_level 10"

fs_cli -x "sofia loglevel all 9"

fs_cli -x "fsctl debug_sql "

fs_cli -x "sofia global debug presence"


Put everything into a jira so we can take a look.


Regards,
António

On 03/10/2017 11:08 AM, Grant Bagdasarian wrote:
Hi,

We’re currently experiencing a weird issue with one of our freeswitch servers where the CPU just goes berserk and stays at 100%.
There are no active calls at that moment, but FS is still using 100% CPU.

I was unable to get a debug using gdb, since freeswitch does not crash.

We had this issue on 1.6.13 and then I upgraded to 1.6.15 this morning, but the problem still persists.
There is currently only one active application running on this box which is using LUA. Maybe that is causing the issue?
We checked the code but there are no weird loops etc. Perhaps we are missing something?

Any ideas how to trace this?

Regards,

Grant



_________________________________________________________________________

Professional FreeSWITCH Consulting Services:

consulting at freeswitch.org<mailto:consulting at freeswitch.org>

http://www.freeswitchsolutions.com



Official FreeSWITCH Sites

http://www.freeswitch.org

http://confluence.freeswitch.org

http://www.cluecon.com



FreeSWITCH-users mailing list

FreeSWITCH-users at lists.freeswitch.org<mailto:FreeSWITCH-users at lists.freeswitch.org>

http://lists.freeswitch.org/mailman/listinfo/freeswitch-users

UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users

http://www.freeswitch.org


--



Saludos / Regards / Cumprimentos,

António silva

_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting at freeswitch.org<mailto:consulting at freeswitch.org>
http://www.freeswitchsolutions.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users at lists.freeswitch.org<mailto:FreeSWITCH-users at lists.freeswitch.org>
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org



--
Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬

☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞ http://twitter.com/FreeSWITCH
☞ irc.freenode.net<http://irc.freenode.net> #freeswitch ☞ http://freeswitch.org/g+
ClueCon Weekly Development Call
☎ sip:888 at conference.freeswitch.org<mailto:sip%3A888 at conference.freeswitch.org>  ☎ +19193869900

https://www.youtube.com/watch?v=9XXgW34t40s
https://www.youtube.com/watch?v=NLaDpGQuZDA
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20170321/8f9937a4/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 41068 bytes
Desc: image001.png
Url : http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20170321/8f9937a4/attachment-0001.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 01cpu.log
Type: application/octet-stream
Size: 66587 bytes
Desc: 01cpu.log
Url : http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20170321/8f9937a4/attachment-0002.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 02.cpu.log
Type: application/octet-stream
Size: 65482 bytes
Desc: 02.cpu.log
Url : http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20170321/8f9937a4/attachment-0003.obj