[Freeswitch-users] 100% CPU usage

[Anthony Minessale]

top -H -p `pidof freeswitch`
Note the pid using 100% cpu

gcore `pidof freeswitch`

Attach it to a jira.  If you think its a bug in FS you should report it to
Jira not to this list.

https://freeswitch.org/jira





On Wed, Mar 22, 2017 at 10:14 AM Shaun Stokes <
shaun.stokes at itec-support.co.uk> wrote:

> It’s probably a long shot, but if you’re able to replicate the problem it
> might be worth trying to replicate this on physical hardware. That way you
> eliminate your virtual environment from the problem, there are a number of
> factors which exist in virtual environments depending on your hyper-visor
> which can contribute to performance issues of which we’ve experienced in
> the past.
>
>
>
> It would also be worth trying to isolate what the script is doing when the
> high CPU begins, if you’ve got a test environment you can use to replicate
> the issue then you could try tweaking the script so you can eliminate
> certain actions from the list of causes, there’s maybe a FreeSWITCH module
> which is being called which triggers this.
>
>
>
> *From:* freeswitch-users-bounces at lists.freeswitch.org [mailto:
> freeswitch-users-bounces at lists.freeswitch.org] *On Behalf Of *Grant
> Bagdasarian
> *Sent:* 22 March 2017 11:20
>
>
> *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
> *Subject:* Re: [Freeswitch-users] 100% CPU usage
>
>
>
> @Shuan, yes we are using the session:ready() method before we execute.
>
> We are running all of our servers virtually and never had any issues up
> until now.
>
> This same application ran on older versions of freeswitch without issues,
> even during and after high load.
>
>
>
> @Antonio, thanks for the tips! I’ll have a look and try some stuff.
>
>
>
> *From:* freeswitch-users-bounces at lists.freeswitch.org [
> mailto:freeswitch-users-bounces at lists.freeswitch.org
> <freeswitch-users-bounces at lists.freeswitch.org>] *On Behalf Of *Antonio
> Silva
> *Sent:* woensdag 22 maart 2017 10:09
> *To:* freeswitch-users at lists.freeswitch.org
> *Subject:* Re: [Freeswitch-users] 100% CPU usage
>
>
>
> We had some issues using pcall functions like io.popen when trying to
> execute external commands, that block mod_lua, i end up replacing that with
> os.execute.
>
> Recently i also had some issue with cpu hitting 100%,for what i could
> compile from debugging was something related with malloc calls. after
> compiling fs with tcmalloc the issue disappear, you could try this approach
> as well.
>
> You can check jira: https://freeswitch.org/jira/browse/FS-9957 and  also
> in the malling: "CPU sys load issue"  for more info.
>
>
> Regards,
> António
>
> On 03/22/2017 09:52 AM, Shaun Stokes wrote:
>
> Are you using ‘if (session:ready()) then’ in your scripts before you
> execute?
>
>
>
> Are you using virtual or physical hardware and what spec?
>
>
>
> *From:* freeswitch-users-bounces at lists.freeswitch.org [
> mailto:freeswitch-users-bounces at lists.freeswitch.org
> <freeswitch-users-bounces at lists.freeswitch.org>] *On Behalf Of *Grant
> Bagdasarian
> *Sent:* 22 March 2017 07:31
> *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
> <freeswitch-users at lists.freeswitch.org>
> *Subject:* Re: [Freeswitch-users] 100% CPU usage
>
>
>
> Anthony, we triple checked the script and its dependencies. There are no
> loops at all in the script. The imported scripts are also written by us and
> they too do not contain any loops.
>
> It’s just basic call handling with some playback and play_and_get_digits
> calls, and some pcall calls to query a database.
>
>
>
> *From:* freeswitch-users-bounces at lists.freeswitch.org [
> mailto:freeswitch-users-bounces at lists.freeswitch.org
> <freeswitch-users-bounces at lists.freeswitch.org>] *On Behalf Of *Anthony
> Minessale
> *Sent:* dinsdag 21 maart 2017 16:02
> *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
> *Subject:* Re: [Freeswitch-users] 100% CPU usage
>
>
>
> I already told you why a week ago.  Your script most likely has an endless
> loop in it that does not check the condition session:ready()
>
>
>
> However, Feel free to contact consulting at freeswitch.org and we can
> formally diagnose it for a fee.
>
>
>
>
>
>
>
>
>
> On Tue, Mar 21, 2017 at 6:42 AM Grant Bagdasarian <gb at cm.nl> wrote:
>
> Attached are the logs for:
>
> #check thread usage:
>
> top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1
>
>
>
>
>
> #memory mapping:
>
> cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps
>
>
>
> We’re unable to run gcore since freeswitch hogs all available cpu’s at
> 100%.
>
> I tried changing the nice settings real-time but that didn’t work either.
>
>
>
> At one point we had a huge spike in inbound traffic towards our freeswitch
> (valid traffic).
>
> The traffic is handled by a lua application.
>
> What we saw was calls hanging in freeswitch while they’ve already been
> physically disconnected.
>
> After the huge spike freeswitch behaved normally for a couple of minutes
> and then suddenly started eating all the available cpu cores at 100%.
>
>
>
> We have some db queries which are fired in the lua application, so these
> may cause the application to hang, but we haven’t reproduced it yet.
>
>
>
> I’ll setup a test lab with the same configuration and fire sipp at it to
> see if I can reproduce the issue.
>
> However, I probably won’t be able to run gcore for debugging. Any other
> suggestions?
>
>
>
> Also we saw a lot of open udp ports and some had a huge read buffer size
> which did not decrease.
>
>
>
>
>
> *From:* freeswitch-users-bounces at lists.freeswitch.org [mailto:
> freeswitch-users-bounces at lists.freeswitch.org] *On Behalf Of *Grant
> Bagdasarian
> *Sent:* maandag 13 maart 2017 09:04
>
>
> *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
> *Subject:* Re: [Freeswitch-users] 100% CPU usage
>
>
>
> Hi Antony and Antonio,
>
>
>
> Thank you for the tips.
>
> We haven’t had freeswitch going berserk anymore (not sure as to why it did
> perform without issue over the weekend), but will continue to monitor it
> and run the commands to trace the issue.
>
>
>
> Regards,
>
>
>
> Grant
>
>
>
> *From:* freeswitch-users-bounces at lists.freeswitch.org [
> mailto:freeswitch-users-bounces at lists.freeswitch.org
> <freeswitch-users-bounces at lists.freeswitch.org>] *On Behalf Of *Anthony
> Minessale
> *Sent:* zaterdag 11 maart 2017 00:07
> *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
> *Subject:* Re: [Freeswitch-users] 100% CPU usage
>
>
>
> Make sure you lua script has all while loops including session:ready() as
> a condition to continue.  Its a common mistake that leads to loops spinning
> out of control on hungup channels.
>
>
>
>
>
> On Fri, Mar 10, 2017 at 6:11 AM, Antonio Silva <asilva at wirelessmundi.com>
> wrote:
>
> Hi,
>
> You can use the following commands:
>
> #get a core:
>
> gcore -o /home/gcore-$(date +%s) $(cat /var/run/freeswitch/freeswitch.pid)
>
>
>
>
>
> #check thread usage:
>
> top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1
>
>
>
>
>
> #memory mapping:
>
> cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps
>
>
>
> you can get some info from fs:
>
> fs_cli -x "sofia global siptrace on"
>
> fs_cli -x "fsctl debug_level 10"
>
> fs_cli -x "sofia loglevel all 9"
>
> fs_cli -x "fsctl debug_sql "
>
> fs_cli -x "sofia global debug presence"
>
>
>
> Put everything into a jira so we can take a look.
>
>
> Regards,
> António
>
>
>
> On 03/10/2017 11:08 AM, Grant Bagdasarian wrote:
>
> Hi,
>
>
>
> We’re currently experiencing a weird issue with one of our freeswitch
> servers where the CPU just goes berserk and stays at 100%.
>
> There are no active calls at that moment, but FS is still using 100% CPU.
>
>
>
> I was unable to get a debug using gdb, since freeswitch does not crash.
>
>
>
> We had this issue on 1.6.13 and then I upgraded to 1.6.15 this morning,
> but the problem still persists.
>
> There is currently only one active application running on this box which
> is using LUA. Maybe that is causing the issue?
>
> We checked the code but there are no weird loops etc. Perhaps we are
> missing something?
>
>
>
> Any ideas how to trace this?
>
>
>
> Regards,
>
>
>
> Grant
>
>
>
>
>
> _________________________________________________________________________
>
> Professional FreeSWITCH Consulting Services:
>
> consulting at freeswitch.org
>
> http://www.freeswitchsolutions.com
>
>
>
> Official FreeSWITCH Sites
>
> http://www.freeswitch.org
>
> http://confluence.freeswitch.org
>
> http://www.cluecon.com
>
>
>
> FreeSWITCH-users mailing list
>
> FreeSWITCH-users at lists.freeswitch.org
>
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>
> http://www.freeswitch.org
>
>
>
> --
>
>
>
> Saludos / Regards / Cumprimentos,
>
> António silva
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
>
>
>
> --
>
> Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬
>
>
>
> ☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞
> http://twitter.com/FreeSWITCH
>
> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
> <http://freeswitch.org/g+>*
>
> ClueCon Weekly Development Call
>
> ☎ sip:888 at conference.freeswitch.org  ☎ +19193869900
>
>
>
> https://www.youtube.com/watch?v=9XXgW34t40s
>
> https://www.youtube.com/watch?v=NLaDpGQuZDA
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
> --
>
> Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬
>
>
>
> ☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞
> http://twitter.com/FreeSWITCH
>
> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
> <http://freeswitch.org/g+>*
>
> ClueCon Weekly Development Call
>
> ☎ sip:888 at conference.freeswitch.org  ☎ +19193869900
>
>
>
> https://www.youtube.com/watch?v=9XXgW34t40s
>
> https://www.youtube.com/watch?v=NLaDpGQuZDA
>
>
> ______________________________________________________________________
> This message has been checked for all known viruses by MessageLabs Virus
> Scanning Service.
> ______________________________________________________________________
>
> Shaun Stokes - Infrastructure Analyst
>
> T :
>
> 01453 700713
>
> E :
>
> shaun.stokes at itec-support.co.uk
>
> W :
>
> www.itec-support.co.uk
>
> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road,
> Stroud, Gloucestershire GL5 3QF
> Company No. 06908001
>
>
> CONFIDENTIALITY NOTICE
> This communication and the information it contains are intended for the
> person or organisation to which it is addressed. Its contents are
> confidential and may be protected in law. Unauthorised use, copying or
> disclosure of any of it may be unlawful. If you are not the intended
> recipient, please contact us immediately.
> The contents of any attachments in this e-mail may contain software
> viruses, which could damage your own computer system. While ITEC Support
> has taken every reasonable precaution to minimise this risk, we cannot
> accept liability for any damage which you sustain as a result of software
> viruses. You should carry out your own virus checking procedure before
> opening any attachment.
>
>
> ______________________________________________________________________
> This message has been checked for all known viruses by MessageLabs Virus
> Scanning Service.
> ______________________________________________________________________
>
>
> _________________________________________________________________________
>
> Professional FreeSWITCH Consulting Services:
>
> consulting at freeswitch.org
>
> http://www.freeswitchsolutions.com
>
>
>
> Official FreeSWITCH Sites
>
> http://www.freeswitch.org
>
> http://confluence.freeswitch.org
>
> http://www.cluecon.com
>
>
>
> FreeSWITCH-users mailing list
>
> FreeSWITCH-users at lists.freeswitch.org
>
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>
> http://www.freeswitch.org
>
>
>
> --
>
>
>
> Saludos / Regards / Cumprimentos,
>
> António silva
>
>
> ______________________________________________________________________
> This message has been checked for all known viruses by MessageLabs Virus
> Scanning Service.
> ______________________________________________________________________
> Shaun Stokes - Infrastructure Analyst
> T : 01453 700713
> E : shaun.stokes at itec-support.co.uk
> W : www.itec-support.co.uk
> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road,
> Stroud, Gloucestershire GL5 3QF
> Company No. 06908001
>
> CONFIDENTIALITY NOTICE
> This communication and the information it contains are intended for the
> person or organisation to which it is addressed. Its contents are
> confidential and may be protected in law. Unauthorised use, copying or
> disclosure of any of it may be unlawful. If you are not the intended
> recipient, please contact us immediately.
> The contents of any attachments in this e-mail may contain software
> viruses, which could damage your own computer system. While ITEC Support
> has taken every reasonable precaution to minimise this risk, we cannot
> accept liability for any damage which you sustain as a result of software
> viruses. You should carry out your own virus checking procedure before
> opening any attachment.
>
> ______________________________________________________________________
> This message has been checked for all known viruses by MessageLabs Virus
> Scanning Service.
> ______________________________________________________________________
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org

-- 
Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬

☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞
http://twitter.com/FreeSWITCH
☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
<http://freeswitch.org/g+>*

ClueCon Weekly Development Call
☎ sip:888 at conference.freeswitch.org  ☎ +19193869900

https://www.youtube.com/watch?v=9XXgW34t40s
https://www.youtube.com/watch?v=NLaDpGQuZDA
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20170322/58ccf86c/attachment-0001.html