[Freeswitch-users] 100% CPU usage

Wed Mar 22 12:08:40 MSK 2017

We had some issues using pcall functions like io.popen when trying to 
execute external commands, that block mod_lua, i end up replacing that 
with os.execute.

Recently i also had some issue with cpu hitting 100%,for what i could 
compile from debugging was something related with malloc calls. after 
compiling fs with tcmalloc the issue disappear, you could try this 
approach as well.

You can check jira: https://freeswitch.org/jira/browse/FS-9957 and also 
in the malling: "CPU sys load issue"  for more info.

Regards,
António

On 03/22/2017 09:52 AM, Shaun Stokes wrote:
>
> Are you using ‘if (session:ready()) then’ in your scripts before you 
> execute?
>
> Are you using virtual or physical hardware and what spec?
>
> *From:*freeswitch-users-bounces at lists.freeswitch.org 
> [mailto:freeswitch-users-bounces at lists.freeswitch.org] *On Behalf Of 
> *Grant Bagdasarian
> *Sent:* 22 March 2017 07:31
> *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
> *Subject:* Re: [Freeswitch-users] 100% CPU usage
>
> Anthony, we triple checked the script and its dependencies. There are 
> no loops at all in the script. The imported scripts are also written 
> by us and they too do not contain any loops.
>
> It’s just basic call handling with some playback and 
> play_and_get_digits calls, and some pcall calls to query a database.
>
> *From:*freeswitch-users-bounces at lists.freeswitch.org 
> <mailto:freeswitch-users-bounces at lists.freeswitch.org> 
> [mailto:freeswitch-users-bounces at lists.freeswitch.org] *On Behalf Of 
> *Anthony Minessale
> *Sent:* dinsdag 21 maart 2017 16:02
> *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org 
> <mailto:freeswitch-users at lists.freeswitch.org>>
> *Subject:* Re: [Freeswitch-users] 100% CPU usage
>
> I already told you why a week ago.  Your script most likely has an 
> endless loop in it that does not check the condition session:ready()
>
> However, Feel free to contact consulting at freeswitch.org 
> <mailto:consulting at freeswitch.org> and we can formally diagnose it for 
> a fee.
>
> On Tue, Mar 21, 2017 at 6:42 AM Grant Bagdasarian <gb at cm.nl 
> <mailto:gb at cm.nl>> wrote:
>
>     Attached are the logs for:
>
>     #check thread usage:
>
>     top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1
>
>     #memory mapping:
>
>     cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps
>
>     We’re unable to run gcore since freeswitch hogs all available
>     cpu’s at 100%.
>
>     I tried changing the nice settings real-time but that didn’t work
>     either.
>
>     At one point we had a huge spike in inbound traffic towards our
>     freeswitch (valid traffic).
>
>     The traffic is handled by a lua application.
>
>     What we saw was calls hanging in freeswitch while they’ve already
>     been physically disconnected.
>
>     After the huge spike freeswitch behaved normally for a couple of
>     minutes and then suddenly started eating all the available cpu
>     cores at 100%.
>
>     We have some db queries which are fired in the lua application, so
>     these may cause the application to hang, but we haven’t reproduced
>     it yet.
>
>     I’ll setup a test lab with the same configuration and fire sipp at
>     it to see if I can reproduce the issue.
>
>     However, I probably won’t be able to run gcore for debugging. Any
>     other suggestions?
>
>     Also we saw a lot of open udp ports and some had a huge read
>     buffer size which did not decrease.
>
>     *From:*freeswitch-users-bounces at lists.freeswitch.org
>     <mailto:freeswitch-users-bounces at lists.freeswitch.org>
>     [mailto:freeswitch-users-bounces at lists.freeswitch.org
>     <mailto:freeswitch-users-bounces at lists.freeswitch.org>] *On Behalf
>     Of *Grant Bagdasarian
>     *Sent:* maandag 13 maart 2017 09:04
>
>
>     *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org
>     <mailto:freeswitch-users at lists.freeswitch.org>>
>     *Subject:* Re: [Freeswitch-users] 100% CPU usage
>
>     Hi Antony and Antonio,
>
>     Thank you for the tips.
>
>     We haven’t had freeswitch going berserk anymore (not sure as to
>     why it did perform without issue over the weekend), but will
>     continue to monitor it and run the commands to trace the issue.
>
>     Regards,
>
>     Grant
>
>     *From:*freeswitch-users-bounces at lists.freeswitch.org
>     <mailto:freeswitch-users-bounces at lists.freeswitch.org>
>     [mailto:freeswitch-users-bounces at lists.freeswitch.org] *On Behalf
>     Of *Anthony Minessale
>     *Sent:* zaterdag 11 maart 2017 00:07
>     *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org
>     <mailto:freeswitch-users at lists.freeswitch.org>>
>     *Subject:* Re: [Freeswitch-users] 100% CPU usage
>
>     Make sure you lua script has all while loops including
>     session:ready() as a condition to continue.  Its a common mistake
>     that leads to loops spinning out of control on hungup channels.
>
>     On Fri, Mar 10, 2017 at 6:11 AM, Antonio Silva
>     <asilva at wirelessmundi.com <mailto:asilva at wirelessmundi.com>> wrote:
>
>         Hi,
>
>         You can use the following commands:
>
>         #get a core:
>
>         gcore -o /home/gcore-$(date +%s) $(cat
>         /var/run/freeswitch/freeswitch.pid)
>
>         #check thread usage:
>
>         top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1
>
>         #memory mapping:
>
>         cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps
>
>
>
>         you can get some info from fs:
>
>         fs_cli -x "sofia global siptrace on"
>
>         fs_cli -x "fsctl debug_level 10"
>
>         fs_cli -x "sofia loglevel all 9"
>
>         fs_cli -x "fsctl debug_sql "
>
>         fs_cli -x "sofia global debug presence"
>
>
>
>         Put everything into a jira so we can take a look.
>
>
>         Regards,
>         António
>
>         On 03/10/2017 11:08 AM, Grant Bagdasarian wrote:
>
>             Hi,
>
>             We’re currently experiencing a weird issue with one of our
>             freeswitch servers where the CPU just goes berserk and
>             stays at 100%.
>
>             There are no active calls at that moment, but FS is still
>             using 100% CPU.
>
>             I was unable to get a debug using gdb, since freeswitch
>             does not crash.
>
>             We had this issue on 1.6.13 and then I upgraded to 1.6.15
>             this morning, but the problem still persists.
>
>             There is currently only one active application running on
>             this box which is using LUA. Maybe that is causing the issue?
>
>             We checked the code but there are no weird loops etc.
>             Perhaps we are missing something?
>
>             Any ideas how to trace this?
>
>             Regards,
>
>             Grant
>
>             _________________________________________________________________________
>
>             Professional FreeSWITCH Consulting Services:
>
>             consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>
>             http://www.freeswitchsolutions.com
>
>             Official FreeSWITCH Sites
>
>             http://www.freeswitch.org
>
>             http://confluence.freeswitch.org
>
>             http://www.cluecon.com
>
>             FreeSWITCH-users mailing list
>
>             FreeSWITCH-users at lists.freeswitch.org
>             <mailto:FreeSWITCH-users at lists.freeswitch.org>
>
>             http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>
>             UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>
>             http://www.freeswitch.org
>
>         -- 
>
>         Saludos / Regards / Cumprimentos,
>
>         António silva
>
>
>         _________________________________________________________________________
>         Professional FreeSWITCH Consulting Services:
>         consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>         http://www.freeswitchsolutions.com
>
>         Official FreeSWITCH Sites
>         http://www.freeswitch.org
>         http://confluence.freeswitch.org
>         http://www.cluecon.com
>
>         FreeSWITCH-users mailing list
>         FreeSWITCH-users at lists.freeswitch.org
>         <mailto:FreeSWITCH-users at lists.freeswitch.org>
>         http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>         UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>         http://www.freeswitch.org
>
>
>
>     -- 
>
>     Anthony Minessale II ♬@anthmfs ♬@FreeSWITCH ♬
>
>     ☞http://freeswitch.org/ ☞http://cluecon.com/
>     ☞http://twitter.com/FreeSWITCH
>
>     ☞irc.freenode.net <http://irc.freenode.net> #freeswitch
>     ☞_http://freeswitch.org/g+_
>
>     ClueCon Weekly Development Call
>
>     ☎sip:888 at conference.freeswitch.org
>     <mailto:sip%3A888 at conference.freeswitch.org> ☎+19193869900
>
>     https://www.youtube.com/watch?v=9XXgW34t40s
>
>     https://www.youtube.com/watch?v=NLaDpGQuZDA
>
>     _________________________________________________________________________
>     Professional FreeSWITCH Consulting Services:
>     consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>     http://www.freeswitchsolutions.com
>
>     Official FreeSWITCH Sites
>     http://www.freeswitch.org
>     http://confluence.freeswitch.org
>     http://www.cluecon.com
>
>     FreeSWITCH-users mailing list
>     FreeSWITCH-users at lists.freeswitch.org
>     <mailto:FreeSWITCH-users at lists.freeswitch.org>
>     http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>     http://www.freeswitch.org
>
> -- 
>
> Anthony Minessale II ♬@anthmfs ♬@FreeSWITCH ♬
>
> ☞http://freeswitch.org/ ☞http://cluecon.com/ 
> ☞http://twitter.com/FreeSWITCH
>
> ☞irc.freenode.net <http://irc.freenode.net> #freeswitch 
> ☞_http://freeswitch.org/g+_
>
> ClueCon Weekly Development Call
>
> ☎sip:888 at conference.freeswitch.org 
> <mailto:sip%3A888 at conference.freeswitch.org> ☎+19193869900
>
> https://www.youtube.com/watch?v=9XXgW34t40s
>
> https://www.youtube.com/watch?v=NLaDpGQuZDA
>
>
> ______________________________________________________________________
> This message has been checked for all known viruses by MessageLabs 
> Virus Scanning Service.
> ______________________________________________________________________
>
> 	
> Shaun Stokes - Infrastructure Analyst
>
> 	
> T : 	01453 700713
> E : 	shaun.stokes at itec-support.co.uk
> W : 	www.itec-support.co.uk
>
> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, 
> Stroud, Gloucestershire GL5 3QF
> Company No. 06908001
>
> CONFIDENTIALITY NOTICE
> This communication and the information it contains are intended for 
> the person or organisation to which it is addressed. Its contents are 
> confidential and may be protected in law. Unauthorised use, copying or 
> disclosure of any of it may be unlawful. If you are not the intended 
> recipient, please contact us immediately.
> The contents of any attachments in this e-mail may contain software 
> viruses, which could damage your own computer system. While ITEC 
> Support has taken every reasonable precaution to minimise this risk, 
> we cannot accept liability for any damage which you sustain as a 
> result of software viruses. You should carry out your own virus 
> checking procedure before opening any attachment.
>
> ______________________________________________________________________
> This message has been checked for all known viruses by MessageLabs 
> Virus Scanning Service.
> ______________________________________________________________________
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org

-- 

Saludos / Regards / Cumprimentos,
António silva

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20170322/83e5caec/attachment-0001.html 