[Freeswitch-users] 100% CPU usage

Anthony Minessale anthony.minessale at gmail.com
Wed Mar 29 21:28:57 MSD 2017 

Try to avoid doing things like pointing out how frustrated you are when
seeking help here.  It makes those trying to help you feel like you are
being entitled.

The suggestion on what to do was clearly outlined in previous posts in this
thread including the exact commands to run when filing a jira.








On Wed, Mar 29, 2017 at 12:20 PM, Cj B <blackc2004 at gmail.com> wrote:

> Hi All,
>
> I’m having a very similar issue however it’s not staying at 100% CPU on
> physical hardware. I’ve created a JIRA but wanted to see if anyone else had
> any feedback on possible causes? I’m using freeswitch 1.6.15 (previously
> 1.6.13). What happens is that FS will jump to 100% CPU and stay there for
> 1-2 seconds then drop back down but this is then causing massive
> jitter/call quality issues for about 10-15 seconds after the spike. The
> strange thing that I’ve noticed is that even with 0 active calls, FS is
> using 15-30% cpu constantly with only 40-60 registered endpoints.
>
> I’m watching the logs in DEBUG mode and not seeing anything there that’s
> indicative of any issue, no loops that I can see, and it seems to be
> happening when there’s even just 1 or 2 active calls on the system.
>
> I’m wondering if there’s any other suggestions on where to look because
> this is getting very frustrating for me and my users.
>
> Thanks much for any pointers!
> Cj B
>
> On Mar 22, 2017, at 8:13 AM, Anthony Minessale <
> anthony.minessale at gmail.com> wrote:
>
> top -H -p `pidof freeswitch`
> Note the pid using 100% cpu
>
> gcore `pidof freeswitch`
>
> Attach it to a jira.  If you think its a bug in FS you should report it to
> Jira not to this list.
>
> https://freeswitch.org/jira
>
>
>
>
>
> On Wed, Mar 22, 2017 at 10:14 AM Shaun Stokes <
> shaun.stokes at itec-support.co.uk> wrote:
>
>> It’s probably a long shot, but if you’re able to replicate the problem it
>> might be worth trying to replicate this on physical hardware. That way you
>> eliminate your virtual environment from the problem, there are a number of
>> factors which exist in virtual environments depending on your hyper-visor
>> which can contribute to performance issues of which we’ve experienced in
>> the past.
>>
>>
>> It would also be worth trying to isolate what the script is doing when
>> the high CPU begins, if you’ve got a test environment you can use to
>> replicate the issue then you could try tweaking the script so you can
>> eliminate certain actions from the list of causes, there’s maybe a
>> FreeSWITCH module which is being called which triggers this.
>>
>>
>> *From:* freeswitch-users-bounces at lists.freeswitch.org [mailto:
>> freeswitch-users-bounces at lists.freeswitch.org] *On Behalf Of *Grant
>> Bagdasarian
>> *Sent:* 22 March 2017 11:20
>>
>>
>> *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
>> *Subject:* Re: [Freeswitch-users] 100% CPU usage
>>
>>
>> @Shuan, yes we are using the session:ready() method before we execute.
>>
>> We are running all of our servers virtually and never had any issues up
>> until now.
>>
>> This same application ran on older versions of freeswitch without issues,
>> even during and after high load.
>>
>>
>> @Antonio, thanks for the tips! I’ll have a look and try some stuff.
>>
>>
>> *From:* freeswitch-users-bounces at lists.freeswitch.org [
>> mailto:freeswitch-users-bounces at lists.freeswitch.org
>> <freeswitch-users-bounces at lists.freeswitch.org>] *On Behalf Of *Antonio
>> Silva
>> *Sent:* woensdag 22 maart 2017 10:09
>> *To:* freeswitch-users at lists.freeswitch.org
>> *Subject:* Re: [Freeswitch-users] 100% CPU usage
>>
>>
>> We had some issues using pcall functions like io.popen when trying to
>> execute external commands, that block mod_lua, i end up replacing that with
>> os.execute.
>>
>> Recently i also had some issue with cpu hitting 100%,for what i could
>> compile from debugging was something related with malloc calls. after
>> compiling fs with tcmalloc the issue disappear, you could try this approach
>> as well.
>>
>> You can check jira: https://freeswitch.org/jira/browse/FS-9957 and  also
>> in the malling: "CPU sys load issue"  for more info.
>>
>>
>> Regards,
>> António
>>
>> On 03/22/2017 09:52 AM, Shaun Stokes wrote:
>>
>> Are you using ‘if (session:ready()) then’ in your scripts before you
>> execute?
>>
>>
>> Are you using virtual or physical hardware and what spec?
>>
>>
>> *From:* freeswitch-users-bounces at lists.freeswitch.org [
>> mailto:freeswitch-users-bounces at lists.freeswitch.org
>> <freeswitch-users-bounces at lists.freeswitch.org>] *On Behalf Of *Grant
>> Bagdasarian
>> *Sent:* 22 March 2017 07:31
>> *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
>> <freeswitch-users at lists.freeswitch.org>
>> *Subject:* Re: [Freeswitch-users] 100% CPU usage
>>
>>
>> Anthony, we triple checked the script and its dependencies. There are no
>> loops at all in the script. The imported scripts are also written by us and
>> they too do not contain any loops.
>>
>> It’s just basic call handling with some playback and play_and_get_digits
>> calls, and some pcall calls to query a database.
>>
>>
>> *From:* freeswitch-users-bounces at lists.freeswitch.org [
>> mailto:freeswitch-users-bounces at lists.freeswitch.org
>> <freeswitch-users-bounces at lists.freeswitch.org>] *On Behalf Of *Anthony
>> Minessale
>> *Sent:* dinsdag 21 maart 2017 16:02
>> *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
>> *Subject:* Re: [Freeswitch-users] 100% CPU usage
>>
>>
>> I already told you why a week ago.  Your script most likely has an
>> endless loop in it that does not check the condition session:ready()
>>
>>
>> However, Feel free to contact consulting at freeswitch.org and we can
>> formally diagnose it for a fee.
>>
>>
>>
>>
>>
>> On Tue, Mar 21, 2017 at 6:42 AM Grant Bagdasarian <gb at cm.nl> wrote:
>>
>> Attached are the logs for:
>>
>> #check thread usage:
>>
>> top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1
>>
>>
>>
>>
>>
>> #memory mapping:
>>
>> cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps
>>
>>
>>
>> We’re unable to run gcore since freeswitch hogs all available cpu’s at
>> 100%.
>>
>> I tried changing the nice settings real-time but that didn’t work either.
>>
>>
>> At one point we had a huge spike in inbound traffic towards our
>> freeswitch (valid traffic).
>>
>> The traffic is handled by a lua application.
>>
>> What we saw was calls hanging in freeswitch while they’ve already been
>> physically disconnected.
>>
>> After the huge spike freeswitch behaved normally for a couple of minutes
>> and then suddenly started eating all the available cpu cores at 100%.
>>
>>
>> We have some db queries which are fired in the lua application, so these
>> may cause the application to hang, but we haven’t reproduced it yet.
>>
>>
>> I’ll setup a test lab with the same configuration and fire sipp at it to
>> see if I can reproduce the issue.
>>
>> However, I probably won’t be able to run gcore for debugging. Any other
>> suggestions?
>>
>>
>> Also we saw a lot of open udp ports and some had a huge read buffer size
>> which did not decrease.
>>
>>
>>
>>
>> *From:* freeswitch-users-bounces at lists.freeswitch.org [mailto:
>> freeswitch-users-bounces at lists.freeswitch.org] *On Behalf Of *Grant
>> Bagdasarian
>> *Sent:* maandag 13 maart 2017 09:04
>>
>>
>> *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
>> *Subject:* Re: [Freeswitch-users] 100% CPU usage
>>
>>
>> Hi Antony and Antonio,
>>
>>
>> Thank you for the tips.
>>
>> We haven’t had freeswitch going berserk anymore (not sure as to why it
>> did perform without issue over the weekend), but will continue to monitor
>> it and run the commands to trace the issue.
>>
>>
>> Regards,
>>
>>
>> Grant
>>
>>
>> *From:* freeswitch-users-bounces at lists.freeswitch.org [
>> mailto:freeswitch-users-bounces at lists.freeswitch.org
>> <freeswitch-users-bounces at lists.freeswitch.org>] *On Behalf Of *Anthony
>> Minessale
>> *Sent:* zaterdag 11 maart 2017 00:07
>> *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
>> *Subject:* Re: [Freeswitch-users] 100% CPU usage
>>
>>
>> Make sure you lua script has all while loops including session:ready() as
>> a condition to continue.  Its a common mistake that leads to loops spinning
>> out of control on hungup channels.
>>
>>
>>
>> On Fri, Mar 10, 2017 at 6:11 AM, Antonio Silva <asilva at wirelessmundi.com>
>> wrote:
>>
>> Hi,
>>
>> You can use the following commands:
>>
>> #get a core:
>>
>> gcore -o /home/gcore-$(date +%s) $(cat /var/run/freeswitch/freeswitch.pid)
>>
>>
>>
>>
>>
>> #check thread usage:
>>
>> top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1
>>
>>
>>
>>
>>
>> #memory mapping:
>>
>> cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps
>>
>>
>>
>> you can get some info from fs:
>>
>> fs_cli -x "sofia global siptrace on"
>>
>> fs_cli -x "fsctl debug_level 10"
>>
>> fs_cli -x "sofia loglevel all 9"
>>
>> fs_cli -x "fsctl debug_sql "
>>
>> fs_cli -x "sofia global debug presence"
>>
>>
>>
>> Put everything into a jira so we can take a look.
>>
>>
>> Regards,
>> António
>>
>>
>> On 03/10/2017 11:08 AM, Grant Bagdasarian wrote:
>>
>> Hi,
>>
>>
>> We’re currently experiencing a weird issue with one of our freeswitch
>> servers where the CPU just goes berserk and stays at 100%.
>>
>> There are no active calls at that moment, but FS is still using 100% CPU.
>>
>>
>> I was unable to get a debug using gdb, since freeswitch does not crash.
>>
>>
>> We had this issue on 1.6.13 and then I upgraded to 1.6.15 this morning,
>> but the problem still persists.
>>
>> There is currently only one active application running on this box which
>> is using LUA. Maybe that is causing the issue?
>>
>> We checked the code but there are no weird loops etc. Perhaps we are
>> missing something?
>>
>>
>> Any ideas how to trace this?
>>
>>
>> Regards,
>>
>>
>> Grant
>>
>>
>>
>> _________________________________________________________________________
>>
>> Professional FreeSWITCH Consulting Services:
>>
>> consulting at freeswitch.org
>>
>> http://www.freeswitchsolutions.com
>>
>>
>>
>> Official FreeSWITCH Sites
>>
>> http://www.freeswitch.org
>>
>> http://confluence.freeswitch.org
>>
>> http://www.cluecon.com
>>
>>
>>
>> FreeSWITCH-users mailing list
>>
>> FreeSWITCH-users at lists.freeswitch.org
>>
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>
>> http://www.freeswitch.org
>>
>>
>>
>> --
>>
>>
>>
>> Saludos / Regards / Cumprimentos,
>>
>> António silva
>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://confluence.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>>
>>
>>
>> --
>>
>> Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬
>>
>>
>> ☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞
>> http://twitter.com/FreeSWITCH
>>
>> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
>> <http://freeswitch.org/g+>*
>>
>> ClueCon Weekly Development Call
>>
>> ☎ sip:888 at conference.freeswitch.org  ☎ +19193869900 <(919)%20386-9900>
>>
>>
>> https://www.youtube.com/watch?v=9XXgW34t40s
>>
>> https://www.youtube.com/watch?v=NLaDpGQuZDA
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://confluence.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>> --
>>
>> Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬
>>
>>
>> ☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞
>> http://twitter.com/FreeSWITCH
>>
>> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
>> <http://freeswitch.org/g+>*
>>
>> ClueCon Weekly Development Call
>>
>> ☎ sip:888 at conference.freeswitch.org  ☎ +19193869900 <(919)%20386-9900>
>>
>>
>> https://www.youtube.com/watch?v=9XXgW34t40s
>>
>> https://www.youtube.com/watch?v=NLaDpGQuZDA
>>
>>
>> ______________________________________________________________________
>> This message has been checked for all known viruses by MessageLabs Virus
>> Scanning Service.
>> ______________________________________________________________________
>>
>> Shaun Stokes - Infrastructure Analyst
>>
>> T :
>>
>> 01453 700713
>>
>> E :
>>
>> shaun.stokes at itec-support.co.uk
>>
>> W :
>>
>> www.itec-support.co.uk
>>
>> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road,
>> Stroud, Gloucestershire GL5 3QF
>> Company No. 06908001
>>
>>
>> CONFIDENTIALITY NOTICE
>> This communication and the information it contains are intended for the
>> person or organisation to which it is addressed. Its contents are
>> confidential and may be protected in law. Unauthorised use, copying or
>> disclosure of any of it may be unlawful. If you are not the intended
>> recipient, please contact us immediately.
>> The contents of any attachments in this e-mail may contain software
>> viruses, which could damage your own computer system. While ITEC Support
>> has taken every reasonable precaution to minimise this risk, we cannot
>> accept liability for any damage which you sustain as a result of software
>> viruses. You should carry out your own virus checking procedure before
>> opening any attachment.
>>
>>
>> ______________________________________________________________________
>> This message has been checked for all known viruses by MessageLabs Virus
>> Scanning Service.
>> ______________________________________________________________________
>>
>>
>> _________________________________________________________________________
>>
>> Professional FreeSWITCH Consulting Services:
>>
>> consulting at freeswitch.org
>>
>> http://www.freeswitchsolutions.com
>>
>>
>>
>> Official FreeSWITCH Sites
>>
>> http://www.freeswitch.org
>>
>> http://confluence.freeswitch.org
>>
>> http://www.cluecon.com
>>
>>
>>
>> FreeSWITCH-users mailing list
>>
>> FreeSWITCH-users at lists.freeswitch.org
>>
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>
>> http://www.freeswitch.org
>>
>>
>>
>> --
>>
>>
>>
>> Saludos / Regards / Cumprimentos,
>>
>> António silva
>>
>>
>> ______________________________________________________________________
>> This message has been checked for all known viruses by MessageLabs Virus
>> Scanning Service.
>> ______________________________________________________________________
>> Shaun Stokes - Infrastructure Analyst
>> T : 01453 700713
>> E : shaun.stokes at itec-support.co.uk
>> W : www.itec-support.co.uk
>> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road,
>> Stroud, Gloucestershire GL5 3QF
>> Company No. 06908001
>>
>> CONFIDENTIALITY NOTICE
>> This communication and the information it contains are intended for the
>> person or organisation to which it is addressed. Its contents are
>> confidential and may be protected in law. Unauthorised use, copying or
>> disclosure of any of it may be unlawful. If you are not the intended
>> recipient, please contact us immediately.
>> The contents of any attachments in this e-mail may contain software
>> viruses, which could damage your own computer system. While ITEC Support
>> has taken every reasonable precaution to minimise this risk, we cannot
>> accept liability for any damage which you sustain as a result of software
>> viruses. You should carry out your own virus checking procedure before
>> opening any attachment.
>>
>> ______________________________________________________________________
>> This message has been checked for all known viruses by MessageLabs Virus
>> Scanning Service.
>> ______________________________________________________________________
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://confluence.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>
> --
> Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬
>
> ☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞
> http://twitter.com/FreeSWITCH
> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
> <http://freeswitch.org/g+>*
>
> ClueCon Weekly Development Call
> ☎ sip:888 at conference.freeswitch.org  ☎ +19193869900 <(919)%20386-9900>
>
> https://www.youtube.com/watch?v=9XXgW34t40s
> https://www.youtube.com/watch?v=NLaDpGQuZDA
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>



-- 
Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬

☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞
http://twitter.com/FreeSWITCH
☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
<http://freeswitch.org/g+>*

ClueCon Weekly Development Call
☎ sip:888 at conference.freeswitch.org  ☎ +19193869900

https://www.youtube.com/watch?v=9XXgW34t40s
https://www.youtube.com/watch?v=NLaDpGQuZDA
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20170329/1c03c5ef/attachment-0001.html

Join us at ClueCon 2016 Aug 8-12, 2016
More information about the FreeSWITCH-users mailing list