[Freeswitch-users] 100% CPU usage

Thu Mar 30 14:34:38 MSD 2017

>  The strange thing that I’ve noticed is that even with 0 active calls, FS
is using 15-30% cpu constantly with only 40-60 registered endpoints.

How are you handling the registrations? Those endpoints will send REGISTER
packets frequently which will individually get processed. So if they're
calling an API though mod_xml_curl that'll be some extra load without
active calls. mod_sofia will be updating the registration database which is
sqlite by default and can be moved to a ramdisk or to ODBC. You might also
find clients are registering unusually frequently.

On 29 March 2017 at 18:20, Cj B <blackc2004 at gmail.com> wrote:

> Hi All,
>
> I’m having a very similar issue however it’s not staying at 100% CPU on
> physical hardware. I’ve created a JIRA but wanted to see if anyone else had
> any feedback on possible causes? I’m using freeswitch 1.6.15 (previously
> 1.6.13). What happens is that FS will jump to 100% CPU and stay there for
> 1-2 seconds then drop back down but this is then causing massive
> jitter/call quality issues for about 10-15 seconds after the spike. The
> strange thing that I’ve noticed is that even with 0 active calls, FS is
> using 15-30% cpu constantly with only 40-60 registered endpoints.
>
> I’m watching the logs in DEBUG mode and not seeing anything there that’s
> indicative of any issue, no loops that I can see, and it seems to be
> happening when there’s even just 1 or 2 active calls on the system.
>
> I’m wondering if there’s any other suggestions on where to look because
> this is getting very frustrating for me and my users.
>
> Thanks much for any pointers!
> Cj B
>
> On Mar 22, 2017, at 8:13 AM, Anthony Minessale <
> anthony.minessale at gmail.com> wrote:
>
> top -H -p `pidof freeswitch`
> Note the pid using 100% cpu
>
> gcore `pidof freeswitch`
>
> Attach it to a jira.  If you think its a bug in FS you should report it to
> Jira not to this list.
>
> https://freeswitch.org/jira
>
>
>
>
>
> On Wed, Mar 22, 2017 at 10:14 AM Shaun Stokes <
> shaun.stokes at itec-support.co.uk> wrote:
>
>> It’s probably a long shot, but if you’re able to replicate the problem it
>> might be worth trying to replicate this on physical hardware. That way you
>> eliminate your virtual environment from the problem, there are a number of
>> factors which exist in virtual environments depending on your hyper-visor
>> which can contribute to performance issues of which we’ve experienced in
>> the past.
>>
>>
>> It would also be worth trying to isolate what the script is doing when
>> the high CPU begins, if you’ve got a test environment you can use to
>> replicate the issue then you could try tweaking the script so you can
>> eliminate certain actions from the list of causes, there’s maybe a
>> FreeSWITCH module which is being called which triggers this.
>>
>>
>> *From:* freeswitch-users-bounces at lists.freeswitch.org [mailto:
>> freeswitch-users-bounces at lists.freeswitch.org] *On Behalf Of *Grant
>> Bagdasarian
>> *Sent:* 22 March 2017 11:20
>>
>>
>> *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
>> *Subject:* Re: [Freeswitch-users] 100% CPU usage
>>
>>
>> @Shuan, yes we are using the session:ready() method before we execute.
>>
>> We are running all of our servers virtually and never had any issues up
>> until now.
>>
>> This same application ran on older versions of freeswitch without issues,
>> even during and after high load.
>>
>>
>> @Antonio, thanks for the tips! I’ll have a look and try some stuff.
>>
>>
>> *From:* freeswitch-users-bounces at lists.freeswitch.org [
>> mailto:freeswitch-users-bounces at lists.freeswitch.org
>> <freeswitch-users-bounces at lists.freeswitch.org>] *On Behalf Of *Antonio
>> Silva
>> *Sent:* woensdag 22 maart 2017 10:09
>> *To:* freeswitch-users at lists.freeswitch.org
>> *Subject:* Re: [Freeswitch-users] 100% CPU usage
>>
>>
>> We had some issues using pcall functions like io.popen when trying to
>> execute external commands, that block mod_lua, i end up replacing that with
>> os.execute.
>>
>> Recently i also had some issue with cpu hitting 100%,for what i could
>> compile from debugging was something related with malloc calls. after
>> compiling fs with tcmalloc the issue disappear, you could try this approach
>> as well.
>>
>> You can check jira: https://freeswitch.org/jira/browse/FS-9957 and  also
>> in the malling: "CPU sys load issue"  for more info.
>>
>>
>> Regards,
>> António
>>
>> On 03/22/2017 09:52 AM, Shaun Stokes wrote:
>>
>> Are you using ‘if (session:ready()) then’ in your scripts before you
>> execute?
>>
>>
>> Are you using virtual or physical hardware and what spec?
>>
>>
>> *From:* freeswitch-users-bounces at lists.freeswitch.org [
>> mailto:freeswitch-users-bounces at lists.freeswitch.org
>> <freeswitch-users-bounces at lists.freeswitch.org>] *On Behalf Of *Grant
>> Bagdasarian
>> *Sent:* 22 March 2017 07:31
>> *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
>> <freeswitch-users at lists.freeswitch.org>
>> *Subject:* Re: [Freeswitch-users] 100% CPU usage
>>
>>
>> Anthony, we triple checked the script and its dependencies. There are no
>> loops at all in the script. The imported scripts are also written by us and
>> they too do not contain any loops.
>>
>> It’s just basic call handling with some playback and play_and_get_digits
>> calls, and some pcall calls to query a database.
>>
>>
>> *From:* freeswitch-users-bounces at lists.freeswitch.org [
>> mailto:freeswitch-users-bounces at lists.freeswitch.org
>> <freeswitch-users-bounces at lists.freeswitch.org>] *On Behalf Of *Anthony
>> Minessale
>> *Sent:* dinsdag 21 maart 2017 16:02
>> *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
>> *Subject:* Re: [Freeswitch-users] 100% CPU usage
>>
>>
>> I already told you why a week ago.  Your script most likely has an
>> endless loop in it that does not check the condition session:ready()
>>
>>
>> However, Feel free to contact consulting at freeswitch.org and we can
>> formally diagnose it for a fee.
>>
>>
>>
>>
>>
>> On Tue, Mar 21, 2017 at 6:42 AM Grant Bagdasarian <gb at cm.nl> wrote:
>>
>> Attached are the logs for:
>>
>> #check thread usage:
>>
>> top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1
>>
>>
>>
>>
>>
>> #memory mapping:
>>
>> cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps
>>
>>
>>
>> We’re unable to run gcore since freeswitch hogs all available cpu’s at
>> 100%.
>>
>> I tried changing the nice settings real-time but that didn’t work either.
>>
>>
>> At one point we had a huge spike in inbound traffic towards our
>> freeswitch (valid traffic).
>>
>> The traffic is handled by a lua application.
>>
>> What we saw was calls hanging in freeswitch while they’ve already been
>> physically disconnected.
>>
>> After the huge spike freeswitch behaved normally for a couple of minutes
>> and then suddenly started eating all the available cpu cores at 100%.
>>
>>
>> We have some db queries which are fired in the lua application, so these
>> may cause the application to hang, but we haven’t reproduced it yet.
>>
>>
>> I’ll setup a test lab with the same configuration and fire sipp at it to
>> see if I can reproduce the issue.
>>
>> However, I probably won’t be able to run gcore for debugging. Any other
>> suggestions?
>>
>>
>> Also we saw a lot of open udp ports and some had a huge read buffer size
>> which did not decrease.
>>
>>
>>
>>
>> *From:* freeswitch-users-bounces at lists.freeswitch.org [mailto:
>> freeswitch-users-bounces at lists.freeswitch.org] *On Behalf Of *Grant
>> Bagdasarian
>> *Sent:* maandag 13 maart 2017 09:04
>>
>>
>> *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
>> *Subject:* Re: [Freeswitch-users] 100% CPU usage
>>
>>
>> Hi Antony and Antonio,
>>
>>
>> Thank you for the tips.
>>
>> We haven’t had freeswitch going berserk anymore (not sure as to why it
>> did perform without issue over the weekend), but will continue to monitor
>> it and run the commands to trace the issue.
>>
>>
>> Regards,
>>
>>
>> Grant
>>
>>
>> *From:* freeswitch-users-bounces at lists.freeswitch.org [
>> mailto:freeswitch-users-bounces at lists.freeswitch.org
>> <freeswitch-users-bounces at lists.freeswitch.org>] *On Behalf Of *Anthony
>> Minessale
>> *Sent:* zaterdag 11 maart 2017 00:07
>> *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
>> *Subject:* Re: [Freeswitch-users] 100% CPU usage
>>
>>
>> Make sure you lua script has all while loops including session:ready() as
>> a condition to continue.  Its a common mistake that leads to loops spinning
>> out of control on hungup channels.
>>
>>
>>
>> On Fri, Mar 10, 2017 at 6:11 AM, Antonio Silva <asilva at wirelessmundi.com>
>> wrote:
>>
>> Hi,
>>
>> You can use the following commands:
>>
>> #get a core:
>>
>> gcore -o /home/gcore-$(date +%s) $(cat /var/run/freeswitch/freeswitch.pid)
>>
>>
>>
>>
>>
>> #check thread usage:
>>
>> top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1
>>
>>
>>
>>
>>
>> #memory mapping:
>>
>> cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps
>>
>>
>>
>> you can get some info from fs:
>>
>> fs_cli -x "sofia global siptrace on"
>>
>> fs_cli -x "fsctl debug_level 10"
>>
>> fs_cli -x "sofia loglevel all 9"
>>
>> fs_cli -x "fsctl debug_sql "
>>
>> fs_cli -x "sofia global debug presence"
>>
>>
>>
>> Put everything into a jira so we can take a look.
>>
>>
>> Regards,
>> António
>>
>>
>> On 03/10/2017 11:08 AM, Grant Bagdasarian wrote:
>>
>> Hi,
>>
>>
>> We’re currently experiencing a weird issue with one of our freeswitch
>> servers where the CPU just goes berserk and stays at 100%.
>>
>> There are no active calls at that moment, but FS is still using 100% CPU.
>>
>>
>> I was unable to get a debug using gdb, since freeswitch does not crash.
>>
>>
>> We had this issue on 1.6.13 and then I upgraded to 1.6.15 this morning,
>> but the problem still persists.
>>
>> There is currently only one active application running on this box which
>> is using LUA. Maybe that is causing the issue?
>>
>> We checked the code but there are no weird loops etc. Perhaps we are
>> missing something?
>>
>>
>> Any ideas how to trace this?
>>
>>
>> Regards,
>>
>>
>> Grant
>>
>>
>>
>> _________________________________________________________________________
>>
>> Professional FreeSWITCH Consulting Services:
>>
>> consulting at freeswitch.org
>>
>> http://www.freeswitchsolutions.com
>>
>>
>>
>> Official FreeSWITCH Sites
>>
>> http://www.freeswitch.org
>>
>> http://confluence.freeswitch.org
>>
>> http://www.cluecon.com
>>
>>
>>
>> FreeSWITCH-users mailing list
>>
>> FreeSWITCH-users at lists.freeswitch.org
>>
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>
>> http://www.freeswitch.org
>>
>>
>>
>> --
>>
>>
>>
>> Saludos / Regards / Cumprimentos,
>>
>> António silva
>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://confluence.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>>
>>
>>
>> --
>>
>> Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬
>>
>>
>> ☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞
>> http://twitter.com/FreeSWITCH
>>
>> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
>> <http://freeswitch.org/g+>*
>>
>> ClueCon Weekly Development Call
>>
>> ☎ sip:888 at conference.freeswitch.org  ☎ +19193869900 <(919)%20386-9900>
>>
>>
>> https://www.youtube.com/watch?v=9XXgW34t40s
>>
>> https://www.youtube.com/watch?v=NLaDpGQuZDA
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://confluence.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>> --
>>
>> Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬
>>
>>
>> ☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞
>> http://twitter.com/FreeSWITCH
>>
>> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
>> <http://freeswitch.org/g+>*
>>
>> ClueCon Weekly Development Call
>>
>> ☎ sip:888 at conference.freeswitch.org  ☎ +19193869900 <(919)%20386-9900>
>>
>>
>> https://www.youtube.com/watch?v=9XXgW34t40s
>>
>> https://www.youtube.com/watch?v=NLaDpGQuZDA
>>
>>
>> ______________________________________________________________________
>> This message has been checked for all known viruses by MessageLabs Virus
>> Scanning Service.
>> ______________________________________________________________________
>>
>> Shaun Stokes - Infrastructure Analyst
>>
>> T :
>>
>> 01453 700713
>>
>> E :
>>
>> shaun.stokes at itec-support.co.uk
>>
>> W :
>>
>> www.itec-support.co.uk
>>
>> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road,
>> Stroud, Gloucestershire GL5 3QF
>> Company No. 06908001
>>
>>
>> CONFIDENTIALITY NOTICE
>> This communication and the information it contains are intended for the
>> person or organisation to which it is addressed. Its contents are
>> confidential and may be protected in law. Unauthorised use, copying or
>> disclosure of any of it may be unlawful. If you are not the intended
>> recipient, please contact us immediately.
>> The contents of any attachments in this e-mail may contain software
>> viruses, which could damage your own computer system. While ITEC Support
>> has taken every reasonable precaution to minimise this risk, we cannot
>> accept liability for any damage which you sustain as a result of software
>> viruses. You should carry out your own virus checking procedure before
>> opening any attachment.
>>
>>
>> ______________________________________________________________________
>> This message has been checked for all known viruses by MessageLabs Virus
>> Scanning Service.
>> ______________________________________________________________________
>>
>>
>> _________________________________________________________________________
>>
>> Professional FreeSWITCH Consulting Services:
>>
>> consulting at freeswitch.org
>>
>> http://www.freeswitchsolutions.com
>>
>>
>>
>> Official FreeSWITCH Sites
>>
>> http://www.freeswitch.org
>>
>> http://confluence.freeswitch.org
>>
>> http://www.cluecon.com
>>
>>
>>
>> FreeSWITCH-users mailing list
>>
>> FreeSWITCH-users at lists.freeswitch.org
>>
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>
>> http://www.freeswitch.org
>>
>>
>>
>> --
>>
>>
>>
>> Saludos / Regards / Cumprimentos,
>>
>> António silva
>>
>>
>> ______________________________________________________________________
>> This message has been checked for all known viruses by MessageLabs Virus
>> Scanning Service.
>> ______________________________________________________________________
>> Shaun Stokes - Infrastructure Analyst
>> T : 01453 700713
>> E : shaun.stokes at itec-support.co.uk
>> W : www.itec-support.co.uk
>> Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road,
>> Stroud, Gloucestershire GL5 3QF
>> Company No. 06908001
>>
>> CONFIDENTIALITY NOTICE
>> This communication and the information it contains are intended for the
>> person or organisation to which it is addressed. Its contents are
>> confidential and may be protected in law. Unauthorised use, copying or
>> disclosure of any of it may be unlawful. If you are not the intended
>> recipient, please contact us immediately.
>> The contents of any attachments in this e-mail may contain software
>> viruses, which could damage your own computer system. While ITEC Support
>> has taken every reasonable precaution to minimise this risk, we cannot
>> accept liability for any damage which you sustain as a result of software
>> viruses. You should carry out your own virus checking procedure before
>> opening any attachment.
>>
>> ______________________________________________________________________
>> This message has been checked for all known viruses by MessageLabs Virus
>> Scanning Service.
>> ______________________________________________________________________
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://confluence.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>
> --
> Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬
>
> ☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞
> http://twitter.com/FreeSWITCH
> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
> <http://freeswitch.org/g+>*
>
> ClueCon Weekly Development Call
> ☎ sip:888 at conference.freeswitch.org  ☎ +19193869900 <(919)%20386-9900>
>
> https://www.youtube.com/watch?v=9XXgW34t40s
> https://www.youtube.com/watch?v=NLaDpGQuZDA
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20170330/980b7881/attachment-0001.html 