[Freeswitch-users] CPU sys load issue

Antonio Silva asilva at wirelessmundi.com
Fri Feb 24 01:51:47 MSK 2017


Hi Brian,

Not for today, the first time it happens was my first though something 
with the traffic, but i couldn't find any association with the type of 
traffic. Actually after restart fs i receive traffic from the same 
endpoints, same type of calls and the issue don't reproduce until next 
morning...

There was something i was doing at 00 every day, the command "fsctl 
sync_clock"... well i just disabled it today. I've to wait until 
tomorrow to check the results.

I just install fs compiled with tcmalloc, so if there is no problems i 
will put back the "sync_clock " command to really check if it could be 
the cause.

Right now i'm quite lost on what to do more...


On 02/23/2017 11:37 PM, Brian West wrote:
> Do you have pcaps of the traffic that may trigger this behavior?
>
> /b
>
>
> On Thu, Feb 23, 2017 at 2:58 PM, Antonio Silva 
> <asilva at wirelessmundi.com <mailto:asilva at wirelessmundi.com>> wrote:
>
>     Thanks Juan, that's it, i was missing the dev pkg.
>
>
>
>     On 02/23/2017 08:34 PM, Juan Pablo Godoy Macari wrote:
>>     Antonio, you should install this:
>>
>>     apt-get install libgoogle-perftools-dev
>>
>>     Regards,
>>
>>     2017-02-23 16:26 GMT-03:00 Juan Pablo Godoy Macari
>>     <jpgodoy at redvoiss.net <mailto:jpgodoy at redvoiss.net>>:
>>
>>         Try this: ./configure LIBS="-ltcmalloc"
>>
>>         2017-02-23 16:19 GMT-03:00 Antonio Silva
>>         <asilva at wirelessmundi.com <mailto:asilva at wirelessmundi.com>>:
>>
>>             Hi,
>>
>>             Did you compile fs in debian with tcmalloc?
>>
>>             didn't work for me... do i miss something?
>>
>>
>>             root at dev:/usr/src/freeswitch.git#
>>             <mailto:root at dev:/usr/src/freeswitch.git#> apt-get
>>             install libtcmalloc-minimal4
>>             root at dev:/usr/src/freeswitch.git#
>>             <mailto:root at dev:/usr/src/freeswitch.git#> export
>>             LIBS=-ltcmalloc
>>             root at dev:/usr/src/freeswitch.git#
>>             <mailto:root at dev:/usr/src/freeswitch.git#> ./configure
>>             checking for a BSD-compatible install... /usr/bin/install -c
>>             checking whether build environment is sane... yes
>>             checking for a thread-safe mkdir -p... /bin/mkdir -p
>>             checking for gawk... no
>>             checking for mawk... mawk
>>             checking whether make sets $(MAKE)... yes
>>             checking whether make supports nested variables... yes
>>             checking whether make supports nested variables...
>>             (cached) yes
>>             checking build system type... x86_64-unknown-linux-gnu
>>             checking host system type... x86_64-unknown-linux-gnu
>>             checking for style of include used by make... GNU
>>             checking for gcc... gcc
>>             checking whether the C compiler works... no
>>             configure: error: in `/usr/src/freeswitch.git':
>>             configure: error: C compiler cannot create executables
>>
>>
>>
>>
>>
>>             On 02/23/2017 07:56 PM, Antonio Silva wrote:
>>>             Hi Juan,
>>>
>>>             Thanks for the tips!!
>>>
>>>             Unfortunately i cannot test with -nosql option, i need
>>>             to have the sip_registration fill to be able to send
>>>             options to natted endpoints. But i don't think the issue
>>>             is in db, i use postgres and to get better performance
>>>             from it i run it on tmpfs.
>>>
>>>
>>>             I read about tcmalloc, but from jira FS-7315, i get the
>>>             idea that it didn't change anything.. but i guess i can
>>>             try it and post the results, from the bt i do see some
>>>             calls been done to malloc, but not sure if they are
>>>             locking or not.
>>>
>>>             Regards,
>>>
>>>
>>>             On 02/23/2017 06:52 PM, Juan Pablo Godoy Macari wrote:
>>>>             Hello Antonio,
>>>>
>>>>             I had the same issue, to fix it i tried many things
>>>>             written in this mail list. Try running FS with -nosql
>>>>             option, if that doesn't work recompile FS with tcmalloc
>>>>             (from google's perfomance tools).
>>>>
>>>>             That did it for me.
>>>>
>>>>             Regards,
>>>>             Juan Pablo.
>>>>
>>>>             2017-02-23 13:26 GMT-03:00 Antonio Silva
>>>>             <asilva at wirelessmundi.com
>>>>             <mailto:asilva at wirelessmundi.com>>:
>>>>
>>>>                 Hi,
>>>>
>>>>                 i'm having an issue with FS that gets lock every
>>>>                 morning with cpu sys
>>>>                 hitting 80%, because is limited in fs with
>>>>                 min-idle, it could reach the
>>>>                 100%, so the calls are rejected by load, to solve
>>>>                 the problem i kill fs
>>>>                 and start it again.
>>>>
>>>>                 I discard a load issue, because after restarting
>>>>                 freeswitch the system
>>>>                 increases in number of sessions/cps and no problems
>>>>                 at all until next
>>>>                 day.... it's given me crazy... i've enough memory,
>>>>                 lots of cores...
>>>>                 after restart fs the system stays at less than 30%
>>>>                 of cpu usage.
>>>>
>>>>
>>>>                 Right now i have hpet as clocksource, transcoding
>>>>                 is minimal (use
>>>>                 inheric_codec, only happens because of different
>>>>                 ptimes with some
>>>>                 endpoints) .
>>>>
>>>>
>>>>                 anyone having similar problem? or any tip i can do
>>>>                 to debug this issue?
>>>>
>>>>
>>>>                 For info the Jira ticket can be located at
>>>>                 https://freeswitch.org/jira/browse/FS-9957
>>>>                 <https://freeswitch.org/jira/browse/FS-9957>,
>>>>
>>>>                 i've done a gcore, top, htop when the system is
>>>>                 with high cpu sys but i
>>>>                 can't figure the thread that is causing the
>>>>                 issue... initial i thought
>>>>                 it was because of module lua (most of my logic is
>>>>                 done there), but today
>>>>                 only 2 threads where running mod_lua when it happen.
>>>>
>>>>
>>>>                 Appreciate the help.
>>>>
>>>>
>>>>                 --
>>>>
>>>>                 Saludos / Regards / Cumprimentos,
>>>>                 António silva
>>>>
>>>>
>>>>                 _________________________________________________________________________
>>>>                 Professional FreeSWITCH Consulting Services:
>>>>                 consulting at freeswitch.org
>>>>                 <mailto:consulting at freeswitch.org>
>>>>                 http://www.freeswitchsolutions.com
>>>>                 <http://www.freeswitchsolutions.com>
>>>>
>>>>                 Official FreeSWITCH Sites
>>>>                 http://www.freeswitch.org
>>>>                 http://confluence.freeswitch.org
>>>>                 <http://confluence.freeswitch.org>
>>>>                 http://www.cluecon.com
>>>>
>>>>                 FreeSWITCH-users mailing list
>>>>                 FreeSWITCH-users at lists.freeswitch.org
>>>>                 <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>>                 http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>                 <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
>>>>                 UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>                 <http://lists.freeswitch.org/mailman/options/freeswitch-users>
>>>>                 http://www.freeswitch.org
>>>>
>>>>
>>>>
>>>>
>>>>             _________________________________________________________________________
>>>>             Professional FreeSWITCH Consulting Services:
>>>>             consulting at freeswitch.org
>>>>             <mailto:consulting at freeswitch.org>
>>>>             http://www.freeswitchsolutions.com
>>>>             <http://www.freeswitchsolutions.com>
>>>>
>>>>             Official FreeSWITCH Sites
>>>>             http://www.freeswitch.org
>>>>             http://confluence.freeswitch.org
>>>>             <http://confluence.freeswitch.org>
>>>>             http://www.cluecon.com
>>>>
>>>>             FreeSWITCH-users mailing list
>>>>             FreeSWITCH-users at lists.freeswitch.org
>>>>             <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>>             http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>             <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
>>>>             UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>             <http://lists.freeswitch.org/mailman/options/freeswitch-users>
>>>>             http://www.freeswitch.org
>>>
>>>             _________________________________________________________________________
>>>             Professional FreeSWITCH Consulting Services:
>>>             consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>>             http://www.freeswitchsolutions.com
>>>             <http://www.freeswitchsolutions.com>
>>>
>>>             Official FreeSWITCH Sites
>>>             http://www.freeswitch.org
>>>             http://confluence.freeswitch.org
>>>             <http://confluence.freeswitch.org>
>>>             http://www.cluecon.com
>>>
>>>             FreeSWITCH-users mailing list
>>>             FreeSWITCH-users at lists.freeswitch.org
>>>             <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>             http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>             <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
>>>             UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>             <http://lists.freeswitch.org/mailman/options/freeswitch-users>
>>>             http://www.freeswitch.org
>>             _________________________________________________________________________
>>             Professional FreeSWITCH Consulting Services:
>>             consulting at freeswitch.org
>>             <mailto:consulting at freeswitch.org>
>>             http://www.freeswitchsolutions.com
>>             <http://www.freeswitchsolutions.com> Official FreeSWITCH
>>             Sites http://www.freeswitch.org
>>             http://confluence.freeswitch.org
>>             <http://confluence.freeswitch.org> http://www.cluecon.com
>>             FreeSWITCH-users mailing list
>>             FreeSWITCH-users at lists.freeswitch.org
>>             <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>             http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>             <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
>>             UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>             <http://lists.freeswitch.org/mailman/options/freeswitch-users>
>>             http://www.freeswitch.org 
>>
>>     _________________________________________________________________________
>>     Professional FreeSWITCH Consulting Services:
>>     consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>     http://www.freeswitchsolutions.com
>>     <http://www.freeswitchsolutions.com>
>>
>>     Official FreeSWITCH Sites
>>     http://www.freeswitch.org
>>     http://confluence.freeswitch.org <http://confluence.freeswitch.org>
>>     http://www.cluecon.com
>>
>>     FreeSWITCH-users mailing list
>>     FreeSWITCH-users at lists.freeswitch.org
>>     <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>     http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>     <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
>>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>     <http://lists.freeswitch.org/mailman/options/freeswitch-users>
>>     http://www.freeswitch.org
>     _________________________________________________________________________
>     Professional FreeSWITCH Consulting Services:
>     consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>     http://www.freeswitchsolutions.com
>     <http://www.freeswitchsolutions.com> Official FreeSWITCH Sites
>     http://www.freeswitch.org http://confluence.freeswitch.org
>     <http://confluence.freeswitch.org> http://www.cluecon.com
>     FreeSWITCH-users mailing list
>     FreeSWITCH-users at lists.freeswitch.org
>     <mailto:FreeSWITCH-users at lists.freeswitch.org>
>     http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>     <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>     <http://lists.freeswitch.org/mailman/options/freeswitch-users>
>     http://www.freeswitch.org 
>
> -- 
>
> */Brian West/* brian at freeswitch.org <mailto:brian at freeswitch.org>
>
> */Twitter: @FreeSWITCH , @briankwest/*
>
> http://www.freeswitchbook.com http://www.freeswitchcookbook.com 
> Allison prompts for FreeSWITCH:
>
> *https://www.gofundme.com/allison-prompts-for-freeswitch*
>
> Wish to schedule a meeting?
>
> http://app.timebridge.com/#/meet/freeswitch
>
> Got Bugs? Report them here <https://freeswitch.org/jira>! | Reddit: 
> /r/freeswitch <https://www.reddit.com/r/freeswitch>
>
> *T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378) 
> *Skype:*briankwest
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20170223/c3e225c5/attachment-0001.html 


Join us at ClueCon 2016 Aug 8-12, 2016
More information about the FreeSWITCH-users mailing list