[Freeswitch-users] CPU sys load issue

Antonio Silva asilva at wirelessmundi.com
Fri Feb 24 02:14:58 MSK 2017


i had a issue with cdr timestamp and the system date, after uptime of 
140 days fs clock was behind 1min.
It has something between system clock and ntp, so to force fs to keep up 
i force the sync_clock.


On 02/24/2017 12:06 AM, Brian West wrote:
> First why did you ever call sync_clock?
>
> On Thu, Feb 23, 2017 at 4:51 PM, Antonio Silva 
> <asilva at wirelessmundi.com <mailto:asilva at wirelessmundi.com>> wrote:
>
>     Hi Brian,
>
>     Not for today, the first time it happens was my first though
>     something with the traffic, but i couldn't find any association
>     with the type of traffic. Actually after restart fs i receive
>     traffic from the same endpoints, same type of calls and the issue
>     don't reproduce until next morning...
>
>     There was something i was doing at 00 every day, the command
>     "fsctl sync_clock"... well i just disabled it today. I've to wait
>     until tomorrow to check the results.
>
>     I just install fs compiled with tcmalloc, so if there is no
>     problems i will put back the "sync_clock " command to really check
>     if it could be the cause.
>
>     Right now i'm quite lost on what to do more...
>
>
>
>     On 02/23/2017 11:37 PM, Brian West wrote:
>>     Do you have pcaps of the traffic that may trigger this behavior?
>>
>>     /b
>>
>>
>>     On Thu, Feb 23, 2017 at 2:58 PM, Antonio Silva
>>     <asilva at wirelessmundi.com <mailto:asilva at wirelessmundi.com>> wrote:
>>
>>         Thanks Juan, that's it, i was missing the dev pkg.
>>
>>
>>
>>         On 02/23/2017 08:34 PM, Juan Pablo Godoy Macari wrote:
>>>         Antonio, you should install this:
>>>
>>>         apt-get install libgoogle-perftools-dev
>>>
>>>         Regards,
>>>
>>>         2017-02-23 16:26 GMT-03:00 Juan Pablo Godoy Macari
>>>         <jpgodoy at redvoiss.net <mailto:jpgodoy at redvoiss.net>>:
>>>
>>>             Try this: ./configure LIBS="-ltcmalloc"
>>>
>>>             2017-02-23 16:19 GMT-03:00 Antonio Silva
>>>             <asilva at wirelessmundi.com
>>>             <mailto:asilva at wirelessmundi.com>>:
>>>
>>>                 Hi,
>>>
>>>                 Did you compile fs in debian with tcmalloc?
>>>
>>>                 didn't work for me... do i miss something?
>>>
>>>
>>>                 root at dev:/usr/src/freeswitch.git#
>>>                 <mailto:root at dev:/usr/src/freeswitch.git#> apt-get
>>>                 install libtcmalloc-minimal4
>>>                 root at dev:/usr/src/freeswitch.git#
>>>                 <mailto:root at dev:/usr/src/freeswitch.git#> export
>>>                 LIBS=-ltcmalloc
>>>                 root at dev:/usr/src/freeswitch.git#
>>>                 <mailto:root at dev:/usr/src/freeswitch.git#> ./configure
>>>                 checking for a BSD-compatible install...
>>>                 /usr/bin/install -c
>>>                 checking whether build environment is sane... yes
>>>                 checking for a thread-safe mkdir -p... /bin/mkdir -p
>>>                 checking for gawk... no
>>>                 checking for mawk... mawk
>>>                 checking whether make sets $(MAKE)... yes
>>>                 checking whether make supports nested variables... yes
>>>                 checking whether make supports nested variables...
>>>                 (cached) yes
>>>                 checking build system type... x86_64-unknown-linux-gnu
>>>                 checking host system type... x86_64-unknown-linux-gnu
>>>                 checking for style of include used by make... GNU
>>>                 checking for gcc... gcc
>>>                 checking whether the C compiler works... no
>>>                 configure: error: in `/usr/src/freeswitch.git':
>>>                 configure: error: C compiler cannot create executables
>>>
>>>
>>>
>>>
>>>
>>>                 On 02/23/2017 07:56 PM, Antonio Silva wrote:
>>>>                 Hi Juan,
>>>>
>>>>                 Thanks for the tips!!
>>>>
>>>>                 Unfortunately i cannot test with -nosql option, i
>>>>                 need to have the sip_registration fill to be able
>>>>                 to send options to natted endpoints. But i don't
>>>>                 think the issue is in db, i use postgres and to get
>>>>                 better performance from it i run it on tmpfs.
>>>>
>>>>
>>>>                 I read about tcmalloc, but from jira FS-7315, i get
>>>>                 the idea that it didn't change anything.. but i
>>>>                 guess i can try it and post the results, from the
>>>>                 bt i do see some calls been done to malloc, but not
>>>>                 sure if they are locking or not.
>>>>
>>>>                 Regards,
>>>>
>>>>
>>>>                 On 02/23/2017 06:52 PM, Juan Pablo Godoy Macari wrote:
>>>>>                 Hello Antonio,
>>>>>
>>>>>                 I had the same issue, to fix it i tried many
>>>>>                 things written in this mail list. Try running FS
>>>>>                 with -nosql option, if that doesn't work recompile
>>>>>                 FS with tcmalloc (from google's perfomance tools).
>>>>>
>>>>>                 That did it for me.
>>>>>
>>>>>                 Regards,
>>>>>                 Juan Pablo.
>>>>>
>>>>>                 2017-02-23 13:26 GMT-03:00 Antonio Silva
>>>>>                 <asilva at wirelessmundi.com
>>>>>                 <mailto:asilva at wirelessmundi.com>>:
>>>>>
>>>>>                     Hi,
>>>>>
>>>>>                     i'm having an issue with FS that gets lock
>>>>>                     every morning with cpu sys
>>>>>                     hitting 80%, because is limited in fs with
>>>>>                     min-idle, it could reach the
>>>>>                     100%, so the calls are rejected by load, to
>>>>>                     solve the problem i kill fs
>>>>>                     and start it again.
>>>>>
>>>>>                     I discard a load issue, because after
>>>>>                     restarting freeswitch the system
>>>>>                     increases in number of sessions/cps and no
>>>>>                     problems at all until next
>>>>>                     day.... it's given me crazy... i've enough
>>>>>                     memory, lots of cores...
>>>>>                     after restart fs the system stays at less than
>>>>>                     30% of cpu usage.
>>>>>
>>>>>
>>>>>                     Right now i have hpet as clocksource,
>>>>>                     transcoding is minimal (use
>>>>>                     inheric_codec, only happens because of
>>>>>                     different ptimes with some
>>>>>                     endpoints) .
>>>>>
>>>>>
>>>>>                     anyone having similar problem? or any tip i
>>>>>                     can do to debug this issue?
>>>>>
>>>>>
>>>>>                     For info the Jira ticket can be located at
>>>>>                     https://freeswitch.org/jira/browse/FS-9957
>>>>>                     <https://freeswitch.org/jira/browse/FS-9957>,
>>>>>
>>>>>                     i've done a gcore, top, htop when the system
>>>>>                     is with high cpu sys but i
>>>>>                     can't figure the thread that is causing the
>>>>>                     issue... initial i thought
>>>>>                     it was because of module lua (most of my logic
>>>>>                     is done there), but today
>>>>>                     only 2 threads where running mod_lua when it
>>>>>                     happen.
>>>>>
>>>>>
>>>>>                     Appreciate the help.
>>>>>
>>>>>
>>>>>                     --
>>>>>
>>>>>                     Saludos / Regards / Cumprimentos,
>>>>>                     António silva
>>>>>
>>>>>
>>>>>                     _________________________________________________________________________
>>>>>                     Professional FreeSWITCH Consulting Services:
>>>>>                     consulting at freeswitch.org
>>>>>                     <mailto:consulting at freeswitch.org>
>>>>>                     http://www.freeswitchsolutions.com
>>>>>                     <http://www.freeswitchsolutions.com>
>>>>>
>>>>>                     Official FreeSWITCH Sites
>>>>>                     http://www.freeswitch.org
>>>>>                     http://confluence.freeswitch.org
>>>>>                     <http://confluence.freeswitch.org>
>>>>>                     http://www.cluecon.com
>>>>>
>>>>>                     FreeSWITCH-users mailing list
>>>>>                     FreeSWITCH-users at lists.freeswitch.org
>>>>>                     <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>>>                     http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>                     <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
>>>>>                     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>                     <http://lists.freeswitch.org/mailman/options/freeswitch-users>
>>>>>                     http://www.freeswitch.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>                 _________________________________________________________________________
>>>>>                 Professional FreeSWITCH Consulting Services:
>>>>>                 consulting at freeswitch.org
>>>>>                 <mailto:consulting at freeswitch.org>
>>>>>                 http://www.freeswitchsolutions.com
>>>>>                 <http://www.freeswitchsolutions.com>
>>>>>
>>>>>                 Official FreeSWITCH Sites
>>>>>                 http://www.freeswitch.org
>>>>>                 http://confluence.freeswitch.org
>>>>>                 <http://confluence.freeswitch.org>
>>>>>                 http://www.cluecon.com
>>>>>
>>>>>                 FreeSWITCH-users mailing list
>>>>>                 FreeSWITCH-users at lists.freeswitch.org
>>>>>                 <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>>>                 http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>                 <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
>>>>>                 UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>                 <http://lists.freeswitch.org/mailman/options/freeswitch-users>
>>>>>                 http://www.freeswitch.org
>>>>
>>>>                 _________________________________________________________________________
>>>>                 Professional FreeSWITCH Consulting Services:
>>>>                 consulting at freeswitch.org
>>>>                 <mailto:consulting at freeswitch.org>
>>>>                 http://www.freeswitchsolutions.com
>>>>                 <http://www.freeswitchsolutions.com>
>>>>
>>>>                 Official FreeSWITCH Sites
>>>>                 http://www.freeswitch.org
>>>>                 http://confluence.freeswitch.org
>>>>                 <http://confluence.freeswitch.org>
>>>>                 http://www.cluecon.com
>>>>
>>>>                 FreeSWITCH-users mailing list
>>>>                 FreeSWITCH-users at lists.freeswitch.org
>>>>                 <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>>                 http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>                 <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
>>>>                 UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>                 <http://lists.freeswitch.org/mailman/options/freeswitch-users>
>>>>                 http://www.freeswitch.org
>>>                 _________________________________________________________________________
>>>                 Professional FreeSWITCH Consulting Services:
>>>                 consulting at freeswitch.org
>>>                 <mailto:consulting at freeswitch.org>
>>>                 http://www.freeswitchsolutions.com
>>>                 <http://www.freeswitchsolutions.com> Official
>>>                 FreeSWITCH Sites http://www.freeswitch.org
>>>                 http://confluence.freeswitch.org
>>>                 <http://confluence.freeswitch.org>
>>>                 http://www.cluecon.com FreeSWITCH-users mailing list
>>>                 FreeSWITCH-users at lists.freeswitch.org
>>>                 <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>                 http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>                 <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
>>>                 UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>                 <http://lists.freeswitch.org/mailman/options/freeswitch-users>
>>>                 http://www.freeswitch.org 
>>>
>>>         _________________________________________________________________________
>>>         Professional FreeSWITCH Consulting Services:
>>>         consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>>         http://www.freeswitchsolutions.com
>>>         <http://www.freeswitchsolutions.com>
>>>
>>>         Official FreeSWITCH Sites
>>>         http://www.freeswitch.org
>>>         http://confluence.freeswitch.org
>>>         <http://confluence.freeswitch.org>
>>>         http://www.cluecon.com
>>>
>>>         FreeSWITCH-users mailing list
>>>         FreeSWITCH-users at lists.freeswitch.org
>>>         <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>         http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>         <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
>>>         UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>         <http://lists.freeswitch.org/mailman/options/freeswitch-users>
>>>         http://www.freeswitch.org
>>         _________________________________________________________________________
>>         Professional FreeSWITCH Consulting Services:
>>         consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>         http://www.freeswitchsolutions.com
>>         <http://www.freeswitchsolutions.com> Official FreeSWITCH
>>         Sites http://www.freeswitch.org
>>         http://confluence.freeswitch.org
>>         <http://confluence.freeswitch.org> http://www.cluecon.com
>>         FreeSWITCH-users mailing list
>>         FreeSWITCH-users at lists.freeswitch.org
>>         <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>         http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>         <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
>>         UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>         <http://lists.freeswitch.org/mailman/options/freeswitch-users>
>>         http://www.freeswitch.org 
>>
>>     -- 
>>
>>     */Brian West/* brian at freeswitch.org <mailto:brian at freeswitch.org>
>>
>>     */Twitter: @FreeSWITCH , @briankwest/*
>>
>>     http://www.freeswitchbook.com http://www.freeswitchcookbook.com
>>     <http://www.freeswitchcookbook.com> Allison prompts for FreeSWITCH:
>>
>>     *https://www.gofundme.com/allison-prompts-for-freeswitch*
>>     <https://www.gofundme.com/allison-prompts-for-freeswitch>
>>
>>     Wish to schedule a meeting?
>>
>>     http://app.timebridge.com/#/meet/freeswitch
>>     <http://app.timebridge.com/#/meet/freeswitch>
>>
>>     Got Bugs? Report them here <https://freeswitch.org/jira>! |
>>     Reddit: /r/freeswitch <https://www.reddit.com/r/freeswitch>
>>
>>     *T:*+19184209001 <tel:%28918%29%20420-9001> | *F:*+19184209002
>>     <tel:%28918%29%20420-9002> | *M:*+1918424WEST (9378)
>>     *Skype:*briankwest
>>
>>     _________________________________________________________________________
>>     Professional FreeSWITCH Consulting Services:
>>     consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>     http://www.freeswitchsolutions.com
>>     <http://www.freeswitchsolutions.com>
>>
>>     Official FreeSWITCH Sites
>>     http://www.freeswitch.org
>>     http://confluence.freeswitch.org <http://confluence.freeswitch.org>
>>     http://www.cluecon.com
>>
>>     FreeSWITCH-users mailing list
>>     FreeSWITCH-users at lists.freeswitch.org
>>     <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>     http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>     <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
>>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>     <http://lists.freeswitch.org/mailman/options/freeswitch-users>
>>     http://www.freeswitch.org
>     _________________________________________________________________________
>     Professional FreeSWITCH Consulting Services:
>     consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>     http://www.freeswitchsolutions.com
>     <http://www.freeswitchsolutions.com> Official FreeSWITCH Sites
>     http://www.freeswitch.org http://confluence.freeswitch.org
>     <http://confluence.freeswitch.org> http://www.cluecon.com
>     FreeSWITCH-users mailing list
>     FreeSWITCH-users at lists.freeswitch.org
>     <mailto:FreeSWITCH-users at lists.freeswitch.org>
>     http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>     <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>     <http://lists.freeswitch.org/mailman/options/freeswitch-users>
>     http://www.freeswitch.org 
>
> -- 
>
> */Brian West/* brian at freeswitch.org <mailto:brian at freeswitch.org>
>
> */Twitter: @FreeSWITCH , @briankwest/*
>
> http://www.freeswitchbook.com http://www.freeswitchcookbook.com 
> Allison prompts for FreeSWITCH:
>
> *https://www.gofundme.com/allison-prompts-for-freeswitch*
>
> Wish to schedule a meeting?
>
> http://app.timebridge.com/#/meet/freeswitch
>
> Got Bugs? Report them here <https://freeswitch.org/jira>! | Reddit: 
> /r/freeswitch <https://www.reddit.com/r/freeswitch>
>
> *T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378) 
> *Skype:*briankwest
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20170224/8740173a/attachment-0001.html 


Join us at ClueCon 2016 Aug 8-12, 2016
More information about the FreeSWITCH-users mailing list