[Freeswitch-users] CPU sys load issue

Fri Feb 24 16:31:15 MSK 2017

NTP adjusts the system clock while FS initialises a monotonic clock on
start. FS doesn't see adjustments for drift by NTP.

sync_clock_when_idle is better than sync_clock.

You can also use the enable-use-system-time option which'll get it to use
the system (and therefore NTP) time.

On 23 February 2017 at 23:14, Antonio Silva <asilva at wirelessmundi.com>
wrote:

> i had a issue with cdr timestamp and the system date, after uptime of 140
> days fs clock was behind 1min.
> It has something between system clock and ntp, so to force fs to keep up i
> force the sync_clock.
>
>
>
> On 02/24/2017 12:06 AM, Brian West wrote:
>
> First why did you ever call sync_clock?
>
> On Thu, Feb 23, 2017 at 4:51 PM, Antonio Silva <asilva at wirelessmundi.com>
> wrote:
>
>> Hi Brian,
>>
>> Not for today, the first time it happens was my first though something
>> with the traffic, but i couldn't find any association with the type of
>> traffic. Actually after restart fs i receive traffic from the same
>> endpoints, same type of calls and the issue don't reproduce until next
>> morning...
>>
>> There was something i was doing at 00 every day, the command "fsctl
>> sync_clock"... well i just disabled it today. I've to wait until tomorrow
>> to check the results.
>>
>> I just install fs compiled with tcmalloc, so if there is no problems i
>> will put back the "sync_clock " command to really check if it could be the
>> cause.
>>
>> Right now i'm quite lost on what to do more...
>>
>>
>>
>> On 02/23/2017 11:37 PM, Brian West wrote:
>>
>> Do you have pcaps of the traffic that may trigger this behavior?
>>
>> /b
>>
>>
>> On Thu, Feb 23, 2017 at 2:58 PM, Antonio Silva <asilva at wirelessmundi.com>
>> wrote:
>>
>>> Thanks Juan, that's it, i was missing the dev pkg.
>>>
>>>
>>>
>>> On 02/23/2017 08:34 PM, Juan Pablo Godoy Macari wrote:
>>>
>>> Antonio, you should install this:
>>>
>>> apt-get install libgoogle-perftools-dev
>>>
>>> Regards,
>>>
>>> 2017-02-23 16:26 GMT-03:00 Juan Pablo Godoy Macari <jpgodoy at redvoiss.net
>>> >:
>>>
>>>> Try this: ./configure LIBS="-ltcmalloc"
>>>>
>>>> 2017-02-23 16:19 GMT-03:00 Antonio Silva <asilva at wirelessmundi.com>:
>>>>
>>>>> Hi,
>>>>>
>>>>> Did you compile fs in debian with tcmalloc?
>>>>>
>>>>> didn't work for me... do i miss something?
>>>>>
>>>>>
>>>>> root at dev:/usr/src/freeswitch.git# apt-get install libtcmalloc-minimal4
>>>>> root at dev:/usr/src/freeswitch.git# export LIBS=-ltcmalloc
>>>>> root at dev:/usr/src/freeswitch.git# ./configure
>>>>> checking for a BSD-compatible install... /usr/bin/install -c
>>>>> checking whether build environment is sane... yes
>>>>> checking for a thread-safe mkdir -p... /bin/mkdir -p
>>>>> checking for gawk... no
>>>>> checking for mawk... mawk
>>>>> checking whether make sets $(MAKE)... yes
>>>>> checking whether make supports nested variables... yes
>>>>> checking whether make supports nested variables... (cached) yes
>>>>> checking build system type... x86_64-unknown-linux-gnu
>>>>> checking host system type... x86_64-unknown-linux-gnu
>>>>> checking for style of include used by make... GNU
>>>>> checking for gcc... gcc
>>>>> checking whether the C compiler works... no
>>>>> configure: error: in `/usr/src/freeswitch.git':
>>>>> configure: error: C compiler cannot create executables
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 02/23/2017 07:56 PM, Antonio Silva wrote:
>>>>>
>>>>> Hi Juan,
>>>>>
>>>>> Thanks for the tips!!
>>>>>
>>>>> Unfortunately i cannot test with -nosql option, i need to have the
>>>>> sip_registration fill to be able to send options to natted endpoints. But i
>>>>> don't think the issue is in db, i use postgres and to get better
>>>>> performance from it i run it on tmpfs.
>>>>>
>>>>>
>>>>> I read about tcmalloc, but from jira FS-7315, i get the idea that it
>>>>> didn't change anything.. but i guess i can try it and post the results,
>>>>> from the bt i do see some calls been done to malloc, but not sure if they
>>>>> are locking or not.
>>>>>
>>>>> Regards,
>>>>>
>>>>>
>>>>> On 02/23/2017 06:52 PM, Juan Pablo Godoy Macari wrote:
>>>>>
>>>>> Hello Antonio,
>>>>>
>>>>> I had the same issue, to fix it i tried many things written in this
>>>>> mail list. Try running FS with -nosql option, if that doesn't work
>>>>> recompile FS with tcmalloc (from google's perfomance tools).
>>>>>
>>>>> That did it for me.
>>>>>
>>>>> Regards,
>>>>> Juan Pablo.
>>>>>
>>>>> 2017-02-23 13:26 GMT-03:00 Antonio Silva <asilva at wirelessmundi.com>:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> i'm having an issue with FS that gets lock every morning  with cpu sys
>>>>>> hitting 80%, because is limited in fs with min-idle, it could reach
>>>>>> the
>>>>>> 100%, so the calls are rejected by load, to solve the problem i kill
>>>>>> fs
>>>>>> and start it again.
>>>>>>
>>>>>> I discard a load issue, because after restarting freeswitch the system
>>>>>> increases in number of sessions/cps and no problems at all until next
>>>>>> day.... it's given me crazy... i've enough memory, lots of cores...
>>>>>> after restart fs the system stays at less than 30% of cpu usage.
>>>>>>
>>>>>>
>>>>>> Right now i have hpet as clocksource, transcoding is minimal (use
>>>>>> inheric_codec, only happens because of different ptimes with some
>>>>>> endpoints) .
>>>>>>
>>>>>>
>>>>>> anyone having similar problem? or any tip i can do to debug this
>>>>>> issue?
>>>>>>
>>>>>>
>>>>>> For info the Jira ticket can be located at
>>>>>> https://freeswitch.org/jira/browse/FS-9957,
>>>>>>
>>>>>> i've done a gcore, top, htop when the system is with high cpu sys but
>>>>>> i
>>>>>> can't figure the thread that is causing the issue... initial i thought
>>>>>> it was because of module lua (most of my logic is done there), but
>>>>>> today
>>>>>> only 2 threads where running mod_lua when it happen.
>>>>>>
>>>>>>
>>>>>> Appreciate the help.
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Saludos / Regards / Cumprimentos,
>>>>>> António silva
>>>>>>
>>>>>>
>>>>>> ____________________________________________________________
>>>>>> _____________
>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>> consulting at freeswitch.org
>>>>>> http://www.freeswitchsolutions.com
>>>>>>
>>>>>> Official FreeSWITCH Sites
>>>>>> http://www.freeswitch.org
>>>>>> http://confluence.freeswitch.org
>>>>>> http://www.cluecon.com
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/free
>>>>>> switch-users
>>>>>> http://www.freeswitch.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services: consulting at freeswitch.orghttp://www.freeswitchsolutions.com
>>>>>
>>>>> Official FreeSWITCH Siteshttp://www.freeswitch.orghttp://confluence.freeswitch.orghttp://www.cluecon.com
>>>>>
>>>>> FreeSWITCH-users mailing listFreeSWITCH-users at lists.freeswitch.orghttp://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-usershttp://www.freeswitch.org
>>>>>
>>>>> _________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services: consulting at freeswitch.orghttp://www.freeswitchsolutions.com
>>>>>
>>>>> Official FreeSWITCH Siteshttp://www.freeswitch.orghttp://confluence.freeswitch.orghttp://www.cluecon.com
>>>>>
>>>>> FreeSWITCH-users mailing listFreeSWITCH-users at lists.freeswitch.orghttp://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-usershttp://www.freeswitch.org
>>>>>
>>>>> _________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services: consulting at freeswitch.org
>>>>> http://www.freeswitchsolutions.com Official FreeSWITCH Sites
>>>>> http://www.freeswitch.org http://confluence.freeswitch.org
>>>>> http://www.cluecon.com FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org http://lists.freeswitch.org/ma
>>>>> ilman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.frees
>>>>> witch.org/mailman/options/freeswitch-users http://www.freeswitch.org
>>>>
>>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services: consulting at freeswitch.orghttp://www.freeswitchsolutions.com
>>>
>>> Official FreeSWITCH Siteshttp://www.freeswitch.orghttp://confluence.freeswitch.orghttp://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing listFreeSWITCH-users at lists.freeswitch.orghttp://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-usershttp://www.freeswitch.org
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services: consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com Official FreeSWITCH Sites
>>> http://www.freeswitch.org http://confluence.freeswitch.org
>>> http://www.cluecon.com FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org http://lists.freeswitch.org/ma
>>> ilman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.frees
>>> witch.org/mailman/options/freeswitch-users http://www.freeswitch.org
>>
>> --
>>
>> *Brian West* brian at freeswitch.org
>>
>> *Twitter: @FreeSWITCH , @briankwest*
>>
>> http://www.freeswitchbook.com  http://www.freeswitchcookbook.com Allison
>> prompts for FreeSWITCH:
>>
>> *https://www.gofundme.com/allison-prompts-for-freeswitch*
>> <https://www.gofundme.com/allison-prompts-for-freeswitch>
>>
>> Wish to schedule a meeting?
>>
>> http://app.timebridge.com/#/meet/freeswitch
>>
>> Got Bugs? Report them here <https://freeswitch.org/jira>! | Reddit:
>> /r/freeswitch <https://www.reddit.com/r/freeswitch>
>>
>> *T:*+19184209001 <%28918%29%20420-9001> | *F:*+19184209002
>> <%28918%29%20420-9002> | *M:*+1918424WEST (9378) *Skype:*briankwest
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services: consulting at freeswitch.orghttp://www.freeswitchsolutions.com
>>
>> Official FreeSWITCH Siteshttp://www.freeswitch.orghttp://confluence.freeswitch.orghttp://www.cluecon.com
>>
>> FreeSWITCH-users mailing listFreeSWITCH-users at lists.freeswitch.orghttp://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-usershttp://www.freeswitch.org
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services: consulting at freeswitch.org
>> http://www.freeswitchsolutions.com Official FreeSWITCH Sites
>> http://www.freeswitch.org http://confluence.freeswitch.org
>> http://www.cluecon.com FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org http://lists.freeswitch.org/ma
>> ilman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.frees
>> witch.org/mailman/options/freeswitch-users http://www.freeswitch.org
>
> --
>
> *Brian West* brian at freeswitch.org
>
> *Twitter: @FreeSWITCH , @briankwest*
>
> http://www.freeswitchbook.com  http://www.freeswitchcookbook.com Allison
> prompts for FreeSWITCH:
>
> *https://www.gofundme.com/allison-prompts-for-freeswitch*
> <https://www.gofundme.com/allison-prompts-for-freeswitch>
>
> Wish to schedule a meeting?
>
> http://app.timebridge.com/#/meet/freeswitch
>
> Got Bugs? Report them here <https://freeswitch.org/jira>! | Reddit:
> /r/freeswitch <https://www.reddit.com/r/freeswitch>
>
> *T:*+19184209001 <(918)%20420-9001> | *F:*+19184209002 <(918)%20420-9002>
> | *M:*+1918424WEST (9378) *Skype:*briankwest
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services: consulting at freeswitch.orghttp://www.freeswitchsolutions.com
>
> Official FreeSWITCH Siteshttp://www.freeswitch.orghttp://confluence.freeswitch.orghttp://www.cluecon.com
>
> FreeSWITCH-users mailing listFreeSWITCH-users at lists.freeswitch.orghttp://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-usershttp://www.freeswitch.org
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20170224/affd8067/attachment-0001.html 