Hi Juan, Thanks for the tips!! Unfortunately i cannot test with -nosql option, i need to have the sip_registration fill to be able to send options to natted endpoints. But i don't think the issue is in db, i use postgres and to get better performance from it i run it on tmpfs. I read about tcmalloc, but from jira FS-7315, i get the idea that it didn't change anything.. but i guess i can try it and post the results, from the bt i do see some calls been done to malloc, but not sure if they are locking or not. Regards, On 02/23/2017 06:52 PM, Juan Pablo Godoy Macari wrote:

Hello Antonio,

I had the same issue, to fix it i tried many things written in this mail list. Try running FS with -nosql option, if that doesn't work recompile FS with tcmalloc (from google's perfomance tools).

That did it for me.

Regards, Juan Pablo.

2017-02-23 13:26 GMT-03:00 Antonio Silva <asilva@wirelessmundi.com <mailto:asilva@wirelessmundi.com>>:

Hi,

i'm having an issue with FS that gets lock every morning with cpu sys hitting 80%, because is limited in fs with min-idle, it could reach the 100%, so the calls are rejected by load, to solve the problem i kill fs and start it again.

I discard a load issue, because after restarting freeswitch the system increases in number of sessions/cps and no problems at all until next day.... it's given me crazy... i've enough memory, lots of cores... after restart fs the system stays at less than 30% of cpu usage.

Right now i have hpet as clocksource, transcoding is minimal (use inheric_codec, only happens because of different ptimes with some endpoints) .

anyone having similar problem? or any tip i can do to debug this issue?

For info the Jira ticket can be located at https://freeswitch.org/jira/browse/FS-9957 <https://freeswitch.org/jira/browse/FS-9957>,

i've done a gcore, top, htop when the system is with high cpu sys but i can't figure the thread that is causing the issue... initial i thought it was because of module lua (most of my logic is done there), but today only 2 threads where running mod_lua when it happen.

Appreciate the help.

--

Saludos / Regards / Cumprimentos, António silva

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com>

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

Try this: ./configure LIBS="-ltcmalloc" 2017-02-23 16:19 GMT-03:00 Antonio Silva <asilva@wirelessmundi.com>:

Hi,

Did you compile fs in debian with tcmalloc?

didn't work for me... do i miss something?

root@dev:/usr/src/freeswitch.git# apt-get install libtcmalloc-minimal4 root@dev:/usr/src/freeswitch.git# export LIBS=-ltcmalloc root@dev:/usr/src/freeswitch.git# ./configure checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /bin/mkdir -p checking for gawk... no checking for mawk... mawk checking whether make sets $(MAKE)... yes checking whether make supports nested variables... yes checking whether make supports nested variables... (cached) yes checking build system type... x86_64-unknown-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking for style of include used by make... GNU checking for gcc... gcc checking whether the C compiler works... no configure: error: in `/usr/src/freeswitch.git': configure: error: C compiler cannot create executables

On 02/23/2017 07:56 PM, Antonio Silva wrote:

Hi Juan,

Thanks for the tips!!

Unfortunately i cannot test with -nosql option, i need to have the sip_registration fill to be able to send options to natted endpoints. But i don't think the issue is in db, i use postgres and to get better performance from it i run it on tmpfs.

I read about tcmalloc, but from jira FS-7315, i get the idea that it didn't change anything.. but i guess i can try it and post the results, from the bt i do see some calls been done to malloc, but not sure if they are locking or not.

Regards,

On 02/23/2017 06:52 PM, Juan Pablo Godoy Macari wrote:

Hello Antonio,

I had the same issue, to fix it i tried many things written in this mail list. Try running FS with -nosql option, if that doesn't work recompile FS with tcmalloc (from google's perfomance tools).

That did it for me.

Regards, Juan Pablo.

2017-02-23 13:26 GMT-03:00 Antonio Silva <asilva@wirelessmundi.com>:

...

Hi,

i'm having an issue with FS that gets lock every morning with cpu sys hitting 80%, because is limited in fs with min-idle, it could reach the 100%, so the calls are rejected by load, to solve the problem i kill fs and start it again.

I discard a load issue, because after restarting freeswitch the system increases in number of sessions/cps and no problems at all until next day.... it's given me crazy... i've enough memory, lots of cores... after restart fs the system stays at less than 30% of cpu usage.

Right now i have hpet as clocksource, transcoding is minimal (use inheric_codec, only happens because of different ptimes with some endpoints) .

anyone having similar problem? or any tip i can do to debug this issue?

For info the Jira ticket can be located at https://freeswitch.org/jira/browse/FS-9957,

i've done a gcore, top, htop when the system is with high cpu sys but i can't figure the thread that is causing the issue... initial i thought it was because of module lua (most of my logic is done there), but today only 2 threads where running mod_lua when it happen.

Appreciate the help.

--

Saludos / Regards / Cumprimentos, António silva

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.orghttp://www.freeswitchsolutions.com

Official FreeSWITCH Siteshttp://www.freeswitch.orghttp://confluence.freeswitch.orghttp://www.cluecon....

FreeSWITCH-users mailing listFreeSWITCH-users@lists.freeswitch.orghttp://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-usershttp://www.frees...

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.orghttp://www.freeswitchsolutions.com

Official FreeSWITCH Siteshttp://www.freeswitch.orghttp://confluence.freeswitch.orghttp://www.cluecon....

FreeSWITCH-users mailing listFreeSWITCH-users@lists.freeswitch.orghttp://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-usershttp://www.frees...

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

Thanks Juan, that's it, i was missing the dev pkg. On 02/23/2017 08:34 PM, Juan Pablo Godoy Macari wrote:

Antonio, you should install this:

apt-get install libgoogle-perftools-dev

Regards,

2017-02-23 16:26 GMT-03:00 Juan Pablo Godoy Macari <jpgodoy@redvoiss.net <mailto:jpgodoy@redvoiss.net>>:

Try this: ./configure LIBS="-ltcmalloc"

2017-02-23 16:19 GMT-03:00 Antonio Silva <asilva@wirelessmundi.com <mailto:asilva@wirelessmundi.com>>:

Hi,

Did you compile fs in debian with tcmalloc?

didn't work for me... do i miss something?

root@dev:/usr/src/freeswitch.git# <mailto:root@dev:/usr/src/freeswitch.git#> apt-get install libtcmalloc-minimal4 root@dev:/usr/src/freeswitch.git# <mailto:root@dev:/usr/src/freeswitch.git#> export LIBS=-ltcmalloc root@dev:/usr/src/freeswitch.git# <mailto:root@dev:/usr/src/freeswitch.git#> ./configure checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /bin/mkdir -p checking for gawk... no checking for mawk... mawk checking whether make sets $(MAKE)... yes checking whether make supports nested variables... yes checking whether make supports nested variables... (cached) yes checking build system type... x86_64-unknown-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking for style of include used by make... GNU checking for gcc... gcc checking whether the C compiler works... no configure: error: in `/usr/src/freeswitch.git': configure: error: C compiler cannot create executables

On 02/23/2017 07:56 PM, Antonio Silva wrote:

...

Hi Juan,

Thanks for the tips!!

Unfortunately i cannot test with -nosql option, i need to have the sip_registration fill to be able to send options to natted endpoints. But i don't think the issue is in db, i use postgres and to get better performance from it i run it on tmpfs.

I read about tcmalloc, but from jira FS-7315, i get the idea that it didn't change anything.. but i guess i can try it and post the results, from the bt i do see some calls been done to malloc, but not sure if they are locking or not.

Regards,

On 02/23/2017 06:52 PM, Juan Pablo Godoy Macari wrote:

...

Hello Antonio,

I had the same issue, to fix it i tried many things written in this mail list. Try running FS with -nosql option, if that doesn't work recompile FS with tcmalloc (from google's perfomance tools).

That did it for me.

Regards, Juan Pablo.

2017-02-23 13:26 GMT-03:00 Antonio Silva <asilva@wirelessmundi.com <mailto:asilva@wirelessmundi.com>>:

Hi,

i'm having an issue with FS that gets lock every morning with cpu sys hitting 80%, because is limited in fs with min-idle, it could reach the 100%, so the calls are rejected by load, to solve the problem i kill fs and start it again.

I discard a load issue, because after restarting freeswitch the system increases in number of sessions/cps and no problems at all until next day.... it's given me crazy... i've enough memory, lots of cores... after restart fs the system stays at less than 30% of cpu usage.

Right now i have hpet as clocksource, transcoding is minimal (use inheric_codec, only happens because of different ptimes with some endpoints) .

anyone having similar problem? or any tip i can do to debug this issue?

For info the Jira ticket can be located at https://freeswitch.org/jira/browse/FS-9957 <https://freeswitch.org/jira/browse/FS-9957>,

i've done a gcore, top, htop when the system is with high cpu sys but i can't figure the thread that is causing the issue... initial i thought it was because of module lua (most of my logic is done there), but today only 2 threads where running mod_lua when it happen.

Appreciate the help.

--

Saludos / Regards / Cumprimentos, António silva

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com>

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com>

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com>

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com> Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

Hi Brian, Not for today, the first time it happens was my first though something with the traffic, but i couldn't find any association with the type of traffic. Actually after restart fs i receive traffic from the same endpoints, same type of calls and the issue don't reproduce until next morning... There was something i was doing at 00 every day, the command "fsctl sync_clock"... well i just disabled it today. I've to wait until tomorrow to check the results. I just install fs compiled with tcmalloc, so if there is no problems i will put back the "sync_clock " command to really check if it could be the cause. Right now i'm quite lost on what to do more... On 02/23/2017 11:37 PM, Brian West wrote:

Do you have pcaps of the traffic that may trigger this behavior?

/b

On Thu, Feb 23, 2017 at 2:58 PM, Antonio Silva <asilva@wirelessmundi.com <mailto:asilva@wirelessmundi.com>> wrote:

Thanks Juan, that's it, i was missing the dev pkg.

On 02/23/2017 08:34 PM, Juan Pablo Godoy Macari wrote:

...

Antonio, you should install this:

apt-get install libgoogle-perftools-dev

Regards,

2017-02-23 16:26 GMT-03:00 Juan Pablo Godoy Macari <jpgodoy@redvoiss.net <mailto:jpgodoy@redvoiss.net>>:

Try this: ./configure LIBS="-ltcmalloc"

2017-02-23 16:19 GMT-03:00 Antonio Silva <asilva@wirelessmundi.com <mailto:asilva@wirelessmundi.com>>:

Hi,

Did you compile fs in debian with tcmalloc?

didn't work for me... do i miss something?

root@dev:/usr/src/freeswitch.git# <mailto:root@dev:/usr/src/freeswitch.git#> apt-get install libtcmalloc-minimal4 root@dev:/usr/src/freeswitch.git# <mailto:root@dev:/usr/src/freeswitch.git#> export LIBS=-ltcmalloc root@dev:/usr/src/freeswitch.git# <mailto:root@dev:/usr/src/freeswitch.git#> ./configure checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /bin/mkdir -p checking for gawk... no checking for mawk... mawk checking whether make sets $(MAKE)... yes checking whether make supports nested variables... yes checking whether make supports nested variables... (cached) yes checking build system type... x86_64-unknown-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking for style of include used by make... GNU checking for gcc... gcc checking whether the C compiler works... no configure: error: in `/usr/src/freeswitch.git': configure: error: C compiler cannot create executables

On 02/23/2017 07:56 PM, Antonio Silva wrote:

...

Hi Juan,

Thanks for the tips!!

Unfortunately i cannot test with -nosql option, i need to have the sip_registration fill to be able to send options to natted endpoints. But i don't think the issue is in db, i use postgres and to get better performance from it i run it on tmpfs.

I read about tcmalloc, but from jira FS-7315, i get the idea that it didn't change anything.. but i guess i can try it and post the results, from the bt i do see some calls been done to malloc, but not sure if they are locking or not.

Regards,

On 02/23/2017 06:52 PM, Juan Pablo Godoy Macari wrote:

...

Hello Antonio,

I had the same issue, to fix it i tried many things written in this mail list. Try running FS with -nosql option, if that doesn't work recompile FS with tcmalloc (from google's perfomance tools).

That did it for me.

Regards, Juan Pablo.

2017-02-23 13:26 GMT-03:00 Antonio Silva <asilva@wirelessmundi.com <mailto:asilva@wirelessmundi.com>>:

Hi,

i'm having an issue with FS that gets lock every morning with cpu sys hitting 80%, because is limited in fs with min-idle, it could reach the 100%, so the calls are rejected by load, to solve the problem i kill fs and start it again.

I discard a load issue, because after restarting freeswitch the system increases in number of sessions/cps and no problems at all until next day.... it's given me crazy... i've enough memory, lots of cores... after restart fs the system stays at less than 30% of cpu usage.

Right now i have hpet as clocksource, transcoding is minimal (use inheric_codec, only happens because of different ptimes with some endpoints) .

anyone having similar problem? or any tip i can do to debug this issue?

For info the Jira ticket can be located at https://freeswitch.org/jira/browse/FS-9957 <https://freeswitch.org/jira/browse/FS-9957>,

i've done a gcore, top, htop when the system is with high cpu sys but i can't figure the thread that is causing the issue... initial i thought it was because of module lua (most of my logic is done there), but today only 2 threads where running mod_lua when it happen.

Appreciate the help.

--

Saludos / Regards / Cumprimentos, António silva

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com>

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com>

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com>

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com> Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com>

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com> Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

--

*/Brian West/* brian@freeswitch.org <mailto:brian@freeswitch.org>

*/Twitter: @FreeSWITCH , @briankwest/*

http://www.freeswitchbook.com http://www.freeswitchcookbook.com Allison prompts for FreeSWITCH:

*https://www.gofundme.com/allison-prompts-for-freeswitch*

Wish to schedule a meeting?

http://app.timebridge.com/#/meet/freeswitch

Got Bugs? Report them here <https://freeswitch.org/jira>! | Reddit: /r/freeswitch <https://www.reddit.com/r/freeswitch>

*T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378) *Skype:*briankwest

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

i had a issue with cdr timestamp and the system date, after uptime of 140 days fs clock was behind 1min. It has something between system clock and ntp, so to force fs to keep up i force the sync_clock. On 02/24/2017 12:06 AM, Brian West wrote:

First why did you ever call sync_clock?

On Thu, Feb 23, 2017 at 4:51 PM, Antonio Silva <asilva@wirelessmundi.com <mailto:asilva@wirelessmundi.com>> wrote:

Hi Brian,

Not for today, the first time it happens was my first though something with the traffic, but i couldn't find any association with the type of traffic. Actually after restart fs i receive traffic from the same endpoints, same type of calls and the issue don't reproduce until next morning...

There was something i was doing at 00 every day, the command "fsctl sync_clock"... well i just disabled it today. I've to wait until tomorrow to check the results.

I just install fs compiled with tcmalloc, so if there is no problems i will put back the "sync_clock " command to really check if it could be the cause.

Right now i'm quite lost on what to do more...

On 02/23/2017 11:37 PM, Brian West wrote:

...

Do you have pcaps of the traffic that may trigger this behavior?

/b

On Thu, Feb 23, 2017 at 2:58 PM, Antonio Silva <asilva@wirelessmundi.com <mailto:asilva@wirelessmundi.com>> wrote:

Thanks Juan, that's it, i was missing the dev pkg.

On 02/23/2017 08:34 PM, Juan Pablo Godoy Macari wrote:

...

Antonio, you should install this:

apt-get install libgoogle-perftools-dev

Regards,

2017-02-23 16:26 GMT-03:00 Juan Pablo Godoy Macari <jpgodoy@redvoiss.net <mailto:jpgodoy@redvoiss.net>>:

Try this: ./configure LIBS="-ltcmalloc"

2017-02-23 16:19 GMT-03:00 Antonio Silva <asilva@wirelessmundi.com <mailto:asilva@wirelessmundi.com>>:

Hi,

Did you compile fs in debian with tcmalloc?

didn't work for me... do i miss something?

root@dev:/usr/src/freeswitch.git# <mailto:root@dev:/usr/src/freeswitch.git#> apt-get install libtcmalloc-minimal4 root@dev:/usr/src/freeswitch.git# <mailto:root@dev:/usr/src/freeswitch.git#> export LIBS=-ltcmalloc root@dev:/usr/src/freeswitch.git# <mailto:root@dev:/usr/src/freeswitch.git#> ./configure checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /bin/mkdir -p checking for gawk... no checking for mawk... mawk checking whether make sets $(MAKE)... yes checking whether make supports nested variables... yes checking whether make supports nested variables... (cached) yes checking build system type... x86_64-unknown-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking for style of include used by make... GNU checking for gcc... gcc checking whether the C compiler works... no configure: error: in `/usr/src/freeswitch.git': configure: error: C compiler cannot create executables

On 02/23/2017 07:56 PM, Antonio Silva wrote:

...

Hi Juan,

Thanks for the tips!!

Unfortunately i cannot test with -nosql option, i need to have the sip_registration fill to be able to send options to natted endpoints. But i don't think the issue is in db, i use postgres and to get better performance from it i run it on tmpfs.

I read about tcmalloc, but from jira FS-7315, i get the idea that it didn't change anything.. but i guess i can try it and post the results, from the bt i do see some calls been done to malloc, but not sure if they are locking or not.

Regards,

On 02/23/2017 06:52 PM, Juan Pablo Godoy Macari wrote:

...

Hello Antonio,

I had the same issue, to fix it i tried many things written in this mail list. Try running FS with -nosql option, if that doesn't work recompile FS with tcmalloc (from google's perfomance tools).

That did it for me.

Regards, Juan Pablo.

2017-02-23 13:26 GMT-03:00 Antonio Silva <asilva@wirelessmundi.com <mailto:asilva@wirelessmundi.com>>:

Hi,

i'm having an issue with FS that gets lock every morning with cpu sys hitting 80%, because is limited in fs with min-idle, it could reach the 100%, so the calls are rejected by load, to solve the problem i kill fs and start it again.

I discard a load issue, because after restarting freeswitch the system increases in number of sessions/cps and no problems at all until next day.... it's given me crazy... i've enough memory, lots of cores... after restart fs the system stays at less than 30% of cpu usage.

Right now i have hpet as clocksource, transcoding is minimal (use inheric_codec, only happens because of different ptimes with some endpoints) .

anyone having similar problem? or any tip i can do to debug this issue?

For info the Jira ticket can be located at https://freeswitch.org/jira/browse/FS-9957 <https://freeswitch.org/jira/browse/FS-9957>,

i've done a gcore, top, htop when the system is with high cpu sys but i can't figure the thread that is causing the issue... initial i thought it was because of module lua (most of my logic is done there), but today only 2 threads where running mod_lua when it happen.

Appreciate the help.

--

Saludos / Regards / Cumprimentos, António silva

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com>

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com>

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com>

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com> Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com>

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com> Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

--

*/Brian West/* brian@freeswitch.org <mailto:brian@freeswitch.org>

*/Twitter: @FreeSWITCH , @briankwest/*

http://www.freeswitchbook.com http://www.freeswitchcookbook.com <http://www.freeswitchcookbook.com> Allison prompts for FreeSWITCH:

*https://www.gofundme.com/allison-prompts-for-freeswitch* <https://www.gofundme.com/allison-prompts-for-freeswitch>

Wish to schedule a meeting?

http://app.timebridge.com/#/meet/freeswitch <http://app.timebridge.com/#/meet/freeswitch>

Got Bugs? Report them here <https://freeswitch.org/jira>! | Reddit: /r/freeswitch <https://www.reddit.com/r/freeswitch>

*T:*+19184209001 <tel:%28918%29%20420-9001> | *F:*+19184209002 <tel:%28918%29%20420-9002> | *M:*+1918424WEST (9378) *Skype:*briankwest

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com>

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com> Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

--

*/Brian West/* brian@freeswitch.org <mailto:brian@freeswitch.org>

*/Twitter: @FreeSWITCH , @briankwest/*

http://www.freeswitchbook.com http://www.freeswitchcookbook.com Allison prompts for FreeSWITCH:

*https://www.gofundme.com/allison-prompts-for-freeswitch*

Wish to schedule a meeting?

http://app.timebridge.com/#/meet/freeswitch

Got Bugs? Report them here <https://freeswitch.org/jira>! | Reddit: /r/freeswitch <https://www.reddit.com/r/freeswitch>

*T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378) *Skype:*briankwest

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

Hi Steven, Thanks for the tips. i don't use sync_clock_when_idle because it waits until there is 0 calls, if you never hit 0 calls then the clock is never sync, i wanted to be sure that is done. didn't know about "enable-use-system-time", just see the commit and there is a note from Anthony: "add enable-use-system-time param to switch.conf.xml *use at your own risk*", are you using it? does it affect performance? I've to tried it. About cpu sys to high and getting block, using tcmalloc seams to solve it, today everything went ok, no peaks in cpu sys. Next week i'll put back the sync_clock. Regards, On 02/24/2017 02:31 PM, Steven Ayre wrote:

NTP adjusts the system clock while FS initialises a monotonic clock on start. FS doesn't see adjustments for drift by NTP.

sync_clock_when_idle is better than sync_clock.

You can also use the enable-use-system-time option which'll get it to use the system (and therefore NTP) time.

On 23 February 2017 at 23:14, Antonio Silva <asilva@wirelessmundi.com <mailto:asilva@wirelessmundi.com>> wrote:

i had a issue with cdr timestamp and the system date, after uptime of 140 days fs clock was behind 1min. It has something between system clock and ntp, so to force fs to keep up i force the sync_clock.

On 02/24/2017 12:06 AM, Brian West wrote:

...

First why did you ever call sync_clock?

On Thu, Feb 23, 2017 at 4:51 PM, Antonio Silva <asilva@wirelessmundi.com <mailto:asilva@wirelessmundi.com>> wrote:

Hi Brian,

Not for today, the first time it happens was my first though something with the traffic, but i couldn't find any association with the type of traffic. Actually after restart fs i receive traffic from the same endpoints, same type of calls and the issue don't reproduce until next morning...

There was something i was doing at 00 every day, the command "fsctl sync_clock"... well i just disabled it today. I've to wait until tomorrow to check the results.

I just install fs compiled with tcmalloc, so if there is no problems i will put back the "sync_clock " command to really check if it could be the cause.

Right now i'm quite lost on what to do more...

On 02/23/2017 11:37 PM, Brian West wrote:

...

Do you have pcaps of the traffic that may trigger this behavior?

/b

On Thu, Feb 23, 2017 at 2:58 PM, Antonio Silva <asilva@wirelessmundi.com <mailto:asilva@wirelessmundi.com>> wrote:

Thanks Juan, that's it, i was missing the dev pkg.

On 02/23/2017 08:34 PM, Juan Pablo Godoy Macari wrote:

...

Antonio, you should install this:

apt-get install libgoogle-perftools-dev

Regards,

2017-02-23 16:26 GMT-03:00 Juan Pablo Godoy Macari <jpgodoy@redvoiss.net <mailto:jpgodoy@redvoiss.net>>:

Try this: ./configure LIBS="-ltcmalloc"

2017-02-23 16:19 GMT-03:00 Antonio Silva <asilva@wirelessmundi.com <mailto:asilva@wirelessmundi.com>>:

Hi,

Did you compile fs in debian with tcmalloc?

didn't work for me... do i miss something?

root@dev:/usr/src/freeswitch.git# <mailto:root@dev:/usr/src/freeswitch.git#> apt-get install libtcmalloc-minimal4 root@dev:/usr/src/freeswitch.git# <mailto:root@dev:/usr/src/freeswitch.git#> export LIBS=-ltcmalloc root@dev:/usr/src/freeswitch.git# <mailto:root@dev:/usr/src/freeswitch.git#> ./configure checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /bin/mkdir -p checking for gawk... no checking for mawk... mawk checking whether make sets $(MAKE)... yes checking whether make supports nested variables... yes checking whether make supports nested variables... (cached) yes checking build system type... x86_64-unknown-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking for style of include used by make... GNU checking for gcc... gcc checking whether the C compiler works... no configure: error: in `/usr/src/freeswitch.git': configure: error: C compiler cannot create executables

On 02/23/2017 07:56 PM, Antonio Silva wrote:

...

Hi Juan,

Thanks for the tips!!

Unfortunately i cannot test with -nosql option, i need to have the sip_registration fill to be able to send options to natted endpoints. But i don't think the issue is in db, i use postgres and to get better performance from it i run it on tmpfs.

I read about tcmalloc, but from jira FS-7315, i get the idea that it didn't change anything.. but i guess i can try it and post the results, from the bt i do see some calls been done to malloc, but not sure if they are locking or not.

Regards,

On 02/23/2017 06:52 PM, Juan Pablo Godoy Macari wrote:

...

Hello Antonio,

I had the same issue, to fix it i tried many things written in this mail list. Try running FS with -nosql option, if that doesn't work recompile FS with tcmalloc (from google's perfomance tools).

That did it for me.

Regards, Juan Pablo.

2017-02-23 13:26 GMT-03:00 Antonio Silva <asilva@wirelessmundi.com <mailto:asilva@wirelessmundi.com>>:

Hi,

i'm having an issue with FS that gets lock every morning with cpu sys hitting 80%, because is limited in fs with min-idle, it could reach the 100%, so the calls are rejected by load, to solve the problem i kill fs and start it again.

I discard a load issue, because after restarting freeswitch the system increases in number of sessions/cps and no problems at all until next day.... it's given me crazy... i've enough memory, lots of cores... after restart fs the system stays at less than 30% of cpu usage.

Right now i have hpet as clocksource, transcoding is minimal (use inheric_codec, only happens because of different ptimes with some endpoints) .

anyone having similar problem? or any tip i can do to debug this issue?

For info the Jira ticket can be located at https://freeswitch.org/jira/browse/FS-9957 <https://freeswitch.org/jira/browse/FS-9957>,

i've done a gcore, top, htop when the system is with high cpu sys but i can't figure the thread that is causing the issue... initial i thought it was because of module lua (most of my logic is done there), but today only 2 threads where running mod_lua when it happen.

Appreciate the help.

--

Saludos / Regards / Cumprimentos, António silva

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com>

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com>

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com>

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com> Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com>

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com> Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

--

*/Brian West/* brian@freeswitch.org <mailto:brian@freeswitch.org>

*/Twitter: @FreeSWITCH , @briankwest/*

http://www.freeswitchbook.com http://www.freeswitchcookbook.com <http://www.freeswitchcookbook.com> Allison prompts for FreeSWITCH:

*https://www.gofundme.com/allison-prompts-for-freeswitch* <https://www.gofundme.com/allison-prompts-for-freeswitch>

Wish to schedule a meeting?

http://app.timebridge.com/#/meet/freeswitch <http://app.timebridge.com/#/meet/freeswitch>

Got Bugs? Report them here <https://freeswitch.org/jira>! | Reddit: /r/freeswitch <https://www.reddit.com/r/freeswitch>

*T:*+19184209001 <tel:%28918%29%20420-9001> | *F:*+19184209002 <tel:%28918%29%20420-9002> | *M:*+1918424WEST (9378) *Skype:*briankwest

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com>

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com> Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

--

*/Brian West/* brian@freeswitch.org <mailto:brian@freeswitch.org>

*/Twitter: @FreeSWITCH , @briankwest/*

http://www.freeswitchbook.com http://www.freeswitchcookbook.com <http://www.freeswitchcookbook.com> Allison prompts for FreeSWITCH:

*https://www.gofundme.com/allison-prompts-for-freeswitch* <https://www.gofundme.com/allison-prompts-for-freeswitch>

Wish to schedule a meeting?

http://app.timebridge.com/#/meet/freeswitch <http://app.timebridge.com/#/meet/freeswitch>

Got Bugs? Report them here <https://freeswitch.org/jira>! | Reddit: /r/freeswitch <https://www.reddit.com/r/freeswitch>

*T:*+19184209001 <tel:%28918%29%20420-9001> | *F:*+19184209002 <tel:%28918%29%20420-9002> | *M:*+1918424WEST (9378) *Skype:*briankwest

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com>

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com> Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

thsk steven, i've to do some test first as well. As for the main problem, i put back the sync clock every night and do far no peaks on cpu sys. So the solution for me was to put fs compiled with tcmalloc. Regards, António On 03/01/2017 01:02 PM, Steven Ayre wrote:

We're planning on rolling it out but haven't yet done so. Need to schedule some testing on it first.

I suspect the risk he refers to is if you're manually changing the system clock you can cause large clock jumps. That can affect CDR timestamps, either billing for longer periods than you should or getting negative call durations.

If you're using the NTP daemon to get constant adjustments, it's configured correctly, it starts on boot, is trustworthy, refuses to make large clock changes and you're monitoring it properly you'll probably be fine. I'd suggest using UTC for the system clock too, to guarantee you're not seeing any DST jumps.

Steve

On 24 February 2017 at 18:02, Antonio Silva <asilva@wirelessmundi.com <mailto:asilva@wirelessmundi.com>> wrote:

Hi Steven,

Thanks for the tips.

i don't use sync_clock_when_idle because it waits until there is 0 calls, if you never hit 0 calls then the clock is never sync, i wanted to be sure that is done.

didn't know about "enable-use-system-time", just see the commit and there is a note from Anthony: "add enable-use-system-time param to switch.conf.xml *use at your own risk*", are you using it? does it affect performance?

I've to tried it.

About cpu sys to high and getting block, using tcmalloc seams to solve it, today everything went ok, no peaks in cpu sys. Next week i'll put back the sync_clock.

Regards,

On 02/24/2017 02:31 PM, Steven Ayre wrote:

...

NTP adjusts the system clock while FS initialises a monotonic clock on start. FS doesn't see adjustments for drift by NTP.

sync_clock_when_idle is better than sync_clock.

You can also use the enable-use-system-time option which'll get it to use the system (and therefore NTP) time.

On 23 February 2017 at 23:14, Antonio Silva <asilva@wirelessmundi.com <mailto:asilva@wirelessmundi.com>> wrote:

i had a issue with cdr timestamp and the system date, after uptime of 140 days fs clock was behind 1min. It has something between system clock and ntp, so to force fs to keep up i force the sync_clock.

On 02/24/2017 12:06 AM, Brian West wrote:

...

First why did you ever call sync_clock?

On Thu, Feb 23, 2017 at 4:51 PM, Antonio Silva <asilva@wirelessmundi.com <mailto:asilva@wirelessmundi.com>> wrote:

Hi Brian,

Not for today, the first time it happens was my first though something with the traffic, but i couldn't find any association with the type of traffic. Actually after restart fs i receive traffic from the same endpoints, same type of calls and the issue don't reproduce until next morning...

There was something i was doing at 00 every day, the command "fsctl sync_clock"... well i just disabled it today. I've to wait until tomorrow to check the results.

I just install fs compiled with tcmalloc, so if there is no problems i will put back the "sync_clock " command to really check if it could be the cause.

Right now i'm quite lost on what to do more...

On 02/23/2017 11:37 PM, Brian West wrote:

...

Do you have pcaps of the traffic that may trigger this behavior?

/b

On Thu, Feb 23, 2017 at 2:58 PM, Antonio Silva <asilva@wirelessmundi.com <mailto:asilva@wirelessmundi.com>> wrote:

Thanks Juan, that's it, i was missing the dev pkg.

On 02/23/2017 08:34 PM, Juan Pablo Godoy Macari wrote:

...

Antonio, you should install this:

apt-get install libgoogle-perftools-dev

Regards,

2017-02-23 16:26 GMT-03:00 Juan Pablo Godoy Macari <jpgodoy@redvoiss.net <mailto:jpgodoy@redvoiss.net>>:

Try this: ./configure LIBS="-ltcmalloc"

2017-02-23 16:19 GMT-03:00 Antonio Silva <asilva@wirelessmundi.com <mailto:asilva@wirelessmundi.com>>:

Hi,

Did you compile fs in debian with tcmalloc?

didn't work for me... do i miss something?

root@dev:/usr/src/freeswitch.git# <mailto:root@dev:/usr/src/freeswitch.git#> apt-get install libtcmalloc-minimal4 root@dev:/usr/src/freeswitch.git# <mailto:root@dev:/usr/src/freeswitch.git#> export LIBS=-ltcmalloc root@dev:/usr/src/freeswitch.git# <mailto:root@dev:/usr/src/freeswitch.git#> ./configure checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /bin/mkdir -p checking for gawk... no checking for mawk... mawk checking whether make sets $(MAKE)... yes checking whether make supports nested variables... yes checking whether make supports nested variables... (cached) yes checking build system type... x86_64-unknown-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking for style of include used by make... GNU checking for gcc... gcc checking whether the C compiler works... no configure: error: in `/usr/src/freeswitch.git': configure: error: C compiler cannot create executables

On 02/23/2017 07:56 PM, Antonio Silva wrote:

...

Hi Juan,

Thanks for the tips!!

Unfortunately i cannot test with -nosql option, i need to have the sip_registration fill to be able to send options to natted endpoints. But i don't think the issue is in db, i use postgres and to get better performance from it i run it on tmpfs.

I read about tcmalloc, but from jira FS-7315, i get the idea that it didn't change anything.. but i guess i can try it and post the results, from the bt i do see some calls been done to malloc, but not sure if they are locking or not.

Regards,

On 02/23/2017 06:52 PM, Juan Pablo Godoy Macari wrote: > Hello Antonio, > > I had the same issue, to fix it i tried > many things written in this mail list. > Try running FS with -nosql option, if > that doesn't work recompile FS with > tcmalloc (from google's perfomance tools). > > That did it for me. > > Regards, > Juan Pablo. > > 2017-02-23 13:26 GMT-03:00 Antonio Silva > <asilva@wirelessmundi.com > <mailto:asilva@wirelessmundi.com>>: > > Hi, > > i'm having an issue with FS that > gets lock every morning with cpu sys > hitting 80%, because is limited in > fs with min-idle, it could reach the > 100%, so the calls are rejected by > load, to solve the problem i kill fs > and start it again. > > I discard a load issue, because > after restarting freeswitch the system > increases in number of sessions/cps > and no problems at all until next > day.... it's given me crazy... i've > enough memory, lots of cores... > after restart fs the system stays at > less than 30% of cpu usage. > > > Right now i have hpet as > clocksource, transcoding is minimal (use > inheric_codec, only happens because > of different ptimes with some > endpoints) . > > > anyone having similar problem? or > any tip i can do to debug this issue? > > > For info the Jira ticket can be > located at > https://freeswitch.org/jira/browse/FS-9957 > <https://freeswitch.org/jira/browse/FS-9957>, > > i've done a gcore, top, htop when > the system is with high cpu sys but i > can't figure the thread that is > causing the issue... initial i thought > it was because of module lua (most > of my logic is done there), but today > only 2 threads where running mod_lua > when it happen. > > > Appreciate the help. > > > -- > > Saludos / Regards / Cumprimentos, > António silva > > > _________________________________________________________________________ > Professional FreeSWITCH Consulting > Services: > consulting@freeswitch.org > <mailto:consulting@freeswitch.org> > http://www.freeswitchsolutions.com > <http://www.freeswitchsolutions.com> > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://confluence.freeswitch.org > <http://confluence.freeswitch.org> > http://www.cluecon.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org > <mailto:FreeSWITCH-users@lists.freeswitch.org> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users > <http://lists.freeswitch.org/mailman/options/freeswitch-users> > http://www.freeswitch.org > > > > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org > <mailto:consulting@freeswitch.org> > http://www.freeswitchsolutions.com > <http://www.freeswitchsolutions.com> > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://confluence.freeswitch.org > <http://confluence.freeswitch.org> > http://www.cluecon.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org > <mailto:FreeSWITCH-users@lists.freeswitch.org> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users > <http://lists.freeswitch.org/mailman/options/freeswitch-users> > http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com>

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com> Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com>

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com> Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

--

*/Brian West/* brian@freeswitch.org <mailto:brian@freeswitch.org>

*/Twitter: @FreeSWITCH , @briankwest/*

http://www.freeswitchbook.com http://www.freeswitchcookbook.com <http://www.freeswitchcookbook.com> Allison prompts for FreeSWITCH:

*https://www.gofundme.com/allison-prompts-for-freeswitch* <https://www.gofundme.com/allison-prompts-for-freeswitch>

Wish to schedule a meeting?

http://app.timebridge.com/#/meet/freeswitch <http://app.timebridge.com/#/meet/freeswitch>

Got Bugs? Report them here <https://freeswitch.org/jira>! | Reddit: /r/freeswitch <https://www.reddit.com/r/freeswitch>

*T:*+19184209001 <tel:%28918%29%20420-9001> | *F:*+19184209002 <tel:%28918%29%20420-9002> | *M:*+1918424WEST (9378) *Skype:*briankwest

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com>

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com> Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

--

*/Brian West/* brian@freeswitch.org <mailto:brian@freeswitch.org>

*/Twitter: @FreeSWITCH , @briankwest/*

http://www.freeswitchbook.com http://www.freeswitchcookbook.com <http://www.freeswitchcookbook.com> Allison prompts for FreeSWITCH:

*https://www.gofundme.com/allison-prompts-for-freeswitch* <https://www.gofundme.com/allison-prompts-for-freeswitch>

Wish to schedule a meeting?

http://app.timebridge.com/#/meet/freeswitch <http://app.timebridge.com/#/meet/freeswitch>

Got Bugs? Report them here <https://freeswitch.org/jira>! | Reddit: /r/freeswitch <https://www.reddit.com/r/freeswitch>

*T:*+19184209001 <tel:%28918%29%20420-9001> | *F:*+19184209002 <tel:%28918%29%20420-9002> | *M:*+1918424WEST (9378) *Skype:*briankwest

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com>

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com> Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com>

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com> Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org --

Saludos / Regards / Cumprimentos, António silva

Yes, to clarify the sync_clock was a different issue it appear in this tread to discard that could be the cause behind from the main problem on cpu peaks, and it doesn't. My initial problem off cpu peaks, the main reason behind looks like that malloc calls are locking the cpu after a while of running fs, or something is triggering block in fs. I've putt all the info in jira, I'm still searching and trying to reproduce the issue in a local machine. using tcmalloc implementation solves the issue, and so far as been quite mornings. :) Regards, António On 03/02/2017 10:07 PM, Steven Ayre wrote:

Should add that we don't see the cpu behaviour you're seeing. We only sync clock because the monotonic clock drifts away from the NTP one on some of our servers by up to 300ms. Usually it's only an issue if FreeSWITCH starts on boot before NTP has synced.

tcmalloc is presumably solving your cpu issue and sync clock working around your separate clock sync issue. If you're running sync clock frequently enough it's probably safe. enable-use-system-time is essentially the extreme end of that. The reason to normally prefer sync_clock_when_idle is so as not to affect CDRs of current calls by creating big jumps in timestamp/duration, but if you're never letting them drift far apart that shouldn't become an issue.

On 1 March 2017 at 13:57, Antonio Silva <asilva@wirelessmundi.com <mailto:asilva@wirelessmundi.com>> wrote:

thsk steven, i've to do some test first as well.

As for the main problem, i put back the sync clock every night and do far no peaks on cpu sys. So the solution for me was to put fs compiled with tcmalloc.

Regards, António

On 03/01/2017 01:02 PM, Steven Ayre wrote:

...

We're planning on rolling it out but haven't yet done so. Need to schedule some testing on it first.

I suspect the risk he refers to is if you're manually changing the system clock you can cause large clock jumps. That can affect CDR timestamps, either billing for longer periods than you should or getting negative call durations.

If you're using the NTP daemon to get constant adjustments, it's configured correctly, it starts on boot, is trustworthy, refuses to make large clock changes and you're monitoring it properly you'll probably be fine. I'd suggest using UTC for the system clock too, to guarantee you're not seeing any DST jumps.

Steve

On 24 February 2017 at 18:02, Antonio Silva <asilva@wirelessmundi.com <mailto:asilva@wirelessmundi.com>> wrote:

Hi Steven,

Thanks for the tips.

i don't use sync_clock_when_idle because it waits until there is 0 calls, if you never hit 0 calls then the clock is never sync, i wanted to be sure that is done.

didn't know about "enable-use-system-time", just see the commit and there is a note from Anthony: "add enable-use-system-time param to switch.conf.xml *use at your own risk*", are you using it? does it affect performance?

I've to tried it.

About cpu sys to high and getting block, using tcmalloc seams to solve it, today everything went ok, no peaks in cpu sys. Next week i'll put back the sync_clock.

Regards,

On 02/24/2017 02:31 PM, Steven Ayre wrote:

...

NTP adjusts the system clock while FS initialises a monotonic clock on start. FS doesn't see adjustments for drift by NTP.

sync_clock_when_idle is better than sync_clock.

You can also use the enable-use-system-time option which'll get it to use the system (and therefore NTP) time.

On 23 February 2017 at 23:14, Antonio Silva <asilva@wirelessmundi.com <mailto:asilva@wirelessmundi.com>> wrote:

i had a issue with cdr timestamp and the system date, after uptime of 140 days fs clock was behind 1min. It has something between system clock and ntp, so to force fs to keep up i force the sync_clock.

On 02/24/2017 12:06 AM, Brian West wrote:

...

First why did you ever call sync_clock?

On Thu, Feb 23, 2017 at 4:51 PM, Antonio Silva <asilva@wirelessmundi.com <mailto:asilva@wirelessmundi.com>> wrote:

Hi Brian,

Not for today, the first time it happens was my first though something with the traffic, but i couldn't find any association with the type of traffic. Actually after restart fs i receive traffic from the same endpoints, same type of calls and the issue don't reproduce until next morning...

There was something i was doing at 00 every day, the command "fsctl sync_clock"... well i just disabled it today. I've to wait until tomorrow to check the results.

I just install fs compiled with tcmalloc, so if there is no problems i will put back the "sync_clock " command to really check if it could be the cause.

Right now i'm quite lost on what to do more...

On 02/23/2017 11:37 PM, Brian West wrote:

...

Do you have pcaps of the traffic that may trigger this behavior?

/b

On Thu, Feb 23, 2017 at 2:58 PM, Antonio Silva <asilva@wirelessmundi.com <mailto:asilva@wirelessmundi.com>> wrote:

Thanks Juan, that's it, i was missing the dev pkg.

On 02/23/2017 08:34 PM, Juan Pablo Godoy Macari wrote:

...

Antonio, you should install this:

apt-get install libgoogle-perftools-dev

Regards,

2017-02-23 16:26 GMT-03:00 Juan Pablo Godoy Macari <jpgodoy@redvoiss.net <mailto:jpgodoy@redvoiss.net>>:

Try this: ./configure LIBS="-ltcmalloc"

2017-02-23 16:19 GMT-03:00 Antonio Silva <asilva@wirelessmundi.com <mailto:asilva@wirelessmundi.com>>:

Hi,

Did you compile fs in debian with tcmalloc?

didn't work for me... do i miss something?

root@dev:/usr/src/freeswitch.git# <mailto:root@dev:/usr/src/freeswitch.git#> apt-get install libtcmalloc-minimal4 root@dev:/usr/src/freeswitch.git# <mailto:root@dev:/usr/src/freeswitch.git#> export LIBS=-ltcmalloc root@dev:/usr/src/freeswitch.git# <mailto:root@dev:/usr/src/freeswitch.git#> ./configure checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /bin/mkdir -p checking for gawk... no checking for mawk... mawk checking whether make sets $(MAKE)... yes checking whether make supports nested variables... yes checking whether make supports nested variables... (cached) yes checking build system type... x86_64-unknown-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking for style of include used by make... GNU checking for gcc... gcc checking whether the C compiler works... no configure: error: in `/usr/src/freeswitch.git': configure: error: C compiler cannot create executables

On 02/23/2017 07:56 PM, Antonio Silva wrote: > Hi Juan, > > Thanks for the tips!! > > Unfortunately i cannot test with > -nosql option, i need to have the > sip_registration fill to be able to > send options to natted endpoints. > But i don't think the issue is in > db, i use postgres and to get better > performance from it i run it on tmpfs. > > > I read about tcmalloc, but from jira > FS-7315, i get the idea that it > didn't change anything.. but i guess > i can try it and post the results, > from the bt i do see some calls been > done to malloc, but not sure if they > are locking or not. > > Regards, > > > On 02/23/2017 06:52 PM, Juan Pablo > Godoy Macari wrote: >> Hello Antonio, >> >> I had the same issue, to fix it i >> tried many things written in this >> mail list. Try running FS with >> -nosql option, if that doesn't work >> recompile FS with tcmalloc (from >> google's perfomance tools). >> >> That did it for me. >> >> Regards, >> Juan Pablo. >> >> 2017-02-23 13:26 GMT-03:00 Antonio >> Silva <asilva@wirelessmundi.com >> <mailto:asilva@wirelessmundi.com>>: >> >> Hi, >> >> i'm having an issue with FS >> that gets lock every morning >> with cpu sys >> hitting 80%, because is limited >> in fs with min-idle, it could >> reach the >> 100%, so the calls are rejected >> by load, to solve the problem i >> kill fs >> and start it again. >> >> I discard a load issue, because >> after restarting freeswitch the >> system >> increases in number of >> sessions/cps and no problems at >> all until next >> day.... it's given me crazy... >> i've enough memory, lots of >> cores... >> after restart fs the system >> stays at less than 30% of cpu >> usage. >> >> >> Right now i have hpet as >> clocksource, transcoding is >> minimal (use >> inheric_codec, only happens >> because of different ptimes >> with some >> endpoints) . >> >> >> anyone having similar problem? >> or any tip i can do to debug >> this issue? >> >> >> For info the Jira ticket can be >> located at >> https://freeswitch.org/jira/browse/FS-9957 >> <https://freeswitch.org/jira/browse/FS-9957>, >> >> i've done a gcore, top, htop >> when the system is with high >> cpu sys but i >> can't figure the thread that is >> causing the issue... initial i >> thought >> it was because of module lua >> (most of my logic is done >> there), but today >> only 2 threads where running >> mod_lua when it happen. >> >> >> Appreciate the help. >> >> >> -- >> >> Saludos / Regards / Cumprimentos, >> António silva >> >> >> _________________________________________________________________________ >> Professional FreeSWITCH >> Consulting Services: >> consulting@freeswitch.org >> <mailto:consulting@freeswitch.org> >> http://www.freeswitchsolutions.com >> <http://www.freeswitchsolutions.com> >> >> Official FreeSWITCH Sites >> http://www.freeswitch.org >> http://confluence.freeswitch.org >> <http://confluence.freeswitch.org> >> http://www.cluecon.com >> >> FreeSWITCH-users mailing list >> FreeSWITCH-users@lists.freeswitch.org >> <mailto:FreeSWITCH-users@lists.freeswitch.org> >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >> <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> >> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users >> <http://lists.freeswitch.org/mailman/options/freeswitch-users> >> http://www.freeswitch.org >> >> >> >> >> _________________________________________________________________________ >> Professional FreeSWITCH Consulting Services: >> consulting@freeswitch.org >> <mailto:consulting@freeswitch.org> >> http://www.freeswitchsolutions.com >> <http://www.freeswitchsolutions.com> >> >> Official FreeSWITCH Sites >> http://www.freeswitch.org >> http://confluence.freeswitch.org >> <http://confluence.freeswitch.org> >> http://www.cluecon.com >> >> FreeSWITCH-users mailing list >> FreeSWITCH-users@lists.freeswitch.org >> <mailto:FreeSWITCH-users@lists.freeswitch.org> >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >> <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> >> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users >> <http://lists.freeswitch.org/mailman/options/freeswitch-users> >> http://www.freeswitch.org > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org > <mailto:consulting@freeswitch.org> > http://www.freeswitchsolutions.com > <http://www.freeswitchsolutions.com> > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://confluence.freeswitch.org > <http://confluence.freeswitch.org> > http://www.cluecon.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org > <mailto:FreeSWITCH-users@lists.freeswitch.org> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users > <http://lists.freeswitch.org/mailman/options/freeswitch-users> > http://www.freeswitch.org _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com> Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com>

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com> Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

--

*/Brian West/* brian@freeswitch.org <mailto:brian@freeswitch.org>

*/Twitter: @FreeSWITCH , @briankwest/*

http://www.freeswitchbook.com http://www.freeswitchcookbook.com <http://www.freeswitchcookbook.com> Allison prompts for FreeSWITCH:

*https://www.gofundme.com/allison-prompts-for-freeswitch* <https://www.gofundme.com/allison-prompts-for-freeswitch>

Wish to schedule a meeting?

http://app.timebridge.com/#/meet/freeswitch <http://app.timebridge.com/#/meet/freeswitch>

Got Bugs? Report them here <https://freeswitch.org/jira>! | Reddit: /r/freeswitch <https://www.reddit.com/r/freeswitch>

*T:*+19184209001 <tel:%28918%29%20420-9001> | *F:*+19184209002 <tel:%28918%29%20420-9002> | *M:*+1918424WEST (9378) *Skype:*briankwest

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com>

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com> Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

--

*/Brian West/* brian@freeswitch.org <mailto:brian@freeswitch.org>

*/Twitter: @FreeSWITCH , @briankwest/*

http://www.freeswitchbook.com http://www.freeswitchcookbook.com <http://www.freeswitchcookbook.com> Allison prompts for FreeSWITCH:

*https://www.gofundme.com/allison-prompts-for-freeswitch* <https://www.gofundme.com/allison-prompts-for-freeswitch>

Wish to schedule a meeting?

http://app.timebridge.com/#/meet/freeswitch <http://app.timebridge.com/#/meet/freeswitch>

Got Bugs? Report them here <https://freeswitch.org/jira>! | Reddit: /r/freeswitch <https://www.reddit.com/r/freeswitch>

*T:*+19184209001 <tel:%28918%29%20420-9001> | *F:*+19184209002 <tel:%28918%29%20420-9002> | *M:*+1918424WEST (9378) *Skype:*briankwest

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com>

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com> Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com>

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com> Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com>

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

--

Saludos / Regards / Cumprimentos, António silva

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com> Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org <http://confluence.freeswitch.org> http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org --

Saludos / Regards / Cumprimentos, António silva