[Freeswitch-users] Freeswitch 1.4.15 and a sudden jump in CPU usage

Thu Feb 12 13:31:52 MSK 2015

Hi Ali,

2 years ago here, in mail-list, was a discussion
<http://lists.freeswitch.org/pipermail/freeswitch-users/2013-March/093257.html>.
(memleak can be omitted)
Not sure, but It seems the symptoms was similar.

Best Wishes,
 Dmitry.

2015-02-12 2:53 GMT+02:00 Ali Pey <alipey at gmail.com>:

> Hi Dmitry,
>
> Can you elaborate a bit please? Are you saying that it won't happen with
> -np? Or it's just a suggestion?
>
> What's the relationship?
>
> Thanks,
> Ali Pey
>
>
> On Wed, Feb 11, 2015 at 6:30 PM, Dmitry Lysenko <dvl36.ripe.nick at gmail.com
> > wrote:
>
>> Hello guys,
>>
>> Try to reproduce this bug with normal priority ("-np") command line
>> switch.
>>
>> Best Wishes,
>>  Dmitry.
>>
>> 2015-02-11 22:54 GMT+02:00 Ali Pey <alipey at gmail.com>:
>>
>>> Thanks Tristan.
>>>
>>> I have had rtp-timer-name set to none for a long time but I still see
>>> the sudden and random jumps in CPU.
>>>
>>> I haven't been able to reproduce it in our Dev/QA environment as well.
>>> If I can reproduce it, I will definitely do the logs as well.
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Feb 11, 2015 at 3:31 PM, Tristan Mahé <t.mahe at b-and-c.net>
>>> wrote:
>>>
>>>>  Hi Ali,
>>>>
>>>> What I see is kinda the same thing, I still need to provide Anthony
>>>> with the debug logs ( still had no chance to do the requested tests on the
>>>> production, could be related to some of our users UA/callflow and cannot
>>>> reproduce the issue in a lab ). The interesting part as discussed in
>>>> FS-7066 is the timer change is correcting the trouble, and without the
>>>> timer change, we eventually experience blocking of sofia profile ( seems
>>>> related to media playback ).
>>>>
>>>> I hope to have a chance soon to be able to furnish the debug logs, if
>>>> you can, do the same, Anthony gave the steps to provide him some usefull
>>>> logs on the bug comments.
>>>>
>>>>
>>>> Le 11/02/2015 10:58, Ali Pey a écrit :
>>>>
>>>> Hi Tristan,
>>>>
>>>>  Do you see a sudden jump or just more CPU usage?
>>>>
>>>>  In my case, everything is good for a few days with less than %10 CPU
>>>> usage, then there is a sudden jump to ~%98 CPU usage and it doesn't come
>>>> down until a restart. After restart, it's good again with the same load.
>>>>
>>>>  Thanks,
>>>> Ali Pey
>>>>
>>>>
>>>> On Wed, Feb 11, 2015 at 1:40 PM, Tristan Mahé <t.mahe at b-and-c.net>
>>>> wrote:
>>>>
>>>>> Maybe related to https://freeswitch.org/jira/browse/FS-7066 ?
>>>>>
>>>>> Le 11/02/2015 08:00, Giovanni Maruzzelli a écrit :
>>>>>  > somebody has tested if it's a regression, eg: is 1.4.14 having
>>>>> > same symptoms?
>>>>> >
>>>>> > -giovanni
>>>>> >
>>>>> > On Wed, Feb 11, 2015 at 4:57 PM, Moishe Grunstein <
>>>>> max at nysolutions.com> wrote:
>>>>> >> Why not a Jira?
>>>>> >>
>>>>> >> Thanks,
>>>>> >>
>>>>> >> Moishe Grunstein
>>>>> >> Tornado Computer Systems, Inc.
>>>>> >> 212.400.7650 888.IPPBX.US
>>>>> >> Service Request Email: support at nysolutions.com
>>>>> >>
>>>>> >> Computer Networking * Managed Services * IP Video Surveillance *
>>>>> Network Assessments * Web Solutions * Voice over IP * Disaster Recovery *
>>>>> Network Security * Site Surveys * CMS
>>>>> >>
>>>>> >> -----Original Message-----
>>>>> >> From: freeswitch-users-bounces at lists.freeswitch.org [mailto:
>>>>> freeswitch-users-bounces at lists.freeswitch.org] On Behalf Of ??
>>>>> >> Sent: Wednesday, February 11, 2015 10:49 AM
>>>>> >> To: FreeSWITCH Users Help
>>>>> >> Subject: Re: [Freeswitch-users] Freeswitch 1.4.15 and a sudden jump
>>>>> in CPU usage
>>>>> >>
>>>>> >> i think it's bug of freeswitch:
>>>>> >>
>>>>> >> -- the Freeswitch using more than 1000% CPU, and there is no other
>>>>> process eating CPU
>>>>> >> -- we move all SIP traffic from the server away, the cpu is low.
>>>>> and after sevral hours, if we move the SIP traffic back again, the FS
>>>>> eating all CPU again. the only way to recovery is to restart the FS.
>>>>> >> -- there is no DB such as mysql on the server, and we have add
>>>>> "nosql"
>>>>> >> option, no help
>>>>> >>
>>>>> >> On Wed, Feb 11, 2015 at 10:03 PM, John Nash <john.nash778 at gmail.com>
>>>>> wrote:
>>>>> >>> Dear Muhammad,
>>>>> >>>
>>>>> >>> I think your issue is different than Ali Pey because CPU rises at a
>>>>> >>> specific time (approx) while running fine most of the time. In my
>>>>> >>> experience you should check following ..
>>>>> >>> - Check using top commands if its indeed freeswitch taking up CPU.
>>>>> It
>>>>> >>> may be logrotate or something.
>>>>> >>> - If you are using DB on same server, also check DB connections,
>>>>> May
>>>>> >>> be some long running query causing it (Like someone fires some
>>>>> billing
>>>>> >>> report at approx same time)
>>>>> >>>
>>>>> >>> On the other hand when you say "it runs fine all day". You may
>>>>> want to
>>>>> >>> check that using top command at different time of day, may be CPU
>>>>> >>> usage keep on increasing and in your peak time it increases so
>>>>> much that you see symptoms.
>>>>> >>> (If this is true may be you and Ali pey have same issue perhaps
>>>>> some
>>>>> >>> CPU leak).
>>>>> >>>
>>>>> >>> P.S. All points I mentioned are from my VOIP experiences in
>>>>> general. I
>>>>> >>> am NOT freeswitch expert. I have recently started experiments with
>>>>> >>> freeswitch so people who are using freeswitch or experts will give
>>>>> >>> more meaningful insight.
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>> On Wed, Feb 11, 2015 at 7:16 PM, 彭勇 <ppyy at pubyun.com> wrote:
>>>>> >>>> we have add this option to grub and reboot one months ago. no
>>>>> help.
>>>>> >>>>
>>>>> >>>> we have media-pass-through  option, and have more than 10
>>>>> thousands
>>>>> >>>> sessions. it works fine all day and the CPU will rise suddenly
>>>>> about
>>>>> >>>> 9:00 am every day.  i should restart the FS to recovery.
>>>>> >>>>
>>>>> >>>> On Wed, Feb 11, 2015 at 9:25 PM, Muhammad Naseer Bhatti
>>>>> >>>> <nbhatti at gmail.com> wrote:
>>>>> >>>>> Try nohz=off reboot and see if that helps.
>>>>> >>>>>
>>>>> >>>>> Sent from my iPad
>>>>> >>>>>
>>>>> >>>>> On Feb 11, 2015, at 16:18, Aqs Younas <aqsyounas at gmail.com>
>>>>> wrote:
>>>>> >>>>>
>>>>> >>>>> I am facing same issue. Freeswitch taking more than 100% cpu
>>>>> usage
>>>>> >>>>> in htop even on 200 concurrent calls. For 1000 concurrent calls i
>>>>> >>>>> have to install 3 freeswitch instances due to this much cpu usage
>>>>> >>>>> taking by a single freeswitch node. My Dist is Debian 7.
>>>>> >>>>>
>>>>> >>>>> On 11 February 2015 at 17:35, Muhammad Naseer Bhatti
>>>>> >>>>> <nbhatti at gmail.com>
>>>>> >>>>> wrote:
>>>>> >>>>>>
>>>>> >>>>>> It looks like this is the same tickless kernel issue. I am using
>>>>> >>>>>> Debian
>>>>> >>>>>> 7
>>>>> >>>>>> and the CPU usage shows very high. Adding nohz=off to grub
>>>>> options
>>>>> >>>>>> does makes it lower. I am gathering more information in order to
>>>>> >>>>>> file a Jira.
>>>>> >>>>>> Debian 6 on the other hand works very quietly on the same
>>>>> >>>>>> workload. We are talking about 200+% CPU utilization on Debian 7
>>>>> >>>>>> and around 50% on Debian 6 showing in htop. I know so many other
>>>>> >>>>>> factors also involved but I am still digging out more.
>>>>> >>>>>>
>>>>> >>>>>> —
>>>>> >>>>>> Thanks,
>>>>> >>>>>> Muhammad Naseer Bhatti
>>>>> >>>>>>
>>>>> >>>>>> From: 彭勇 <ppyy at pubyun.com>
>>>>> >>>>>> Reply: FreeSWITCH Users Help
>>>>> >>>>>> <freeswitch-users at lists.freeswitch.org>>
>>>>> >>>>>> Date: February 11, 2015 at 3:20:46 PM
>>>>> >>>>>> To: FreeSWITCH Users Help <
>>>>> freeswitch-users at lists.freeswitch.org>>
>>>>> >>>>>> Subject:  Re: [Freeswitch-users] Freeswitch 1.4.15 and a sudden
>>>>> >>>>>> jump in CPU usage
>>>>> >>>>>>
>>>>> >>>>>> same issue.
>>>>> >>>>>>
>>>>> >>>>>> I have several FS 1.4.15 running on CentOS 6.5 x86_64, the CPU
>>>>> >>>>>> will be very high about 9:00 am every morning. there is no
>>>>> crontab
>>>>> >>>>>> at this time.
>>>>> >>>>>>
>>>>> >>>>>> i will downgrade to 1.4.14 to see if there is such issue.
>>>>> >>>>>>
>>>>> >>>>>> On Wed, Feb 11, 2015 at 1:14 AM, Ali Pey <alipey at gmail.com>
>>>>> wrote:
>>>>> >>>>>>> Hi Anthony,
>>>>> >>>>>>>
>>>>> >>>>>>> I do not know if I am hitting a bug and if this is a bug, I
>>>>> >>>>>>> don't have much info. I'm using this mailing list to see if
>>>>> >>>>>>> anyone else is experiencing this problem and if there is a
>>>>> >>>>>>> configuration change on Linux or Freeswitch that can fix this
>>>>> >>>>>>> problem. Jira would be to report bugs when you know there is a
>>>>> >>>>>>> bug. My problem could be (very likely) an environment related
>>>>> issue.
>>>>> >>>>>>>
>>>>> >>>>>>> Thanks.
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>> On Tue, Feb 10, 2015 at 11:55 AM, Anthony Minessale
>>>>> >>>>>>> <anthony.minessale at gmail.com> wrote:
>>>>> >>>>>>>> Also you will have better luck using JIRA. The mailing list is
>>>>> >>>>>>>> expressly NOT for reporting issues.
>>>>> >>>>>>>>
>>>>> >>>>>>>>
>>>>> >>>>>>>> On Tue, Feb 10, 2015 at 10:51 AM, Michael Jerris
>>>>> >>>>>>>> <mike at jerris.com>
>>>>> >>>>>>>> wrote:
>>>>> >>>>>>>>> you might want to use gcore to look at the threads and see
>>>>> >>>>>>>>> what is different when it in this state and not.
>>>>> >>>>>>>>>
>>>>> >>>>>>>>>> On Feb 10, 2015, at 11:28 AM, Ali Pey <alipey at gmail.com>
>>>>> wrote:
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> Hello,
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> Normally our CPU usage during peak hours are less than %10;
>>>>> >>>>>>>>>> however, time to time, freeswitch uses more than %95 of the
>>>>> >>>>>>>>>> CPU and it stays there forever. The only way to get it back
>>>>> >>>>>>>>>> to normal is to restart freeswitch service. During the high
>>>>> >>>>>>>>>> CPU usage, freeswitch continues to operate but very very
>>>>> >>>>>>>>>> slow.
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> If I lower the traffic quite a bit, the CPU comes down, but
>>>>> >>>>>>>>>> adding back the traffic, the CPU usage goes right back up.
>>>>> >>>>>>>>>> After freeswitch restart, it becomes normal again with the
>>>>> >>>>>>>>>> same traffic.
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> Any ideas or suggestions?
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> Much appreciated.
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> Thanks,
>>>>> >>>>>>>>>> Ali Pey
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>
>>>>> >>>>>>>>>
>>>>> >>>>>>>>>
>>>>> >>>>>>>>>
>>>>> ______________________________________________________________
>>>>> >>>>>>>>> ___________ Professional FreeSWITCH Consulting Services:
>>>>> >>>>>>>>> consulting at freeswitch.org
>>>>> >>>>>>>>> http://www.freeswitchsolutions.com
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> Official FreeSWITCH Sites
>>>>> >>>>>>>>> http://www.freeswitch.org
>>>>> >>>>>>>>> http://confluence.freeswitch.org http://www.cluecon.com
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> FreeSWITCH-users mailing list
>>>>> >>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> >>>>>>>>>
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> >>>>>>>>>
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freesw
>>>>> >>>>>>>>> itch-users
>>>>> >>>>>>>>> http://www.freeswitch.org
>>>>> >>>>>>>>
>>>>> >>>>>>>>
>>>>> >>>>>>>>
>>>>> >>>>>>>> --
>>>>> >>>>>>>> Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
>>>>> >>>>>>>>
>>>>> >>>>>>>> ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞
>>>>> >>>>>>>> http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch
>>>>> ☞
>>>>> >>>>>>>> http://freeswitch.org/g+
>>>>> >>>>>>>>
>>>>> >>>>>>>> ClueCon Weekly Development Call ☎
>>>>> >>>>>>>> sip:888 at conference.freeswitch.org ☎ +19193869900
>>>>> >>>>>>>>
>>>>> >>>>>>>>
>>>>> >>>>>>>>
>>>>> >>>>>>>>
>>>>> >>>>>>>>
>>>>> _______________________________________________________________
>>>>> >>>>>>>> __________ Professional FreeSWITCH Consulting Services:
>>>>> >>>>>>>> consulting at freeswitch.org
>>>>> >>>>>>>> http://www.freeswitchsolutions.com
>>>>> >>>>>>>>
>>>>> >>>>>>>> Official FreeSWITCH Sites
>>>>> >>>>>>>> http://www.freeswitch.org
>>>>> >>>>>>>> http://confluence.freeswitch.org http://www.cluecon.com
>>>>> >>>>>>>>
>>>>> >>>>>>>> FreeSWITCH-users mailing list
>>>>> >>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> >>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> >>>>>>>>
>>>>> >>>>>>>>
>>>>> >>>>>>>> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswi
>>>>> >>>>>>>> tch-users
>>>>> >>>>>>>> http://www.freeswitch.org
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> ________________________________________________________________
>>>>> >>>>>>> _________ Professional FreeSWITCH Consulting Services:
>>>>> >>>>>>> consulting at freeswitch.org
>>>>> >>>>>>> http://www.freeswitchsolutions.com
>>>>> >>>>>>>
>>>>> >>>>>>> Official FreeSWITCH Sites
>>>>> >>>>>>> http://www.freeswitch.org
>>>>> >>>>>>> http://confluence.freeswitch.org http://www.cluecon.com
>>>>> >>>>>>>
>>>>> >>>>>>> FreeSWITCH-users mailing list
>>>>> >>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> >>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> >>>>>>>
>>>>> >>>>>>> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswit
>>>>> >>>>>>> ch-users
>>>>> >>>>>>> http://www.freeswitch.org
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>> --
>>>>> >>>>>> Peng Yong
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> __________________________________________________________________
>>>>> >>>>>> _______ Professional FreeSWITCH Consulting Services:
>>>>> >>>>>> consulting at freeswitch.org
>>>>> >>>>>> http://www.freeswitchsolutions.com
>>>>> >>>>>>
>>>>> >>>>>> Official FreeSWITCH Sites
>>>>> >>>>>> http://www.freeswitch.org
>>>>> >>>>>> http://confluence.freeswitch.org
>>>>> >>>>>> http://www.cluecon.com
>>>>> >>>>>>
>>>>> >>>>>> FreeSWITCH-users mailing list
>>>>> >>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> >>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> >>>>>>
>>>>> >>>>>> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch
>>>>> >>>>>> -users
>>>>> >>>>>> http://www.freeswitch.org
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> __________________________________________________________________
>>>>> >>>>>> _______ Professional FreeSWITCH Consulting Services:
>>>>> >>>>>> consulting at freeswitch.org
>>>>> >>>>>> http://www.freeswitchsolutions.com
>>>>> >>>>>>
>>>>> >>>>>> Official FreeSWITCH Sites
>>>>> >>>>>> http://www.freeswitch.org
>>>>> >>>>>> http://confluence.freeswitch.org
>>>>> >>>>>> http://www.cluecon.com
>>>>> >>>>>>
>>>>> >>>>>> FreeSWITCH-users mailing list
>>>>> >>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> >>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> >>>>>>
>>>>> >>>>>> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch
>>>>> >>>>>> -users
>>>>> >>>>>> http://www.freeswitch.org
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>>
>>>>> ___________________________________________________________________
>>>>> >>>>> ______ Professional FreeSWITCH Consulting Services:
>>>>> >>>>> consulting at freeswitch.org
>>>>> >>>>> http://www.freeswitchsolutions.com
>>>>> >>>>>
>>>>> >>>>> Official FreeSWITCH Sites
>>>>> >>>>> http://www.freeswitch.org
>>>>> >>>>> http://confluence.freeswitch.org
>>>>> >>>>> http://www.cluecon.com
>>>>> >>>>>
>>>>> >>>>> FreeSWITCH-users mailing list
>>>>> >>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> >>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> >>>>> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-
>>>>> >>>>> users
>>>>> >>>>> http://www.freeswitch.org
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>>
>>>>> ___________________________________________________________________
>>>>> >>>>> ______ Professional FreeSWITCH Consulting Services:
>>>>> >>>>> consulting at freeswitch.org
>>>>> >>>>> http://www.freeswitchsolutions.com
>>>>> >>>>>
>>>>> >>>>> Official FreeSWITCH Sites
>>>>> >>>>> http://www.freeswitch.org
>>>>> >>>>> http://confluence.freeswitch.org
>>>>> >>>>> http://www.cluecon.com
>>>>> >>>>>
>>>>> >>>>> FreeSWITCH-users mailing list
>>>>> >>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> >>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> >>>>> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-
>>>>> >>>>> users
>>>>> >>>>> http://www.freeswitch.org
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> --
>>>>> >>>> Peng Yong
>>>>> >>>>
>>>>> >>>>
>>>>> _____________________________________________________________________
>>>>> >>>> ____ Professional FreeSWITCH Consulting Services:
>>>>> >>>> consulting at freeswitch.org
>>>>> >>>> http://www.freeswitchsolutions.com
>>>>> >>>>
>>>>> >>>> Official FreeSWITCH Sites
>>>>> >>>> http://www.freeswitch.org
>>>>> >>>> http://confluence.freeswitch.org
>>>>> >>>> http://www.cluecon.com
>>>>> >>>>
>>>>> >>>> FreeSWITCH-users mailing list
>>>>> >>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> >>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> >>>> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-us
>>>>> >>>> ers
>>>>> >>>> http://www.freeswitch.org
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> ______________________________________________________________________
>>>>> >>> ___ Professional FreeSWITCH Consulting Services:
>>>>> >>> consulting at freeswitch.org
>>>>> >>> http://www.freeswitchsolutions.com
>>>>> >>>
>>>>> >>> Official FreeSWITCH Sites
>>>>> >>> http://www.freeswitch.org
>>>>> >>> http://confluence.freeswitch.org
>>>>> >>> http://www.cluecon.com
>>>>> >>>
>>>>> >>> FreeSWITCH-users mailing list
>>>>> >>> FreeSWITCH-users at lists.freeswitch.org
>>>>> >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> >>> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-use
>>>>> >>> rs
>>>>> >>> http://www.freeswitch.org
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> Peng Yong
>>>>> >>
>>>>> >>
>>>>> _________________________________________________________________________
>>>>> >> Professional FreeSWITCH Consulting Services:
>>>>> >> consulting at freeswitch.org
>>>>> >> http://www.freeswitchsolutions.com
>>>>> >>
>>>>> >> Official FreeSWITCH Sites
>>>>> >> http://www.freeswitch.org
>>>>> >> http://confluence.freeswitch.org
>>>>> >> http://www.cluecon.com
>>>>> >>
>>>>> >> FreeSWITCH-users mailing list
>>>>> >> FreeSWITCH-users at lists.freeswitch.org
>>>>> >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> >> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> >> http://www.freeswitch.org
>>>>> >>
>>>>> _________________________________________________________________________
>>>>> >> Professional FreeSWITCH Consulting Services:
>>>>> >> consulting at freeswitch.org
>>>>> >> http://www.freeswitchsolutions.com
>>>>> >>
>>>>> >> Official FreeSWITCH Sites
>>>>> >> http://www.freeswitch.org
>>>>> >> http://confluence.freeswitch.org
>>>>> >> http://www.cluecon.com
>>>>> >>
>>>>> >> FreeSWITCH-users mailing list
>>>>> >> FreeSWITCH-users at lists.freeswitch.org
>>>>> >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> >> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> >> http://www.freeswitch.org
>>>>> >
>>>>> >
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services:
>>>>> consulting at freeswitch.org
>>>>> http://www.freeswitchsolutions.com
>>>>>
>>>>> Official FreeSWITCH Sites
>>>>> http://www.freeswitch.org
>>>>> http://confluence.freeswitch.org
>>>>> http://www.cluecon.com
>>>>>
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> http://www.freeswitch.org
>>>>>
>>>>
>>>>
>>>>
>>>> _________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services: consulting at freeswitch.orghttp://www.freeswitchsolutions.com
>>>>
>>>> Official FreeSWITCH Siteshttp://www.freeswitch.orghttp://confluence.freeswitch.orghttp://www.cluecon.com
>>>>
>>>> FreeSWITCH-users mailing listFreeSWITCH-users at lists.freeswitch.orghttp://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-usershttp://www.freeswitch.org
>>>>
>>>>
>>>>
>>>>
>>>> _________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services:
>>>> consulting at freeswitch.org
>>>> http://www.freeswitchsolutions.com
>>>>
>>>> Official FreeSWITCH Sites
>>>> http://www.freeswitch.org
>>>> http://confluence.freeswitch.org
>>>> http://www.cluecon.com
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:
>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>>>
>>>
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://confluence.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://confluence.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20150212/482984dc/attachment-0001.html 