[Freeswitch-users] Thread failure with low CPU utilization?

Luis Daniel Lucio Quiroz luis.daniel.lucio at gmail.com
Thu Dec 17 20:24:30 MSK 2015 

Just to give more info about this,

I had an issue with simultaneous calls and db connection. Seems that adding
entries in limit was the bootle neck.
ulimit command was used as well.

Luis Daniel Lucio Quiroz
CISSP, CISM, CISA
Linux, VoIP and much more fun
www.okay.com.mx

Need LCR? Check out LCR for FusionPBX with FreeSWITCH
Need Billing? Check out Billing for FusionPBX with FreeSWITCH

2015-12-15 16:38 GMT-05:00 Vladislav Ivanov <deforceczt at gmail.com>:

> I'm using the systemd file from
> https://freeswitch.org/confluence/display/FREESWITCH/FreeSWITCH+1.6+Video
>
> Just calling calls with 10 cps, low duration, should it slowly use the
> cpu? It really slowly eats all CPU, then drops all calls, repeats again...
> Shouldn't it be 10% cpu, nothing more for the same cps, no logs writing.
> Just simple minimal.
>
> 2015-12-15 22:22 GMT+02:00 Anthony Minessale <anthony.minessale at gmail.com>
> :
>
>> More issues related to resource limits.
>> Check your allowed max sessions and sps in your config and any other
>> ulimits on the system that you may have too low.
>>
>> Its wise to have limits low enough that the calls start failing before
>> your box does.
>>
>>
>> On Tue, Dec 15, 2015 at 12:15 PM, Vladislav Ivanov <deforceczt at gmail.com>
>> wrote:
>>
>>> Hey guys,
>>>
>>> I have a new problem that is happening with CPU load now.
>>> I have been testing freeswitch with sipp, and i'm encountering strange
>>> issue with same load.
>>> 1st time I test with 10 cps, 5 second duration calls, for just cps load.
>>> CPU idle is min idle cpu 10.00/84.67
>>> Then I add CPS by 5, it should be enough power to be 15 cps on the
>>> machine when I have idle 84%...
>>> But my CPU goes heavily down, more and more down until it goes to 0 and
>>> spams:
>>> 2015-12-15 20:13:43.199451 [CRIT] mod_sofia.c:4406 Error Creating Session
>>> 2015-12-15 20:13:43.259702 [CRIT] mod_sofia.c:4406 Error Creating Session
>>> 2015-12-15 20:13:43.279578 [CRIT] mod_sofia.c:4406 Error Creating Session
>>> 2015-12-15 20:13:43.299562 [CRIT] mod_sofia.c:4406 Error Creating Session
>>> 2015-12-15 20:13:43.299562 [CRIT] mod_sofia.c:4406 Error Creating Session
>>> 2015-12-15 20:13:43.399496 [CRIT] mod_sofia.c:4406 Error Creating Session
>>> 2015-12-15 20:13:43.419667 [CRIT] mod_sofia.c:4406 Error Creating Session
>>> 2015-12-15 20:14:32.819449 [CRIT] mod_sofia.c:4406 Error Creating Session
>>>
>>> Anything I can do to debug or check why it is causing this?
>>> (Same is happening on Centos 6, Debian)
>>> OS: "Debian GNU/Linux 8 (jessie)
>>> FS: FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit)
>>>
>>> I'm kinda lost at this point...
>>>
>>> 2015-12-15 19:21 GMT+02:00 Anthony Minessale <
>>> anthony.minessale at gmail.com>:
>>>
>>>> 1000 threads is not really a show stopper =D
>>>> That is what the artoo message, it tells you that the new current
>>>> thread limit has been auto detected and that is the new max calls you can
>>>> handle henceforth.
>>>>
>>>>
>>>> On Tue, Dec 15, 2015 at 5:16 AM, Michael Giagnocavo <mgg at giagnocavo.net
>>>> > wrote:
>>>>
>>>>> Perhaps write a patch to detect these limits and then CRIT on start if
>>>>> they conflict with settings.
>>>>>
>>>>>
>>>>>
>>>>> *From:* freeswitch-users-bounces at lists.freeswitch.org [mailto:
>>>>> freeswitch-users-bounces at lists.freeswitch.org] *On Behalf Of *Vladislav
>>>>> Ivanov
>>>>> *Sent:* Tuesday, 15 December, 2015 2:15
>>>>> *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
>>>>> *Subject:* Re: [Freeswitch-users] Thread failure with low CPU
>>>>> utilization?
>>>>>
>>>>>
>>>>>
>>>>> Hey guys,
>>>>>
>>>>> Anthony was correct in this case, I ran freeswitch not from root, for
>>>>> some reason my /etc/security/limits.conf file line:
>>>>>
>>>>> *          soft     nproc          32768
>>>>>
>>>>> *          hard     nproc          32768
>>>>>
>>>>> was not propagated on freeswitch user...
>>>>>
>>>>> Added new line:
>>>>>
>>>>> freeswitch          soft     nproc          32768
>>>>>
>>>>> freeswitch          hard     nproc          32768
>>>>>
>>>>> and it worked fine from that.
>>>>> Thanks to you all!
>>>>>
>>>>>
>>>>>
>>>>> 2015-12-14 23:06 GMT+02:00 Anthony Minessale <
>>>>> anthony.minessale at gmail.com>:
>>>>>
>>>>> Are you running FS as root or a regular user, you need to check the
>>>>> ulimits of concurrent threads and some boxes have a hard limit of threads,
>>>>> its not all tied to CPU usage.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Dec 14, 2015 at 1:30 PM, Kevin Wormington <kworm at sofnet.com>
>>>>> wrote:
>>>>>
>>>>> This post
>>>>> http://unix.stackexchange.com/questions/47595/linux-max-threads-count
>>>>> has a simple program to check out how many threads can be created.  You
>>>>> can run the test program as root and then as FS user and see if it's a
>>>>> ulimit type of problem.
>>>>>
>>>>> On 12/14/2015 01:08 PM, Michael Giagnocavo wrote:
>>>>> > Is there a reason to think that CentOS 6 would cause threads to fail
>>>>> to
>>>>> > spawn? This is basically just calling libapr, which is code from
>>>>> like 8
>>>>> > years ago.
>>>>> >
>>>>> > Vladislav: This means it failed to create a thread
>>>>> (switch_thread_create
>>>>> > in switch_apr.c). FS generates a ton of threads. The only time I’ve
>>>>> seen
>>>>> > this was a long time ago on 32-bit Windows, since default stack of
>>>>> 1MB
>>>>> > means you hit address space limits pretty quickly. But on 64-bit, you
>>>>> > should be able to go all out.
>>>>> >
>>>>> > I’d start by adding some if’d printf’s to check the various things
>>>>> that
>>>>> > switch_thread_create does and try to pinpoint what is failing. If the
>>>>> > libapr call is failing, then you should be able to create a minimal
>>>>> > program that just spawns threads and see if that one is able to get
>>>>> > 1000+. From there I’d guess troubleshooting is gonna be the same as
>>>>> > figuring out why pthread_create fails and that should be generic to
>>>>> your
>>>>> > system. (Please post if you figure out why.)
>>>>> >
>>>>> > If other apps are able to spawn and keep thousands of threads and
>>>>> just
>>>>> > FS isn’t then I guess you’re unlucky?
>>>>> >
>>>>> > -Michael
>>>>> >
>>>>> > *From:*freeswitch-users-bounces at lists.freeswitch.org
>>>>> > [mailto:freeswitch-users-bounces at lists.freeswitch.org] *On Behalf Of
>>>>> > *Giovanni Maruzzelli
>>>>> > *Sent:* Monday, 14 December, 2015 10:57
>>>>> > *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
>>>>> > *Subject:* Re: [Freeswitch-users] Thread failure with low CPU
>>>>> utilization?
>>>>> >
>>>>> > centos 6 is not supported by FreeSWITCH 1.6 !
>>>>> >
>>>>> > -giovanni
>>>>> >
>>>>> > On Mon, Dec 14, 2015 at 5:47 PM, Vladislav Ivanov <
>>>>> deforceczt at gmail.com
>>>>> > <mailto:deforceczt at gmail.com>> wrote:
>>>>> >
>>>>> >     Centos 6,latest update. Via init script
>>>>> >
>>>>> >     On 14 Dec 2015 6:25 p.m., "Ken Rice" <krice at freeswitch.org
>>>>> >     <mailto:krice at freeswitch.org>> wrote:
>>>>> >
>>>>> >         A)What platform are you running FreeSWITCH on? (eg: Debian 8
>>>>> etc)
>>>>> >
>>>>> >         b)How are you starting FreeSWITCH? (eg: in the foreground or
>>>>> >         background with an init script)
>>>>> >
>>>>> >         *From:*freeswitch-users-bounces at lists.freeswitch.org
>>>>> >         <mailto:freeswitch-users-bounces at lists.freeswitch.org>
>>>>> >         [mailto:freeswitch-users-bounces at lists.freeswitch.org
>>>>> >         <mailto:freeswitch-users-bounces at lists.freeswitch.org>] *On
>>>>> >         Behalf Of *Vladislav Ivanov
>>>>> >         *Sent:* Monday, December 14, 2015 9:47 AM
>>>>> >         *To:* freeswitch-users at lists.freeswitch.org
>>>>> >         <mailto:freeswitch-users at lists.freeswitch.org>
>>>>> >         *Subject:* [Freeswitch-users] Thread failure with low CPU
>>>>>
>>>>> >         utilization?
>>>>> >
>>>>> >         Hello,
>>>>> >
>>>>> >         I have 1.6.5 Freeswitch and I'm trying to test how much
>>>>> >         lines/cps it can handle on my test machine:
>>>>> >
>>>>> >         CPU: Intel(R) Xeon(R) CPU E5450  @ 3.00GHz x2 (8 cores, 3
>>>>> GHz)
>>>>> >         RAM: 32GB
>>>>> >         SSD: 500 GB
>>>>> >
>>>>> >         I'm using call generator to check flood the freeswitch, but
>>>>> when
>>>>> >         FS gets 1k sessions it sends
>>>>> >         [CRIT] switch_core_session.c:1781 Thread Failure!
>>>>> >         and the Star Wars quote... But the CPU load is less than 10%
>>>>> >
>>>>> >         Ulimit is as follows:
>>>>> >
>>>>> >         core file size          (blocks, -c) 0
>>>>> >
>>>>> >         data seg size           (kbytes, -d) unlimited
>>>>> >
>>>>> >         scheduling priority             (-e) 0
>>>>> >
>>>>> >         file size               (blocks, -f) unlimited
>>>>> >
>>>>> >         pending signals                 (-i) 128330
>>>>> >
>>>>> >         max locked memory       (kbytes, -l) 64
>>>>> >
>>>>> >         max memory size         (kbytes, -m) unlimited
>>>>> >
>>>>> >         open files                      (-n) 32768
>>>>> >
>>>>> >         pipe size            (512 bytes, -p) 8
>>>>> >
>>>>> >         POSIX message queues     (bytes, -q) 819200
>>>>> >
>>>>> >         real-time priority              (-r) 0
>>>>> >
>>>>> >         stack size              (kbytes, -s) 2048
>>>>> >
>>>>> >         cpu time               (seconds, -t) unlimited
>>>>> >
>>>>> >         max user processes              (-u) 32768
>>>>> >
>>>>> >         virtual memory          (kbytes, -v) unlimited
>>>>> >
>>>>> >         file locks                      (-x) unlimited
>>>>> >
>>>>> >
>>>>> >         Server is physical, not virtual, but I dont have the same
>>>>> issue
>>>>> >         with virtual freeswitch.
>>>>> >
>>>>> >         Here is the "status" before the session limit:
>>>>> >
>>>>> >         freeswitch at internal> status
>>>>> >
>>>>> >         UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 137
>>>>> >         milliseconds, 603 microseconds
>>>>> >
>>>>> >         FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z
>>>>> >         64bit) is ready
>>>>> >
>>>>> >         994 session(s) since startup
>>>>> >
>>>>> >         994 session(s) - peak 994, last 5min 994
>>>>> >
>>>>> >         40 session(s) per Sec out of max 200, peak 41, last 5min 41
>>>>> >
>>>>> >         10000 session(s) max
>>>>> >
>>>>> >         min idle cpu 10.00/96.77
>>>>> >
>>>>> >         Current Stack Size/Max 2048K/2048K
>>>>> >
>>>>> >         freeswitch at internal> status
>>>>> >
>>>>> >         UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 370
>>>>> >         milliseconds, 877 microseconds
>>>>> >
>>>>> >         FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z
>>>>> >         64bit) is ready
>>>>> >
>>>>> >         1003 session(s) since startup
>>>>> >
>>>>> >         1003 session(s) - peak 1003, last 5min 1003
>>>>> >
>>>>> >         40 session(s) per Sec out of max 200, peak 41, last 5min 41
>>>>> >
>>>>> >         993 session(s) max
>>>>> >
>>>>> >         min idle cpu 10.00/96.77
>>>>> >
>>>>> >         Current Stack Size/Max 2048K/2048K
>>>>> >
>>>>> >         freeswitch at internal> 2015-12-14 15:37:37.436968 [CRIT]
>>>>> >         switch_core_session.c:1781 Thread Failure!
>>>>> >
>>>>> >         2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1737
>>>>> >         LUKE: I'm hit, but not bad.
>>>>> >
>>>>> >         2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1738
>>>>> >         LUKE'S VOICE: Artoo, see what you can do with it. Hang on
>>>>> back
>>>>> >         there....
>>>>> >
>>>>> >         Green laserfire moves past the beeping little robot as his
>>>>> head
>>>>> >         turns.  After a few beeps and a twist of his mechanical arm,
>>>>> >
>>>>> >         Artoo reduces the max sessions to 993 thus, saving the switch
>>>>> >         from certain doom.
>>>>> >
>>>>> >         Any help with that? Really strange error...
>>>>> >
>>>>> >
>>>>>  _________________________________________________________________________
>>>>> >         Professional FreeSWITCH Consulting Services:
>>>>>
>>>>> >         consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>>>> >         http://www.freeswitchsolutions.com
>>>>> >
>>>>> >         Official FreeSWITCH Sites
>>>>> >         http://www.freeswitch.org
>>>>> >         http://confluence.freeswitch.org
>>>>> >         http://www.cluecon.com
>>>>> >
>>>>> >         FreeSWITCH-users mailing list
>>>>> >         FreeSWITCH-users at lists.freeswitch.org
>>>>> >         <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>>> >
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> >         UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> >         http://www.freeswitch.org
>>>>> >
>>>>> >
>>>>> >
>>>>>  _________________________________________________________________________
>>>>> >     Professional FreeSWITCH Consulting Services:
>>>>> >     consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>>>> >     http://www.freeswitchsolutions.com
>>>>> >
>>>>> >     Official FreeSWITCH Sites
>>>>> >     http://www.freeswitch.org
>>>>> >     http://confluence.freeswitch.org
>>>>> >     http://www.cluecon.com
>>>>> >
>>>>> >     FreeSWITCH-users mailing list
>>>>> >     FreeSWITCH-users at lists.freeswitch.org
>>>>> >     <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>>>
>>>>> >     http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> >     UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> >     http://www.freeswitch.org
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> >
>>>>> > Sincerely,
>>>>> >
>>>>> > Giovanni Maruzzelli
>>>>> > Cell : +39-347-2665618
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> _________________________________________________________________________
>>>>> > Professional FreeSWITCH Consulting Services:
>>>>> > consulting at freeswitch.org
>>>>> > http://www.freeswitchsolutions.com
>>>>> >
>>>>> > Official FreeSWITCH Sites
>>>>> > http://www.freeswitch.org
>>>>> > http://confluence.freeswitch.org
>>>>> > http://www.cluecon.com
>>>>> >
>>>>> > FreeSWITCH-users mailing list
>>>>> > FreeSWITCH-users at lists.freeswitch.org
>>>>> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> > UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> > http://www.freeswitch.org
>>>>> >
>>>>>
>>>>>
>>>>> _________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services:
>>>>> consulting at freeswitch.org
>>>>> http://www.freeswitchsolutions.com
>>>>>
>>>>> Official FreeSWITCH Sites
>>>>> http://www.freeswitch.org
>>>>> http://confluence.freeswitch.org
>>>>> http://www.cluecon.com
>>>>>
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> http://www.freeswitch.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬
>>>>>
>>>>>
>>>>>
>>>>> ☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞
>>>>> http://twitter.com/FreeSWITCH
>>>>>
>>>>> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
>>>>> <http://freeswitch.org/g+>*
>>>>>
>>>>> ClueCon Weekly Development Call
>>>>>
>>>>> ☎ sip:888 at conference.freeswitch.org  ☎ +19193869900
>>>>>
>>>>>
>>>>>
>>>>> https://www.youtube.com/watch?v=9XXgW34t40s
>>>>>
>>>>> https://www.youtube.com/watch?v=NLaDpGQuZDA
>>>>>
>>>>>
>>>>>
>>>>> _________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services:
>>>>> consulting at freeswitch.org
>>>>> http://www.freeswitchsolutions.com
>>>>>
>>>>> Official FreeSWITCH Sites
>>>>> http://www.freeswitch.org
>>>>> http://confluence.freeswitch.org
>>>>> http://www.cluecon.com
>>>>>
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> http://www.freeswitch.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services:
>>>>> consulting at freeswitch.org
>>>>> http://www.freeswitchsolutions.com
>>>>>
>>>>> Official FreeSWITCH Sites
>>>>> http://www.freeswitch.org
>>>>> http://confluence.freeswitch.org
>>>>> http://www.cluecon.com
>>>>>
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> http://www.freeswitch.org
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬
>>>>
>>>> ☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞
>>>> http://twitter.com/FreeSWITCH
>>>> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
>>>> <http://freeswitch.org/g+>*
>>>>
>>>> ClueCon Weekly Development Call
>>>> ☎ sip:888 at conference.freeswitch.org  ☎ +19193869900
>>>>
>>>> https://www.youtube.com/watch?v=9XXgW34t40s
>>>> https://www.youtube.com/watch?v=NLaDpGQuZDA
>>>>
>>>>
>>>> _________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services:
>>>> consulting at freeswitch.org
>>>> http://www.freeswitchsolutions.com
>>>>
>>>> Official FreeSWITCH Sites
>>>> http://www.freeswitch.org
>>>> http://confluence.freeswitch.org
>>>> http://www.cluecon.com
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:
>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>>>
>>>
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://confluence.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>
>>
>>
>> --
>> Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬
>>
>> ☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞
>> http://twitter.com/FreeSWITCH
>> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
>> <http://freeswitch.org/g+>*
>>
>> ClueCon Weekly Development Call
>> ☎ sip:888 at conference.freeswitch.org  ☎ +19193869900
>>
>> https://www.youtube.com/watch?v=9XXgW34t40s
>> https://www.youtube.com/watch?v=NLaDpGQuZDA
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://confluence.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20151217/4df34cb7/attachment-0001.html

Join us at ClueCon 2016 Aug 8-12, 2016
More information about the FreeSWITCH-users mailing list