<div dir="ltr">1000 threads is not really a show stopper =D<div>That is what the artoo message, it tells you that the new current thread limit has been auto detected and that is the new max calls you can handle henceforth.</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Dec 15, 2015 at 5:16 AM, Michael Giagnocavo <span dir="ltr"><<a href="mailto:mgg@giagnocavo.net" target="_blank">mgg@giagnocavo.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Perhaps write a patch to detect these limits and then CRIT on start if they conflict with settings.
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a> [mailto:<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a>]
<b>On Behalf Of </b>Vladislav Ivanov<br>
<b>Sent:</b> Tuesday, 15 December, 2015 2:15<span class=""><br>
<b>To:</b> FreeSWITCH Users Help <<a href="mailto:freeswitch-users@lists.freeswitch.org" target="_blank">freeswitch-users@lists.freeswitch.org</a>><br>
</span><b>Subject:</b> Re: [Freeswitch-users] Thread failure with low CPU utilization?<u></u><u></u></span></p><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Hey guys,<br>
<br>
Anthony was correct in this case, I ran freeswitch not from root, for some reason my /etc/security/limits.conf file line:<u></u><u></u></p>
<div>
<p class="MsoNormal">* soft nproc 32768<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">* hard nproc 32768<br>
<br>
was not propagated on freeswitch user... <br>
<br>
Added new line:<u></u><u></u></p>
<div>
<p class="MsoNormal">freeswitch soft nproc 32768<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">freeswitch hard nproc 32768<br>
<br>
and it worked fine from that.<br>
Thanks to you all!<u></u><u></u></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">2015-12-14 23:06 GMT+02:00 Anthony Minessale <<a href="mailto:anthony.minessale@gmail.com" target="_blank">anthony.minessale@gmail.com</a>>:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">Are you running FS as root or a regular user, you need to check the ulimits of concurrent threads and some boxes have a hard limit of threads, its not all tied to CPU usage.<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Mon, Dec 14, 2015 at 1:30 PM, Kevin Wormington <<a href="mailto:kworm@sofnet.com" target="_blank">kworm@sofnet.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">This post<br>
<a href="http://unix.stackexchange.com/questions/47595/linux-max-threads-count" target="_blank">http://unix.stackexchange.com/questions/47595/linux-max-threads-count</a><br>
has a simple program to check out how many threads can be created. You<br>
can run the test program as root and then as FS user and see if it's a<br>
ulimit type of problem.<br>
<br>
On 12/14/2015 01:08 PM, Michael Giagnocavo wrote:<br>
> Is there a reason to think that CentOS 6 would cause threads to fail to<br>
> spawn? This is basically just calling libapr, which is code from like 8<br>
> years ago.<br>
><br>
> Vladislav: This means it failed to create a thread (switch_thread_create<br>
> in switch_apr.c). FS generates a ton of threads. The only time I’ve seen<br>
> this was a long time ago on 32-bit Windows, since default stack of 1MB<br>
> means you hit address space limits pretty quickly. But on 64-bit, you<br>
> should be able to go all out.<br>
><br>
> I’d start by adding some if’d printf’s to check the various things that<br>
> switch_thread_create does and try to pinpoint what is failing. If the<br>
> libapr call is failing, then you should be able to create a minimal<br>
> program that just spawns threads and see if that one is able to get<br>
> 1000+. From there I’d guess troubleshooting is gonna be the same as<br>
> figuring out why pthread_create fails and that should be generic to your<br>
> system. (Please post if you figure out why.)<br>
><br>
> If other apps are able to spawn and keep thousands of threads and just<br>
> FS isn’t then I guess you’re unlucky?<br>
><br>
> -Michael<br>
><br>
> *From:*<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a><br>
> [mailto:<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a>] *On Behalf Of<br>
> *Giovanni Maruzzelli<br>
> *Sent:* Monday, 14 December, 2015 10:57<br>
> *To:* FreeSWITCH Users Help <<a href="mailto:freeswitch-users@lists.freeswitch.org" target="_blank">freeswitch-users@lists.freeswitch.org</a>><br>
> *Subject:* Re: [Freeswitch-users] Thread failure with low CPU utilization?<br>
><br>
> centos 6 is not supported by FreeSWITCH 1.6 !<br>
><br>
> -giovanni<br>
><br>
> On Mon, Dec 14, 2015 at 5:47 PM, Vladislav Ivanov <<a href="mailto:deforceczt@gmail.com" target="_blank">deforceczt@gmail.com</a><br>
> <mailto:<a href="mailto:deforceczt@gmail.com" target="_blank">deforceczt@gmail.com</a>>> wrote:<br>
><br>
> Centos 6,latest update. Via init script<br>
><br>
> On 14 Dec 2015 6:25 p.m., "Ken Rice" <<a href="mailto:krice@freeswitch.org" target="_blank">krice@freeswitch.org</a><br>
> <mailto:<a href="mailto:krice@freeswitch.org" target="_blank">krice@freeswitch.org</a>>> wrote:<br>
><br>
> A)What platform are you running FreeSWITCH on? (eg: Debian 8 etc)<br>
><br>
> b)How are you starting FreeSWITCH? (eg: in the foreground or<br>
> background with an init script)<br>
><br>
> *From:*<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a><br>
> <mailto:<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a>><br>
> [mailto:<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a><br>
> <mailto:<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a>>] *On<br>
> Behalf Of *Vladislav Ivanov<br>
> *Sent:* Monday, December 14, 2015 9:47 AM<br>
> *To:* <a href="mailto:freeswitch-users@lists.freeswitch.org" target="_blank">
freeswitch-users@lists.freeswitch.org</a><br>
> <mailto:<a href="mailto:freeswitch-users@lists.freeswitch.org" target="_blank">freeswitch-users@lists.freeswitch.org</a>><br>
> *Subject:* [Freeswitch-users] Thread failure with low CPU<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">> utilization?<br>
><br>
> Hello,<br>
><br>
> I have 1.6.5 Freeswitch and I'm trying to test how much<br>
> lines/cps it can handle on my test machine:<br>
><br>
> CPU: Intel(R) Xeon(R) CPU E5450 @ 3.00GHz x2 (8 cores, 3 GHz)<br>
> RAM: 32GB<br>
> SSD: 500 GB<br>
><br>
> I'm using call generator to check flood the freeswitch, but when<br>
> FS gets 1k sessions it sends<br>
> [CRIT] switch_core_session.c:1781 Thread Failure!<br>
> and the Star Wars quote... But the CPU load is less than 10%<br>
><br>
> Ulimit is as follows:<br>
><br>
> core file size (blocks, -c) 0<br>
><br>
> data seg size (kbytes, -d) unlimited<br>
><br>
> scheduling priority (-e) 0<br>
><br>
> file size (blocks, -f) unlimited<br>
><br>
> pending signals (-i) 128330<br>
><br>
> max locked memory (kbytes, -l) 64<br>
><br>
> max memory size (kbytes, -m) unlimited<br>
><br>
> open files (-n) 32768<br>
><br>
> pipe size (512 bytes, -p) 8<br>
><br>
> POSIX message queues (bytes, -q) 819200<br>
><br>
> real-time priority (-r) 0<br>
><br>
> stack size (kbytes, -s) 2048<br>
><br>
> cpu time (seconds, -t) unlimited<br>
><br>
> max user processes (-u) 32768<br>
><br>
> virtual memory (kbytes, -v) unlimited<br>
><br>
> file locks (-x) unlimited<br>
><br>
><br>
> Server is physical, not virtual, but I dont have the same issue<br>
> with virtual freeswitch.<br>
><br>
> Here is the "status" before the session limit:<br>
><br>
> freeswitch@internal> status<br>
><br>
> UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 137<br>
> milliseconds, 603 microseconds<br>
><br>
> FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z<br>
> 64bit) is ready<br>
><br>
> 994 session(s) since startup<br>
><br>
> 994 session(s) - peak 994, last 5min 994<br>
><br>
> 40 session(s) per Sec out of max 200, peak 41, last 5min 41<br>
><br>
> 10000 session(s) max<br>
><br>
> min idle cpu 10.00/96.77<br>
><br>
> Current Stack Size/Max 2048K/2048K<br>
><br>
> freeswitch@internal> status<br>
><br>
> UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 370<br>
> milliseconds, 877 microseconds<br>
><br>
> FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z<br>
> 64bit) is ready<br>
><br>
> 1003 session(s) since startup<br>
><br>
> 1003 session(s) - peak 1003, last 5min 1003<br>
><br>
> 40 session(s) per Sec out of max 200, peak 41, last 5min 41<br>
><br>
> 993 session(s) max<br>
><br>
> min idle cpu 10.00/96.77<br>
><br>
> Current Stack Size/Max 2048K/2048K<br>
><br>
> freeswitch@internal> 2015-12-14 15:37:37.436968 [CRIT]<br>
> switch_core_session.c:1781 Thread Failure!<br>
><br>
> 2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1737<br>
> LUKE: I'm hit, but not bad.<br>
><br>
> 2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1738<br>
> LUKE'S VOICE: Artoo, see what you can do with it. Hang on back<br>
> there....<br>
><br>
> Green laserfire moves past the beeping little robot as his head<br>
> turns. After a few beeps and a twist of his mechanical arm,<br>
><br>
> Artoo reduces the max sessions to 993 thus, saving the switch<br>
> from certain doom.<br>
><br>
> Any help with that? Really strange error...<br>
><br>
> _________________________________________________________________________<br>
> Professional FreeSWITCH Consulting Services:<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal">> <a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a> <mailto:<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a>><br>
> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
><br>
> Official FreeSWITCH Sites<br>
> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
> <a href="http://confluence.freeswitch.org" target="_blank">http://confluence.freeswitch.org</a><br>
> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
><br>
> FreeSWITCH-users mailing list<br>
> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
> <mailto:<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a>><br>
> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
><br>
><br>
> _________________________________________________________________________<br>
> Professional FreeSWITCH Consulting Services:<br>
> <a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a> <mailto:<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a>><br>
> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
><br>
> Official FreeSWITCH Sites<br>
> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
> <a href="http://confluence.freeswitch.org" target="_blank">http://confluence.freeswitch.org</a><br>
> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
><br>
> FreeSWITCH-users mailing list<br>
> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
> <mailto:<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a>><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
><br>
><br>
><br>
><br>
> --<br>
><br>
> Sincerely,<br>
><br>
> Giovanni Maruzzelli<br>
> Cell : <a href="tel:%2B39-347-2665618" target="_blank">+39-347-2665618</a><br>
><br>
><br>
><br>
> _________________________________________________________________________<br>
> Professional FreeSWITCH Consulting Services:<br>
> <a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
><br>
> Official FreeSWITCH Sites<br>
> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
> <a href="http://confluence.freeswitch.org" target="_blank">http://confluence.freeswitch.org</a><br>
> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
><br>
> FreeSWITCH-users mailing list<br>
> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
><br>
<br>
_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://confluence.freeswitch.org" target="_blank">http://confluence.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><u></u><u></u></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
<p class="MsoNormal"><span><span style="color:#888888">-- <u></u><u></u></span></span></p>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="color:#888888">Anthony Minessale II </span>
<span style="font-family:"Segoe UI Symbol",sans-serif;color:#888888">♬</span><span style="color:#888888"> @anthmfs </span><span style="font-family:"Segoe UI Symbol",sans-serif;color:#888888">♬</span><span style="color:#888888"> @FreeSWITCH </span><span style="font-family:"Segoe UI Symbol",sans-serif;color:#888888">♬</span><u></u><u></u></p>
<div>
<p class="MsoNormal"><span style="color:#888888"><u></u> <u></u></span></p>
<div>
<p class="MsoNormal"><span style="font-family:"Segoe UI Symbol",sans-serif;color:#888888">☞</span><span style="color:#888888">
<a href="http://freeswitch.org/" target="_blank">http://freeswitch.org/</a> </span><span style="font-family:"Segoe UI Symbol",sans-serif;color:#888888">☞</span><span style="color:#888888">
<a href="http://cluecon.com/" target="_blank">http://cluecon.com/</a> </span><span style="font-family:"Segoe UI Symbol",sans-serif;color:#888888">☞</span><span style="color:#888888">
<a href="http://twitter.com/FreeSWITCH" target="_blank">http://twitter.com/FreeSWITCH</a><u></u><u></u></span></p>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-family:"Segoe UI Symbol",sans-serif;color:#888888">☞</span><span style="color:#888888">
<a href="http://irc.freenode.net" target="_blank">irc.freenode.net</a> #freeswitch </span><span style="font-family:"Segoe UI Symbol",sans-serif;color:#888888">☞</span><span style="color:#888888">
<u><a href="http://freeswitch.org/g+" target="_blank">http://freeswitch.org/g+</a></u><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#888888">ClueCon Weekly Development Call <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Segoe UI Symbol",sans-serif;color:#888888">☎</span><span style="color:#888888"> <a href="mailto:sip%3A888@conference.freeswitch.org" target="_blank">sip:888@conference.freeswitch.org</a> </span><span style="font-family:"Segoe UI Symbol",sans-serif;color:#888888">☎</span><span style="color:#888888">
<a href="tel:%2B19193869900" target="_blank">+19193869900</a> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#888888"><u></u> <u></u></span></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><span style="color:#888888"><a href="https://www.youtube.com/watch?v=9XXgW34t40s" target="_blank"><span style="font-size:9.5pt;color:#1155cc">https://www.youtube.com/watch?v=9XXgW34t40s</span></a><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#888888"><a href="https://www.youtube.com/watch?v=NLaDpGQuZDA" target="_blank">https://www.youtube.com/watch?v=NLaDpGQuZDA</a><u></u><u></u></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><br>
_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://confluence.freeswitch.org" target="_blank">http://confluence.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div></div></div>
</div>
<br>_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" rel="noreferrer" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://confluence.freeswitch.org" rel="noreferrer" target="_blank">http://confluence.freeswitch.org</a><br>
<a href="http://www.cluecon.com" rel="noreferrer" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬<div><br><div>☞ <a href="http://freeswitch.org/" target="_blank">http://freeswitch.org/</a> ☞ <a href="http://cluecon.com/" target="_blank">http://cluecon.com/</a> ☞ <a href="http://twitter.com/FreeSWITCH" target="_blank">http://twitter.com/FreeSWITCH</a></div><div><div>☞ <a href="http://irc.freenode.net" target="_blank">irc.freenode.net</a> #freeswitch ☞ <u><a href="http://freeswitch.org/g+" target="_blank">http://freeswitch.org/g+</a></u><br><br></div><div>ClueCon Weekly Development Call <br></div><div>☎ <a href="mailto:sip%3A888@conference.freeswitch.org" target="_blank">sip:888@conference.freeswitch.org</a> ☎ +19193869900 </div><div><br></div></div></div><div><a href="https://www.youtube.com/watch?v=9XXgW34t40s" style="color:rgb(17,85,204);font-size:12.8000001907349px" target="_blank">https://www.youtube.com/watch?v=9XXgW34t40s</a></div><div><a href="https://www.youtube.com/watch?v=NLaDpGQuZDA" target="_blank">https://www.youtube.com/watch?v=NLaDpGQuZDA</a><br></div></div></div></div></div></div></div>
</div>