<div>I already told you why a week ago. Your script most likely has an endless loop in it that does not check the condition session:ready() </div><div><br></div><div>However, Feel free to contact <a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a> and we can formally diagnose it for a fee.</div><div><br></div><div><br></div><div><br></div><div><br><div class="gmail_quote"><div>On Tue, Mar 21, 2017 at 6:42 AM Grant Bagdasarian <<a href="mailto:gb@cm.nl">gb@cm.nl</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple" class="gmail_msg">
<div class="m_3623806029287245330WordSection1 gmail_msg">
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg">Attached are the logs for:<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p></div></div><div lang="EN-US" link="blue" vlink="purple" class="gmail_msg"><div class="m_3623806029287245330WordSection1 gmail_msg">
<pre class="gmail_msg">#check thread usage:<u class="gmail_msg"></u><u class="gmail_msg"></u></pre>
<pre class="gmail_msg">top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1<u class="gmail_msg"></u><u class="gmail_msg"></u></pre>
<pre class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></pre>
<pre class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></pre>
<pre class="gmail_msg">#memory mapping:<u class="gmail_msg"></u><u class="gmail_msg"></u></pre>
<pre class="gmail_msg">cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps<u class="gmail_msg"></u><u class="gmail_msg"></u></pre>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></p>
</div></div><div lang="EN-US" link="blue" vlink="purple" class="gmail_msg"><div class="m_3623806029287245330WordSection1 gmail_msg"><p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg">We’re unable to run gcore since freeswitch hogs all available cpu’s at 100%.
<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg">I tried changing the nice settings real-time but that didn’t work either.<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg">At one point we had a huge spike in inbound traffic towards our freeswitch (valid traffic).<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg">The traffic is handled by a lua application.<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg">What we saw was calls hanging in freeswitch while they’ve already been physically disconnected.
<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg">After the huge spike freeswitch behaved normally for a couple of minutes and then suddenly started eating all the available cpu cores at 100%.
<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg">We have some db queries which are fired in the lua application, so these may cause the application to hang, but we haven’t reproduced it yet.
<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg">I’ll setup a test lab with the same configuration and fire sipp at it to see if I can reproduce the issue.
<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg">However, I probably won’t be able to run gcore for debugging. Any other suggestions?<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg">Also we saw a lot of open udp ports and some had a huge read buffer size which did not decrease.
<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg"><img width="924" height="436" style="width:9.625in;height:4.5416in" id="m_3623806029287245330Picture_x0020_2" src="cid:image001.png@01D2A238.14965330" class="gmail_msg"></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg"><u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></p>
<div class="gmail_msg">
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0cm 0cm 0cm" class="gmail_msg">
<p class="MsoNormal gmail_msg"><b class="gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg"> <a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" class="gmail_msg" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a> [mailto:<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" class="gmail_msg" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a>]
<b class="gmail_msg">On Behalf Of </b>Grant Bagdasarian<br class="gmail_msg">
<b class="gmail_msg">Sent:</b> maandag 13 maart 2017 09:04</span></p></div></div></div></div><div lang="EN-US" link="blue" vlink="purple" class="gmail_msg"><div class="m_3623806029287245330WordSection1 gmail_msg"><div class="gmail_msg"><div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0cm 0cm 0cm" class="gmail_msg"><p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg"><br class="gmail_msg">
<b class="gmail_msg">To:</b> FreeSWITCH Users Help <<a href="mailto:freeswitch-users@lists.freeswitch.org" class="gmail_msg" target="_blank">freeswitch-users@lists.freeswitch.org</a>><br class="gmail_msg">
<b class="gmail_msg">Subject:</b> Re: [Freeswitch-users] 100% CPU usage<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p></div></div></div></div><div lang="EN-US" link="blue" vlink="purple" class="gmail_msg"><div class="m_3623806029287245330WordSection1 gmail_msg">
<p class="MsoNormal gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></p>
<p class="MsoNormal gmail_msg"><span lang="NL" style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg">Hi Antony and Antonio,<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span lang="NL" style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg">Thank you for the tips.<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg">We haven’t had freeswitch going berserk anymore (not sure as to why it did perform without issue over the weekend), but will continue to monitor it and run the commands to
trace the issue.<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg">Regards,<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg">Grant<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><b class="gmail_msg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" class="gmail_msg">
<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" class="gmail_msg" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a> [<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" class="gmail_msg" target="_blank">mailto:freeswitch-users-bounces@lists.freeswitch.org</a>]
<b class="gmail_msg">On Behalf Of </b>Anthony Minessale<br class="gmail_msg">
<b class="gmail_msg">Sent:</b> zaterdag 11 maart 2017 00:07<br class="gmail_msg">
<b class="gmail_msg">To:</b> FreeSWITCH Users Help <<a href="mailto:freeswitch-users@lists.freeswitch.org" class="gmail_msg" target="_blank">freeswitch-users@lists.freeswitch.org</a>><br class="gmail_msg">
<b class="gmail_msg">Subject:</b> Re: [Freeswitch-users] 100% CPU usage<u class="gmail_msg"></u><u class="gmail_msg"></u></span></p>
<p class="MsoNormal gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></p>
<div class="gmail_msg">
<p class="MsoNormal gmail_msg">Make sure you lua script has all while loops including session:ready() as a condition to continue. Its a common mistake that leads to loops spinning out of control on hungup channels.<u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<div class="gmail_msg">
<p class="MsoNormal gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></p>
</div>
</div>
<div class="gmail_msg">
<p class="MsoNormal gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></p>
<div class="gmail_msg">
<p class="MsoNormal gmail_msg">On Fri, Mar 10, 2017 at 6:11 AM, Antonio Silva <<a href="mailto:asilva@wirelessmundi.com" class="gmail_msg" target="_blank">asilva@wirelessmundi.com</a>> wrote:<u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt" class="gmail_msg">
<div class="gmail_msg">
<p class="MsoNormal gmail_msg" style="margin-bottom:12.0pt">Hi,<br class="gmail_msg">
<br class="gmail_msg">
You can use the following commands:<u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<pre class="gmail_msg">#get a core:<u class="gmail_msg"></u><u class="gmail_msg"></u></pre>
<pre class="gmail_msg">gcore -o /home/gcore-$(date +%s) $(cat /var/run/freeswitch/freeswitch.pid)<u class="gmail_msg"></u><u class="gmail_msg"></u></pre>
<pre class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></pre>
<pre class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></pre>
<pre class="gmail_msg">#check thread usage:<u class="gmail_msg"></u><u class="gmail_msg"></u></pre>
<pre class="gmail_msg">top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1<u class="gmail_msg"></u><u class="gmail_msg"></u></pre>
<pre class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></pre>
<pre class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></pre>
<pre class="gmail_msg">#memory mapping:<u class="gmail_msg"></u><u class="gmail_msg"></u></pre>
<pre class="gmail_msg">cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps<u class="gmail_msg"></u><u class="gmail_msg"></u></pre>
<p class="MsoNormal gmail_msg" style="margin-bottom:12.0pt"><br class="gmail_msg">
<br class="gmail_msg">
you can get some info from fs:<u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<pre class="gmail_msg">fs_cli -x "sofia global siptrace on"<u class="gmail_msg"></u><u class="gmail_msg"></u></pre>
<pre class="gmail_msg">fs_cli -x "fsctl debug_level 10"<u class="gmail_msg"></u><u class="gmail_msg"></u></pre>
<pre class="gmail_msg">fs_cli -x "sofia loglevel all 9"<u class="gmail_msg"></u><u class="gmail_msg"></u></pre>
<pre class="gmail_msg">fs_cli -x "fsctl debug_sql "<u class="gmail_msg"></u><u class="gmail_msg"></u></pre>
<pre class="gmail_msg">fs_cli -x "sofia global debug presence"<u class="gmail_msg"></u><u class="gmail_msg"></u></pre>
<p class="MsoNormal gmail_msg"><br class="gmail_msg">
<br class="gmail_msg">
Put everything into a jira so we can take a look.<br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
Regards,<br class="gmail_msg">
António<u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<div class="gmail_msg">
<div class="gmail_msg">
<p class="MsoNormal gmail_msg" style="margin-bottom:12.0pt"><u class="gmail_msg"></u> <u class="gmail_msg"></u></p>
<div class="gmail_msg">
<p class="MsoNormal gmail_msg">On 03/10/2017 11:08 AM, Grant Bagdasarian wrote:<u class="gmail_msg"></u><u class="gmail_msg"></u></p>
</div>
</div>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt" class="gmail_msg">
<div class="gmail_msg">
<div class="gmail_msg">
<div class="gmail_msg">
<p class="MsoNormal gmail_msg"><span lang="NL" class="gmail_msg">Hi,</span><u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<p class="MsoNormal gmail_msg"><span lang="NL" class="gmail_msg"> </span><u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<p class="MsoNormal gmail_msg">We’re currently experiencing a weird issue with one of our freeswitch servers where the CPU just goes berserk and stays at 100%.
<u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<p class="MsoNormal gmail_msg">There are no active calls at that moment, but FS is still using 100% CPU.
<u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<p class="MsoNormal gmail_msg"> <u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<p class="MsoNormal gmail_msg">I was unable to get a debug using gdb, since freeswitch does not crash.
<u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<p class="MsoNormal gmail_msg"> <u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<p class="MsoNormal gmail_msg">We had this issue on 1.6.13 and then I upgraded to 1.6.15 this morning, but the problem still persists.
<u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<p class="MsoNormal gmail_msg">There is currently only one active application running on this box which is using LUA. Maybe that is causing the issue?<u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<p class="MsoNormal gmail_msg">We checked the code but there are no weird loops etc. Perhaps we are missing something?<u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<p class="MsoNormal gmail_msg"> <u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<p class="MsoNormal gmail_msg">Any ideas how to trace this?<u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<p class="MsoNormal gmail_msg"> <u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<p class="MsoNormal gmail_msg">Regards,<u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<p class="MsoNormal gmail_msg"> <u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<p class="MsoNormal gmail_msg">Grant<u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<p class="MsoNormal gmail_msg"> <u class="gmail_msg"></u><u class="gmail_msg"></u></p>
</div>
<p class="MsoNormal gmail_msg" style="margin-bottom:12.0pt"><u class="gmail_msg"></u> <u class="gmail_msg"></u></p>
</div>
</div>
<pre class="gmail_msg">_________________________________________________________________________<u class="gmail_msg"></u><u class="gmail_msg"></u></pre>
<pre class="gmail_msg">Professional FreeSWITCH Consulting Services: <u class="gmail_msg"></u><u class="gmail_msg"></u></pre>
<pre class="gmail_msg"><a href="mailto:consulting@freeswitch.org" class="gmail_msg" target="_blank">consulting@freeswitch.org</a><u class="gmail_msg"></u><u class="gmail_msg"></u></pre>
<pre class="gmail_msg"><a href="http://www.freeswitchsolutions.com" class="gmail_msg" target="_blank">http://www.freeswitchsolutions.com</a><u class="gmail_msg"></u><u class="gmail_msg"></u></pre>
<pre class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></pre>
<pre class="gmail_msg">Official FreeSWITCH Sites<u class="gmail_msg"></u><u class="gmail_msg"></u></pre>
<pre class="gmail_msg"><a href="http://www.freeswitch.org" class="gmail_msg" target="_blank">http://www.freeswitch.org</a><u class="gmail_msg"></u><u class="gmail_msg"></u></pre>
<pre class="gmail_msg"><a href="http://confluence.freeswitch.org" class="gmail_msg" target="_blank">http://confluence.freeswitch.org</a><u class="gmail_msg"></u><u class="gmail_msg"></u></pre>
<pre class="gmail_msg"><a href="http://www.cluecon.com" class="gmail_msg" target="_blank">http://www.cluecon.com</a><u class="gmail_msg"></u><u class="gmail_msg"></u></pre>
<pre class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></pre>
<pre class="gmail_msg">FreeSWITCH-users mailing list<u class="gmail_msg"></u><u class="gmail_msg"></u></pre>
<pre class="gmail_msg"><a href="mailto:FreeSWITCH-users@lists.freeswitch.org" class="gmail_msg" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><u class="gmail_msg"></u><u class="gmail_msg"></u></pre>
<pre class="gmail_msg"><a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" class="gmail_msg" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><u class="gmail_msg"></u><u class="gmail_msg"></u></pre>
<pre class="gmail_msg">UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" class="gmail_msg" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><u class="gmail_msg"></u><u class="gmail_msg"></u></pre>
<pre class="gmail_msg"><a href="http://www.freeswitch.org" class="gmail_msg" target="_blank">http://www.freeswitch.org</a><u class="gmail_msg"></u><u class="gmail_msg"></u></pre>
</blockquote>
<p class="MsoNormal gmail_msg" style="margin-bottom:12.0pt"><span class="m_3623806029287245330hoenzb gmail_msg"><span style="color:#888888" class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></span></p>
<pre class="gmail_msg"><span style="color:#888888" class="gmail_msg">-- </span><u class="gmail_msg"></u><u class="gmail_msg"></u></pre>
<pre class="gmail_msg"><span style="color:#888888" class="gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></span></pre>
<pre class="gmail_msg"><span style="color:#888888" class="gmail_msg">Saludos / Regards / Cumprimentos,<u class="gmail_msg"></u><u class="gmail_msg"></u></span></pre>
<pre class="gmail_msg"><span style="color:#888888" class="gmail_msg">António silva</span><u class="gmail_msg"></u><u class="gmail_msg"></u></pre>
</div>
<p class="MsoNormal gmail_msg"><br class="gmail_msg">
_________________________________________________________________________<br class="gmail_msg">
Professional FreeSWITCH Consulting Services:<br class="gmail_msg">
<a href="mailto:consulting@freeswitch.org" class="gmail_msg" target="_blank">consulting@freeswitch.org</a><br class="gmail_msg">
<a href="http://www.freeswitchsolutions.com" class="gmail_msg" target="_blank">http://www.freeswitchsolutions.com</a><br class="gmail_msg">
<br class="gmail_msg">
Official FreeSWITCH Sites<br class="gmail_msg">
<a href="http://www.freeswitch.org" class="gmail_msg" target="_blank">http://www.freeswitch.org</a><br class="gmail_msg">
<a href="http://confluence.freeswitch.org" class="gmail_msg" target="_blank">http://confluence.freeswitch.org</a><br class="gmail_msg">
<a href="http://www.cluecon.com" class="gmail_msg" target="_blank">http://www.cluecon.com</a><br class="gmail_msg">
<br class="gmail_msg">
FreeSWITCH-users mailing list<br class="gmail_msg">
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" class="gmail_msg" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br class="gmail_msg">
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" class="gmail_msg" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br class="gmail_msg">
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" class="gmail_msg" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br class="gmail_msg">
<a href="http://www.freeswitch.org" class="gmail_msg" target="_blank">http://www.freeswitch.org</a><u class="gmail_msg"></u><u class="gmail_msg"></u></p>
</blockquote>
</div>
<p class="MsoNormal gmail_msg"><br class="gmail_msg">
<br clear="all" class="gmail_msg">
<u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<div class="gmail_msg">
<p class="MsoNormal gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></p>
</div>
<p class="MsoNormal gmail_msg">-- <u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<div class="gmail_msg">
<div class="gmail_msg">
<div class="gmail_msg">
<div class="gmail_msg">
<div class="gmail_msg">
<div class="gmail_msg">
<p class="MsoNormal gmail_msg">Anthony Minessale II <span style="font-family:"Segoe UI Symbol",sans-serif" class="gmail_msg">
♬</span> @anthmfs <span style="font-family:"Segoe UI Symbol",sans-serif" class="gmail_msg">♬</span> @FreeSWITCH <span style="font-family:"Segoe UI Symbol",sans-serif" class="gmail_msg">♬</span><u class="gmail_msg"></u><u class="gmail_msg"></u></p>
<div class="gmail_msg">
<p class="MsoNormal gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></p>
<div class="gmail_msg">
<p class="MsoNormal gmail_msg"><span style="font-family:"Segoe UI Symbol",sans-serif" class="gmail_msg">☞</span>
<a href="http://freeswitch.org/" class="gmail_msg" target="_blank">http://freeswitch.org/</a> <span style="font-family:"Segoe UI Symbol",sans-serif" class="gmail_msg">☞</span>
<a href="http://cluecon.com/" class="gmail_msg" target="_blank">http://cluecon.com/</a> <span style="font-family:"Segoe UI Symbol",sans-serif" class="gmail_msg">☞</span>
<a href="http://twitter.com/FreeSWITCH" class="gmail_msg" target="_blank">http://twitter.com/FreeSWITCH</a><u class="gmail_msg"></u><u class="gmail_msg"></u></p>
</div>
<div class="gmail_msg">
<div class="gmail_msg">
<p class="MsoNormal gmail_msg" style="margin-bottom:12.0pt"><span style="font-family:"Segoe UI Symbol",sans-serif" class="gmail_msg">☞</span>
<a href="http://irc.freenode.net" class="gmail_msg" target="_blank">irc.freenode.net</a> #freeswitch <span style="font-family:"Segoe UI Symbol",sans-serif" class="gmail_msg">☞</span>
<u class="gmail_msg"><a href="http://freeswitch.org/g+" class="gmail_msg" target="_blank">http://freeswitch.org/g+</a></u><u class="gmail_msg"></u><u class="gmail_msg"></u></p>
</div>
<div class="gmail_msg">
<p class="MsoNormal gmail_msg">ClueCon Weekly Development Call <u class="gmail_msg"></u><u class="gmail_msg"></u></p>
</div>
<div class="gmail_msg">
<p class="MsoNormal gmail_msg"><span style="font-family:"Segoe UI Emoji",sans-serif" class="gmail_msg">☎</span> <a href="mailto:sip%3A888@conference.freeswitch.org" class="gmail_msg" target="_blank">sip:888@conference.freeswitch.org</a> <span style="font-family:"Segoe UI Emoji",sans-serif" class="gmail_msg">☎</span> +19193869900 <u class="gmail_msg"></u><u class="gmail_msg"></u></p>
</div>
<div class="gmail_msg">
<p class="MsoNormal gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></p>
</div>
</div>
</div>
<div class="gmail_msg">
<p class="MsoNormal gmail_msg"><a href="https://www.youtube.com/watch?v=9XXgW34t40s" class="gmail_msg" target="_blank"><span style="font-size:9.5pt;color:#1155cc" class="gmail_msg">https://www.youtube.com/watch?v=9XXgW34t40s</span></a><u class="gmail_msg"></u><u class="gmail_msg"></u></p>
</div>
<div class="gmail_msg">
<p class="MsoNormal gmail_msg"><a href="https://www.youtube.com/watch?v=NLaDpGQuZDA" class="gmail_msg" target="_blank">https://www.youtube.com/watch?v=NLaDpGQuZDA</a><u class="gmail_msg"></u><u class="gmail_msg"></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div></div>
_________________________________________________________________________<br class="gmail_msg">
Professional FreeSWITCH Consulting Services:<br class="gmail_msg">
<a href="mailto:consulting@freeswitch.org" class="gmail_msg" target="_blank">consulting@freeswitch.org</a><br class="gmail_msg">
<a href="http://www.freeswitchsolutions.com" rel="noreferrer" class="gmail_msg" target="_blank">http://www.freeswitchsolutions.com</a><br class="gmail_msg">
<br class="gmail_msg">
Official FreeSWITCH Sites<br class="gmail_msg">
<a href="http://www.freeswitch.org" rel="noreferrer" class="gmail_msg" target="_blank">http://www.freeswitch.org</a><br class="gmail_msg">
<a href="http://confluence.freeswitch.org" rel="noreferrer" class="gmail_msg" target="_blank">http://confluence.freeswitch.org</a><br class="gmail_msg">
<a href="http://www.cluecon.com" rel="noreferrer" class="gmail_msg" target="_blank">http://www.cluecon.com</a><br class="gmail_msg">
<br class="gmail_msg">
FreeSWITCH-users mailing list<br class="gmail_msg">
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" class="gmail_msg" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br class="gmail_msg">
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br class="gmail_msg">
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br class="gmail_msg">
<a href="http://www.freeswitch.org" rel="noreferrer" class="gmail_msg" target="_blank">http://www.freeswitch.org</a></blockquote></div></div><div dir="ltr">-- <br></div><div data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬<div><br><div>☞ <a href="http://freeswitch.org/" target="_blank">http://freeswitch.org/</a> ☞ <a href="http://cluecon.com/" target="_blank">http://cluecon.com/</a> ☞ <a href="http://twitter.com/FreeSWITCH" target="_blank">http://twitter.com/FreeSWITCH</a></div><div><div>☞ <a href="http://irc.freenode.net" target="_blank">irc.freenode.net</a> #freeswitch ☞ <u><a href="http://freeswitch.org/g+" target="_blank">http://freeswitch.org/g+</a></u><br><br></div><div>ClueCon Weekly Development Call <br></div><div>☎ <a href="mailto:sip%3A888@conference.freeswitch.org" target="_blank">sip:888@conference.freeswitch.org</a> ☎ +19193869900 </div><div><br></div></div></div><div><a href="https://www.youtube.com/watch?v=9XXgW34t40s" style="color:rgb(17,85,204);font-size:12.8000001907349px" target="_blank">https://www.youtube.com/watch?v=9XXgW34t40s</a></div><div><a href="https://www.youtube.com/watch?v=NLaDpGQuZDA" target="_blank">https://www.youtube.com/watch?v=NLaDpGQuZDA</a><br></div></div></div></div></div></div></div>