<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI Emoji";
panose-1:2 11 5 2 4 2 4 2 2 3;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI Symbol";
panose-1:2 11 5 2 4 2 4 2 2 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.gmailmsg
{mso-style-name:gmail_msg;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.m3623806029287245330hoenzb
{mso-style-name:m_3623806029287245330hoenzb;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Anthony, we triple checked the script and its dependencies. There are no loops at all in the script. The imported scripts are also written by us and they too do not contain
any loops. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">It’s just basic call handling with some playback and play_and_get_digits calls, and some pcall calls to query a database.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org]
<b>On Behalf Of </b>Anthony Minessale<br>
<b>Sent:</b> dinsdag 21 maart 2017 16:02<br>
<b>To:</b> FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org><br>
<b>Subject:</b> Re: [Freeswitch-users] 100% CPU usage<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">I already told you why a week ago. Your script most likely has an endless loop in it that does not check the condition session:ready() <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">However, Feel free to contact <a href="mailto:consulting@freeswitch.org">
consulting@freeswitch.org</a> and we can formally diagnose it for a fee.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Tue, Mar 21, 2017 at 6:42 AM Grant Bagdasarian <<a href="mailto:gb@cm.nl">gb@cm.nl</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Attached are the logs for:</span></span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<pre>#check thread usage:<o:p></o:p></pre>
<pre>top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>#memory mapping:<o:p></o:p></pre>
<pre>cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps<o:p></o:p></pre>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span></span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">We’re unable to run gcore since freeswitch hogs all available cpu’s at 100%.
</span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">I tried changing the nice settings real-time but that didn’t work either.</span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">At one point we had a huge spike in inbound traffic towards our freeswitch (valid traffic).</span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">The traffic is handled by a lua application.</span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">What we saw was calls hanging in freeswitch while they’ve already been physically disconnected.
</span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">After the huge spike freeswitch behaved normally for a couple of minutes and then suddenly
started eating all the available cpu cores at 100%. </span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">We have some db queries which are fired in the lua application, so these may cause the application
to hang, but we haven’t reproduced it yet. </span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">I’ll setup a test lab with the same configuration and fire sipp at it to see if I can reproduce
the issue. </span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">However, I probably won’t be able to run gcore for debugging. Any other suggestions?</span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Also we saw a lot of open udp ports and some had a huge read buffer size which did not decrease.
</span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><img border="0" width="924" height="436" style="width:9.625in;height:4.5416in" id="m_3623806029287245330Picture_x0020_2" src="cid:image001.png@01D2A238.14965330"></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span></span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b></span><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">
<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a> [mailto:<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a>]
<b>On Behalf Of </b>Grant Bagdasarian</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><br>
<span class="gmailmsg"><b>Sent:</b> maandag 13 maart 2017 09:04</span></span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><br>
<span class="gmailmsg"><b>To:</b> FreeSWITCH Users Help <<a href="mailto:freeswitch-users@lists.freeswitch.org" target="_blank">freeswitch-users@lists.freeswitch.org</a>></span><br>
<span class="gmailmsg"><b>Subject:</b> Re: [Freeswitch-users] 100% CPU usage</span></span><o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span lang="NL" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Hi Antony and Antonio,</span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span lang="NL" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Thank you for the tips.</span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">We haven’t had freeswitch going berserk anymore (not sure as to why it did perform without
issue over the weekend), but will continue to monitor it and run the commands to trace the issue.</span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Regards,</span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Grant</span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b></span><span class="gmailmsg"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">
<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a> [<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">mailto:freeswitch-users-bounces@lists.freeswitch.org</a>]
<b>On Behalf Of </b>Anthony Minessale</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><br>
<span class="gmailmsg"><b>Sent:</b> zaterdag 11 maart 2017 00:07</span><br>
<span class="gmailmsg"><b>To:</b> FreeSWITCH Users Help <<a href="mailto:freeswitch-users@lists.freeswitch.org" target="_blank">freeswitch-users@lists.freeswitch.org</a>></span><br>
<span class="gmailmsg"><b>Subject:</b> Re: [Freeswitch-users] 100% CPU usage</span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Make sure you lua script has all while loops including session:ready() as a condition to continue. Its a common mistake that leads to loops spinning out of control on hungup channels.<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Fri, Mar 10, 2017 at 6:11 AM, Antonio Silva <<a href="mailto:asilva@wirelessmundi.com" target="_blank">asilva@wirelessmundi.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">Hi,<br>
<br>
You can use the following commands:<o:p></o:p></p>
<pre>#get a core:<o:p></o:p></pre>
<pre>gcore -o /home/gcore-$(date +%s) $(cat /var/run/freeswitch/freeswitch.pid)<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>#check thread usage:<o:p></o:p></pre>
<pre>top -p $(cat /var/run/freeswitch/freeswitch.pid) -H -b -n 1<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>#memory mapping:<o:p></o:p></pre>
<pre>cat /proc/$(cat /var/run/freeswitch/freeswitch.pid)/maps<o:p></o:p></pre>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><br>
<br>
you can get some info from fs:<o:p></o:p></p>
<pre>fs_cli -x "sofia global siptrace on"<o:p></o:p></pre>
<pre>fs_cli -x "fsctl debug_level 10"<o:p></o:p></pre>
<pre>fs_cli -x "sofia loglevel all 9"<o:p></o:p></pre>
<pre>fs_cli -x "fsctl debug_sql "<o:p></o:p></pre>
<pre>fs_cli -x "sofia global debug presence"<o:p></o:p></pre>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br>
<br>
Put everything into a jira so we can take a look.<br>
<br>
<br>
Regards,<br>
António<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On 03/10/2017 11:08 AM, Grant Bagdasarian wrote:<o:p></o:p></p>
</div>
</div>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span lang="NL">Hi,</span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span lang="NL"> </span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">We’re currently experiencing a weird issue with one of our freeswitch servers where the CPU just goes berserk and stays at 100%.
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">There are no active calls at that moment, but FS is still using 100% CPU.
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I was unable to get a debug using gdb, since freeswitch does not crash.
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">We had this issue on 1.6.13 and then I upgraded to 1.6.15 this morning, but the problem still persists.
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">There is currently only one active application running on this box which is using LUA. Maybe that is causing the issue?<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">We checked the code but there are no weird loops etc. Perhaps we are missing something?<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Any ideas how to trace this?<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Regards,<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Grant<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"> <o:p></o:p></p>
</div>
</div>
<pre>_________________________________________________________________________<o:p></o:p></pre>
<pre>Professional FreeSWITCH Consulting Services: <o:p></o:p></pre>
<pre><a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><o:p></o:p></pre>
<pre><a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>Official FreeSWITCH Sites<o:p></o:p></pre>
<pre><a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><o:p></o:p></pre>
<pre><a href="http://confluence.freeswitch.org" target="_blank">http://confluence.freeswitch.org</a><o:p></o:p></pre>
<pre><a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>FreeSWITCH-users mailing list<o:p></o:p></pre>
<pre><a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><o:p></o:p></pre>
<pre><a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><o:p></o:p></pre>
<pre>UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><o:p></o:p></pre>
<pre><a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span class="gmailmsg"><span style="color:#888888"> </span></span><o:p></o:p></p>
<pre><span class="gmailmsg"><span style="color:#888888">-- </span></span><o:p></o:p></pre>
<pre><span class="gmailmsg"><span style="color:#888888"> </span></span><o:p></o:p></pre>
<pre><span class="gmailmsg"><span style="color:#888888">Saludos / Regards / Cumprimentos,</span></span><o:p></o:p></pre>
<pre><span class="gmailmsg"><span style="color:#888888">António silva</span></span><o:p></o:p></pre>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br>
_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://confluence.freeswitch.org" target="_blank">http://confluence.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br>
<br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">--
<o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Anthony Minessale II
<span class="gmailmsg"><span style="font-family:"Segoe UI Symbol",sans-serif">♬</span></span> @anthmfs <span class="gmailmsg"><span style="font-family:"Segoe UI Symbol",sans-serif">♬</span></span> @FreeSWITCH <span class="gmailmsg"><span style="font-family:"Segoe UI Symbol",sans-serif">♬</span></span><o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-family:"Segoe UI Symbol",sans-serif">☞</span></span>
<a href="http://freeswitch.org/" target="_blank">http://freeswitch.org/</a> <span class="gmailmsg"><span style="font-family:"Segoe UI Symbol",sans-serif">☞</span></span>
<a href="http://cluecon.com/" target="_blank">http://cluecon.com/</a> <span class="gmailmsg"><span style="font-family:"Segoe UI Symbol",sans-serif">☞</span></span>
<a href="http://twitter.com/FreeSWITCH" target="_blank">http://twitter.com/FreeSWITCH</a><o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span class="gmailmsg"><span style="font-family:"Segoe UI Symbol",sans-serif">☞</span></span>
<a href="http://irc.freenode.net" target="_blank">irc.freenode.net</a> #freeswitch <span class="gmailmsg"><span style="font-family:"Segoe UI Symbol",sans-serif">☞</span></span>
<u><a href="http://freeswitch.org/g+" target="_blank">http://freeswitch.org/g+</a></u><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">ClueCon Weekly Development Call <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="gmailmsg"><span style="font-family:"Segoe UI Emoji",sans-serif">☎</span></span> <a href="mailto:sip%3A888@conference.freeswitch.org" target="_blank">sip:888@conference.freeswitch.org</a>
<span class="gmailmsg"><span style="font-family:"Segoe UI Emoji",sans-serif">☎</span></span> +19193869900 <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><a href="https://www.youtube.com/watch?v=9XXgW34t40s" target="_blank"><span class="gmailmsg"><span style="font-size:9.5pt;color:#1155CC">https://www.youtube.com/watch?v=9XXgW34t40s</span></span></a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><a href="https://www.youtube.com/watch?v=NLaDpGQuZDA" target="_blank">https://www.youtube.com/watch?v=NLaDpGQuZDA</a><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal">_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://confluence.freeswitch.org" target="_blank">http://confluence.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><o:p></o:p></p>
</blockquote>
</div>
</div>
<div>
<p class="MsoNormal">-- <o:p></o:p></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Anthony Minessale II <span style="font-family:"Segoe UI Symbol",sans-serif">
♬</span> @anthmfs <span style="font-family:"Segoe UI Symbol",sans-serif">♬</span> @FreeSWITCH <span style="font-family:"Segoe UI Symbol",sans-serif">♬</span><o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="font-family:"Segoe UI Symbol",sans-serif">☞</span>
<a href="http://freeswitch.org/" target="_blank">http://freeswitch.org/</a> <span style="font-family:"Segoe UI Symbol",sans-serif">☞</span>
<a href="http://cluecon.com/" target="_blank">http://cluecon.com/</a> <span style="font-family:"Segoe UI Symbol",sans-serif">☞</span>
<a href="http://twitter.com/FreeSWITCH" target="_blank">http://twitter.com/FreeSWITCH</a><o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-family:"Segoe UI Symbol",sans-serif">☞</span>
<a href="http://irc.freenode.net" target="_blank">irc.freenode.net</a> #freeswitch <span style="font-family:"Segoe UI Symbol",sans-serif">☞</span>
<u><a href="http://freeswitch.org/g+" target="_blank">http://freeswitch.org/g+</a></u><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">ClueCon Weekly Development Call <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Segoe UI Emoji",sans-serif">☎</span> <a href="mailto:sip%3A888@conference.freeswitch.org" target="_blank">sip:888@conference.freeswitch.org</a> <span style="font-family:"Segoe UI Emoji",sans-serif">☎</span> +19193869900 <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><a href="https://www.youtube.com/watch?v=9XXgW34t40s" target="_blank"><span style="font-size:9.5pt;color:#1155CC">https://www.youtube.com/watch?v=9XXgW34t40s</span></a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="https://www.youtube.com/watch?v=NLaDpGQuZDA" target="_blank">https://www.youtube.com/watch?v=NLaDpGQuZDA</a><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>