<div dir="ltr"><div>Shaun,</div><div><br></div>800 concurrent with 400cps, expectations are a little out of touch with reality and your test is no where near real world.  Your call duration is ~2000ms, little to no time to actually do anything useful.  You&#39;re burning the bulk of your CPU time on signaling and doing no real work with the session.<div><br></div><div>So yes, you&#39;ll always get the best results out of bare metal, but you can also get good performance on a VM (*)</div><div><br></div><div>(*) This is in no way an endorsement of running FreeSWITCH in a VM, Your mileage may vary depending on your specific application and load requirements.<br></div><div><br></div><div>Thanks,</div><div><br></div><div><div><br></div><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Sep 1, 2015 at 3:08 PM, Shaun Stokes <span dir="ltr">&lt;<a href="mailto:shaun.stokes@itec-support.co.uk" target="_blank">shaun.stokes@itec-support.co.uk</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Update<br>
<br>
We&#39;ve performed various tests and have come to the conclusion as advised by others, that visualization is causing bottlenecks.<br>
<br>
Using Amazon EC2 on an 8 core high performance virtual server this maxed out at around 800 concurrent sessions, 400 sessions per second.<br>
<br>
Using a Dell 1950 8 core server this was able to handle 1000 concurrent sessions, 400 sessions per second and still have plenty of resource available. Average CPU was fluctuating between 30-40% under peak load.<br>
<br>
Conclusion, if you want to run a high performance FreeSwitch server in a production you should stick with physical hardware and avoid visualization where possible.<br>
________________________________________<br>
From: <a href="mailto:freeswitch-users-bounces@lists.freeswitch.org">freeswitch-users-bounces@lists.freeswitch.org</a> [<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org">freeswitch-users-bounces@lists.freeswitch.org</a>] on behalf of Shaun Stokes [<a href="mailto:shaun.stokes@itec-support.co.uk">shaun.stokes@itec-support.co.uk</a>]<br>
Sent: 28 August 2015 15:37<br>
To: &#39;FreeSWITCH Users Help&#39;<br>
<div class="HOEnZb"><div class="h5">Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues<br>
<br>
Hi Nathan,<br>
<br>
Thanks for the suggestion, we still see the CPU spikes when is under load after reducing to 8 cores but I think there may actually be an improvement in the audio quality.<br>
<br>
It&#39;s still not what we&#39;re expecting, so we will have to test our build on physical hardware too.<br>
<br>
Thanks,<br>
Shaun<br>
<br>
-----Original Message-----<br>
From: <a href="mailto:freeswitch-users-bounces@lists.freeswitch.org">freeswitch-users-bounces@lists.freeswitch.org</a> [mailto:<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org">freeswitch-users-bounces@lists.freeswitch.org</a>] On Behalf Of Nathan Neulinger<br>
Sent: 28 August 2015 15:04<br>
To: FreeSWITCH Users Help &lt;<a href="mailto:freeswitch-users@lists.freeswitch.org">freeswitch-users@lists.freeswitch.org</a>&gt;<br>
Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues<br>
<br>
Yes, I know you said Hyper-V, I don&#39;t know if it has the same sort of scheduler behavior or not, just mentioned the vmware for context.<br>
<br>
On 08/28/2015 08:46 AM, Nathan Neulinger wrote:<br>
&gt; One thing that I see jump out - you&#39;ve probably got too many virtual<br>
&gt; cores defined on that VM, particularly since you&#39;ve assigned # vcpu the same as the number of real cores on the host system.<br>
&gt;<br>
&gt; Try your test again dropping that to 8 virtual CPUs and I think you<br>
&gt; might see better results. (Or at least drop to less than 100% of the<br>
&gt; total cores on the hardware.)<br>
&gt;<br>
&gt; Particularly if this host system has other vms running on it, you&#39;re<br>
&gt; hurting the vm scheduler&#39;s ability to hand out resources. With vmware,<br>
&gt; I believe a simple rule of thumb is that scheduler requires N-1 CPUs<br>
&gt; to be locked to schedule execution of an N cpu virtual machine - so even if it doesn&#39;t need all of the CPUs, it has to reserve all of them any time you have more than a 2 vCPU vm.<br>
&gt;<br>
&gt; -- Nathan<br>
&gt;<br>
&gt; On 08/28/2015 08:18 AM, Michael Jerris wrote:<br>
&gt;&gt; The numbers your posting are fairly low, I&#39;m guessing this is a<br>
&gt;&gt; grossly underpowered vm that is causing your issues<br>
&gt;&gt;<br>
&gt;&gt; On Friday, August 28, 2015, Shaun Stokes &lt;<a href="mailto:shaun.stokes@itec-support.co.uk">shaun.stokes@itec-support.co.uk</a> &lt;mailto:<a href="mailto:shaun.stokes@itec-support.co.uk">shaun.stokes@itec-support.co.uk</a>&gt;&gt; wrote:<br>
&gt;&gt;<br>
&gt;&gt;      Hi Peter,____<br>
&gt;&gt;<br>
&gt;&gt;      __ __<br>
&gt;&gt;<br>
&gt;&gt;      Thanks for the advice, this is something we&#39;re already looking into but we don&#39;t have the new hardware available<br>
&gt;&gt;      yet.____<br>
&gt;&gt;<br>
&gt;&gt;      __ __<br>
&gt;&gt;<br>
&gt;&gt;      Debian 8 has great integration with Hyper-V on 2012R2 and operates as a Generation 2 VM, obviously it&#39;s never going<br>
&gt;&gt;      to be as good as running directly on the hardware but we were<br>
&gt;&gt; hoping for better performance.____<br>
&gt;&gt;<br>
&gt;&gt;      __ __<br>
&gt;&gt;<br>
&gt;&gt;      Thanks,____<br>
&gt;&gt;<br>
&gt;&gt;      Shaun____<br>
&gt;&gt;<br>
&gt;&gt;      __ __<br>
&gt;&gt;<br>
&gt;&gt;      *From:*<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org">freeswitch-users-bounces@lists.freeswitch.org</a><br>
&gt;&gt;      &lt;javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org">freeswitch-users-bounces@lists.freeswitch.org</a>&#39;);&gt;<br>
&gt;&gt;      [mailto:<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org">freeswitch-users-bounces@lists.freeswitch.org</a><br>
&gt;&gt;      &lt;javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org">freeswitch-users-bounces@lists.freeswitch.org</a>&#39;);&gt;] *On Behalf Of *Peter Olsson<br>
&gt;&gt;      *Sent:* 28 August 2015 10:43<br>
&gt;&gt;      *To:* FreeSWITCH Users Help &lt;<a href="mailto:freeswitch-users@lists.freeswitch.org">freeswitch-users@lists.freeswitch.org</a><br>
&gt;&gt;      &lt;javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;<a href="mailto:freeswitch-users@lists.freeswitch.org">freeswitch-users@lists.freeswitch.org</a>&#39;);&gt;&gt;<br>
&gt;&gt;      *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance<br>
&gt;&gt; issues____<br>
&gt;&gt;<br>
&gt;&gt;      __ __<br>
&gt;&gt;<br>
&gt;&gt;      If you want high performance, I recommend using real hardware, not a virtual machine setup - it will cause you<br>
&gt;&gt;      issues.____<br>
&gt;&gt;<br>
&gt;&gt;      __ __<br>
&gt;&gt;<br>
&gt;&gt;      /Peter____<br>
&gt;&gt;<br>
&gt;&gt;      __ __<br>
&gt;&gt;<br>
&gt;&gt;      2015-08-28 9:44 GMT+02:00 Shaun Stokes &lt;<a href="mailto:shaun.stokes@itec-support.co.uk">shaun.stokes@itec-support.co.uk</a><br>
&gt;&gt;<br>
&gt;&gt; &lt;javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;<a href="mailto:shaun.stokes@itec-support.co.uk">shaun.stokes@itec-support.co.uk</a>&#39;);&gt;&gt;:__<br>
&gt;&gt; __<br>
&gt;&gt;<br>
&gt;&gt;          Hi Michael,____<br>
&gt;&gt;<br>
&gt;&gt;          ____<br>
&gt;&gt;<br>
&gt;&gt;          Thanks for the response and recommendation.____<br>
&gt;&gt;<br>
&gt;&gt;          ____<br>
&gt;&gt;<br>
&gt;&gt;          Our new build is now on Debian 8 Jessie, performance on the system is noticeably better and any audio problems<br>
&gt;&gt;          while the system is under load are significantly reduced but we do still experience spikes along with reduced<br>
&gt;&gt;          audio quality when the system is processing 20 calls per<br>
&gt;&gt; second while already maintaining 80 existing calls.____<br>
&gt;&gt;<br>
&gt;&gt;          ____<br>
&gt;&gt;<br>
&gt;&gt;          We&#39;re still hitting some kind of bottleneck, what should we expect to be able to support on a single piece of<br>
&gt;&gt;          hardware? We&#39;ve kept our dialplans as small as possible and we&#39;re using Memcache, but something tells me it&#39;s<br>
&gt;&gt;          the call setup process which is triggering the spikes, is this possibly a limitation on the SIP profiles which<br>
&gt;&gt;          are single threaded?____<br>
&gt;&gt;<br>
&gt;&gt;          ____<br>
&gt;&gt;<br>
&gt;&gt;          I also see FreeSWITCH supports clustering but it&#39;s in relation to high availability, is it possible to cluster a<br>
&gt;&gt;          single FreeSWITCH instance across multiple hardware or should we be looking elsewhere to solve these<br>
&gt;&gt;          bottlenecks?____<br>
&gt;&gt;<br>
&gt;&gt;          ____<br>
&gt;&gt;<br>
&gt;&gt;          We switched to TCP Vegas on our new build as recommended on the FreeSWITCH Wiki, this has provided a noticeable<br>
&gt;&gt;          reduction in audio latency but doesn&#39;t solve the issue with<br>
&gt;&gt; CPU spikes.____<br>
&gt;&gt;<br>
&gt;&gt;          ____<br>
&gt;&gt;<br>
&gt;&gt;          Many Thanks,____<br>
&gt;&gt;<br>
&gt;&gt;          Shaun____<br>
&gt;&gt;<br>
&gt;&gt;          ____<br>
&gt;&gt;<br>
&gt;&gt;          *From:*<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org">freeswitch-users-bounces@lists.freeswitch.org</a><br>
&gt;&gt;          &lt;javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org">freeswitch-users-bounces@lists.freeswitch.org</a>&#39;);&gt;<br>
&gt;&gt;          [mailto:<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org">freeswitch-users-bounces@lists.freeswitch.org</a><br>
&gt;&gt;          &lt;javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org">freeswitch-users-bounces@lists.freeswitch.org</a>&#39;);&gt;] *On Behalf Of *Michael Jerris<br>
&gt;&gt;          *Sent:* 15 August 2015 23:00<br>
&gt;&gt;          *To:* FreeSWITCH Users Help &lt;<a href="mailto:freeswitch-users@lists.freeswitch.org">freeswitch-users@lists.freeswitch.org</a><br>
&gt;&gt;          &lt;javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;<a href="mailto:freeswitch-users@lists.freeswitch.org">freeswitch-users@lists.freeswitch.org</a>&#39;);&gt;&gt;<br>
&gt;&gt;          *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance<br>
&gt;&gt; issues____<br>
&gt;&gt;<br>
&gt;&gt;          ____<br>
&gt;&gt;<br>
&gt;&gt;          There are multiple known issues with Ubuntu 12.04.  We reccomend Debian 8 Jessie.<br>
&gt;&gt;<br>
&gt;&gt;          On Friday, August 14, 2015, Shaun Stokes &lt;<a href="mailto:shaun.stokes@itec-support.co.uk">shaun.stokes@itec-support.co.uk</a><br>
&gt;&gt;<br>
&gt;&gt; &lt;javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;<a href="mailto:shaun.stokes@itec-support.co.uk">shaun.stokes@itec-support.co.uk</a>&#39;);&gt;&gt;<br>
&gt;&gt; wrote:____<br>
&gt;&gt;<br>
&gt;&gt;              Hi,____<br>
&gt;&gt;<br>
&gt;&gt;              ____<br>
&gt;&gt;<br>
&gt;&gt;              We&#39;re experiencing performance issues with FreeSwitch, our target is 500 concurrent sessions, but at the<br>
&gt;&gt;              moment this starts to bottleneck around 30.____<br>
&gt;&gt;<br>
&gt;&gt;              ____<br>
&gt;&gt;<br>
&gt;&gt;              Host system:____<br>
&gt;&gt;<br>
&gt;&gt;              Windows Server 2012 Hyper-V Host____<br>
&gt;&gt;<br>
&gt;&gt;              AMD Opteron 4386 (2 processors) - 16 cores total____<br>
&gt;&gt;<br>
&gt;&gt;              128GB DDR3____<br>
&gt;&gt;<br>
&gt;&gt;              2TB RAID 5 (700MB/s tested read and write<br>
&gt;&gt; throughput)____<br>
&gt;&gt;<br>
&gt;&gt;              ____<br>
&gt;&gt;<br>
&gt;&gt;              FreeSwitch Virtual Machine:____<br>
&gt;&gt;<br>
&gt;&gt;              FreeSwitch 1.4.15____<br>
&gt;&gt;<br>
&gt;&gt;              Ubuntu 12.04 LTS____<br>
&gt;&gt;<br>
&gt;&gt;              16 Virtual cores (high priority)____<br>
&gt;&gt;<br>
&gt;&gt;              2GB RAM (would assign more but FreeSwitch never seems to<br>
&gt;&gt; use much)____<br>
&gt;&gt;<br>
&gt;&gt;              500GB HD (on VHDX)____<br>
&gt;&gt;<br>
&gt;&gt;              ____<br>
&gt;&gt;<br>
&gt;&gt;              After around 30 concurrent sessions we begin to see CPU spikes almost every time a new call comes in, as the<br>
&gt;&gt;              sessions increase the size and frequency of the CPU spikes also increase. The system seems to be able to sit<br>
&gt;&gt;              comfortably with over 100 concurrent sessions and 80% idle CPU, providing we don&#39;t have any new calls<br>
&gt;&gt;              hitting the platform. The spikes are causing audio (RTP) to stutter or in some cases drop completely for a<br>
&gt;&gt;              few seconds.____<br>
&gt;&gt;<br>
&gt;&gt;              ____<br>
&gt;&gt;<br>
&gt;&gt;              The FreeSwitch spikes are occurring on all 16 cores, we have been monitoring the system using htop and<br>
&gt;&gt;              mpstat.____<br>
&gt;&gt;<br>
&gt;&gt;              ____<br>
&gt;&gt;<br>
&gt;&gt;              This is an example of when we receive an inbound call<br>
&gt;&gt; while we have 34 concurrent sessions:____<br>
&gt;&gt;<br>
&gt;&gt;              14:26:04     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle____<br>
&gt;&gt;<br>
&gt;&gt;              14:26:05     all    1.67    0.00   27.34    0.00    0.25    0.00    0.00    0.00   70.74____<br>
&gt;&gt;<br>
&gt;&gt;              14:26:05       0    1.00    0.00   28.00    0.00    0.00    0.00    0.00    0.00   71.00____<br>
&gt;&gt;<br>
&gt;&gt;              14:26:05       1    0.99    0.00   26.73    0.00    0.00    0.00    0.00    0.00   72.28____<br>
&gt;&gt;<br>
&gt;&gt;              14:26:05       2    0.99    0.00   26.73    0.00    0.00    0.00    0.00    0.00   72.28____<br>
&gt;&gt;<br>
&gt;&gt;              14:26:05       3    3.92    0.00   26.47    0.00    0.00    0.00    0.00    0.00   69.61____<br>
&gt;&gt;<br>
&gt;&gt;              14:26:05       4    0.99    0.00   25.74    0.00    1.98    0.00    0.00    0.00   71.29____<br>
&gt;&gt;<br>
&gt;&gt;              14:26:05       5    0.99    0.00   26.73    0.00    0.99    0.00    0.00    0.00   71.29____<br>
&gt;&gt;<br>
&gt;&gt;              14:26:05       6    0.00    0.00   26.47    0.00    0.98    0.00    0.00    0.00   72.55____<br>
&gt;&gt;<br>
&gt;&gt;              14:26:05       7    0.00    0.00   26.73    0.00    0.00    0.00    0.00    0.00   73.27____<br>
&gt;&gt;<br>
&gt;&gt;              14:26:05       8    3.96    0.00   27.72    0.00    0.00    0.00    0.00    0.00   68.32____<br>
&gt;&gt;<br>
&gt;&gt;              14:26:05       9    2.00    0.00   27.00    0.00    0.00    0.00    0.00    0.00   71.00____<br>
&gt;&gt;<br>
&gt;&gt;              14:26:05      10    6.00    0.00   32.00    0.00    0.00    0.00    0.00    0.00   62.00____<br>
&gt;&gt;<br>
&gt;&gt;              14:26:05      11    1.96    0.00   27.45    0.00    0.00    0.00    0.00    0.00   70.59____<br>
&gt;&gt;<br>
&gt;&gt;              14:26:05      12    0.99    0.00   26.73    0.00    0.00    0.00    0.00    0.00   72.28____<br>
&gt;&gt;<br>
&gt;&gt;              14:26:05      13    1.00    0.00   28.00    0.00    0.00    0.00    0.00    0.00   71.00____<br>
&gt;&gt;<br>
&gt;&gt;              14:26:05      14    0.00    0.00   27.00    0.00    0.00    0.00    0.00    0.00   73.00____<br>
&gt;&gt;<br>
&gt;&gt;              ____<br>
&gt;&gt;<br>
&gt;&gt;              This is when the system is not receiving an inbound call but is sitting comfortably at 34 concurrent<br>
&gt;&gt;              sessions:____<br>
&gt;&gt;<br>
&gt;&gt;              14:25:57     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle____<br>
&gt;&gt;<br>
&gt;&gt;              14:25:58     all    0.87    0.00    0.62    0.00    0.12    0.00    0.00    0.00   98.39____<br>
&gt;&gt;<br>
&gt;&gt;              14:25:58       0    0.99    0.00    0.99    0.00    0.00    0.00    0.00    0.00   98.02____<br>
&gt;&gt;<br>
&gt;&gt;              14:25:58       1    0.99    0.00    0.99    0.00    0.00    0.00    0.00    0.00   98.02____<br>
&gt;&gt;<br>
&gt;&gt;              14:25:58       2    0.98    0.00    0.98    0.00    0.00    0.00    0.00    0.00   98.04____<br>
&gt;&gt;<br>
&gt;&gt;              14:25:58       3    0.00    0.00    0.99    0.00    0.00    0.00    0.00    0.00   99.01____<br>
&gt;&gt;<br>
&gt;&gt;              14:25:58       4    0.00    0.00    0.00    0.00    1.98    0.00    0.00    0.00   98.02____<br>
&gt;&gt;<br>
&gt;&gt;              14:25:58       5    0.00    0.00    0.00    0.00    0.99    0.00    0.00    0.00   99.01____<br>
&gt;&gt;<br>
&gt;&gt;              14:25:58       6    0.00    0.00    0.99    0.00    0.00    0.00    0.00    0.00   99.01____<br>
&gt;&gt;<br>
&gt;&gt;              14:25:58       7    0.00    0.00    0.99    0.00    0.00    0.00    0.00    0.00   99.01____<br>
&gt;&gt;<br>
&gt;&gt;              14:25:58       8    5.00    0.00    1.00    0.00    0.00    0.00    0.00    0.00   94.00____<br>
&gt;&gt;<br>
&gt;&gt;              14:25:58       9    1.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   99.00____<br>
&gt;&gt;<br>
&gt;&gt;              14:25:58      10    1.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   99.00____<br>
&gt;&gt;<br>
&gt;&gt;              14:25:58      11    0.99    0.00    0.99    0.00    0.00    0.00    0.00    0.00   98.02____<br>
&gt;&gt;<br>
&gt;&gt;              14:25:58      12    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00____<br>
&gt;&gt;<br>
&gt;&gt;              14:25:58      13    1.98    0.00    0.99    0.00    0.00    0.00    0.00    0.00   97.03____<br>
&gt;&gt;<br>
&gt;&gt;              14:25:58      14    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00____<br>
&gt;&gt;<br>
&gt;&gt;              14:25:58      15    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00____<br>
&gt;&gt;<br>
&gt;&gt;              ____<br>
&gt;&gt;<br>
&gt;&gt;              We&#39;re using FreeSwitch in multi-tenant mode, we have tried dedicating a single VM for one tenancy but still<br>
&gt;&gt;              experience the issue. My assumption has been that this will be due to the dialplan, I&#39;ve optimized this<br>
&gt;&gt;              slightly by writing LUA scripts to handle some of the inbound calls this appears to take away some of the<br>
&gt;&gt;              load but we&#39;re still using the internal dialplans for each tenancy (tenancy dialplans have an average of 300<br>
&gt;&gt;              entries).____<br>
&gt;&gt;<br>
&gt;&gt;              ____<br>
&gt;&gt;<br>
&gt;&gt;              We are using the following arguments when running<br>
&gt;&gt; FreeSwitch:____<br>
&gt;&gt;<br>
&gt;&gt;              -rp -nc -nonat____<br>
&gt;&gt;<br>
&gt;&gt;              ____<br>
&gt;&gt;<br>
&gt;&gt;              I&#39;ve seen another post which says we should operate FreeSwitch in High Performance mode using -hp but can&#39;t<br>
&gt;&gt;              find anything information about this, is this still a<br>
&gt;&gt; valid argument to use with FreeSwitch?____<br>
&gt;&gt;<br>
&gt;&gt;              ____<br>
&gt;&gt;<br>
&gt;&gt;              Has anyone experienced similar performance issues before<br>
&gt;&gt; or have any advice?____<br>
&gt;&gt;<br>
&gt;&gt;              ____<br>
&gt;&gt;<br>
&gt;&gt;              Hope someone may be able help.____<br>
&gt;&gt;<br>
&gt;&gt;              ____<br>
&gt;&gt;<br>
&gt;&gt;              Thanks,____<br>
&gt;&gt;<br>
&gt;&gt;              Shaun____<br>
&gt;&gt;<br>
&gt;&gt;<br>
&gt;&gt;              ______________________________________________________________________<br>
&gt;&gt;              This message has been checked for all known viruses by MessageLabs Virus Scanning Service.<br>
&gt;&gt;<br>
&gt;&gt; _____________________________________________________________________<br>
&gt;&gt; _____<br>
&gt;&gt;<br>
&gt;&gt;<br>
&gt;&gt;          ______________________________________________________________________<br>
&gt;&gt;          This message has been checked for all known viruses by MessageLabs Virus Scanning Service.<br>
&gt;&gt;<br>
&gt;&gt; _____________________________________________________________________<br>
&gt;&gt; _____<br>
&gt;&gt;<br>
&gt;&gt;<br>
&gt;&gt;          ______________________________________________________________________<br>
&gt;&gt;          This message has been checked for all known viruses by MessageLabs Virus Scanning Service.<br>
&gt;&gt;<br>
&gt;&gt; _____________________________________________________________________<br>
&gt;&gt; _____<br>
&gt;&gt;<br>
&gt;&gt;<br>
&gt;&gt;          _________________________________________________________________________<br>
&gt;&gt;          Professional FreeSWITCH Consulting Services:<br>
&gt;&gt;          <a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a> &lt;javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;<a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a>&#39;);&gt;<br>
&gt;&gt;          <a href="http://www.freeswitchsolutions.com" rel="noreferrer" target="_blank">http://www.freeswitchsolutions.com</a><br>
&gt;&gt;<br>
&gt;&gt;          Official FreeSWITCH Sites<br>
&gt;&gt;          <a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
&gt;&gt;          <a href="http://confluence.freeswitch.org" rel="noreferrer" target="_blank">http://confluence.freeswitch.org</a><br>
&gt;&gt;          <a href="http://www.cluecon.com" rel="noreferrer" target="_blank">http://www.cluecon.com</a><br>
&gt;&gt;<br>
&gt;&gt;          FreeSWITCH-users mailing list<br>
&gt;&gt;          <a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a> &lt;javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;<a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a>&#39;);&gt;<br>
&gt;&gt;          <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
&gt;&gt;          UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
&gt;&gt;          <a href="http://www.freeswitch.org____" rel="noreferrer" target="_blank">http://www.freeswitch.org____</a><br>
&gt;&gt;<br>
&gt;&gt;      __ __<br>
&gt;&gt;<br>
&gt;&gt;<br>
&gt;&gt;      ______________________________________________________________________<br>
&gt;&gt;      This message has been checked for all known viruses by MessageLabs Virus Scanning Service.<br>
&gt;&gt;<br>
&gt;&gt; _____________________________________________________________________<br>
&gt;&gt; _____<br>
&gt;&gt;<br>
&gt;&gt;<br>
&gt;&gt;      ______________________________________________________________________<br>
&gt;&gt;      This message has been checked for all known viruses by MessageLabs Virus Scanning Service.<br>
&gt;&gt;<br>
&gt;&gt; _____________________________________________________________________<br>
&gt;&gt; _<br>
&gt;&gt;<br>
&gt;&gt;<br>
&gt;&gt;<br>
&gt;&gt; _____________________________________________________________________<br>
&gt;&gt; ____ Professional FreeSWITCH Consulting Services:<br>
&gt;&gt; <a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
&gt;&gt; <a href="http://www.freeswitchsolutions.com" rel="noreferrer" target="_blank">http://www.freeswitchsolutions.com</a><br>
&gt;&gt;<br>
&gt;&gt; Official FreeSWITCH Sites<br>
&gt;&gt; <a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
&gt;&gt; <a href="http://confluence.freeswitch.org" rel="noreferrer" target="_blank">http://confluence.freeswitch.org</a><br>
&gt;&gt; <a href="http://www.cluecon.com" rel="noreferrer" target="_blank">http://www.cluecon.com</a><br>
&gt;&gt;<br>
&gt;&gt; FreeSWITCH-users mailing list<br>
&gt;&gt; <a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
&gt;&gt; <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
&gt;&gt; UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-us" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-us</a><br>
&gt;&gt; ers<br>
&gt;&gt; <a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
&gt;&gt;<br>
&gt;<br>
<br>
--<br>
------------------------------------------------------------<br>
Nathan Neulinger                       <a href="mailto:nneul@mst.edu">nneul@mst.edu</a><br>
Missouri S&amp;T Information Technology    <a href="tel:%28573%29%20612-1412" value="+15736121412">(573) 612-1412</a><br>
System Administrator - Architect<br>
<br>
_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" rel="noreferrer" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://confluence.freeswitch.org" rel="noreferrer" target="_blank">http://confluence.freeswitch.org</a><br>
<a href="http://www.cluecon.com" rel="noreferrer" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
<br>
______________________________________________________________________<br>
This message has been checked for all known viruses by MessageLabs Virus Scanning Service.<br>
______________________________________________________________________<br>
<br>
______________________________________________________________________<br>
This message has been checked for all known viruses by  MessageLabs Virus Scanning Service.<br>
______________________________________________________________________<br>
<br>
_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" rel="noreferrer" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://confluence.freeswitch.org" rel="noreferrer" target="_blank">http://confluence.freeswitch.org</a><br>
<a href="http://www.cluecon.com" rel="noreferrer" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
<br>
______________________________________________________________________<br>
This message has been checked for all known viruses by MessageLabs Virus Scanning Service.<br>
______________________________________________________________________<br>
<br>
______________________________________________________________________<br>
This message has been checked for all known viruses by  MessageLabs Virus Scanning Service.<br>
______________________________________________________________________<br>
<br>
_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" rel="noreferrer" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://confluence.freeswitch.org" rel="noreferrer" target="_blank">http://confluence.freeswitch.org</a><br>
<a href="http://www.cluecon.com" rel="noreferrer" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">







<p><font face="courier new, monospace"><b><i><font size="4">Brian West</font></i></b><br><span style="font-size:x-small"><a href="mailto:brian@freeswitch.org" target="_blank">brian@freeswitch.org</a></span></font></p>
<p><font size="1" face="courier new, monospace"><img src="http://billing.freeswitch.org/templates/default/img/whmcslogo.png"><br></font></p><p><font size="2" face="monospace, monospace"><b><i>Twitter: @FreeSWITCH , @briankwest</i></b><br><a href="http://www.freeswitchbook.com" target="_blank">http://www.freeswitchbook.com</a><br><a href="http://www.freeswitchcookbook.com" target="_blank">http://www.freeswitchcookbook.com</a></font></p><p><font face="monospace, monospace">Got Bugs? Report them <a href="https://freeswitch.org/jira" target="_blank">here</a>! | Reddit: <a href="https://www.reddit.com/r/freeswitch" target="_blank">/r/freeswitch</a></font></p>
<p><font size="2" face="monospace, monospace"><b>T:</b>+19184209001 | <b>F:</b>+19184209002 | <b>M:</b>+1918424WEST (9378)<br><b>iNUM:</b>+883 5100 1420 9001 | <b>ISN:</b>410*543 | <b>Skype:</b>briankwest</font></p></div></div></div></div></div></div></div></div></div></div>
</div>