<div dir="ltr">That's assuming 100% ASR. It might be acceptable to return 503 when too busy but still need to process those INVITEs to reject the call at that rate.</div><div class="gmail_extra"><br><div class="gmail_quote">On 2 September 2015 at 15:01, Brian West <span dir="ltr"><<a href="mailto:brian@freeswitch.org" target="_blank">brian@freeswitch.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">There is absolutely no way you can have 800 concurrent calls at 400cps, and have them last 10 seconds, Your math doesn't add up. 400cps with 10 seconds acd would be around 4000 concurrent sessions. <div><br><div>This is the exact reason we don't endorse any types of load testing. </div><div class="gmail_extra"><br></div><div class="gmail_extra">While we can do crazy things like that in specific test cases in a lab, real world is never that structured. Sorry but the glove doesn't fit!</div><span class="HOEnZb"><font color="#888888"><div class="gmail_extra"><br></div><div class="gmail_extra">/b</div></font></span><div><div class="h5"><div class="gmail_extra"><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Sep 2, 2015 at 3:53 AM, Shaun Stokes <span dir="ltr"><<a href="mailto:shaun.stokes@itec-support.co.uk" target="_blank">shaun.stokes@itec-support.co.uk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-GB" link="blue" vlink="purple">
<div>
<p class="MsoNormal">Hi Brian,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I’m a huge fan of virtualization, but in our environment it simply doesn’t provide the performance we’re looking for. The tests carried out were 10 second calls which were routed to a Call Centre Queue with one registered SIP extension
as a member.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">There was a significant improvement when we tested on an Amazon VM in comparison to Hyper-V, but we’d like a system which can handle large volumes of call traffic, based on our tests this will need to be physical hardware to achieve the
best performance.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Further to this, we have managed to resolve our initial problem with CPU spikes on Hyper-V following these steps:<u></u><u></u></p>
<p><u></u><span style="font-family:Symbol"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u>Disable VMQ on all physical and virtual adapters on the Hyper-V host which are linked to the FreeSwitch VM<u></u><u></u></p>
<p><u></u><span style="font-family:Symbol"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u>Reduce the virtual cores on all VMs to at least half of the available physical cores<u></u><u></u></p>
<p><u></u><span style="font-family:Symbol"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u>Increase the Processor Relative weight on the FreeSwitch VMs to 200 (default 100)<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
<p class="MsoNormal">Shaun<u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a> [mailto:<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a>]
<b>On Behalf Of </b>Brian West<br>
<b>Sent:</b> 01 September 2015 21:21</span></p><div><div><br>
<b>To:</b> FreeSWITCH Users Help <<a href="mailto:freeswitch-users@lists.freeswitch.org" target="_blank">freeswitch-users@lists.freeswitch.org</a>><br>
<b>Subject:</b> Re: [Freeswitch-users] FreeSwitch - Performance issues<u></u><u></u></div></div><p></p><div><div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">Shaun,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">800 concurrent with 400cps, expectations are a little out of touch with reality and your test is no where near real world. Your call duration is ~2000ms, little to no time to actually do anything useful. You're burning the bulk of your
CPU time on signaling and doing no real work with the session.<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">So yes, you'll always get the best results out of bare metal, but you can also get good performance on a VM (*)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">(*) This is in no way an endorsement of running FreeSWITCH in a VM, Your mileage may vary depending on your specific application and load requirements.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Tue, Sep 1, 2015 at 3:08 PM, Shaun Stokes <<a href="mailto:shaun.stokes@itec-support.co.uk" target="_blank">shaun.stokes@itec-support.co.uk</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal">Update<br>
<br>
We've performed various tests and have come to the conclusion as advised by others, that visualization is causing bottlenecks.<br>
<br>
Using Amazon EC2 on an 8 core high performance virtual server this maxed out at around 800 concurrent sessions, 400 sessions per second.<br>
<br>
Using a Dell 1950 8 core server this was able to handle 1000 concurrent sessions, 400 sessions per second and still have plenty of resource available. Average CPU was fluctuating between 30-40% under peak load.<br>
<br>
Conclusion, if you want to run a high performance FreeSwitch server in a production you should stick with physical hardware and avoid visualization where possible.<br>
________________________________________<br>
From: <a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a> [<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a>] on behalf of Shaun Stokes
[<a href="mailto:shaun.stokes@itec-support.co.uk" target="_blank">shaun.stokes@itec-support.co.uk</a>]<br>
Sent: 28 August 2015 15:37<br>
To: 'FreeSWITCH Users Help'<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues<br>
<br>
Hi Nathan,<br>
<br>
Thanks for the suggestion, we still see the CPU spikes when is under load after reducing to 8 cores but I think there may actually be an improvement in the audio quality.<br>
<br>
It's still not what we're expecting, so we will have to test our build on physical hardware too.<br>
<br>
Thanks,<br>
Shaun<br>
<br>
-----Original Message-----<br>
From: <a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a> [mailto:<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a>] On Behalf Of Nathan
Neulinger<br>
Sent: 28 August 2015 15:04<br>
To: FreeSWITCH Users Help <<a href="mailto:freeswitch-users@lists.freeswitch.org" target="_blank">freeswitch-users@lists.freeswitch.org</a>><br>
Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues<br>
<br>
Yes, I know you said Hyper-V, I don't know if it has the same sort of scheduler behavior or not, just mentioned the vmware for context.<br>
<br>
On 08/28/2015 08:46 AM, Nathan Neulinger wrote:<br>
> One thing that I see jump out - you've probably got too many virtual<br>
> cores defined on that VM, particularly since you've assigned # vcpu the same as the number of real cores on the host system.<br>
><br>
> Try your test again dropping that to 8 virtual CPUs and I think you<br>
> might see better results. (Or at least drop to less than 100% of the<br>
> total cores on the hardware.)<br>
><br>
> Particularly if this host system has other vms running on it, you're<br>
> hurting the vm scheduler's ability to hand out resources. With vmware,<br>
> I believe a simple rule of thumb is that scheduler requires N-1 CPUs<br>
> to be locked to schedule execution of an N cpu virtual machine - so even if it doesn't need all of the CPUs, it has to reserve all of them any time you have more than a 2 vCPU vm.<br>
><br>
> -- Nathan<br>
><br>
> On 08/28/2015 08:18 AM, Michael Jerris wrote:<br>
>> The numbers your posting are fairly low, I'm guessing this is a<br>
>> grossly underpowered vm that is causing your issues<br>
>><br>
>> On Friday, August 28, 2015, Shaun Stokes <<a href="mailto:shaun.stokes@itec-support.co.uk" target="_blank">shaun.stokes@itec-support.co.uk</a> <mailto:<a href="mailto:shaun.stokes@itec-support.co.uk" target="_blank">shaun.stokes@itec-support.co.uk</a>>> wrote:<br>
>><br>
>> Hi Peter,____<br>
>><br>
>> __ __<br>
>><br>
>> Thanks for the advice, this is something we're already looking into but we don't have the new hardware available<br>
>> yet.____<br>
>><br>
>> __ __<br>
>><br>
>> Debian 8 has great integration with Hyper-V on 2012R2 and operates as a Generation 2 VM, obviously it's never going<br>
>> to be as good as running directly on the hardware but we were<br>
>> hoping for better performance.____<br>
>><br>
>> __ __<br>
>><br>
>> Thanks,____<br>
>><br>
>> Shaun____<br>
>><br>
>> __ __<br>
>><br>
>> *From:*<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a><br>
>> <javascript:_e(%7B%7D,'cvml','<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a>');><br>
>> [mailto:<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a><br>
>> <javascript:_e(%7B%7D,'cvml','<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a>');>] *On Behalf Of *Peter Olsson<br>
>> *Sent:* 28 August 2015 10:43<br>
>> *To:* FreeSWITCH Users Help <<a href="mailto:freeswitch-users@lists.freeswitch.org" target="_blank">freeswitch-users@lists.freeswitch.org</a><br>
>> <javascript:_e(%7B%7D,'cvml','<a href="mailto:freeswitch-users@lists.freeswitch.org" target="_blank">freeswitch-users@lists.freeswitch.org</a>');>><br>
>> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance<br>
>> issues____<br>
>><br>
>> __ __<br>
>><br>
>> If you want high performance, I recommend using real hardware, not a virtual machine setup - it will cause you<br>
>> issues.____<br>
>><br>
>> __ __<br>
>><br>
>> /Peter____<br>
>><br>
>> __ __<br>
>><br>
>> 2015-08-28 9:44 GMT+02:00 Shaun Stokes <<a href="mailto:shaun.stokes@itec-support.co.uk" target="_blank">shaun.stokes@itec-support.co.uk</a><br>
>><br>
>> <javascript:_e(%7B%7D,'cvml','<a href="mailto:shaun.stokes@itec-support.co.uk" target="_blank">shaun.stokes@itec-support.co.uk</a>');>>:__<br>
>> __<br>
>><br>
>> Hi Michael,____<br>
>><br>
>> ____<br>
>><br>
>> Thanks for the response and recommendation.____<br>
>><br>
>> ____<br>
>><br>
>> Our new build is now on Debian 8 Jessie, performance on the system is noticeably better and any audio problems<br>
>> while the system is under load are significantly reduced but we do still experience spikes along with reduced<br>
>> audio quality when the system is processing 20 calls per<br>
>> second while already maintaining 80 existing calls.____<br>
>><br>
>> ____<br>
>><br>
>> We're still hitting some kind of bottleneck, what should we expect to be able to support on a single piece of<br>
>> hardware? We've kept our dialplans as small as possible and we're using Memcache, but something tells me it's<br>
>> the call setup process which is triggering the spikes, is this possibly a limitation on the SIP profiles which<br>
>> are single threaded?____<br>
>><br>
>> ____<br>
>><br>
>> I also see FreeSWITCH supports clustering but it's in relation to high availability, is it possible to cluster a<br>
>> single FreeSWITCH instance across multiple hardware or should we be looking elsewhere to solve these<br>
>> bottlenecks?____<br>
>><br>
>> ____<br>
>><br>
>> We switched to TCP Vegas on our new build as recommended on the FreeSWITCH Wiki, this has provided a noticeable<br>
>> reduction in audio latency but doesn't solve the issue with<br>
>> CPU spikes.____<br>
>><br>
>> ____<br>
>><br>
>> Many Thanks,____<br>
>><br>
>> Shaun____<br>
>><br>
>> ____<br>
>><br>
>> *From:*<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a><br>
>> <javascript:_e(%7B%7D,'cvml','<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a>');><br>
>> [mailto:<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a><br>
>> <javascript:_e(%7B%7D,'cvml','<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a>');>] *On Behalf Of *Michael Jerris<br>
>> *Sent:* 15 August 2015 23:00<br>
>> *To:* FreeSWITCH Users Help <<a href="mailto:freeswitch-users@lists.freeswitch.org" target="_blank">freeswitch-users@lists.freeswitch.org</a><br>
>> <javascript:_e(%7B%7D,'cvml','<a href="mailto:freeswitch-users@lists.freeswitch.org" target="_blank">freeswitch-users@lists.freeswitch.org</a>');>><br>
>> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance<br>
>> issues____<br>
>><br>
>> ____<br>
>><br>
>> There are multiple known issues with Ubuntu 12.04. We reccomend Debian 8 Jessie.<br>
>><br>
>> On Friday, August 14, 2015, Shaun Stokes <<a href="mailto:shaun.stokes@itec-support.co.uk" target="_blank">shaun.stokes@itec-support.co.uk</a><br>
>><br>
>> <javascript:_e(%7B%7D,'cvml','<a href="mailto:shaun.stokes@itec-support.co.uk" target="_blank">shaun.stokes@itec-support.co.uk</a>');>><br>
>> wrote:____<br>
>><br>
>> Hi,____<br>
>><br>
>> ____<br>
>><br>
>> We're experiencing performance issues with FreeSwitch, our target is 500 concurrent sessions, but at the<br>
>> moment this starts to bottleneck around 30.____<br>
>><br>
>> ____<br>
>><br>
>> Host system:____<br>
>><br>
>> Windows Server 2012 Hyper-V Host____<br>
>><br>
>> AMD Opteron 4386 (2 processors) - 16 cores total____<br>
>><br>
>> 128GB DDR3____<br>
>><br>
>> 2TB RAID 5 (700MB/s tested read and write<br>
>> throughput)____<br>
>><br>
>> ____<br>
>><br>
>> FreeSwitch Virtual Machine:____<br>
>><br>
>> FreeSwitch 1.4.15____<br>
>><br>
>> Ubuntu 12.04 LTS____<br>
>><br>
>> 16 Virtual cores (high priority)____<br>
>><br>
>> 2GB RAM (would assign more but FreeSwitch never seems to<br>
>> use much)____<br>
>><br>
>> 500GB HD (on VHDX)____<br>
>><br>
>> ____<br>
>><br>
>> After around 30 concurrent sessions we begin to see CPU spikes almost every time a new call comes in, as the<br>
>> sessions increase the size and frequency of the CPU spikes also increase. The system seems to be able to sit<br>
>> comfortably with over 100 concurrent sessions and 80% idle CPU, providing we don't have any new calls<br>
>> hitting the platform. The spikes are causing audio (RTP) to stutter or in some cases drop completely for a<br>
>> few seconds.____<br>
>><br>
>> ____<br>
>><br>
>> The FreeSwitch spikes are occurring on all 16 cores, we have been monitoring the system using htop and<br>
>> mpstat.____<br>
>><br>
>> ____<br>
>><br>
>> This is an example of when we receive an inbound call<br>
>> while we have 34 concurrent sessions:____<br>
>><br>
>> 14:26:04 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle____<br>
>><br>
>> 14:26:05 all 1.67 0.00 27.34 0.00 0.25 0.00 0.00 0.00 70.74____<br>
>><br>
>> 14:26:05 0 1.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 71.00____<br>
>><br>
>> 14:26:05 1 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____<br>
>><br>
>> 14:26:05 2 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____<br>
>><br>
>> 14:26:05 3 3.92 0.00 26.47 0.00 0.00 0.00 0.00 0.00 69.61____<br>
>><br>
>> 14:26:05 4 0.99 0.00 25.74 0.00 1.98 0.00 0.00 0.00 71.29____<br>
>><br>
>> 14:26:05 5 0.99 0.00 26.73 0.00 0.99 0.00 0.00 0.00 71.29____<br>
>><br>
>> 14:26:05 6 0.00 0.00 26.47 0.00 0.98 0.00 0.00 0.00 72.55____<br>
>><br>
>> 14:26:05 7 0.00 0.00 26.73 0.00 0.00 0.00 0.00 0.00 73.27____<br>
>><br>
>> 14:26:05 8 3.96 0.00 27.72 0.00 0.00 0.00 0.00 0.00 68.32____<br>
>><br>
>> 14:26:05 9 2.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00 71.00____<br>
>><br>
>> 14:26:05 10 6.00 0.00 32.00 0.00 0.00 0.00 0.00 0.00 62.00____<br>
>><br>
>> 14:26:05 11 1.96 0.00 27.45 0.00 0.00 0.00 0.00 0.00 70.59____<br>
>><br>
>> 14:26:05 12 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____<br>
>><br>
>> 14:26:05 13 1.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 71.00____<br>
>><br>
>> 14:26:05 14 0.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00 73.00____<br>
>><br>
>> ____<br>
>><br>
>> This is when the system is not receiving an inbound call but is sitting comfortably at 34 concurrent<br>
>> sessions:____<br>
>><br>
>> 14:25:57 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle____<br>
>><br>
>> 14:25:58 all 0.87 0.00 0.62 0.00 0.12 0.00 0.00 0.00 98.39____<br>
>><br>
>> 14:25:58 0 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____<br>
>><br>
>> 14:25:58 1 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____<br>
>><br>
>> 14:25:58 2 0.98 0.00 0.98 0.00 0.00 0.00 0.00 0.00 98.04____<br>
>><br>
>> 14:25:58 3 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____<br>
>><br>
>> 14:25:58 4 0.00 0.00 0.00 0.00 1.98 0.00 0.00 0.00 98.02____<br>
>><br>
>> 14:25:58 5 0.00 0.00 0.00 0.00 0.99 0.00 0.00 0.00 99.01____<br>
>><br>
>> 14:25:58 6 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____<br>
>><br>
>> 14:25:58 7 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____<br>
>><br>
>> 14:25:58 8 5.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 94.00____<br>
>><br>
>> 14:25:58 9 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00____<br>
>><br>
>> 14:25:58 10 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00____<br>
>><br>
>> 14:25:58 11 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____<br>
>><br>
>> 14:25:58 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____<br>
>><br>
>> 14:25:58 13 1.98 0.00 0.99 0.00 0.00 0.00 0.00 0.00 97.03____<br>
>><br>
>> 14:25:58 14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____<br>
>><br>
>> 14:25:58 15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____<br>
>><br>
>> ____<br>
>><br>
>> We're using FreeSwitch in multi-tenant mode, we have tried dedicating a single VM for one tenancy but still<br>
>> experience the issue. My assumption has been that this will be due to the dialplan, I've optimized this<br>
>> slightly by writing LUA scripts to handle some of the inbound calls this appears to take away some of the<br>
>> load but we're still using the internal dialplans for each tenancy (tenancy dialplans have an average of 300<br>
>> entries).____<br>
>><br>
>> ____<br>
>><br>
>> We are using the following arguments when running<br>
>> FreeSwitch:____<br>
>><br>
>> -rp -nc -nonat____<br>
>><br>
>> ____<br>
>><br>
>> I've seen another post which says we should operate FreeSwitch in High Performance mode using -hp but can't<br>
>> find anything information about this, is this still a<br>
>> valid argument to use with FreeSwitch?____<br>
>><br>
>> ____<br>
>><br>
>> Has anyone experienced similar performance issues before<br>
>> or have any advice?____<br>
>><br>
>> ____<br>
>><br>
>> Hope someone may be able help.____<br>
>><br>
>> ____<br>
>><br>
>> Thanks,____<br>
>><br>
>> Shaun____<br>
>><br>
>><br>
>> ______________________________________________________________________<br>
>> This message has been checked for all known viruses by MessageLabs Virus Scanning Service.<br>
>><br>
>> _____________________________________________________________________<br>
>> _____<br>
>><br>
>><br>
>> ______________________________________________________________________<br>
>> This message has been checked for all known viruses by MessageLabs Virus Scanning Service.<br>
>><br>
>> _____________________________________________________________________<br>
>> _____<br>
>><br>
>><br>
>> ______________________________________________________________________<br>
>> This message has been checked for all known viruses by MessageLabs Virus Scanning Service.<br>
>><br>
>> _____________________________________________________________________<br>
>> _____<br>
>><br>
>><br>
>> _________________________________________________________________________<br>
>> Professional FreeSWITCH Consulting Services:<br>
>> <a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a> <javascript:_e(%7B%7D,'cvml','<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a>');><br>
>> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
>><br>
>> Official FreeSWITCH Sites<br>
>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
>> <a href="http://confluence.freeswitch.org" target="_blank">http://confluence.freeswitch.org</a><br>
>> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
>><br>
>> FreeSWITCH-users mailing list<br>
>> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a> <javascript:_e(%7B%7D,'cvml','<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a>');><br>
>> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
>> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
>> <a href="http://www.freeswitch.org____" target="_blank">http://www.freeswitch.org____</a><br>
>><br>
>> __ __<br>
>><br>
>><br>
>> ______________________________________________________________________<br>
>> This message has been checked for all known viruses by MessageLabs Virus Scanning Service.<br>
>><br>
>> _____________________________________________________________________<br>
>> _____<br>
>><br>
>><br>
>> ______________________________________________________________________<br>
>> This message has been checked for all known viruses by MessageLabs Virus Scanning Service.<br>
>><br>
>> _____________________________________________________________________<br>
>> _<br>
>><br>
>><br>
>><br>
>> _____________________________________________________________________<br>
>> ____ Professional FreeSWITCH Consulting Services:<br>
>> <a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
>> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
>><br>
>> Official FreeSWITCH Sites<br>
>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
>> <a href="http://confluence.freeswitch.org" target="_blank">http://confluence.freeswitch.org</a><br>
>> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
>><br>
>> FreeSWITCH-users mailing list<br>
>> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
>> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
>> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-us" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-us</a><br>
>> ers<br>
>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
>><br>
><br>
<br>
--<br>
------------------------------------------------------------<br>
Nathan Neulinger <a href="mailto:nneul@mst.edu" target="_blank">nneul@mst.edu</a><br>
Missouri S&T Information Technology <a href="tel:%28573%29%20612-1412" target="_blank">(573) 612-1412</a><br>
System Administrator - Architect<br>
<br>
_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://confluence.freeswitch.org" target="_blank">http://confluence.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<br>
______________________________________________________________________<br>
This message has been checked for all known viruses by MessageLabs Virus Scanning Service.<br>
______________________________________________________________________<br>
<br>
______________________________________________________________________<br>
This message has been checked for all known viruses by MessageLabs Virus Scanning Service.<br>
______________________________________________________________________<br>
<br>
_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://confluence.freeswitch.org" target="_blank">http://confluence.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<br>
______________________________________________________________________<br>
This message has been checked for all known viruses by MessageLabs Virus Scanning Service.<br>
______________________________________________________________________<br>
<br>
______________________________________________________________________<br>
This message has been checked for all known viruses by MessageLabs Virus Scanning Service.<br>
______________________________________________________________________<br>
<br>
_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://confluence.freeswitch.org" target="_blank">http://confluence.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><u></u><u></u></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">-- <u></u><u></u></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p><b><i><span style="font-size:13.5pt;font-family:"Courier New"">Brian West</span></i></b><span style="font-family:"Courier New""><br>
</span><span style="font-size:10.0pt;font-family:"Courier New""><a href="mailto:brian@freeswitch.org" target="_blank">brian@freeswitch.org</a></span><u></u><u></u></p>
<p><span style="font-size:7.5pt;font-family:"Courier New""><img border="0" src="http://billing.freeswitch.org/templates/default/img/whmcslogo.png"></span><u></u><u></u></p>
<p><b><i><span style="font-size:10.0pt;font-family:"Courier New"">Twitter: @FreeSWITCH , @briankwest</span></i></b><span style="font-size:10.0pt;font-family:"Courier New""><br>
<a href="http://www.freeswitchbook.com" target="_blank">http://www.freeswitchbook.com</a><br>
<a href="http://www.freeswitchcookbook.com" target="_blank">http://www.freeswitchcookbook.com</a></span><u></u><u></u></p>
<p><span style="font-family:"Courier New"">Got Bugs? Report them <a href="https://freeswitch.org/jira" target="_blank">
here</a>! | Reddit: <a href="https://www.reddit.com/r/freeswitch" target="_blank">/r/freeswitch</a></span><u></u><u></u></p>
<p><b><span style="font-size:10.0pt;font-family:"Courier New"">T:</span></b><span style="font-size:10.0pt;font-family:"Courier New""><a href="tel:%2B19184209001" value="+19184209001" target="_blank">+19184209001</a> |
<b>F:</b><a href="tel:%2B19184209002" value="+19184209002" target="_blank">+19184209002</a> | <b>M:</b>+1918424WEST (9378)<br>
<b>iNUM:</b>+883 5100 1420 9001 | <b>ISN:</b>410*543 | <b>Skype:</b>briankwest</span><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><br>
______________________________________________________________________<br>
This message has been checked for all known viruses by MessageLabs Virus Scanning Service.<br>
______________________________________________________________________<u></u><u></u></p>
</div></div></div><div><div>
<br clear="both">
______________________________________________________________________<br>
This message has been checked for all known viruses by MessageLabs Virus Scanning Service.<br>
______________________________________________________________________<br>
</div></div></div>
<br>_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" rel="noreferrer" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://confluence.freeswitch.org" rel="noreferrer" target="_blank">http://confluence.freeswitch.org</a><br>
<a href="http://www.cluecon.com" rel="noreferrer" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">
<p><font face="courier new, monospace"><b><i><font size="4">Brian West</font></i></b><br><span style="font-size:x-small"><a href="mailto:brian@freeswitch.org" target="_blank">brian@freeswitch.org</a></span></font></p>
<p><font size="1" face="courier new, monospace"><img src="http://billing.freeswitch.org/templates/default/img/whmcslogo.png"><br></font></p><p><font size="2" face="monospace, monospace"><b><i>Twitter: @FreeSWITCH , @briankwest</i></b><br><a href="http://www.freeswitchbook.com" target="_blank">http://www.freeswitchbook.com</a><br><a href="http://www.freeswitchcookbook.com" target="_blank">http://www.freeswitchcookbook.com</a></font></p><p><font face="monospace, monospace">Got Bugs? Report them <a href="https://freeswitch.org/jira" target="_blank">here</a>! | Reddit: <a href="https://www.reddit.com/r/freeswitch" target="_blank">/r/freeswitch</a></font></p>
<p><font size="2" face="monospace, monospace"><b>T:</b><a href="tel:%2B19184209001" value="+19184209001" target="_blank">+19184209001</a> | <b>F:</b><a href="tel:%2B19184209002" value="+19184209002" target="_blank">+19184209002</a> | <b>M:</b>+1918424WEST (9378)<br><b>iNUM:</b>+883 5100 1420 9001 | <b>ISN:</b>410*543 | <b>Skype:</b>briankwest</font></p></div></div></div></div></div></div></div></div></div></div>
</div></div></div></div></div>
<br>_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" rel="noreferrer" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://confluence.freeswitch.org" rel="noreferrer" target="_blank">http://confluence.freeswitch.org</a><br>
<a href="http://www.cluecon.com" rel="noreferrer" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br></blockquote></div><br></div>