Hi Michael, Thanks for the response and recommendation. Our new build is now on Debian 8 Jessie, performance on the system is noticeably better and any audio problems while the system is under load are significantly reduced but we do still experience spikes along with reduced audio quality when the system is processing 20 calls per second while already maintaining 80 existing calls. We’re still hitting some kind of bottleneck, what should we expect to be able to support on a single piece of hardware? We’ve kept our dialplans as small as possible and we’re using Memcache, but something tells me it’s the call setup process which is triggering the spikes, is this possibly a limitation on the SIP profiles which are single threaded? I also see FreeSWITCH supports clustering but it’s in relation to high availability, is it possible to cluster a single FreeSWITCH instance across multiple hardware or should we be looking elsewhere to solve these bottlenecks? We switched to TCP Vegas on our new build as recommended on the FreeSWITCH Wiki, this has provided a noticeable reduction in audio latency but doesn’t solve the issue with CPU spikes. Many Thanks, Shaun From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Michael Jerris Sent: 15 August 2015 23:00 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues There are multiple known issues with Ubuntu 12.04. We reccomend Debian 8 Jessie. On Friday, August 14, 2015, Shaun Stokes <shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>> wrote: Hi, We’re experiencing performance issues with FreeSwitch, our target is 500 concurrent sessions, but at the moment this starts to bottleneck around 30. Host system: Windows Server 2012 Hyper-V Host AMD Opteron 4386 (2 processors) – 16 cores total 128GB DDR3 2TB RAID 5 (700MB/s tested read and write throughput) FreeSwitch Virtual Machine: FreeSwitch 1.4.15 Ubuntu 12.04 LTS 16 Virtual cores (high priority) 2GB RAM (would assign more but FreeSwitch never seems to use much) 500GB HD (on VHDX) After around 30 concurrent sessions we begin to see CPU spikes almost every time a new call comes in, as the sessions increase the size and frequency of the CPU spikes also increase. The system seems to be able to sit comfortably with over 100 concurrent sessions and 80% idle CPU, providing we don’t have any new calls hitting the platform. The spikes are causing audio (RTP) to stutter or in some cases drop completely for a few seconds. The FreeSwitch spikes are occurring on all 16 cores, we have been monitoring the system using htop and mpstat. This is an example of when we receive an inbound call while we have 34 concurrent sessions: 14:26:04 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 14:26:05 all 1.67 0.00 27.34 0.00 0.25 0.00 0.00 0.00 70.74 14:26:05 0 1.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 71.00 14:26:05 1 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28 14:26:05 2 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28 14:26:05 3 3.92 0.00 26.47 0.00 0.00 0.00 0.00 0.00 69.61 14:26:05 4 0.99 0.00 25.74 0.00 1.98 0.00 0.00 0.00 71.29 14:26:05 5 0.99 0.00 26.73 0.00 0.99 0.00 0.00 0.00 71.29 14:26:05 6 0.00 0.00 26.47 0.00 0.98 0.00 0.00 0.00 72.55 14:26:05 7 0.00 0.00 26.73 0.00 0.00 0.00 0.00 0.00 73.27 14:26:05 8 3.96 0.00 27.72 0.00 0.00 0.00 0.00 0.00 68.32 14:26:05 9 2.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00 71.00 14:26:05 10 6.00 0.00 32.00 0.00 0.00 0.00 0.00 0.00 62.00 14:26:05 11 1.96 0.00 27.45 0.00 0.00 0.00 0.00 0.00 70.59 14:26:05 12 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28 14:26:05 13 1.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 71.00 14:26:05 14 0.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00 73.00 This is when the system is not receiving an inbound call but is sitting comfortably at 34 concurrent sessions: 14:25:57 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 14:25:58 all 0.87 0.00 0.62 0.00 0.12 0.00 0.00 0.00 98.39 14:25:58 0 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02 14:25:58 1 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02 14:25:58 2 0.98 0.00 0.98 0.00 0.00 0.00 0.00 0.00 98.04 14:25:58 3 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01 14:25:58 4 0.00 0.00 0.00 0.00 1.98 0.00 0.00 0.00 98.02 14:25:58 5 0.00 0.00 0.00 0.00 0.99 0.00 0.00 0.00 99.01 14:25:58 6 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01 14:25:58 7 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01 14:25:58 8 5.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 94.00 14:25:58 9 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00 14:25:58 10 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00 14:25:58 11 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02 14:25:58 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 14:25:58 13 1.98 0.00 0.99 0.00 0.00 0.00 0.00 0.00 97.03 14:25:58 14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 14:25:58 15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 We’re using FreeSwitch in multi-tenant mode, we have tried dedicating a single VM for one tenancy but still experience the issue. My assumption has been that this will be due to the dialplan, I’ve optimized this slightly by writing LUA scripts to handle some of the inbound calls this appears to take away some of the load but we’re still using the internal dialplans for each tenancy (tenancy dialplans have an average of 300 entries). We are using the following arguments when running FreeSwitch: -rp –nc –nonat I’ve seen another post which says we should operate FreeSwitch in High Performance mode using –hp but can’t find anything information about this, is this still a valid argument to use with FreeSwitch? Has anyone experienced similar performance issues before or have any advice? Hope someone may be able help. Thanks, Shaun ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________

Hi Peter, Thanks for the advice, this is something we’re already looking into but we don’t have the new hardware available yet. Debian 8 has great integration with Hyper-V on 2012R2 and operates as a Generation 2 VM, obviously it’s never going to be as good as running directly on the hardware but we were hoping for better performance. Thanks, Shaun From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Peter Olsson Sent: 28 August 2015 10:43 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues If you want high performance, I recommend using real hardware, not a virtual machine setup - it will cause you issues. /Peter 2015-08-28 9:44 GMT+02:00 Shaun Stokes <shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>>: Hi Michael, Thanks for the response and recommendation. Our new build is now on Debian 8 Jessie, performance on the system is noticeably better and any audio problems while the system is under load are significantly reduced but we do still experience spikes along with reduced audio quality when the system is processing 20 calls per second while already maintaining 80 existing calls. We’re still hitting some kind of bottleneck, what should we expect to be able to support on a single piece of hardware? We’ve kept our dialplans as small as possible and we’re using Memcache, but something tells me it’s the call setup process which is triggering the spikes, is this possibly a limitation on the SIP profiles which are single threaded? I also see FreeSWITCH supports clustering but it’s in relation to high availability, is it possible to cluster a single FreeSWITCH instance across multiple hardware or should we be looking elsewhere to solve these bottlenecks? We switched to TCP Vegas on our new build as recommended on the FreeSWITCH Wiki, this has provided a noticeable reduction in audio latency but doesn’t solve the issue with CPU spikes. Many Thanks, Shaun From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Michael Jerris Sent: 15 August 2015 23:00 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues There are multiple known issues with Ubuntu 12.04. We reccomend Debian 8 Jessie. On Friday, August 14, 2015, Shaun Stokes <shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>> wrote: Hi, We’re experiencing performance issues with FreeSwitch, our target is 500 concurrent sessions, but at the moment this starts to bottleneck around 30. Host system: Windows Server 2012 Hyper-V Host AMD Opteron 4386 (2 processors) – 16 cores total 128GB DDR3 2TB RAID 5 (700MB/s tested read and write throughput) FreeSwitch Virtual Machine: FreeSwitch 1.4.15 Ubuntu 12.04 LTS 16 Virtual cores (high priority) 2GB RAM (would assign more but FreeSwitch never seems to use much) 500GB HD (on VHDX) After around 30 concurrent sessions we begin to see CPU spikes almost every time a new call comes in, as the sessions increase the size and frequency of the CPU spikes also increase. The system seems to be able to sit comfortably with over 100 concurrent sessions and 80% idle CPU, providing we don’t have any new calls hitting the platform. The spikes are causing audio (RTP) to stutter or in some cases drop completely for a few seconds. The FreeSwitch spikes are occurring on all 16 cores, we have been monitoring the system using htop and mpstat. This is an example of when we receive an inbound call while we have 34 concurrent sessions: 14:26:04 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 14:26:05 all 1.67 0.00 27.34 0.00 0.25 0.00 0.00 0.00 70.74 14:26:05 0 1.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 71.00 14:26:05 1 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28 14:26:05 2 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28 14:26:05 3 3.92 0.00 26.47 0.00 0.00 0.00 0.00 0.00 69.61 14:26:05 4 0.99 0.00 25.74 0.00 1.98 0.00 0.00 0.00 71.29 14:26:05 5 0.99 0.00 26.73 0.00 0.99 0.00 0.00 0.00 71.29 14:26:05 6 0.00 0.00 26.47 0.00 0.98 0.00 0.00 0.00 72.55 14:26:05 7 0.00 0.00 26.73 0.00 0.00 0.00 0.00 0.00 73.27 14:26:05 8 3.96 0.00 27.72 0.00 0.00 0.00 0.00 0.00 68.32 14:26:05 9 2.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00 71.00 14:26:05 10 6.00 0.00 32.00 0.00 0.00 0.00 0.00 0.00 62.00 14:26:05 11 1.96 0.00 27.45 0.00 0.00 0.00 0.00 0.00 70.59 14:26:05 12 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28 14:26:05 13 1.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 71.00 14:26:05 14 0.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00 73.00 This is when the system is not receiving an inbound call but is sitting comfortably at 34 concurrent sessions: 14:25:57 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 14:25:58 all 0.87 0.00 0.62 0.00 0.12 0.00 0.00 0.00 98.39 14:25:58 0 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02 14:25:58 1 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02 14:25:58 2 0.98 0.00 0.98 0.00 0.00 0.00 0.00 0.00 98.04 14:25:58 3 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01 14:25:58 4 0.00 0.00 0.00 0.00 1.98 0.00 0.00 0.00 98.02 14:25:58 5 0.00 0.00 0.00 0.00 0.99 0.00 0.00 0.00 99.01 14:25:58 6 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01 14:25:58 7 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01 14:25:58 8 5.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 94.00 14:25:58 9 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00 14:25:58 10 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00 14:25:58 11 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02 14:25:58 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 14:25:58 13 1.98 0.00 0.99 0.00 0.00 0.00 0.00 0.00 97.03 14:25:58 14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 14:25:58 15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 We’re using FreeSwitch in multi-tenant mode, we have tried dedicating a single VM for one tenancy but still experience the issue. My assumption has been that this will be due to the dialplan, I’ve optimized this slightly by writing LUA scripts to handle some of the inbound calls this appears to take away some of the load but we’re still using the internal dialplans for each tenancy (tenancy dialplans have an average of 300 entries). We are using the following arguments when running FreeSwitch: -rp –nc –nonat I’ve seen another post which says we should operate FreeSwitch in High Performance mode using –hp but can’t find anything information about this, is this still a valid argument to use with FreeSwitch? Has anyone experienced similar performance issues before or have any advice? Hope someone may be able help. Thanks, Shaun ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________

One thing that I see jump out - you've probably got too many virtual cores defined on that VM, particularly since you've assigned # vcpu the same as the number of real cores on the host system. Try your test again dropping that to 8 virtual CPUs and I think you might see better results. (Or at least drop to less than 100% of the total cores on the hardware.) Particularly if this host system has other vms running on it, you're hurting the vm scheduler's ability to hand out resources. With vmware, I believe a simple rule of thumb is that scheduler requires N-1 CPUs to be locked to schedule execution of an N cpu virtual machine - so even if it doesn't need all of the CPUs, it has to reserve all of them any time you have more than a 2 vCPU vm. -- Nathan On 08/28/2015 08:18 AM, Michael Jerris wrote:

The numbers your posting are fairly low, I'm guessing this is a grossly underpowered vm that is causing your issues

On Friday, August 28, 2015, Shaun Stokes <shaun.stokes@itec-support.co.uk <mailto:shaun.stokes@itec-support.co.uk>> wrote:

Hi Peter,____

__ __

Thanks for the advice, this is something we’re already looking into but we don’t have the new hardware available yet.____

__ __

Debian 8 has great integration with Hyper-V on 2012R2 and operates as a Generation 2 VM, obviously it’s never going to be as good as running directly on the hardware but we were hoping for better performance.____

__ __

Thanks,____

Shaun____

__ __

*From:*freeswitch-users-bounces@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces@lists.freeswitch.org');> [mailto:freeswitch-users-bounces@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces@lists.freeswitch.org');>] *On Behalf Of *Peter Olsson *Sent:* 28 August 2015 10:43 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml','freeswitch-users@lists.freeswitch.org');>> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues____

__ __

If you want high performance, I recommend using real hardware, not a virtual machine setup - it will cause you issues.____

__ __

/Peter____

__ __

2015-08-28 9:44 GMT+02:00 Shaun Stokes <shaun.stokes@itec-support.co.uk <javascript:_e(%7B%7D,'cvml','shaun.stokes@itec-support.co.uk');>>:____

Hi Michael,____

____

Thanks for the response and recommendation.____

____

Our new build is now on Debian 8 Jessie, performance on the system is noticeably better and any audio problems while the system is under load are significantly reduced but we do still experience spikes along with reduced audio quality when the system is processing 20 calls per second while already maintaining 80 existing calls.____

____

We’re still hitting some kind of bottleneck, what should we expect to be able to support on a single piece of hardware? We’ve kept our dialplans as small as possible and we’re using Memcache, but something tells me it’s the call setup process which is triggering the spikes, is this possibly a limitation on the SIP profiles which are single threaded?____

____

I also see FreeSWITCH supports clustering but it’s in relation to high availability, is it possible to cluster a single FreeSWITCH instance across multiple hardware or should we be looking elsewhere to solve these bottlenecks?____

____

We switched to TCP Vegas on our new build as recommended on the FreeSWITCH Wiki, this has provided a noticeable reduction in audio latency but doesn’t solve the issue with CPU spikes.____

____

Many Thanks,____

Shaun____

____

*From:*freeswitch-users-bounces@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces@lists.freeswitch.org');> [mailto:freeswitch-users-bounces@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces@lists.freeswitch.org');>] *On Behalf Of *Michael Jerris *Sent:* 15 August 2015 23:00 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml','freeswitch-users@lists.freeswitch.org');>> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues____

____

There are multiple known issues with Ubuntu 12.04. We reccomend Debian 8 Jessie.

On Friday, August 14, 2015, Shaun Stokes <shaun.stokes@itec-support.co.uk <javascript:_e(%7B%7D,'cvml','shaun.stokes@itec-support.co.uk');>> wrote:____

Hi,____

____

We’re experiencing performance issues with FreeSwitch, our target is 500 concurrent sessions, but at the moment this starts to bottleneck around 30.____

____

Host system:____

Windows Server 2012 Hyper-V Host____

AMD Opteron 4386 (2 processors) – 16 cores total____

128GB DDR3____

2TB RAID 5 (700MB/s tested read and write throughput)____

____

FreeSwitch Virtual Machine:____

FreeSwitch 1.4.15____

Ubuntu 12.04 LTS____

16 Virtual cores (high priority)____

2GB RAM (would assign more but FreeSwitch never seems to use much)____

500GB HD (on VHDX)____

____

After around 30 concurrent sessions we begin to see CPU spikes almost every time a new call comes in, as the sessions increase the size and frequency of the CPU spikes also increase. The system seems to be able to sit comfortably with over 100 concurrent sessions and 80% idle CPU, providing we don’t have any new calls hitting the platform. The spikes are causing audio (RTP) to stutter or in some cases drop completely for a few seconds.____

____

The FreeSwitch spikes are occurring on all 16 cores, we have been monitoring the system using htop and mpstat.____

____

This is an example of when we receive an inbound call while we have 34 concurrent sessions:____

14:26:04 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle____

14:26:05 all 1.67 0.00 27.34 0.00 0.25 0.00 0.00 0.00 70.74____

14:26:05 0 1.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 71.00____

14:26:05 1 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____

14:26:05 2 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____

14:26:05 3 3.92 0.00 26.47 0.00 0.00 0.00 0.00 0.00 69.61____

14:26:05 4 0.99 0.00 25.74 0.00 1.98 0.00 0.00 0.00 71.29____

14:26:05 5 0.99 0.00 26.73 0.00 0.99 0.00 0.00 0.00 71.29____

14:26:05 6 0.00 0.00 26.47 0.00 0.98 0.00 0.00 0.00 72.55____

14:26:05 7 0.00 0.00 26.73 0.00 0.00 0.00 0.00 0.00 73.27____

14:26:05 8 3.96 0.00 27.72 0.00 0.00 0.00 0.00 0.00 68.32____

14:26:05 9 2.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00 71.00____

14:26:05 10 6.00 0.00 32.00 0.00 0.00 0.00 0.00 0.00 62.00____

14:26:05 11 1.96 0.00 27.45 0.00 0.00 0.00 0.00 0.00 70.59____

14:26:05 12 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____

14:26:05 13 1.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 71.00____

14:26:05 14 0.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00 73.00____

____

This is when the system is not receiving an inbound call but is sitting comfortably at 34 concurrent sessions:____

14:25:57 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle____

14:25:58 all 0.87 0.00 0.62 0.00 0.12 0.00 0.00 0.00 98.39____

14:25:58 0 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____

14:25:58 1 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____

14:25:58 2 0.98 0.00 0.98 0.00 0.00 0.00 0.00 0.00 98.04____

14:25:58 3 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____

14:25:58 4 0.00 0.00 0.00 0.00 1.98 0.00 0.00 0.00 98.02____

14:25:58 5 0.00 0.00 0.00 0.00 0.99 0.00 0.00 0.00 99.01____

14:25:58 6 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____

14:25:58 7 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____

14:25:58 8 5.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 94.00____

14:25:58 9 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00____

14:25:58 10 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00____

14:25:58 11 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____

14:25:58 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____

14:25:58 13 1.98 0.00 0.99 0.00 0.00 0.00 0.00 0.00 97.03____

14:25:58 14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____

14:25:58 15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____

____

We’re using FreeSwitch in multi-tenant mode, we have tried dedicating a single VM for one tenancy but still experience the issue. My assumption has been that this will be due to the dialplan, I’ve optimized this slightly by writing LUA scripts to handle some of the inbound calls this appears to take away some of the load but we’re still using the internal dialplans for each tenancy (tenancy dialplans have an average of 300 entries).____

____

We are using the following arguments when running FreeSwitch:____

-rp –nc –nonat____

____

I’ve seen another post which says we should operate FreeSwitch in High Performance mode using –hp but can’t find anything information about this, is this still a valid argument to use with FreeSwitch?____

____

Has anyone experienced similar performance issues before or have any advice?____

____

Hope someone may be able help.____

____

Thanks,____

Shaun____

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. __________________________________________________________________________

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. __________________________________________________________________________

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. __________________________________________________________________________

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <javascript:_e(%7B%7D,'cvml','consulting@freeswitch.org');> http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml','FreeSWITCH-users@lists.freeswitch.org');> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org____

__ __

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. __________________________________________________________________________

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- ------------------------------------------------------------ Nathan Neulinger nneul@mst.edu Missouri S&T Information Technology (573) 612-1412 System Administrator - Architect

Hi Nathan, Thanks for the suggestion, we still see the CPU spikes when is under load after reducing to 8 cores but I think there may actually be an improvement in the audio quality. It's still not what we're expecting, so we will have to test our build on physical hardware too. Thanks, Shaun -----Original Message----- From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Nathan Neulinger Sent: 28 August 2015 15:04 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues Yes, I know you said Hyper-V, I don't know if it has the same sort of scheduler behavior or not, just mentioned the vmware for context. On 08/28/2015 08:46 AM, Nathan Neulinger wrote:

One thing that I see jump out - you've probably got too many virtual cores defined on that VM, particularly since you've assigned # vcpu the same as the number of real cores on the host system.

Try your test again dropping that to 8 virtual CPUs and I think you might see better results. (Or at least drop to less than 100% of the total cores on the hardware.)

Particularly if this host system has other vms running on it, you're hurting the vm scheduler's ability to hand out resources. With vmware, I believe a simple rule of thumb is that scheduler requires N-1 CPUs to be locked to schedule execution of an N cpu virtual machine - so even if it doesn't need all of the CPUs, it has to reserve all of them any time you have more than a 2 vCPU vm.

-- Nathan

On 08/28/2015 08:18 AM, Michael Jerris wrote:

...

The numbers your posting are fairly low, I'm guessing this is a grossly underpowered vm that is causing your issues

On Friday, August 28, 2015, Shaun Stokes <shaun.stokes@itec-support.co.uk <mailto:shaun.stokes@itec-support.co.uk>> wrote:

Hi Peter,____

__ __

Thanks for the advice, this is something we're already looking into but we don't have the new hardware available yet.____

__ __

Debian 8 has great integration with Hyper-V on 2012R2 and operates as a Generation 2 VM, obviously it's never going to be as good as running directly on the hardware but we were hoping for better performance.____

__ __

Thanks,____

Shaun____

__ __

*From:*freeswitch-users-bounces@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces@lists.freeswitch.org');> [mailto:freeswitch-users-bounces@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces@lists.freeswitch.org');>] *On Behalf Of *Peter Olsson *Sent:* 28 August 2015 10:43 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml','freeswitch-users@lists.freeswitch.org');>> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues____

__ __

If you want high performance, I recommend using real hardware, not a virtual machine setup - it will cause you issues.____

__ __

/Peter____

__ __

2015-08-28 9:44 GMT+02:00 Shaun Stokes <shaun.stokes@itec-support.co.uk

<javascript:_e(%7B%7D,'cvml','shaun.stokes@itec-support.co.uk');>>:__ __

Hi Michael,____

____

Thanks for the response and recommendation.____

____

Our new build is now on Debian 8 Jessie, performance on the system is noticeably better and any audio problems while the system is under load are significantly reduced but we do still experience spikes along with reduced audio quality when the system is processing 20 calls per second while already maintaining 80 existing calls.____

____

We're still hitting some kind of bottleneck, what should we expect to be able to support on a single piece of hardware? We've kept our dialplans as small as possible and we're using Memcache, but something tells me it's the call setup process which is triggering the spikes, is this possibly a limitation on the SIP profiles which are single threaded?____

____

I also see FreeSWITCH supports clustering but it's in relation to high availability, is it possible to cluster a single FreeSWITCH instance across multiple hardware or should we be looking elsewhere to solve these bottlenecks?____

____

We switched to TCP Vegas on our new build as recommended on the FreeSWITCH Wiki, this has provided a noticeable reduction in audio latency but doesn't solve the issue with CPU spikes.____

____

Many Thanks,____

Shaun____

____

*From:*freeswitch-users-bounces@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces@lists.freeswitch.org');> [mailto:freeswitch-users-bounces@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces@lists.freeswitch.org');>] *On Behalf Of *Michael Jerris *Sent:* 15 August 2015 23:00 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml','freeswitch-users@lists.freeswitch.org');>> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues____

____

There are multiple known issues with Ubuntu 12.04. We reccomend Debian 8 Jessie.

On Friday, August 14, 2015, Shaun Stokes <shaun.stokes@itec-support.co.uk

<javascript:_e(%7B%7D,'cvml','shaun.stokes@itec-support.co.uk');>> wrote:____

Hi,____

____

We're experiencing performance issues with FreeSwitch, our target is 500 concurrent sessions, but at the moment this starts to bottleneck around 30.____

____

Host system:____

Windows Server 2012 Hyper-V Host____

AMD Opteron 4386 (2 processors) - 16 cores total____

128GB DDR3____

2TB RAID 5 (700MB/s tested read and write throughput)____

____

FreeSwitch Virtual Machine:____

FreeSwitch 1.4.15____

Ubuntu 12.04 LTS____

16 Virtual cores (high priority)____

2GB RAM (would assign more but FreeSwitch never seems to use much)____

500GB HD (on VHDX)____

____

After around 30 concurrent sessions we begin to see CPU spikes almost every time a new call comes in, as the sessions increase the size and frequency of the CPU spikes also increase. The system seems to be able to sit comfortably with over 100 concurrent sessions and 80% idle CPU, providing we don't have any new calls hitting the platform. The spikes are causing audio (RTP) to stutter or in some cases drop completely for a few seconds.____

____

The FreeSwitch spikes are occurring on all 16 cores, we have been monitoring the system using htop and mpstat.____

____

This is an example of when we receive an inbound call while we have 34 concurrent sessions:____

14:26:04 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle____

14:26:05 all 1.67 0.00 27.34 0.00 0.25 0.00 0.00 0.00 70.74____

14:26:05 0 1.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 71.00____

14:26:05 1 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____

14:26:05 2 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____

14:26:05 3 3.92 0.00 26.47 0.00 0.00 0.00 0.00 0.00 69.61____

14:26:05 4 0.99 0.00 25.74 0.00 1.98 0.00 0.00 0.00 71.29____

14:26:05 5 0.99 0.00 26.73 0.00 0.99 0.00 0.00 0.00 71.29____

14:26:05 6 0.00 0.00 26.47 0.00 0.98 0.00 0.00 0.00 72.55____

14:26:05 7 0.00 0.00 26.73 0.00 0.00 0.00 0.00 0.00 73.27____

14:26:05 8 3.96 0.00 27.72 0.00 0.00 0.00 0.00 0.00 68.32____

14:26:05 9 2.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00 71.00____

14:26:05 10 6.00 0.00 32.00 0.00 0.00 0.00 0.00 0.00 62.00____

14:26:05 11 1.96 0.00 27.45 0.00 0.00 0.00 0.00 0.00 70.59____

14:26:05 12 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____

14:26:05 13 1.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 71.00____

14:26:05 14 0.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00 73.00____

____

This is when the system is not receiving an inbound call but is sitting comfortably at 34 concurrent sessions:____

14:25:57 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle____

14:25:58 all 0.87 0.00 0.62 0.00 0.12 0.00 0.00 0.00 98.39____

14:25:58 0 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____

14:25:58 1 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____

14:25:58 2 0.98 0.00 0.98 0.00 0.00 0.00 0.00 0.00 98.04____

14:25:58 3 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____

14:25:58 4 0.00 0.00 0.00 0.00 1.98 0.00 0.00 0.00 98.02____

14:25:58 5 0.00 0.00 0.00 0.00 0.99 0.00 0.00 0.00 99.01____

14:25:58 6 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____

14:25:58 7 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____

14:25:58 8 5.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 94.00____

14:25:58 9 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00____

14:25:58 10 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00____

14:25:58 11 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____

14:25:58 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____

14:25:58 13 1.98 0.00 0.99 0.00 0.00 0.00 0.00 0.00 97.03____

14:25:58 14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____

14:25:58 15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____

____

We're using FreeSwitch in multi-tenant mode, we have tried dedicating a single VM for one tenancy but still experience the issue. My assumption has been that this will be due to the dialplan, I've optimized this slightly by writing LUA scripts to handle some of the inbound calls this appears to take away some of the load but we're still using the internal dialplans for each tenancy (tenancy dialplans have an average of 300 entries).____

____

We are using the following arguments when running FreeSwitch:____

-rp -nc -nonat____

____

I've seen another post which says we should operate FreeSwitch in High Performance mode using -hp but can't find anything information about this, is this still a valid argument to use with FreeSwitch?____

____

Has anyone experienced similar performance issues before or have any advice?____

____

Hope someone may be able help.____

____

Thanks,____

Shaun____

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service.

_____________________________________________________________________ _____

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service.

_____________________________________________________________________ _____

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service.

_____________________________________________________________________ _____

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <javascript:_e(%7B%7D,'cvml','consulting@freeswitch.org');> http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml','FreeSWITCH-users@lists.freeswitch.org');> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org____

__ __

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service.

_____________________________________________________________________ _____

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service.

_____________________________________________________________________ _

_____________________________________________________________________ ____ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-us ers http://www.freeswitch.org

-- ------------------------------------------------------------ Nathan Neulinger nneul@mst.edu Missouri S&T Information Technology (573) 612-1412 System Administrator - Architect _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________

Shaun, 800 concurrent with 400cps, expectations are a little out of touch with reality and your test is no where near real world. Your call duration is ~2000ms, little to no time to actually do anything useful. You're burning the bulk of your CPU time on signaling and doing no real work with the session. So yes, you'll always get the best results out of bare metal, but you can also get good performance on a VM (*) (*) This is in no way an endorsement of running FreeSWITCH in a VM, Your mileage may vary depending on your specific application and load requirements. Thanks, On Tue, Sep 1, 2015 at 3:08 PM, Shaun Stokes < shaun.stokes@itec-support.co.uk> wrote:

Update

We've performed various tests and have come to the conclusion as advised by others, that visualization is causing bottlenecks.

Using Amazon EC2 on an 8 core high performance virtual server this maxed out at around 800 concurrent sessions, 400 sessions per second.

Using a Dell 1950 8 core server this was able to handle 1000 concurrent sessions, 400 sessions per second and still have plenty of resource available. Average CPU was fluctuating between 30-40% under peak load.

Conclusion, if you want to run a high performance FreeSwitch server in a production you should stick with physical hardware and avoid visualization where possible. ________________________________________ From: freeswitch-users-bounces@lists.freeswitch.org [ freeswitch-users-bounces@lists.freeswitch.org] on behalf of Shaun Stokes [ shaun.stokes@itec-support.co.uk] Sent: 28 August 2015 15:37 To: 'FreeSWITCH Users Help' Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues

Hi Nathan,

Thanks for the suggestion, we still see the CPU spikes when is under load after reducing to 8 cores but I think there may actually be an improvement in the audio quality.

It's still not what we're expecting, so we will have to test our build on physical hardware too.

Thanks, Shaun

-----Original Message----- From: freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Nathan Neulinger Sent: 28 August 2015 15:04 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues

Yes, I know you said Hyper-V, I don't know if it has the same sort of scheduler behavior or not, just mentioned the vmware for context.

On 08/28/2015 08:46 AM, Nathan Neulinger wrote:

...

One thing that I see jump out - you've probably got too many virtual cores defined on that VM, particularly since you've assigned # vcpu the same as the number of real cores on the host system.

Try your test again dropping that to 8 virtual CPUs and I think you might see better results. (Or at least drop to less than 100% of the total cores on the hardware.)

Particularly if this host system has other vms running on it, you're hurting the vm scheduler's ability to hand out resources. With vmware, I believe a simple rule of thumb is that scheduler requires N-1 CPUs to be locked to schedule execution of an N cpu virtual machine - so even if it doesn't need all of the CPUs, it has to reserve all of them any time you have more than a 2 vCPU vm.

-- Nathan

On 08/28/2015 08:18 AM, Michael Jerris wrote:

...

The numbers your posting are fairly low, I'm guessing this is a grossly underpowered vm that is causing your issues

On Friday, August 28, 2015, Shaun Stokes < shaun.stokes@itec-support.co.uk <mailto:shaun.stokes@itec-support.co.uk>> wrote:

Hi Peter,____

__ __

Thanks for the advice, this is something we're already looking into but we don't have the new hardware available yet.____

__ __

Debian 8 has great integration with Hyper-V on 2012R2 and operates as a Generation 2 VM, obviously it's never going to be as good as running directly on the hardware but we were hoping for better performance.____

__ __

Thanks,____

Shaun____

__ __

*From:*freeswitch-users-bounces@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml',' freeswitch-users-bounces@lists.freeswitch.org');> [mailto:freeswitch-users-bounces@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml',' freeswitch-users-bounces@lists.freeswitch.org');>] *On Behalf Of *Peter Olsson *Sent:* 28 August 2015 10:43 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml',' freeswitch-users@lists.freeswitch.org');>> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues____

__ __

If you want high performance, I recommend using real hardware, not a virtual machine setup - it will cause you issues.____

__ __

/Peter____

__ __

2015-08-28 9:44 GMT+02:00 Shaun Stokes < shaun.stokes@itec-support.co.uk

<javascript:_e(%7B%7D,'cvml','shaun.stokes@itec-support.co.uk');>>:__ __

Hi Michael,____

____

Thanks for the response and recommendation.____

____

Our new build is now on Debian 8 Jessie, performance on the system is noticeably better and any audio problems while the system is under load are significantly reduced but we do still experience spikes along with reduced audio quality when the system is processing 20 calls per second while already maintaining 80 existing calls.____

____

We're still hitting some kind of bottleneck, what should we expect to be able to support on a single piece of hardware? We've kept our dialplans as small as possible and we're using Memcache, but something tells me it's the call setup process which is triggering the spikes, is this possibly a limitation on the SIP profiles which are single threaded?____

____

I also see FreeSWITCH supports clustering but it's in relation to high availability, is it possible to cluster a single FreeSWITCH instance across multiple hardware or should we be looking elsewhere to solve these bottlenecks?____

____

We switched to TCP Vegas on our new build as recommended on the FreeSWITCH Wiki, this has provided a noticeable reduction in audio latency but doesn't solve the issue with CPU spikes.____

____

Many Thanks,____

Shaun____

____

*From:*freeswitch-users-bounces@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml',' freeswitch-users-bounces@lists.freeswitch.org');> [mailto:freeswitch-users-bounces@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml',' freeswitch-users-bounces@lists.freeswitch.org');>] *On Behalf Of *Michael Jerris *Sent:* 15 August 2015 23:00 *To:* FreeSWITCH Users Help < freeswitch-users@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml',' freeswitch-users@lists.freeswitch.org');>> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues____

____

There are multiple known issues with Ubuntu 12.04. We reccomend Debian 8 Jessie.

On Friday, August 14, 2015, Shaun Stokes < shaun.stokes@itec-support.co.uk

<javascript:_e(%7B%7D,'cvml','shaun.stokes@itec-support.co.uk');>> wrote:____

Hi,____

____

We're experiencing performance issues with FreeSwitch, our target is 500 concurrent sessions, but at the moment this starts to bottleneck around 30.____

____

Host system:____

Windows Server 2012 Hyper-V Host____

AMD Opteron 4386 (2 processors) - 16 cores total____

128GB DDR3____

2TB RAID 5 (700MB/s tested read and write throughput)____

____

FreeSwitch Virtual Machine:____

FreeSwitch 1.4.15____

Ubuntu 12.04 LTS____

16 Virtual cores (high priority)____

2GB RAM (would assign more but FreeSwitch never seems to use much)____

500GB HD (on VHDX)____

____

After around 30 concurrent sessions we begin to see CPU spikes almost every time a new call comes in, as the sessions increase the size and frequency of the CPU spikes also increase. The system seems to be able to sit comfortably with over 100 concurrent sessions and 80% idle CPU, providing we don't have any new calls hitting the platform. The spikes are causing audio (RTP) to stutter or in some cases drop completely for a few seconds.____

____

The FreeSwitch spikes are occurring on all 16 cores, we have been monitoring the system using htop and mpstat.____

____

This is an example of when we receive an inbound call while we have 34 concurrent sessions:____

14:26:04 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle____

14:26:05 all 1.67 0.00 27.34 0.00 0.25 0.00 0.00 0.00 70.74____

14:26:05 0 1.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 71.00____

14:26:05 1 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____

14:26:05 2 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____

14:26:05 3 3.92 0.00 26.47 0.00 0.00 0.00 0.00 0.00 69.61____

14:26:05 4 0.99 0.00 25.74 0.00 1.98 0.00 0.00 0.00 71.29____

14:26:05 5 0.99 0.00 26.73 0.00 0.99 0.00 0.00 0.00 71.29____

14:26:05 6 0.00 0.00 26.47 0.00 0.98 0.00 0.00 0.00 72.55____

14:26:05 7 0.00 0.00 26.73 0.00 0.00 0.00 0.00 0.00 73.27____

14:26:05 8 3.96 0.00 27.72 0.00 0.00 0.00 0.00 0.00 68.32____

14:26:05 9 2.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00 71.00____

14:26:05 10 6.00 0.00 32.00 0.00 0.00 0.00 0.00 0.00 62.00____

14:26:05 11 1.96 0.00 27.45 0.00 0.00 0.00 0.00 0.00 70.59____

14:26:05 12 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____

14:26:05 13 1.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 71.00____

14:26:05 14 0.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00 73.00____

____

This is when the system is not receiving an inbound call but is sitting comfortably at 34 concurrent sessions:____

14:25:57 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle____

14:25:58 all 0.87 0.00 0.62 0.00 0.12 0.00 0.00 0.00 98.39____

14:25:58 0 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____

14:25:58 1 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____

14:25:58 2 0.98 0.00 0.98 0.00 0.00 0.00 0.00 0.00 98.04____

14:25:58 3 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____

14:25:58 4 0.00 0.00 0.00 0.00 1.98 0.00 0.00 0.00 98.02____

14:25:58 5 0.00 0.00 0.00 0.00 0.99 0.00 0.00 0.00 99.01____

14:25:58 6 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____

14:25:58 7 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____

14:25:58 8 5.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 94.00____

14:25:58 9 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00____

14:25:58 10 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00____

14:25:58 11 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____

14:25:58 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____

14:25:58 13 1.98 0.00 0.99 0.00 0.00 0.00 0.00 0.00 97.03____

14:25:58 14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____

14:25:58 15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____

____

We're using FreeSwitch in multi-tenant mode, we have tried dedicating a single VM for one tenancy but still experience the issue. My assumption has been that this will be due to the dialplan, I've optimized this slightly by writing LUA scripts to handle some of the inbound calls this appears to take away some of the load but we're still using the internal dialplans for each tenancy (tenancy dialplans have an average of 300 entries).____

____

We are using the following arguments when running FreeSwitch:____

-rp -nc -nonat____

____

I've seen another post which says we should operate FreeSwitch in High Performance mode using -hp but can't find anything information about this, is this still a valid argument to use with FreeSwitch?____

____

Has anyone experienced similar performance issues before or have any advice?____

____

Hope someone may be able help.____

____

Thanks,____

Shaun____

---

...

...

This message has been checked for all known viruses by

MessageLabs Virus Scanning Service.

...

_____________________________________________________________________ _____

---

...

...

This message has been checked for all known viruses by

MessageLabs Virus Scanning Service.

...

_____________________________________________________________________ _____

---

...

...

This message has been checked for all known viruses by

MessageLabs Virus Scanning Service.

...

_____________________________________________________________________ _____

---

...

...

Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <javascript:_e(%7B%7D,'cvml','

consulting@freeswitch.org');>

...

http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org

<javascript:_e(%7B%7D,'cvml','FreeSWITCH-users@lists.freeswitch.org');>

...

http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:

http://lists.freeswitch.org/mailman/options/freeswitch-users

...

http://www.freeswitch.org____

__ __

---

...

...

This message has been checked for all known viruses by MessageLabs

Virus Scanning Service.

...

_____________________________________________________________________ _____

---

...

...

This message has been checked for all known viruses by MessageLabs

Virus Scanning Service.

...

_____________________________________________________________________ _

_____________________________________________________________________ ____ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-us ers http://www.freeswitch.org

-- ------------------------------------------------------------ Nathan Neulinger nneul@mst.edu Missouri S&T Information Technology (573) 612-1412 System Administrator - Architect

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- *Brian West* brian@freeswitch.org *Twitter: @FreeSWITCH , @briankwest* http://www.freeswitchbook.com http://www.freeswitchcookbook.com Got Bugs? Report them here <https://freeswitch.org/jira>! | Reddit: /r/freeswitch <https://www.reddit.com/r/freeswitch> *T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378) *iNUM:*+883 5100 1420 9001 | *ISN:*410*543 | *Skype:*briankwest

I may have missed it in your previous emails, but im assuming you have the core DB on a tempfs or somewhere other than the default location on spinning rust ? Jay On 2 September 2015 at 18:53, Shaun Stokes <shaun.stokes@itec-support.co.uk> wrote:

Hi Brian,

I’m a huge fan of virtualization, but in our environment it simply doesn’t provide the performance we’re looking for. The tests carried out were 10 second calls which were routed to a Call Centre Queue with one registered SIP extension as a member.

There was a significant improvement when we tested on an Amazon VM in comparison to Hyper-V, but we’d like a system which can handle large volumes of call traffic, based on our tests this will need to be physical hardware to achieve the best performance.

Further to this, we have managed to resolve our initial problem with CPU spikes on Hyper-V following these steps:

· Disable VMQ on all physical and virtual adapters on the Hyper-V host which are linked to the FreeSwitch VM

· Reduce the virtual cores on all VMs to at least half of the available physical cores

· Increase the Processor Relative weight on the FreeSwitch VMs to 200 (default 100)

Thanks,

Shaun

*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Brian West *Sent:* 01 September 2015 21:21

*To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues

Shaun,

800 concurrent with 400cps, expectations are a little out of touch with reality and your test is no where near real world. Your call duration is ~2000ms, little to no time to actually do anything useful. You're burning the bulk of your CPU time on signaling and doing no real work with the session.

So yes, you'll always get the best results out of bare metal, but you can also get good performance on a VM (*)

(*) This is in no way an endorsement of running FreeSWITCH in a VM, Your mileage may vary depending on your specific application and load requirements.

Thanks,

On Tue, Sep 1, 2015 at 3:08 PM, Shaun Stokes < shaun.stokes@itec-support.co.uk> wrote:

Update

We've performed various tests and have come to the conclusion as advised by others, that visualization is causing bottlenecks.

Using Amazon EC2 on an 8 core high performance virtual server this maxed out at around 800 concurrent sessions, 400 sessions per second.

Using a Dell 1950 8 core server this was able to handle 1000 concurrent sessions, 400 sessions per second and still have plenty of resource available. Average CPU was fluctuating between 30-40% under peak load.

Conclusion, if you want to run a high performance FreeSwitch server in a production you should stick with physical hardware and avoid visualization where possible. ________________________________________ From: freeswitch-users-bounces@lists.freeswitch.org [ freeswitch-users-bounces@lists.freeswitch.org] on behalf of Shaun Stokes [ shaun.stokes@itec-support.co.uk] Sent: 28 August 2015 15:37 To: 'FreeSWITCH Users Help'

Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues

Hi Nathan,

Thanks for the suggestion, we still see the CPU spikes when is under load after reducing to 8 cores but I think there may actually be an improvement in the audio quality.

It's still not what we're expecting, so we will have to test our build on physical hardware too.

Thanks, Shaun

-----Original Message----- From: freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Nathan Neulinger Sent: 28 August 2015 15:04 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues

Yes, I know you said Hyper-V, I don't know if it has the same sort of scheduler behavior or not, just mentioned the vmware for context.

On 08/28/2015 08:46 AM, Nathan Neulinger wrote:

...

One thing that I see jump out - you've probably got too many virtual cores defined on that VM, particularly since you've assigned # vcpu the same as the number of real cores on the host system.

Try your test again dropping that to 8 virtual CPUs and I think you might see better results. (Or at least drop to less than 100% of the total cores on the hardware.)

Particularly if this host system has other vms running on it, you're hurting the vm scheduler's ability to hand out resources. With vmware, I believe a simple rule of thumb is that scheduler requires N-1 CPUs to be locked to schedule execution of an N cpu virtual machine - so even if it doesn't need all of the CPUs, it has to reserve all of them any time you have more than a 2 vCPU vm.

-- Nathan

On 08/28/2015 08:18 AM, Michael Jerris wrote:

...

The numbers your posting are fairly low, I'm guessing this is a grossly underpowered vm that is causing your issues

On Friday, August 28, 2015, Shaun Stokes < shaun.stokes@itec-support.co.uk <mailto:shaun.stokes@itec-support.co.uk>> wrote:

Hi Peter,____

__ __

Thanks for the advice, this is something we're already looking into but we don't have the new hardware available yet.____

__ __

Debian 8 has great integration with Hyper-V on 2012R2 and operates as a Generation 2 VM, obviously it's never going to be as good as running directly on the hardware but we were hoping for better performance.____

__ __

Thanks,____

Shaun____

__ __

*From:*freeswitch-users-bounces@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml',' freeswitch-users-bounces@lists.freeswitch.org');> [mailto:freeswitch-users-bounces@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml',' freeswitch-users-bounces@lists.freeswitch.org');>] *On Behalf Of *Peter Olsson *Sent:* 28 August 2015 10:43 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml',' freeswitch-users@lists.freeswitch.org');>> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues____

__ __

If you want high performance, I recommend using real hardware, not a virtual machine setup - it will cause you issues.____

__ __

/Peter____

__ __

2015-08-28 9:44 GMT+02:00 Shaun Stokes < shaun.stokes@itec-support.co.uk

<javascript:_e(%7B%7D,'cvml','shaun.stokes@itec-support.co.uk');>>:__ __

Hi Michael,____

____

Thanks for the response and recommendation.____

____

Our new build is now on Debian 8 Jessie, performance on the system is noticeably better and any audio problems while the system is under load are significantly reduced but we do still experience spikes along with reduced audio quality when the system is processing 20 calls per second while already maintaining 80 existing calls.____

____

We're still hitting some kind of bottleneck, what should we expect to be able to support on a single piece of hardware? We've kept our dialplans as small as possible and we're using Memcache, but something tells me it's the call setup process which is triggering the spikes, is this possibly a limitation on the SIP profiles which are single threaded?____

____

I also see FreeSWITCH supports clustering but it's in relation to high availability, is it possible to cluster a single FreeSWITCH instance across multiple hardware or should we be looking elsewhere to solve these bottlenecks?____

____

We switched to TCP Vegas on our new build as recommended on the FreeSWITCH Wiki, this has provided a noticeable reduction in audio latency but doesn't solve the issue with CPU spikes.____

____

Many Thanks,____

Shaun____

____

*From:*freeswitch-users-bounces@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml',' freeswitch-users-bounces@lists.freeswitch.org');> [mailto:freeswitch-users-bounces@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml',' freeswitch-users-bounces@lists.freeswitch.org');>] *On Behalf Of *Michael Jerris *Sent:* 15 August 2015 23:00 *To:* FreeSWITCH Users Help < freeswitch-users@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml',' freeswitch-users@lists.freeswitch.org');>> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues____

____

There are multiple known issues with Ubuntu 12.04. We reccomend Debian 8 Jessie.

On Friday, August 14, 2015, Shaun Stokes < shaun.stokes@itec-support.co.uk

<javascript:_e(%7B%7D,'cvml','shaun.stokes@itec-support.co.uk');>> wrote:____

Hi,____

____

We're experiencing performance issues with FreeSwitch, our target is 500 concurrent sessions, but at the moment this starts to bottleneck around 30.____

____

Host system:____

Windows Server 2012 Hyper-V Host____

AMD Opteron 4386 (2 processors) - 16 cores total____

128GB DDR3____

2TB RAID 5 (700MB/s tested read and write throughput)____

____

FreeSwitch Virtual Machine:____

FreeSwitch 1.4.15____

Ubuntu 12.04 LTS____

16 Virtual cores (high priority)____

2GB RAM (would assign more but FreeSwitch never seems to use much)____

500GB HD (on VHDX)____

____

After around 30 concurrent sessions we begin to see CPU spikes almost every time a new call comes in, as the sessions increase the size and frequency of the CPU spikes also increase. The system seems to be able to sit comfortably with over 100 concurrent sessions and 80% idle CPU, providing we don't have any new calls hitting the platform. The spikes are causing audio (RTP) to stutter or in some cases drop completely for a few seconds.____

____

The FreeSwitch spikes are occurring on all 16 cores, we have been monitoring the system using htop and mpstat.____

____

This is an example of when we receive an inbound call while we have 34 concurrent sessions:____

14:26:04 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle____

14:26:05 all 1.67 0.00 27.34 0.00 0.25 0.00 0.00 0.00 70.74____

14:26:05 0 1.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 71.00____

14:26:05 1 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____

14:26:05 2 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____

14:26:05 3 3.92 0.00 26.47 0.00 0.00 0.00 0.00 0.00 69.61____

14:26:05 4 0.99 0.00 25.74 0.00 1.98 0.00 0.00 0.00 71.29____

14:26:05 5 0.99 0.00 26.73 0.00 0.99 0.00 0.00 0.00 71.29____

14:26:05 6 0.00 0.00 26.47 0.00 0.98 0.00 0.00 0.00 72.55____

14:26:05 7 0.00 0.00 26.73 0.00 0.00 0.00 0.00 0.00 73.27____

14:26:05 8 3.96 0.00 27.72 0.00 0.00 0.00 0.00 0.00 68.32____

14:26:05 9 2.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00 71.00____

14:26:05 10 6.00 0.00 32.00 0.00 0.00 0.00 0.00 0.00 62.00____

14:26:05 11 1.96 0.00 27.45 0.00 0.00 0.00 0.00 0.00 70.59____

14:26:05 12 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____

14:26:05 13 1.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 71.00____

14:26:05 14 0.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00 73.00____

____

This is when the system is not receiving an inbound call but is sitting comfortably at 34 concurrent sessions:____

14:25:57 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle____

14:25:58 all 0.87 0.00 0.62 0.00 0.12 0.00 0.00 0.00 98.39____

14:25:58 0 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____

14:25:58 1 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____

14:25:58 2 0.98 0.00 0.98 0.00 0.00 0.00 0.00 0.00 98.04____

14:25:58 3 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____

14:25:58 4 0.00 0.00 0.00 0.00 1.98 0.00 0.00 0.00 98.02____

14:25:58 5 0.00 0.00 0.00 0.00 0.99 0.00 0.00 0.00 99.01____

14:25:58 6 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____

14:25:58 7 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____

14:25:58 8 5.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 94.00____

14:25:58 9 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00____

14:25:58 10 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00____

14:25:58 11 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____

14:25:58 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____

14:25:58 13 1.98 0.00 0.99 0.00 0.00 0.00 0.00 0.00 97.03____

14:25:58 14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____

14:25:58 15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____

____

We're using FreeSwitch in multi-tenant mode, we have tried dedicating a single VM for one tenancy but still experience the issue. My assumption has been that this will be due to the dialplan, I've optimized this slightly by writing LUA scripts to handle some of the inbound calls this appears to take away some of the load but we're still using the internal dialplans for each tenancy (tenancy dialplans have an average of 300 entries).____

____

We are using the following arguments when running FreeSwitch:____

-rp -nc -nonat____

____

I've seen another post which says we should operate FreeSwitch in High Performance mode using -hp but can't find anything information about this, is this still a valid argument to use with FreeSwitch?____

____

Has anyone experienced similar performance issues before or have any advice?____

____

Hope someone may be able help.____

____

Thanks,____

Shaun____

---

...

...

This message has been checked for all known viruses by

MessageLabs Virus Scanning Service.

...

_____________________________________________________________________ _____

---

...

...

This message has been checked for all known viruses by

MessageLabs Virus Scanning Service.

...

_____________________________________________________________________ _____

---

...

...

This message has been checked for all known viruses by

MessageLabs Virus Scanning Service.

...

_____________________________________________________________________ _____

---

...

...

Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <javascript:_e(%7B%7D,'cvml','

consulting@freeswitch.org');>

...

http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org

<javascript:_e(%7B%7D,'cvml','FreeSWITCH-users@lists.freeswitch.org');>

...

http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:

http://lists.freeswitch.org/mailman/options/freeswitch-users

...

http://www.freeswitch.org____

__ __

---

...

...

This message has been checked for all known viruses by MessageLabs

Virus Scanning Service.

...

_____________________________________________________________________ _____

---

...

...

This message has been checked for all known viruses by MessageLabs

Virus Scanning Service.

...

_____________________________________________________________________ _

_____________________________________________________________________ ____ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-us ers http://www.freeswitch.org

-- ------------------------------------------------------------ Nathan Neulinger nneul@mst.edu Missouri S&T Information Technology (573) 612-1412 System Administrator - Architect

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

--

*Brian West* brian@freeswitch.org

*Twitter: @FreeSWITCH , @briankwest* http://www.freeswitchbook.com http://www.freeswitchcookbook.com

Got Bugs? Report them here <https://freeswitch.org/jira>! | Reddit: /r/freeswitch <https://www.reddit.com/r/freeswitch>

*T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378) *iNUM:*+883 5100 1420 9001 | *ISN:*410*543 | *Skype:*briankwest

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Sincerely Jay

Years back, I used to run FS *on Windows*, under Hyper-V (2008), on a Q6600 and had considerably higher perf numbers. Hundreds of CPS (shit traffic), with 1000-2000 in progress channels (no media, ASR like 10% maybe). So something is certainly pathologically wrong if you’re seeing things get maxed out at the numbers you show. I didn’t do anything special, and did routing both in-process via mod_managed as well as xml_curl. Do you have a reason not to put the FS core SQLite db on a tmpfs? That’s a big win. Also make sure FS’s temp dir is a tmpfs. Unless you’re persisting and using stuff between sessions, there’s no point in going to disk right? And your random, small, IOPS are gonna be an order of magnitude less than what you posted, at least. One thing to be aware of is CPU pinning and scheduling. First, on multi-socket systems, you have NUMA. This really hurts. Raw memory bandwidth takes a 40% hit. More importantly, you’re toasting your caches (which with tons of threads happens a lot anyways). One number I like to reference is that single ownership transfer of one cache line from socket to another takes ~300 cycles (vs ~10 cycles for L2 and ~40 cycles for L3). Hyper-V has an option to lock servers to a single NUMA node. Make sure this is enabled (it may cause VMs not to start if it cannot allocate all the resources requested – this is a good thing for VoIP apps). Then cap the individual VMs so they aren’t scheduling all over the place (34 should not need more than 1 CPU). FS spawns tons of threads and they’ll end up running on all CPUs even if they don’t need all that time, and that then screws up cache usage even more. But more importantly, it sounds like this is a busy server? Are you sure you’re not just way oversubscribed and the VM is losing out? Your steal% is 0 so I’m guessing HV isn’t reporting that accurately. Only other thing that comes to mind is that Hyper-V does terrible with time, so if you’re relying on HV time service you’re gonna be skewing all over. I don’t think this should hurt FS directly, but you should run ntp anyways to be safe. If you end up needing to scale up on a single server, you’re better off running multiple FS instances, each pinned to specific cores. And you should have OpenSIPS or something in front to do LB (and it provides a ton of flexibility and handiness anyways). This probably only matters if you’re doing high CPS (dialer or shit traffic that mostly isn’t completing). I’m not sure what exactly you’re benching, but it sounds like your actual use case is different than the test, eh? Email me directly and we can chat more about this. -Michael From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Shaun Stokes Sent: Wednesday, September 2, 2015 5:09 AM To: 'FreeSWITCH Users Help' <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues Hi Jay, Thanks for the suggestion. We’re actually using PostgreSQL, the file system is a 2TB RAID5 with 10x 278GB SAS 10K RPM disks, read is 1GB/s (23,000 IOPS) write is 500MB/s (15,000 IOPS) but we’re only using a fraction of this. Thanks, Shaun From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of jay binks Sent: 02 September 2015 11:37 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues I may have missed it in your previous emails, but im assuming you have the core DB on a tempfs or somewhere other than the default location on spinning rust ? Jay On 2 September 2015 at 18:53, Shaun Stokes <shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>> wrote: Hi Brian, I’m a huge fan of virtualization, but in our environment it simply doesn’t provide the performance we’re looking for. The tests carried out were 10 second calls which were routed to a Call Centre Queue with one registered SIP extension as a member. There was a significant improvement when we tested on an Amazon VM in comparison to Hyper-V, but we’d like a system which can handle large volumes of call traffic, based on our tests this will need to be physical hardware to achieve the best performance. Further to this, we have managed to resolve our initial problem with CPU spikes on Hyper-V following these steps: • Disable VMQ on all physical and virtual adapters on the Hyper-V host which are linked to the FreeSwitch VM • Reduce the virtual cores on all VMs to at least half of the available physical cores • Increase the Processor Relative weight on the FreeSwitch VMs to 200 (default 100) Thanks, Shaun From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Brian West Sent: 01 September 2015 21:21 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues Shaun, 800 concurrent with 400cps, expectations are a little out of touch with reality and your test is no where near real world. Your call duration is ~2000ms, little to no time to actually do anything useful. You're burning the bulk of your CPU time on signaling and doing no real work with the session. So yes, you'll always get the best results out of bare metal, but you can also get good performance on a VM (*) (*) This is in no way an endorsement of running FreeSWITCH in a VM, Your mileage may vary depending on your specific application and load requirements. Thanks, On Tue, Sep 1, 2015 at 3:08 PM, Shaun Stokes <shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>> wrote: Update We've performed various tests and have come to the conclusion as advised by others, that visualization is causing bottlenecks. Using Amazon EC2 on an 8 core high performance virtual server this maxed out at around 800 concurrent sessions, 400 sessions per second. Using a Dell 1950 8 core server this was able to handle 1000 concurrent sessions, 400 sessions per second and still have plenty of resource available. Average CPU was fluctuating between 30-40% under peak load. Conclusion, if you want to run a high performance FreeSwitch server in a production you should stick with physical hardware and avoid visualization where possible. ________________________________________ From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] on behalf of Shaun Stokes [shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>] Sent: 28 August 2015 15:37 To: 'FreeSWITCH Users Help' Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues Hi Nathan, Thanks for the suggestion, we still see the CPU spikes when is under load after reducing to 8 cores but I think there may actually be an improvement in the audio quality. It's still not what we're expecting, so we will have to test our build on physical hardware too. Thanks, Shaun -----Original Message----- From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Nathan Neulinger Sent: 28 August 2015 15:04 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues Yes, I know you said Hyper-V, I don't know if it has the same sort of scheduler behavior or not, just mentioned the vmware for context. On 08/28/2015 08:46 AM, Nathan Neulinger wrote:

One thing that I see jump out - you've probably got too many virtual cores defined on that VM, particularly since you've assigned # vcpu the same as the number of real cores on the host system.

Try your test again dropping that to 8 virtual CPUs and I think you might see better results. (Or at least drop to less than 100% of the total cores on the hardware.)

Particularly if this host system has other vms running on it, you're hurting the vm scheduler's ability to hand out resources. With vmware, I believe a simple rule of thumb is that scheduler requires N-1 CPUs to be locked to schedule execution of an N cpu virtual machine - so even if it doesn't need all of the CPUs, it has to reserve all of them any time you have more than a 2 vCPU vm.

-- Nathan

On 08/28/2015 08:18 AM, Michael Jerris wrote:

...

The numbers your posting are fairly low, I'm guessing this is a grossly underpowered vm that is causing your issues

On Friday, August 28, 2015, Shaun Stokes <shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk> <mailto:shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>>> wrote:

Hi Peter,____

__ __

Thanks for the advice, this is something we're already looking into but we don't have the new hardware available yet.____

__ __

Debian 8 has great integration with Hyper-V on 2012R2 and operates as a Generation 2 VM, obviously it's never going to be as good as running directly on the hardware but we were hoping for better performance.____

__ __

Thanks,____

Shaun____

__ __

*From:*freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>');> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>');>] *On Behalf Of *Peter Olsson *Sent:* 28 August 2015 10:43 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org> <javascript:_e(%7B%7D,'cvml','freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>');>> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues____

__ __

If you want high performance, I recommend using real hardware, not a virtual machine setup - it will cause you issues.____

__ __

/Peter____

__ __

2015-08-28 9:44 GMT+02:00 Shaun Stokes <shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>

<javascript:_e(%7B%7D,'cvml','shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>');>>:__ __

Hi Michael,____

____

Thanks for the response and recommendation.____

____

Our new build is now on Debian 8 Jessie, performance on the system is noticeably better and any audio problems while the system is under load are significantly reduced but we do still experience spikes along with reduced audio quality when the system is processing 20 calls per second while already maintaining 80 existing calls.____

____

We're still hitting some kind of bottleneck, what should we expect to be able to support on a single piece of hardware? We've kept our dialplans as small as possible and we're using Memcache, but something tells me it's the call setup process which is triggering the spikes, is this possibly a limitation on the SIP profiles which are single threaded?____

____

I also see FreeSWITCH supports clustering but it's in relation to high availability, is it possible to cluster a single FreeSWITCH instance across multiple hardware or should we be looking elsewhere to solve these bottlenecks?____

____

We switched to TCP Vegas on our new build as recommended on the FreeSWITCH Wiki, this has provided a noticeable reduction in audio latency but doesn't solve the issue with CPU spikes.____

____

Many Thanks,____

Shaun____

____

*From:*freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>');> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>');>] *On Behalf Of *Michael Jerris *Sent:* 15 August 2015 23:00 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org> <javascript:_e(%7B%7D,'cvml','freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>');>> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues____

____

There are multiple known issues with Ubuntu 12.04. We reccomend Debian 8 Jessie.

On Friday, August 14, 2015, Shaun Stokes <shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>

<javascript:_e(%7B%7D,'cvml','shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>');>> wrote:____

Hi,____

____

We're experiencing performance issues with FreeSwitch, our target is 500 concurrent sessions, but at the moment this starts to bottleneck around 30.____

____

Host system:____

Windows Server 2012 Hyper-V Host____

AMD Opteron 4386 (2 processors) - 16 cores total____

128GB DDR3____

2TB RAID 5 (700MB/s tested read and write throughput)____

____

FreeSwitch Virtual Machine:____

FreeSwitch 1.4.15____

Ubuntu 12.04 LTS____

16 Virtual cores (high priority)____

2GB RAM (would assign more but FreeSwitch never seems to use much)____

500GB HD (on VHDX)____

____

After around 30 concurrent sessions we begin to see CPU spikes almost every time a new call comes in, as the sessions increase the size and frequency of the CPU spikes also increase. The system seems to be able to sit comfortably with over 100 concurrent sessions and 80% idle CPU, providing we don't have any new calls hitting the platform. The spikes are causing audio (RTP) to stutter or in some cases drop completely for a few seconds.____

____

The FreeSwitch spikes are occurring on all 16 cores, we have been monitoring the system using htop and mpstat.____

____

This is an example of when we receive an inbound call while we have 34 concurrent sessions:____

14:26:04 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle____

14:26:05 all 1.67 0.00 27.34 0.00 0.25 0.00 0.00 0.00 70.74____

14:26:05 0 1.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 71.00____

14:26:05 1 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____

14:26:05 2 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____

14:26:05 3 3.92 0.00 26.47 0.00 0.00 0.00 0.00 0.00 69.61____

14:26:05 4 0.99 0.00 25.74 0.00 1.98 0.00 0.00 0.00 71.29____

14:26:05 5 0.99 0.00 26.73 0.00 0.99 0.00 0.00 0.00 71.29____

14:26:05 6 0.00 0.00 26.47 0.00 0.98 0.00 0.00 0.00 72.55____

14:26:05 7 0.00 0.00 26.73 0.00 0.00 0.00 0.00 0.00 73.27____

14:26:05 8 3.96 0.00 27.72 0.00 0.00 0.00 0.00 0.00 68.32____

14:26:05 9 2.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00 71.00____

14:26:05 10 6.00 0.00 32.00 0.00 0.00 0.00 0.00 0.00 62.00____

14:26:05 11 1.96 0.00 27.45 0.00 0.00 0.00 0.00 0.00 70.59____

14:26:05 12 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____

14:26:05 13 1.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 71.00____

14:26:05 14 0.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00 73.00____

____

This is when the system is not receiving an inbound call but is sitting comfortably at 34 concurrent sessions:____

14:25:57 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle____

14:25:58 all 0.87 0.00 0.62 0.00 0.12 0.00 0.00 0.00 98.39____

14:25:58 0 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____

14:25:58 1 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____

14:25:58 2 0.98 0.00 0.98 0.00 0.00 0.00 0.00 0.00 98.04____

14:25:58 3 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____

14:25:58 4 0.00 0.00 0.00 0.00 1.98 0.00 0.00 0.00 98.02____

14:25:58 5 0.00 0.00 0.00 0.00 0.99 0.00 0.00 0.00 99.01____

14:25:58 6 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____

14:25:58 7 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____

14:25:58 8 5.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 94.00____

14:25:58 9 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00____

14:25:58 10 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00____

14:25:58 11 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____

14:25:58 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____

14:25:58 13 1.98 0.00 0.99 0.00 0.00 0.00 0.00 0.00 97.03____

14:25:58 14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____

14:25:58 15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____

____

We're using FreeSwitch in multi-tenant mode, we have tried dedicating a single VM for one tenancy but still experience the issue. My assumption has been that this will be due to the dialplan, I've optimized this slightly by writing LUA scripts to handle some of the inbound calls this appears to take away some of the load but we're still using the internal dialplans for each tenancy (tenancy dialplans have an average of 300 entries).____

____

We are using the following arguments when running FreeSwitch:____

-rp -nc -nonat____

____

I've seen another post which says we should operate FreeSwitch in High Performance mode using -hp but can't find anything information about this, is this still a valid argument to use with FreeSwitch?____

____

Has anyone experienced similar performance issues before or have any advice?____

____

Hope someone may be able help.____

____

Thanks,____

Shaun____

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service.

_____________________________________________________________________ _____

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service.

_____________________________________________________________________ _____

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service.

_____________________________________________________________________ _____

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> <javascript:_e(%7B%7D,'cvml','consulting@freeswitch.org<mailto:consulting@freeswitch.org>');> http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> <javascript:_e(%7B%7D,'cvml','FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org>');> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org____

__ __

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service.

_____________________________________________________________________ _____

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service.

_____________________________________________________________________ _

_____________________________________________________________________ ____ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-us ers http://www.freeswitch.org

-- ------------------------------------------------------------ Nathan Neulinger nneul@mst.edu<mailto:nneul@mst.edu> Missouri S&T Information Technology (573) 612-1412<tel:%28573%29%20612-1412> System Administrator - Architect _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Brian West brian@freeswitch.org<mailto:brian@freeswitch.org> [http://billing.freeswitch.org/templates/default/img/whmcslogo.png] Twitter: @FreeSWITCH , @briankwest http://www.freeswitchbook.com http://www.freeswitchcookbook.com Got Bugs? Report them here<https://freeswitch.org/jira>! | Reddit: /r/freeswitch<https://www.reddit.com/r/freeswitch> T:+19184209001<tel:%2B19184209001> | F:+19184209002<tel:%2B19184209002> | M:+1918424WEST (9378) iNUM:+883 5100 1420 9001 | ISN:410*543 | Skype:briankwest ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Sincerely Jay ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________

Hi Michael, Thanks for the response, it's good to know there's someone out there who's used FreeSwitch on Hyper-V with decent performance. SQLite isn't an issue, we use PostgreSQL and our file system shouldn't be a bottleneck, we have fast storage with high IOPS. Although the database does live on a VHDX file, I'm not sure if this could be a bottleneck? Our VMs are currently be spanning multiple NUMA nodes (NUMA spanning), thanks for pointing this out, we will look at locking our VMs to a single NUMA node. The server isn't particularly busy, it's dedicated for Lync Front End, Lync Mediation, Lync Edge and FreeSwitch. The Lync servers are usually around 5% CPU under load, FreeSwitch is 10-30% under load. We'll also look at using a single virtual core and setting NTP, although we had adjusted CPU weight so FreeSwitch should always take priority over other VMs. We're currently doing around 100 concurrent calls (200 sessions) with peaks of roughly 30 calls per second (60 sessions per second), we would like to grow this significantly. Our customers are spread across two FreeSwitch servers on separate physical hardware (same spec). We're not currently doing load balancing as our customers are physically build on separate FreeSwitch VMs, but it would be very nice to have this feature, it makes the platform much more scalable. I'll have to take a look at OpenSIPS and see exactly how this works. I'll let you know how we get on with our Hyper-V optimizations. Thanks, Shaun ________________________________ From: freeswitch-users-bounces@lists.freeswitch.org [freeswitch-users-bounces@lists.freeswitch.org] on behalf of Michael Giagnocavo [mgg@giagnocavo.net] Sent: 02 September 2015 21:19 To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues Years back, I used to run FS *on Windows*, under Hyper-V (2008), on a Q6600 and had considerably higher perf numbers. Hundreds of CPS (shit traffic), with 1000-2000 in progress channels (no media, ASR like 10% maybe). So something is certainly pathologically wrong if you're seeing things get maxed out at the numbers you show. I didn't do anything special, and did routing both in-process via mod_managed as well as xml_curl. Do you have a reason not to put the FS core SQLite db on a tmpfs? That's a big win. Also make sure FS's temp dir is a tmpfs. Unless you're persisting and using stuff between sessions, there's no point in going to disk right? And your random, small, IOPS are gonna be an order of magnitude less than what you posted, at least. One thing to be aware of is CPU pinning and scheduling. First, on multi-socket systems, you have NUMA. This really hurts. Raw memory bandwidth takes a 40% hit. More importantly, you're toasting your caches (which with tons of threads happens a lot anyways). One number I like to reference is that single ownership transfer of one cache line from socket to another takes ~300 cycles (vs ~10 cycles for L2 and ~40 cycles for L3). Hyper-V has an option to lock servers to a single NUMA node. Make sure this is enabled (it may cause VMs not to start if it cannot allocate all the resources requested - this is a good thing for VoIP apps). Then cap the individual VMs so they aren't scheduling all over the place (34 should not need more than 1 CPU). FS spawns tons of threads and they'll end up running on all CPUs even if they don't need all that time, and that then screws up cache usage even more. But more importantly, it sounds like this is a busy server? Are you sure you're not just way oversubscribed and the VM is losing out? Your steal% is 0 so I'm guessing HV isn't reporting that accurately. Only other thing that comes to mind is that Hyper-V does terrible with time, so if you're relying on HV time service you're gonna be skewing all over. I don't think this should hurt FS directly, but you should run ntp anyways to be safe. If you end up needing to scale up on a single server, you're better off running multiple FS instances, each pinned to specific cores. And you should have OpenSIPS or something in front to do LB (and it provides a ton of flexibility and handiness anyways). This probably only matters if you're doing high CPS (dialer or shit traffic that mostly isn't completing). I'm not sure what exactly you're benching, but it sounds like your actual use case is different than the test, eh? Email me directly and we can chat more about this. -Michael From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Shaun Stokes Sent: Wednesday, September 2, 2015 5:09 AM To: 'FreeSWITCH Users Help' <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues Hi Jay, Thanks for the suggestion. We're actually using PostgreSQL, the file system is a 2TB RAID5 with 10x 278GB SAS 10K RPM disks, read is 1GB/s (23,000 IOPS) write is 500MB/s (15,000 IOPS) but we're only using a fraction of this. Thanks, Shaun From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of jay binks Sent: 02 September 2015 11:37 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues I may have missed it in your previous emails, but im assuming you have the core DB on a tempfs or somewhere other than the default location on spinning rust ? Jay On 2 September 2015 at 18:53, Shaun Stokes <shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>> wrote: Hi Brian, I'm a huge fan of virtualization, but in our environment it simply doesn't provide the performance we're looking for. The tests carried out were 10 second calls which were routed to a Call Centre Queue with one registered SIP extension as a member. There was a significant improvement when we tested on an Amazon VM in comparison to Hyper-V, but we'd like a system which can handle large volumes of call traffic, based on our tests this will need to be physical hardware to achieve the best performance. Further to this, we have managed to resolve our initial problem with CPU spikes on Hyper-V following these steps: * Disable VMQ on all physical and virtual adapters on the Hyper-V host which are linked to the FreeSwitch VM * Reduce the virtual cores on all VMs to at least half of the available physical cores * Increase the Processor Relative weight on the FreeSwitch VMs to 200 (default 100) Thanks, Shaun From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Brian West Sent: 01 September 2015 21:21 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues Shaun, 800 concurrent with 400cps, expectations are a little out of touch with reality and your test is no where near real world. Your call duration is ~2000ms, little to no time to actually do anything useful. You're burning the bulk of your CPU time on signaling and doing no real work with the session. So yes, you'll always get the best results out of bare metal, but you can also get good performance on a VM (*) (*) This is in no way an endorsement of running FreeSWITCH in a VM, Your mileage may vary depending on your specific application and load requirements. Thanks, On Tue, Sep 1, 2015 at 3:08 PM, Shaun Stokes <shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>> wrote: Update We've performed various tests and have come to the conclusion as advised by others, that visualization is causing bottlenecks. Using Amazon EC2 on an 8 core high performance virtual server this maxed out at around 800 concurrent sessions, 400 sessions per second. Using a Dell 1950 8 core server this was able to handle 1000 concurrent sessions, 400 sessions per second and still have plenty of resource available. Average CPU was fluctuating between 30-40% under peak load. Conclusion, if you want to run a high performance FreeSwitch server in a production you should stick with physical hardware and avoid visualization where possible. ________________________________________ From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] on behalf of Shaun Stokes [shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>] Sent: 28 August 2015 15:37 To: 'FreeSWITCH Users Help' Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues Hi Nathan, Thanks for the suggestion, we still see the CPU spikes when is under load after reducing to 8 cores but I think there may actually be an improvement in the audio quality. It's still not what we're expecting, so we will have to test our build on physical hardware too. Thanks, Shaun -----Original Message----- From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Nathan Neulinger Sent: 28 August 2015 15:04 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues Yes, I know you said Hyper-V, I don't know if it has the same sort of scheduler behavior or not, just mentioned the vmware for context. On 08/28/2015 08:46 AM, Nathan Neulinger wrote:

One thing that I see jump out - you've probably got too many virtual cores defined on that VM, particularly since you've assigned # vcpu the same as the number of real cores on the host system.

Try your test again dropping that to 8 virtual CPUs and I think you might see better results. (Or at least drop to less than 100% of the total cores on the hardware.)

Particularly if this host system has other vms running on it, you're hurting the vm scheduler's ability to hand out resources. With vmware, I believe a simple rule of thumb is that scheduler requires N-1 CPUs to be locked to schedule execution of an N cpu virtual machine - so even if it doesn't need all of the CPUs, it has to reserve all of them any time you have more than a 2 vCPU vm.

-- Nathan

On 08/28/2015 08:18 AM, Michael Jerris wrote:

...

The numbers your posting are fairly low, I'm guessing this is a grossly underpowered vm that is causing your issues

On Friday, August 28, 2015, Shaun Stokes <shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk> <mailto:shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>>> wrote:

Hi Peter,____

__ __

Thanks for the advice, this is something we're already looking into but we don't have the new hardware available yet.____

__ __

Debian 8 has great integration with Hyper-V on 2012R2 and operates as a Generation 2 VM, obviously it's never going to be as good as running directly on the hardware but we were hoping for better performance.____

__ __

Thanks,____

Shaun____

__ __

*From:*freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>');> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>');>] *On Behalf Of *Peter Olsson *Sent:* 28 August 2015 10:43 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org> <javascript:_e(%7B%7D,'cvml','freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>');>> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues____

__ __

If you want high performance, I recommend using real hardware, not a virtual machine setup - it will cause you issues.____

__ __

/Peter____

__ __

2015-08-28 9:44 GMT+02:00 Shaun Stokes <shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>

<javascript:_e(%7B%7D,'cvml','shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>');>>:__ __

Hi Michael,____

____

Thanks for the response and recommendation.____

____

Our new build is now on Debian 8 Jessie, performance on the system is noticeably better and any audio problems while the system is under load are significantly reduced but we do still experience spikes along with reduced audio quality when the system is processing 20 calls per second while already maintaining 80 existing calls.____

____

We're still hitting some kind of bottleneck, what should we expect to be able to support on a single piece of hardware? We've kept our dialplans as small as possible and we're using Memcache, but something tells me it's the call setup process which is triggering the spikes, is this possibly a limitation on the SIP profiles which are single threaded?____

____

I also see FreeSWITCH supports clustering but it's in relation to high availability, is it possible to cluster a single FreeSWITCH instance across multiple hardware or should we be looking elsewhere to solve these bottlenecks?____

____

We switched to TCP Vegas on our new build as recommended on the FreeSWITCH Wiki, this has provided a noticeable reduction in audio latency but doesn't solve the issue with CPU spikes.____

____

Many Thanks,____

Shaun____

____

*From:*freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>');> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>');>] *On Behalf Of *Michael Jerris *Sent:* 15 August 2015 23:00 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org> <javascript:_e(%7B%7D,'cvml','freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>');>> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues____

____

There are multiple known issues with Ubuntu 12.04. We reccomend Debian 8 Jessie.

On Friday, August 14, 2015, Shaun Stokes <shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>

<javascript:_e(%7B%7D,'cvml','shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>');>> wrote:____

Hi,____

____

We're experiencing performance issues with FreeSwitch, our target is 500 concurrent sessions, but at the moment this starts to bottleneck around 30.____

____

Host system:____

Windows Server 2012 Hyper-V Host____

AMD Opteron 4386 (2 processors) - 16 cores total____

128GB DDR3____

2TB RAID 5 (700MB/s tested read and write throughput)____

____

FreeSwitch Virtual Machine:____

FreeSwitch 1.4.15____

Ubuntu 12.04 LTS____

16 Virtual cores (high priority)____

2GB RAM (would assign more but FreeSwitch never seems to use much)____

500GB HD (on VHDX)____

____

After around 30 concurrent sessions we begin to see CPU spikes almost every time a new call comes in, as the sessions increase the size and frequency of the CPU spikes also increase. The system seems to be able to sit comfortably with over 100 concurrent sessions and 80% idle CPU, providing we don't have any new calls hitting the platform. The spikes are causing audio (RTP) to stutter or in some cases drop completely for a few seconds.____

____

The FreeSwitch spikes are occurring on all 16 cores, we have been monitoring the system using htop and mpstat.____

____

This is an example of when we receive an inbound call while we have 34 concurrent sessions:____

14:26:04 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle____

14:26:05 all 1.67 0.00 27.34 0.00 0.25 0.00 0.00 0.00 70.74____

14:26:05 0 1.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 71.00____

14:26:05 1 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____

14:26:05 2 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____

14:26:05 3 3.92 0.00 26.47 0.00 0.00 0.00 0.00 0.00 69.61____

14:26:05 4 0.99 0.00 25.74 0.00 1.98 0.00 0.00 0.00 71.29____

14:26:05 5 0.99 0.00 26.73 0.00 0.99 0.00 0.00 0.00 71.29____

14:26:05 6 0.00 0.00 26.47 0.00 0.98 0.00 0.00 0.00 72.55____

14:26:05 7 0.00 0.00 26.73 0.00 0.00 0.00 0.00 0.00 73.27____

14:26:05 8 3.96 0.00 27.72 0.00 0.00 0.00 0.00 0.00 68.32____

14:26:05 9 2.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00 71.00____

14:26:05 10 6.00 0.00 32.00 0.00 0.00 0.00 0.00 0.00 62.00____

14:26:05 11 1.96 0.00 27.45 0.00 0.00 0.00 0.00 0.00 70.59____

14:26:05 12 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____

14:26:05 13 1.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 71.00____

14:26:05 14 0.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00 73.00____

____

This is when the system is not receiving an inbound call but is sitting comfortably at 34 concurrent sessions:____

14:25:57 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle____

14:25:58 all 0.87 0.00 0.62 0.00 0.12 0.00 0.00 0.00 98.39____

14:25:58 0 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____

14:25:58 1 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____

14:25:58 2 0.98 0.00 0.98 0.00 0.00 0.00 0.00 0.00 98.04____

14:25:58 3 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____

14:25:58 4 0.00 0.00 0.00 0.00 1.98 0.00 0.00 0.00 98.02____

14:25:58 5 0.00 0.00 0.00 0.00 0.99 0.00 0.00 0.00 99.01____

14:25:58 6 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____

14:25:58 7 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____

14:25:58 8 5.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 94.00____

14:25:58 9 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00____

14:25:58 10 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00____

14:25:58 11 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____

14:25:58 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____

14:25:58 13 1.98 0.00 0.99 0.00 0.00 0.00 0.00 0.00 97.03____

14:25:58 14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____

14:25:58 15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____

____

We're using FreeSwitch in multi-tenant mode, we have tried dedicating a single VM for one tenancy but still experience the issue. My assumption has been that this will be due to the dialplan, I've optimized this slightly by writing LUA scripts to handle some of the inbound calls this appears to take away some of the load but we're still using the internal dialplans for each tenancy (tenancy dialplans have an average of 300 entries).____

____

We are using the following arguments when running FreeSwitch:____

-rp -nc -nonat____

____

I've seen another post which says we should operate FreeSwitch in High Performance mode using -hp but can't find anything information about this, is this still a valid argument to use with FreeSwitch?____

____

Has anyone experienced similar performance issues before or have any advice?____

____

Hope someone may be able help.____

____

Thanks,____

Shaun____

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service.

_____________________________________________________________________ _____

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service.

_____________________________________________________________________ _____

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service.

_____________________________________________________________________ _____

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> <javascript:_e(%7B%7D,'cvml','consulting@freeswitch.org<mailto:consulting@freeswitch.org>');> http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> <javascript:_e(%7B%7D,'cvml','FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org>');> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org____

__ __

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service.

_____________________________________________________________________ _____

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service.

_____________________________________________________________________ _

_____________________________________________________________________ ____ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-us ers http://www.freeswitch.org

-- ------------------------------------------------------------ Nathan Neulinger nneul@mst.edu<mailto:nneul@mst.edu> Missouri S&T Information Technology (573) 612-1412<tel:%28573%29%20612-1412> System Administrator - Architect _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Brian West brian@freeswitch.org<mailto:brian@freeswitch.org> [http://billing.freeswitch.org/templates/default/img/whmcslogo.png] Twitter: @FreeSWITCH , @briankwest http://www.freeswitchbook.com http://www.freeswitchcookbook.com Got Bugs? Report them here<https://freeswitch.org/jira>! | Reddit: /r/freeswitch<https://www.reddit.com/r/freeswitch> T:+19184209001<tel:%2B19184209001> | F:+19184209002<tel:%2B19184209002> | M:+1918424WEST (9378) iNUM:+883 5100 1420 9001 | ISN:410*543 | Skype:briankwest ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Sincerely Jay ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________

Hi Michael, We’re now seeing suitable performance on Hyper-V, but as we’re processing media\RTP we will be moving to physical hardware. If we need FreeSwitch servers to process SIP messages only we will consider these for virtualization. Thanks for your help, Shaun From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Michael Giagnocavo Sent: 02 September 2015 21:19 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues Years back, I used to run FS *on Windows*, under Hyper-V (2008), on a Q6600 and had considerably higher perf numbers. Hundreds of CPS (shit traffic), with 1000-2000 in progress channels (no media, ASR like 10% maybe). So something is certainly pathologically wrong if you’re seeing things get maxed out at the numbers you show. I didn’t do anything special, and did routing both in-process via mod_managed as well as xml_curl. Do you have a reason not to put the FS core SQLite db on a tmpfs? That’s a big win. Also make sure FS’s temp dir is a tmpfs. Unless you’re persisting and using stuff between sessions, there’s no point in going to disk right? And your random, small, IOPS are gonna be an order of magnitude less than what you posted, at least. One thing to be aware of is CPU pinning and scheduling. First, on multi-socket systems, you have NUMA. This really hurts. Raw memory bandwidth takes a 40% hit. More importantly, you’re toasting your caches (which with tons of threads happens a lot anyways). One number I like to reference is that single ownership transfer of one cache line from socket to another takes ~300 cycles (vs ~10 cycles for L2 and ~40 cycles for L3). Hyper-V has an option to lock servers to a single NUMA node. Make sure this is enabled (it may cause VMs not to start if it cannot allocate all the resources requested – this is a good thing for VoIP apps). Then cap the individual VMs so they aren’t scheduling all over the place (34 should not need more than 1 CPU). FS spawns tons of threads and they’ll end up running on all CPUs even if they don’t need all that time, and that then screws up cache usage even more. But more importantly, it sounds like this is a busy server? Are you sure you’re not just way oversubscribed and the VM is losing out? Your steal% is 0 so I’m guessing HV isn’t reporting that accurately. Only other thing that comes to mind is that Hyper-V does terrible with time, so if you’re relying on HV time service you’re gonna be skewing all over. I don’t think this should hurt FS directly, but you should run ntp anyways to be safe. If you end up needing to scale up on a single server, you’re better off running multiple FS instances, each pinned to specific cores. And you should have OpenSIPS or something in front to do LB (and it provides a ton of flexibility and handiness anyways). This probably only matters if you’re doing high CPS (dialer or shit traffic that mostly isn’t completing). I’m not sure what exactly you’re benching, but it sounds like your actual use case is different than the test, eh? Email me directly and we can chat more about this. -Michael From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Shaun Stokes Sent: Wednesday, September 2, 2015 5:09 AM To: 'FreeSWITCH Users Help' <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues Hi Jay, Thanks for the suggestion. We’re actually using PostgreSQL, the file system is a 2TB RAID5 with 10x 278GB SAS 10K RPM disks, read is 1GB/s (23,000 IOPS) write is 500MB/s (15,000 IOPS) but we’re only using a fraction of this. Thanks, Shaun From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of jay binks Sent: 02 September 2015 11:37 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues I may have missed it in your previous emails, but im assuming you have the core DB on a tempfs or somewhere other than the default location on spinning rust ? Jay On 2 September 2015 at 18:53, Shaun Stokes <shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>> wrote: Hi Brian, I’m a huge fan of virtualization, but in our environment it simply doesn’t provide the performance we’re looking for. The tests carried out were 10 second calls which were routed to a Call Centre Queue with one registered SIP extension as a member. There was a significant improvement when we tested on an Amazon VM in comparison to Hyper-V, but we’d like a system which can handle large volumes of call traffic, based on our tests this will need to be physical hardware to achieve the best performance. Further to this, we have managed to resolve our initial problem with CPU spikes on Hyper-V following these steps: • Disable VMQ on all physical and virtual adapters on the Hyper-V host which are linked to the FreeSwitch VM • Reduce the virtual cores on all VMs to at least half of the available physical cores • Increase the Processor Relative weight on the FreeSwitch VMs to 200 (default 100) Thanks, Shaun From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Brian West Sent: 01 September 2015 21:21 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues Shaun, 800 concurrent with 400cps, expectations are a little out of touch with reality and your test is no where near real world. Your call duration is ~2000ms, little to no time to actually do anything useful. You're burning the bulk of your CPU time on signaling and doing no real work with the session. So yes, you'll always get the best results out of bare metal, but you can also get good performance on a VM (*) (*) This is in no way an endorsement of running FreeSWITCH in a VM, Your mileage may vary depending on your specific application and load requirements. Thanks, On Tue, Sep 1, 2015 at 3:08 PM, Shaun Stokes <shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>> wrote: Update We've performed various tests and have come to the conclusion as advised by others, that visualization is causing bottlenecks. Using Amazon EC2 on an 8 core high performance virtual server this maxed out at around 800 concurrent sessions, 400 sessions per second. Using a Dell 1950 8 core server this was able to handle 1000 concurrent sessions, 400 sessions per second and still have plenty of resource available. Average CPU was fluctuating between 30-40% under peak load. Conclusion, if you want to run a high performance FreeSwitch server in a production you should stick with physical hardware and avoid visualization where possible. ________________________________________ From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] on behalf of Shaun Stokes [shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>] Sent: 28 August 2015 15:37 To: 'FreeSWITCH Users Help' Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues Hi Nathan, Thanks for the suggestion, we still see the CPU spikes when is under load after reducing to 8 cores but I think there may actually be an improvement in the audio quality. It's still not what we're expecting, so we will have to test our build on physical hardware too. Thanks, Shaun -----Original Message----- From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Nathan Neulinger Sent: 28 August 2015 15:04 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues Yes, I know you said Hyper-V, I don't know if it has the same sort of scheduler behavior or not, just mentioned the vmware for context. On 08/28/2015 08:46 AM, Nathan Neulinger wrote:

One thing that I see jump out - you've probably got too many virtual cores defined on that VM, particularly since you've assigned # vcpu the same as the number of real cores on the host system.

Try your test again dropping that to 8 virtual CPUs and I think you might see better results. (Or at least drop to less than 100% of the total cores on the hardware.)

Particularly if this host system has other vms running on it, you're hurting the vm scheduler's ability to hand out resources. With vmware, I believe a simple rule of thumb is that scheduler requires N-1 CPUs to be locked to schedule execution of an N cpu virtual machine - so even if it doesn't need all of the CPUs, it has to reserve all of them any time you have more than a 2 vCPU vm.

-- Nathan

On 08/28/2015 08:18 AM, Michael Jerris wrote:

...

The numbers your posting are fairly low, I'm guessing this is a grossly underpowered vm that is causing your issues

On Friday, August 28, 2015, Shaun Stokes <shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk> <mailto:shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>>> wrote:

Hi Peter,____

__ __

Thanks for the advice, this is something we're already looking into but we don't have the new hardware available yet.____

__ __

Debian 8 has great integration with Hyper-V on 2012R2 and operates as a Generation 2 VM, obviously it's never going to be as good as running directly on the hardware but we were hoping for better performance.____

__ __

Thanks,____

Shaun____

__ __

*From:*freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>');> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>');>] *On Behalf Of *Peter Olsson *Sent:* 28 August 2015 10:43 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org> <javascript:_e(%7B%7D,'cvml','freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>');>> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues____

__ __

If you want high performance, I recommend using real hardware, not a virtual machine setup - it will cause you issues.____

__ __

/Peter____

__ __

2015-08-28 9:44 GMT+02:00 Shaun Stokes <shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>

<javascript:_e(%7B%7D,'cvml','shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>');>>:__ __

Hi Michael,____

____

Thanks for the response and recommendation.____

____

Our new build is now on Debian 8 Jessie, performance on the system is noticeably better and any audio problems while the system is under load are significantly reduced but we do still experience spikes along with reduced audio quality when the system is processing 20 calls per second while already maintaining 80 existing calls.____

____

We're still hitting some kind of bottleneck, what should we expect to be able to support on a single piece of hardware? We've kept our dialplans as small as possible and we're using Memcache, but something tells me it's the call setup process which is triggering the spikes, is this possibly a limitation on the SIP profiles which are single threaded?____

____

I also see FreeSWITCH supports clustering but it's in relation to high availability, is it possible to cluster a single FreeSWITCH instance across multiple hardware or should we be looking elsewhere to solve these bottlenecks?____

____

We switched to TCP Vegas on our new build as recommended on the FreeSWITCH Wiki, this has provided a noticeable reduction in audio latency but doesn't solve the issue with CPU spikes.____

____

Many Thanks,____

Shaun____

____

*From:*freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>');> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>');>] *On Behalf Of *Michael Jerris *Sent:* 15 August 2015 23:00 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org> <javascript:_e(%7B%7D,'cvml','freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>');>> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues____

____

There are multiple known issues with Ubuntu 12.04. We reccomend Debian 8 Jessie.

On Friday, August 14, 2015, Shaun Stokes <shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>

<javascript:_e(%7B%7D,'cvml','shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>');>> wrote:____

Hi,____

____

We're experiencing performance issues with FreeSwitch, our target is 500 concurrent sessions, but at the moment this starts to bottleneck around 30.____

____

Host system:____

Windows Server 2012 Hyper-V Host____

AMD Opteron 4386 (2 processors) - 16 cores total____

128GB DDR3____

2TB RAID 5 (700MB/s tested read and write throughput)____

____

FreeSwitch Virtual Machine:____

FreeSwitch 1.4.15____

Ubuntu 12.04 LTS____

16 Virtual cores (high priority)____

2GB RAM (would assign more but FreeSwitch never seems to use much)____

500GB HD (on VHDX)____

____

After around 30 concurrent sessions we begin to see CPU spikes almost every time a new call comes in, as the sessions increase the size and frequency of the CPU spikes also increase. The system seems to be able to sit comfortably with over 100 concurrent sessions and 80% idle CPU, providing we don't have any new calls hitting the platform. The spikes are causing audio (RTP) to stutter or in some cases drop completely for a few seconds.____

____

The FreeSwitch spikes are occurring on all 16 cores, we have been monitoring the system using htop and mpstat.____

____

This is an example of when we receive an inbound call while we have 34 concurrent sessions:____

14:26:04 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle____

14:26:05 all 1.67 0.00 27.34 0.00 0.25 0.00 0.00 0.00 70.74____

14:26:05 0 1.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 71.00____

14:26:05 1 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____

14:26:05 2 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____

14:26:05 3 3.92 0.00 26.47 0.00 0.00 0.00 0.00 0.00 69.61____

14:26:05 4 0.99 0.00 25.74 0.00 1.98 0.00 0.00 0.00 71.29____

14:26:05 5 0.99 0.00 26.73 0.00 0.99 0.00 0.00 0.00 71.29____

14:26:05 6 0.00 0.00 26.47 0.00 0.98 0.00 0.00 0.00 72.55____

14:26:05 7 0.00 0.00 26.73 0.00 0.00 0.00 0.00 0.00 73.27____

14:26:05 8 3.96 0.00 27.72 0.00 0.00 0.00 0.00 0.00 68.32____

14:26:05 9 2.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00 71.00____

14:26:05 10 6.00 0.00 32.00 0.00 0.00 0.00 0.00 0.00 62.00____

14:26:05 11 1.96 0.00 27.45 0.00 0.00 0.00 0.00 0.00 70.59____

14:26:05 12 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____

14:26:05 13 1.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 71.00____

14:26:05 14 0.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00 73.00____

____

This is when the system is not receiving an inbound call but is sitting comfortably at 34 concurrent sessions:____

14:25:57 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle____

14:25:58 all 0.87 0.00 0.62 0.00 0.12 0.00 0.00 0.00 98.39____

14:25:58 0 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____

14:25:58 1 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____

14:25:58 2 0.98 0.00 0.98 0.00 0.00 0.00 0.00 0.00 98.04____

14:25:58 3 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____

14:25:58 4 0.00 0.00 0.00 0.00 1.98 0.00 0.00 0.00 98.02____

14:25:58 5 0.00 0.00 0.00 0.00 0.99 0.00 0.00 0.00 99.01____

14:25:58 6 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____

14:25:58 7 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____

14:25:58 8 5.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 94.00____

14:25:58 9 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00____

14:25:58 10 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00____

14:25:58 11 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____

14:25:58 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____

14:25:58 13 1.98 0.00 0.99 0.00 0.00 0.00 0.00 0.00 97.03____

14:25:58 14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____

14:25:58 15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____

____

We're using FreeSwitch in multi-tenant mode, we have tried dedicating a single VM for one tenancy but still experience the issue. My assumption has been that this will be due to the dialplan, I've optimized this slightly by writing LUA scripts to handle some of the inbound calls this appears to take away some of the load but we're still using the internal dialplans for each tenancy (tenancy dialplans have an average of 300 entries).____

____

We are using the following arguments when running FreeSwitch:____

-rp -nc -nonat____

____

I've seen another post which says we should operate FreeSwitch in High Performance mode using -hp but can't find anything information about this, is this still a valid argument to use with FreeSwitch?____

____

Has anyone experienced similar performance issues before or have any advice?____

____

Hope someone may be able help.____

____

Thanks,____

Shaun____

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service.

_____________________________________________________________________ _____

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service.

_____________________________________________________________________ _____

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service.

_____________________________________________________________________ _____

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> <javascript:_e(%7B%7D,'cvml','consulting@freeswitch.org<mailto:consulting@freeswitch.org>');> http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> <javascript:_e(%7B%7D,'cvml','FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org>');> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org____

__ __

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service.

_____________________________________________________________________ _____

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service.

_____________________________________________________________________ _

_____________________________________________________________________ ____ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-us ers http://www.freeswitch.org

-- ------------------------------------------------------------ Nathan Neulinger nneul@mst.edu<mailto:nneul@mst.edu> Missouri S&T Information Technology (573) 612-1412<tel:%28573%29%20612-1412> System Administrator - Architect _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Brian West brian@freeswitch.org<mailto:brian@freeswitch.org> [http://billing.freeswitch.org/templates/default/img/whmcslogo.png] Twitter: @FreeSWITCH , @briankwest http://www.freeswitchbook.com http://www.freeswitchcookbook.com Got Bugs? Report them here<https://freeswitch.org/jira>! | Reddit: /r/freeswitch<https://www.reddit.com/r/freeswitch> T:+19184209001<tel:%2B19184209001> | F:+19184209002<tel:%2B19184209002> | M:+1918424WEST (9378) iNUM:+883 5100 1420 9001 | ISN:410*543 | Skype:briankwest ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Sincerely Jay ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________

Email me direct (mgg@giagnocavo.net<mailto:mgg@giagnocavo.net>) next week and I’ll help out on this if you’d like. From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Bote Man Sent: Friday, September 4, 2015 7:53 AM To: 'FreeSWITCH Users Help' <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues This thread contains a great deal of useful intelligence that would be a shame to lose. There is a recent discussion of developing a proven High Availability configuration for FreeSWITCH; might there be a better way to track these findings so that they can be documented as part of that effort, both by the Documentation Team and by those doing the research? This is really good stuff. My personal time is being consumed close to 100% right now so I won’t be able to do anything with it until next week. Thanks. Bote Docs Team guy From: Shaun Stokes Sent: Friday, 04 September, 2015 09:25 Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues Hi Michael, We’re now seeing suitable performance on Hyper-V, but as we’re processing media\RTP we will be moving to physical hardware. If we need FreeSwitch servers to process SIP messages only we will consider these for virtualization. Thanks for your help, Shaun From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Michael Giagnocavo Sent: 02 September 2015 21:19 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues Years back, I used to run FS *on Windows*, under Hyper-V (2008), on a Q6600 and had considerably higher perf numbers. Hundreds of CPS (shit traffic), with 1000-2000 in progress channels (no media, ASR like 10% maybe). So something is certainly pathologically wrong if you’re seeing things get maxed out at the numbers you show. I didn’t do anything special, and did routing both in-process via mod_managed as well as xml_curl. Do you have a reason not to put the FS core SQLite db on a tmpfs? That’s a big win. Also make sure FS’s temp dir is a tmpfs. Unless you’re persisting and using stuff between sessions, there’s no point in going to disk right? And your random, small, IOPS are gonna be an order of magnitude less than what you posted, at least. One thing to be aware of is CPU pinning and scheduling. First, on multi-socket systems, you have NUMA. This really hurts. Raw memory bandwidth takes a 40% hit. More importantly, you’re toasting your caches (which with tons of threads happens a lot anyways). One number I like to reference is that single ownership transfer of one cache line from socket to another takes ~300 cycles (vs ~10 cycles for L2 and ~40 cycles for L3). Hyper-V has an option to lock servers to a single NUMA node. Make sure this is enabled (it may cause VMs not to start if it cannot allocate all the resources requested – this is a good thing for VoIP apps). Then cap the individual VMs so they aren’t scheduling all over the place (34 should not need more than 1 CPU). FS spawns tons of threads and they’ll end up running on all CPUs even if they don’t need all that time, and that then screws up cache usage even more. But more importantly, it sounds like this is a busy server? Are you sure you’re not just way oversubscribed and the VM is losing out? Your steal% is 0 so I’m guessing HV isn’t reporting that accurately. Only other thing that comes to mind is that Hyper-V does terrible with time, so if you’re relying on HV time service you’re gonna be skewing all over. I don’t think this should hurt FS directly, but you should run ntp anyways to be safe. If you end up needing to scale up on a single server, you’re better off running multiple FS instances, each pinned to specific cores. And you should have OpenSIPS or something in front to do LB (and it provides a ton of flexibility and handiness anyways). This probably only matters if you’re doing high CPS (dialer or shit traffic that mostly isn’t completing). I’m not sure what exactly you’re benching, but it sounds like your actual use case is different than the test, eh? Email me directly and we can chat more about this. -Michael From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Shaun Stokes Sent: Wednesday, September 2, 2015 5:09 AM To: 'FreeSWITCH Users Help' <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues Hi Jay, Thanks for the suggestion. We’re actually using PostgreSQL, the file system is a 2TB RAID5 with 10x 278GB SAS 10K RPM disks, read is 1GB/s (23,000 IOPS) write is 500MB/s (15,000 IOPS) but we’re only using a fraction of this. Thanks, Shaun From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of jay binks Sent: 02 September 2015 11:37 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues I may have missed it in your previous emails, but im assuming you have the core DB on a tempfs or somewhere other than the default location on spinning rust ? Jay On 2 September 2015 at 18:53, Shaun Stokes <shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>> wrote: Hi Brian, I’m a huge fan of virtualization, but in our environment it simply doesn’t provide the performance we’re looking for. The tests carried out were 10 second calls which were routed to a Call Centre Queue with one registered SIP extension as a member. There was a significant improvement when we tested on an Amazon VM in comparison to Hyper-V, but we’d like a system which can handle large volumes of call traffic, based on our tests this will need to be physical hardware to achieve the best performance. Further to this, we have managed to resolve our initial problem with CPU spikes on Hyper-V following these steps: • Disable VMQ on all physical and virtual adapters on the Hyper-V host which are linked to the FreeSwitch VM • Reduce the virtual cores on all VMs to at least half of the available physical cores • Increase the Processor Relative weight on the FreeSwitch VMs to 200 (default 100) Thanks, Shaun From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Brian West Sent: 01 September 2015 21:21 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues Shaun, 800 concurrent with 400cps, expectations are a little out of touch with reality and your test is no where near real world. Your call duration is ~2000ms, little to no time to actually do anything useful. You're burning the bulk of your CPU time on signaling and doing no real work with the session. So yes, you'll always get the best results out of bare metal, but you can also get good performance on a VM (*) (*) This is in no way an endorsement of running FreeSWITCH in a VM, Your mileage may vary depending on your specific application and load requirements. Thanks, On Tue, Sep 1, 2015 at 3:08 PM, Shaun Stokes <shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>> wrote: Update We've performed various tests and have come to the conclusion as advised by others, that visualization is causing bottlenecks. Using Amazon EC2 on an 8 core high performance virtual server this maxed out at around 800 concurrent sessions, 400 sessions per second. Using a Dell 1950 8 core server this was able to handle 1000 concurrent sessions, 400 sessions per second and still have plenty of resource available. Average CPU was fluctuating between 30-40% under peak load. Conclusion, if you want to run a high performance FreeSwitch server in a production you should stick with physical hardware and avoid visualization where possible. ________________________________________ From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] on behalf of Shaun Stokes [shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>] Sent: 28 August 2015 15:37 To: 'FreeSWITCH Users Help' Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues Hi Nathan, Thanks for the suggestion, we still see the CPU spikes when is under load after reducing to 8 cores but I think there may actually be an improvement in the audio quality. It's still not what we're expecting, so we will have to test our build on physical hardware too. Thanks, Shaun -----Original Message----- From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Nathan Neulinger Sent: 28 August 2015 15:04 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues Yes, I know you said Hyper-V, I don't know if it has the same sort of scheduler behavior or not, just mentioned the vmware for context. On 08/28/2015 08:46 AM, Nathan Neulinger wrote:

One thing that I see jump out - you've probably got too many virtual cores defined on that VM, particularly since you've assigned # vcpu the same as the number of real cores on the host system.

Try your test again dropping that to 8 virtual CPUs and I think you might see better results. (Or at least drop to less than 100% of the total cores on the hardware.)

Particularly if this host system has other vms running on it, you're hurting the vm scheduler's ability to hand out resources. With vmware, I believe a simple rule of thumb is that scheduler requires N-1 CPUs to be locked to schedule execution of an N cpu virtual machine - so even if it doesn't need all of the CPUs, it has to reserve all of them any time you have more than a 2 vCPU vm.

-- Nathan

On 08/28/2015 08:18 AM, Michael Jerris wrote:

...

The numbers your posting are fairly low, I'm guessing this is a grossly underpowered vm that is causing your issues

On Friday, August 28, 2015, Shaun Stokes <shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk> <mailto:shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>>> wrote:

Hi Peter,____

__ __

Thanks for the advice, this is something we're already looking into but we don't have the new hardware available yet.____

__ __

Debian 8 has great integration with Hyper-V on 2012R2 and operates as a Generation 2 VM, obviously it's never going to be as good as running directly on the hardware but we were hoping for better performance.____

__ __

Thanks,____

Shaun____

__ __

*From:*freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>');> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>');>] *On Behalf Of *Peter Olsson *Sent:* 28 August 2015 10:43 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org> <javascript:_e(%7B%7D,'cvml','freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>');>> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues____

__ __

If you want high performance, I recommend using real hardware, not a virtual machine setup - it will cause you issues.____

__ __

/Peter____

__ __

2015-08-28 9:44 GMT+02:00 Shaun Stokes <shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>

<javascript:_e(%7B%7D,'cvml','shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>');>>:__ __

Hi Michael,____

____

Thanks for the response and recommendation.____

____

Our new build is now on Debian 8 Jessie, performance on the system is noticeably better and any audio problems while the system is under load are significantly reduced but we do still experience spikes along with reduced audio quality when the system is processing 20 calls per second while already maintaining 80 existing calls.____

____

We're still hitting some kind of bottleneck, what should we expect to be able to support on a single piece of hardware? We've kept our dialplans as small as possible and we're using Memcache, but something tells me it's the call setup process which is triggering the spikes, is this possibly a limitation on the SIP profiles which are single threaded?____

____

I also see FreeSWITCH supports clustering but it's in relation to high availability, is it possible to cluster a single FreeSWITCH instance across multiple hardware or should we be looking elsewhere to solve these bottlenecks?____

____

We switched to TCP Vegas on our new build as recommended on the FreeSWITCH Wiki, this has provided a noticeable reduction in audio latency but doesn't solve the issue with CPU spikes.____

____

Many Thanks,____

Shaun____

____

*From:*freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>');> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>');>] *On Behalf Of *Michael Jerris *Sent:* 15 August 2015 23:00 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org> <javascript:_e(%7B%7D,'cvml','freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>');>> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues____

____

There are multiple known issues with Ubuntu 12.04. We reccomend Debian 8 Jessie.

On Friday, August 14, 2015, Shaun Stokes <shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>

<javascript:_e(%7B%7D,'cvml','shaun.stokes@itec-support.co.uk<mailto:shaun.stokes@itec-support.co.uk>');>> wrote:____

Hi,____

____

We're experiencing performance issues with FreeSwitch, our target is 500 concurrent sessions, but at the moment this starts to bottleneck around 30.____

____

Host system:____

Windows Server 2012 Hyper-V Host____

AMD Opteron 4386 (2 processors) - 16 cores total____

128GB DDR3____

2TB RAID 5 (700MB/s tested read and write throughput)____

____

FreeSwitch Virtual Machine:____

FreeSwitch 1.4.15____

Ubuntu 12.04 LTS____

16 Virtual cores (high priority)____

2GB RAM (would assign more but FreeSwitch never seems to use much)____

500GB HD (on VHDX)____

____

After around 30 concurrent sessions we begin to see CPU spikes almost every time a new call comes in, as the sessions increase the size and frequency of the CPU spikes also increase. The system seems to be able to sit comfortably with over 100 concurrent sessions and 80% idle CPU, providing we don't have any new calls hitting the platform. The spikes are causing audio (RTP) to stutter or in some cases drop completely for a few seconds.____

____

The FreeSwitch spikes are occurring on all 16 cores, we have been monitoring the system using htop and mpstat.____

____

This is an example of when we receive an inbound call while we have 34 concurrent sessions:____

14:26:04 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle____

14:26:05 all 1.67 0.00 27.34 0.00 0.25 0.00 0.00 0.00 70.74____

14:26:05 0 1.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 71.00____

14:26:05 1 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____

14:26:05 2 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____

14:26:05 3 3.92 0.00 26.47 0.00 0.00 0.00 0.00 0.00 69.61____

14:26:05 4 0.99 0.00 25.74 0.00 1.98 0.00 0.00 0.00 71.29____

14:26:05 5 0.99 0.00 26.73 0.00 0.99 0.00 0.00 0.00 71.29____

14:26:05 6 0.00 0.00 26.47 0.00 0.98 0.00 0.00 0.00 72.55____

14:26:05 7 0.00 0.00 26.73 0.00 0.00 0.00 0.00 0.00 73.27____

14:26:05 8 3.96 0.00 27.72 0.00 0.00 0.00 0.00 0.00 68.32____

14:26:05 9 2.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00 71.00____

14:26:05 10 6.00 0.00 32.00 0.00 0.00 0.00 0.00 0.00 62.00____

14:26:05 11 1.96 0.00 27.45 0.00 0.00 0.00 0.00 0.00 70.59____

14:26:05 12 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____

14:26:05 13 1.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 71.00____

14:26:05 14 0.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00 73.00____

____

This is when the system is not receiving an inbound call but is sitting comfortably at 34 concurrent sessions:____

14:25:57 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle____

14:25:58 all 0.87 0.00 0.62 0.00 0.12 0.00 0.00 0.00 98.39____

14:25:58 0 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____

14:25:58 1 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____

14:25:58 2 0.98 0.00 0.98 0.00 0.00 0.00 0.00 0.00 98.04____

14:25:58 3 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____

14:25:58 4 0.00 0.00 0.00 0.00 1.98 0.00 0.00 0.00 98.02____

14:25:58 5 0.00 0.00 0.00 0.00 0.99 0.00 0.00 0.00 99.01____

14:25:58 6 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____

14:25:58 7 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____

14:25:58 8 5.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 94.00____

14:25:58 9 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00____

14:25:58 10 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00____

14:25:58 11 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____

14:25:58 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____

14:25:58 13 1.98 0.00 0.99 0.00 0.00 0.00 0.00 0.00 97.03____

14:25:58 14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____

14:25:58 15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____

____

We're using FreeSwitch in multi-tenant mode, we have tried dedicating a single VM for one tenancy but still experience the issue. My assumption has been that this will be due to the dialplan, I've optimized this slightly by writing LUA scripts to handle some of the inbound calls this appears to take away some of the load but we're still using the internal dialplans for each tenancy (tenancy dialplans have an average of 300 entries).____

____

We are using the following arguments when running FreeSwitch:____

-rp -nc -nonat____

____

I've seen another post which says we should operate FreeSwitch in High Performance mode using -hp but can't find anything information about this, is this still a valid argument to use with FreeSwitch?____

____

Has anyone experienced similar performance issues before or have any advice?____

____

Hope someone may be able help.____

____

Thanks,____

Shaun____

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service.

_____________________________________________________________________ _____

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service.

_____________________________________________________________________ _____

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service.

_____________________________________________________________________ _____

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> <javascript:_e(%7B%7D,'cvml','consulting@freeswitch.org<mailto:consulting@freeswitch.org>');> http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> <javascript:_e(%7B%7D,'cvml','FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org>');> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org____

__ __

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service.

_____________________________________________________________________ _____

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service.

_____________________________________________________________________ _

_____________________________________________________________________ ____ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-us ers http://www.freeswitch.org

-- ------------------------------------------------------------ Nathan Neulinger nneul@mst.edu<mailto:nneul@mst.edu> Missouri S&T Information Technology (573) 612-1412<tel:%28573%29%20612-1412> System Administrator - Architect _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Brian West brian@freeswitch.org<mailto:brian@freeswitch.org> [http://billing.freeswitch.org/templates/default/img/whmcslogo.png] Twitter: @FreeSWITCH , @briankwest http://www.freeswitchbook.com http://www.freeswitchcookbook.com Got Bugs? Report them here<https://freeswitch.org/jira>! | Reddit: /r/freeswitch<https://www.reddit.com/r/freeswitch> T:+19184209001<tel:%2B19184209001> | F:+19184209002<tel:%2B19184209002> | M:+1918424WEST (9378) iNUM:+883 5100 1420 9001 | ISN:410*543 | Skype:briankwest ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Sincerely Jay ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________

That's assuming 100% ASR. It might be acceptable to return 503 when too busy but still need to process those INVITEs to reject the call at that rate. On 2 September 2015 at 15:01, Brian West <brian@freeswitch.org> wrote:

There is absolutely no way you can have 800 concurrent calls at 400cps, and have them last 10 seconds, Your math doesn't add up. 400cps with 10 seconds acd would be around 4000 concurrent sessions.

This is the exact reason we don't endorse any types of load testing.

While we can do crazy things like that in specific test cases in a lab, real world is never that structured. Sorry but the glove doesn't fit!

/b

On Wed, Sep 2, 2015 at 3:53 AM, Shaun Stokes < shaun.stokes@itec-support.co.uk> wrote:

...

Hi Brian,

I’m a huge fan of virtualization, but in our environment it simply doesn’t provide the performance we’re looking for. The tests carried out were 10 second calls which were routed to a Call Centre Queue with one registered SIP extension as a member.

There was a significant improvement when we tested on an Amazon VM in comparison to Hyper-V, but we’d like a system which can handle large volumes of call traffic, based on our tests this will need to be physical hardware to achieve the best performance.

Further to this, we have managed to resolve our initial problem with CPU spikes on Hyper-V following these steps:

· Disable VMQ on all physical and virtual adapters on the Hyper-V host which are linked to the FreeSwitch VM

· Reduce the virtual cores on all VMs to at least half of the available physical cores

· Increase the Processor Relative weight on the FreeSwitch VMs to 200 (default 100)

Thanks,

Shaun

*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Brian West *Sent:* 01 September 2015 21:21

*To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues

Shaun,

800 concurrent with 400cps, expectations are a little out of touch with reality and your test is no where near real world. Your call duration is ~2000ms, little to no time to actually do anything useful. You're burning the bulk of your CPU time on signaling and doing no real work with the session.

So yes, you'll always get the best results out of bare metal, but you can also get good performance on a VM (*)

(*) This is in no way an endorsement of running FreeSWITCH in a VM, Your mileage may vary depending on your specific application and load requirements.

Thanks,

On Tue, Sep 1, 2015 at 3:08 PM, Shaun Stokes < shaun.stokes@itec-support.co.uk> wrote:

Update

We've performed various tests and have come to the conclusion as advised by others, that visualization is causing bottlenecks.

Using Amazon EC2 on an 8 core high performance virtual server this maxed out at around 800 concurrent sessions, 400 sessions per second.

Using a Dell 1950 8 core server this was able to handle 1000 concurrent sessions, 400 sessions per second and still have plenty of resource available. Average CPU was fluctuating between 30-40% under peak load.

Conclusion, if you want to run a high performance FreeSwitch server in a production you should stick with physical hardware and avoid visualization where possible. ________________________________________ From: freeswitch-users-bounces@lists.freeswitch.org [ freeswitch-users-bounces@lists.freeswitch.org] on behalf of Shaun Stokes [shaun.stokes@itec-support.co.uk] Sent: 28 August 2015 15:37 To: 'FreeSWITCH Users Help'

Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues

Hi Nathan,

Thanks for the suggestion, we still see the CPU spikes when is under load after reducing to 8 cores but I think there may actually be an improvement in the audio quality.

It's still not what we're expecting, so we will have to test our build on physical hardware too.

Thanks, Shaun

-----Original Message----- From: freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Nathan Neulinger Sent: 28 August 2015 15:04 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues

Yes, I know you said Hyper-V, I don't know if it has the same sort of scheduler behavior or not, just mentioned the vmware for context.

On 08/28/2015 08:46 AM, Nathan Neulinger wrote:

...

One thing that I see jump out - you've probably got too many virtual cores defined on that VM, particularly since you've assigned # vcpu the same as the number of real cores on the host system.

Try your test again dropping that to 8 virtual CPUs and I think you might see better results. (Or at least drop to less than 100% of the total cores on the hardware.)

Particularly if this host system has other vms running on it, you're hurting the vm scheduler's ability to hand out resources. With vmware, I believe a simple rule of thumb is that scheduler requires N-1 CPUs to be locked to schedule execution of an N cpu virtual machine - so even if it doesn't need all of the CPUs, it has to reserve all of them any time you have more than a 2 vCPU vm.

-- Nathan

On 08/28/2015 08:18 AM, Michael Jerris wrote:

...

The numbers your posting are fairly low, I'm guessing this is a grossly underpowered vm that is causing your issues

On Friday, August 28, 2015, Shaun Stokes < shaun.stokes@itec-support.co.uk <mailto:shaun.stokes@itec-support.co.uk>> wrote:

Hi Peter,____

__ __

Thanks for the advice, this is something we're already looking into but we don't have the new hardware available yet.____

__ __

Debian 8 has great integration with Hyper-V on 2012R2 and operates as a Generation 2 VM, obviously it's never going to be as good as running directly on the hardware but we were hoping for better performance.____

__ __

Thanks,____

Shaun____

__ __

*From:*freeswitch-users-bounces@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml',' freeswitch-users-bounces@lists.freeswitch.org');> [mailto:freeswitch-users-bounces@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml',' freeswitch-users-bounces@lists.freeswitch.org');>] *On Behalf Of *Peter Olsson *Sent:* 28 August 2015 10:43 *To:* FreeSWITCH Users Help < freeswitch-users@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml',' freeswitch-users@lists.freeswitch.org');>> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues____

__ __

If you want high performance, I recommend using real hardware, not a virtual machine setup - it will cause you issues.____

__ __

/Peter____

__ __

2015-08-28 9:44 GMT+02:00 Shaun Stokes < shaun.stokes@itec-support.co.uk

<javascript:_e(%7B%7D,'cvml','shaun.stokes@itec-support.co.uk');>>:__ __

Hi Michael,____

____

Thanks for the response and recommendation.____

____

Our new build is now on Debian 8 Jessie, performance on the system is noticeably better and any audio problems while the system is under load are significantly reduced but we do still experience spikes along with reduced audio quality when the system is processing 20 calls per second while already maintaining 80 existing calls.____

____

We're still hitting some kind of bottleneck, what should we expect to be able to support on a single piece of hardware? We've kept our dialplans as small as possible and we're using Memcache, but something tells me it's the call setup process which is triggering the spikes, is this possibly a limitation on the SIP profiles which are single threaded?____

____

I also see FreeSWITCH supports clustering but it's in relation to high availability, is it possible to cluster a single FreeSWITCH instance across multiple hardware or should we be looking elsewhere to solve these bottlenecks?____

____

We switched to TCP Vegas on our new build as recommended on the FreeSWITCH Wiki, this has provided a noticeable reduction in audio latency but doesn't solve the issue with CPU spikes.____

____

Many Thanks,____

Shaun____

____

*From:*freeswitch-users-bounces@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml',' freeswitch-users-bounces@lists.freeswitch.org');> [mailto:freeswitch-users-bounces@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml',' freeswitch-users-bounces@lists.freeswitch.org');>] *On Behalf Of *Michael Jerris *Sent:* 15 August 2015 23:00 *To:* FreeSWITCH Users Help < freeswitch-users@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml',' freeswitch-users@lists.freeswitch.org');>> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues____

____

There are multiple known issues with Ubuntu 12.04. We reccomend Debian 8 Jessie.

On Friday, August 14, 2015, Shaun Stokes < shaun.stokes@itec-support.co.uk

<javascript:_e(%7B%7D,'cvml','shaun.stokes@itec-support.co.uk');>> wrote:____

Hi,____

____

We're experiencing performance issues with FreeSwitch, our target is 500 concurrent sessions, but at the moment this starts to bottleneck around 30.____

____

Host system:____

Windows Server 2012 Hyper-V Host____

AMD Opteron 4386 (2 processors) - 16 cores total____

128GB DDR3____

2TB RAID 5 (700MB/s tested read and write throughput)____

____

FreeSwitch Virtual Machine:____

FreeSwitch 1.4.15____

Ubuntu 12.04 LTS____

16 Virtual cores (high priority)____

2GB RAM (would assign more but FreeSwitch never seems to use much)____

500GB HD (on VHDX)____

____

After around 30 concurrent sessions we begin to see CPU spikes almost every time a new call comes in, as the sessions increase the size and frequency of the CPU spikes also increase. The system seems to be able to sit comfortably with over 100 concurrent sessions and 80% idle CPU, providing we don't have any new calls hitting the platform. The spikes are causing audio (RTP) to stutter or in some cases drop completely for a few seconds.____

____

The FreeSwitch spikes are occurring on all 16 cores, we have been monitoring the system using htop and mpstat.____

____

This is an example of when we receive an inbound call while we have 34 concurrent sessions:____

14:26:04 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle____

14:26:05 all 1.67 0.00 27.34 0.00 0.25 0.00 0.00 0.00 70.74____

14:26:05 0 1.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 71.00____

14:26:05 1 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____

14:26:05 2 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____

14:26:05 3 3.92 0.00 26.47 0.00 0.00 0.00 0.00 0.00 69.61____

14:26:05 4 0.99 0.00 25.74 0.00 1.98 0.00 0.00 0.00 71.29____

14:26:05 5 0.99 0.00 26.73 0.00 0.99 0.00 0.00 0.00 71.29____

14:26:05 6 0.00 0.00 26.47 0.00 0.98 0.00 0.00 0.00 72.55____

14:26:05 7 0.00 0.00 26.73 0.00 0.00 0.00 0.00 0.00 73.27____

14:26:05 8 3.96 0.00 27.72 0.00 0.00 0.00 0.00 0.00 68.32____

14:26:05 9 2.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00 71.00____

14:26:05 10 6.00 0.00 32.00 0.00 0.00 0.00 0.00 0.00 62.00____

14:26:05 11 1.96 0.00 27.45 0.00 0.00 0.00 0.00 0.00 70.59____

14:26:05 12 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____

14:26:05 13 1.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 71.00____

14:26:05 14 0.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00 73.00____

____

This is when the system is not receiving an inbound call but is sitting comfortably at 34 concurrent sessions:____

14:25:57 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle____

14:25:58 all 0.87 0.00 0.62 0.00 0.12 0.00 0.00 0.00 98.39____

14:25:58 0 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____

14:25:58 1 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____

14:25:58 2 0.98 0.00 0.98 0.00 0.00 0.00 0.00 0.00 98.04____

14:25:58 3 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____

14:25:58 4 0.00 0.00 0.00 0.00 1.98 0.00 0.00 0.00 98.02____

14:25:58 5 0.00 0.00 0.00 0.00 0.99 0.00 0.00 0.00 99.01____

14:25:58 6 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____

14:25:58 7 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____

14:25:58 8 5.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 94.00____

14:25:58 9 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00____

14:25:58 10 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00____

14:25:58 11 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____

14:25:58 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____

14:25:58 13 1.98 0.00 0.99 0.00 0.00 0.00 0.00 0.00 97.03____

14:25:58 14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____

14:25:58 15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____

____

We're using FreeSwitch in multi-tenant mode, we have tried dedicating a single VM for one tenancy but still experience the issue. My assumption has been that this will be due to the dialplan, I've optimized this slightly by writing LUA scripts to handle some of the inbound calls this appears to take away some of the load but we're still using the internal dialplans for each tenancy (tenancy dialplans have an average of 300 entries).____

____

We are using the following arguments when running FreeSwitch:____

-rp -nc -nonat____

____

I've seen another post which says we should operate FreeSwitch in High Performance mode using -hp but can't find anything information about this, is this still a valid argument to use with FreeSwitch?____

____

Has anyone experienced similar performance issues before or have any advice?____

____

Hope someone may be able help.____

____

Thanks,____

Shaun____

---

...

...

This message has been checked for all known viruses by

MessageLabs Virus Scanning Service.

...

_____________________________________________________________________ _____

---

...

...

This message has been checked for all known viruses by

MessageLabs Virus Scanning Service.

...

_____________________________________________________________________ _____

---

...

...

This message has been checked for all known viruses by

MessageLabs Virus Scanning Service.

...

_____________________________________________________________________ _____

---

...

...

Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <javascript:_e(%7B%7D,'cvml','

consulting@freeswitch.org');>

...

http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org

<javascript:_e(%7B%7D,'cvml','FreeSWITCH-users@lists.freeswitch.org');>

...

http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:

http://lists.freeswitch.org/mailman/options/freeswitch-users

...

http://www.freeswitch.org____

__ __

---

...

...

This message has been checked for all known viruses by

MessageLabs Virus Scanning Service.

...

_____________________________________________________________________ _____

---

...

...

This message has been checked for all known viruses by

MessageLabs Virus Scanning Service.

...

_____________________________________________________________________ _

_____________________________________________________________________ ____ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-us ers http://www.freeswitch.org

-- ------------------------------------------------------------ Nathan Neulinger nneul@mst.edu Missouri S&T Information Technology (573) 612-1412 System Administrator - Architect

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

--

*Brian West* brian@freeswitch.org

*Twitter: @FreeSWITCH , @briankwest* http://www.freeswitchbook.com http://www.freeswitchcookbook.com

Got Bugs? Report them here <https://freeswitch.org/jira>! | Reddit: /r/freeswitch <https://www.reddit.com/r/freeswitch>

*T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378) *iNUM:*+883 5100 1420 9001 | *ISN:*410*543 | *Skype:*briankwest

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

--

*Brian West* brian@freeswitch.org

*Twitter: @FreeSWITCH , @briankwest* http://www.freeswitchbook.com http://www.freeswitchcookbook.com

Got Bugs? Report them here <https://freeswitch.org/jira>! | Reddit: /r/freeswitch <https://www.reddit.com/r/freeswitch>

*T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378) *iNUM:*+883 5100 1420 9001 | *ISN:*410*543 | *Skype:*briankwest

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

one of the issues that I noticed on public VM hosting (tried digitalocean, linode, and a couple of others) is that your CPU resource is unpredictable. So, you may run a perfect test with X parallel sessions, and one hour later you would see distortions under the same load, just because you didn't receive enough CPU time when the session needs to start or an RTP packet needs to be sent. So, any kind of capacity planning needs predictable VM resources. You may get that if you control the physical environment. In a public virtualized hosting, you need to keep in mind that you only get a best-effort resource. If you have enough budget, place your SIP-only services (kamailio and such) on VM's, and real-time media servers (FS) on physical machines. By the way, scaleway.com has dropped the price of physical hosts few days ago. On Thu, Sep 3, 2015 at 7:37 AM, Shaun Stokes < shaun.stokes@itec-support.co.uk> wrote:

Hi Brian,

You're absolutely right, we'd have to double check our configuration on Amazon EC2 to understand why the calls were only lasting 2 minutes. I suspect we didn't increase our sessions per second limit in FreeSwitch, new sessions are rejected when the limit is reached but I've noticed FreeSwitch does still report on these in SIP status.

Obviously it's never going to be a fair comparison to real world as there are many other factors which can play a role in a normal environment, but in our case we seem to be hitting capacity problems with call volume. We can see that both Amazon EC2 and physical hardware perform better than our existing Hyper-V platform.

Thanks, Shaun ------------------------------ *From:* freeswitch-users-bounces@lists.freeswitch.org [ freeswitch-users-bounces@lists.freeswitch.org] on behalf of Brian West [ brian@freeswitch.org] *Sent:* 02 September 2015 15:01

*To:* FreeSWITCH Users Help *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues

There is absolutely no way you can have 800 concurrent calls at 400cps, and have them last 10 seconds, Your math doesn't add up. 400cps with 10 seconds acd would be around 4000 concurrent sessions.

This is the exact reason we don't endorse any types of load testing.

While we can do crazy things like that in specific test cases in a lab, real world is never that structured. Sorry but the glove doesn't fit!

/b

On Wed, Sep 2, 2015 at 3:53 AM, Shaun Stokes < shaun.stokes@itec-support.co.uk> wrote:

...

Hi Brian,

I’m a huge fan of virtualization, but in our environment it simply doesn’t provide the performance we’re looking for. The tests carried out were 10 second calls which were routed to a Call Centre Queue with one registered SIP extension as a member.

There was a significant improvement when we tested on an Amazon VM in comparison to Hyper-V, but we’d like a system which can handle large volumes of call traffic, based on our tests this will need to be physical hardware to achieve the best performance.

Further to this, we have managed to resolve our initial problem with CPU spikes on Hyper-V following these steps:

· Disable VMQ on all physical and virtual adapters on the Hyper-V host which are linked to the FreeSwitch VM

· Reduce the virtual cores on all VMs to at least half of the available physical cores

· Increase the Processor Relative weight on the FreeSwitch VMs to 200 (default 100)

Thanks,

Shaun

*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Brian West *Sent:* 01 September 2015 21:21

*To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues

Shaun,

800 concurrent with 400cps, expectations are a little out of touch with reality and your test is no where near real world. Your call duration is ~2000ms, little to no time to actually do anything useful. You're burning the bulk of your CPU time on signaling and doing no real work with the session.

So yes, you'll always get the best results out of bare metal, but you can also get good performance on a VM (*)

(*) This is in no way an endorsement of running FreeSWITCH in a VM, Your mileage may vary depending on your specific application and load requirements.

Thanks,

On Tue, Sep 1, 2015 at 3:08 PM, Shaun Stokes < shaun.stokes@itec-support.co.uk> wrote:

Update

We've performed various tests and have come to the conclusion as advised by others, that visualization is causing bottlenecks.

Using Amazon EC2 on an 8 core high performance virtual server this maxed out at around 800 concurrent sessions, 400 sessions per second.

Using a Dell 1950 8 core server this was able to handle 1000 concurrent sessions, 400 sessions per second and still have plenty of resource available. Average CPU was fluctuating between 30-40% under peak load.

Conclusion, if you want to run a high performance FreeSwitch server in a production you should stick with physical hardware and avoid visualization where possible. ________________________________________ From: freeswitch-users-bounces@lists.freeswitch.org [ freeswitch-users-bounces@lists.freeswitch.org] on behalf of Shaun Stokes [shaun.stokes@itec-support.co.uk] Sent: 28 August 2015 15:37 To: 'FreeSWITCH Users Help'

Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues

Hi Nathan,

Thanks for the suggestion, we still see the CPU spikes when is under load after reducing to 8 cores but I think there may actually be an improvement in the audio quality.

It's still not what we're expecting, so we will have to test our build on physical hardware too.

Thanks, Shaun

-----Original Message----- From: freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Nathan Neulinger Sent: 28 August 2015 15:04 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues

Yes, I know you said Hyper-V, I don't know if it has the same sort of scheduler behavior or not, just mentioned the vmware for context.

On 08/28/2015 08:46 AM, Nathan Neulinger wrote:

...

One thing that I see jump out - you've probably got too many virtual cores defined on that VM, particularly since you've assigned # vcpu the same as the number of real cores on the host system.

Try your test again dropping that to 8 virtual CPUs and I think you might see better results. (Or at least drop to less than 100% of the total cores on the hardware.)

Particularly if this host system has other vms running on it, you're hurting the vm scheduler's ability to hand out resources. With vmware, I believe a simple rule of thumb is that scheduler requires N-1 CPUs to be locked to schedule execution of an N cpu virtual machine - so even if it doesn't need all of the CPUs, it has to reserve all of them any time you have more than a 2 vCPU vm.

-- Nathan

On 08/28/2015 08:18 AM, Michael Jerris wrote:

...

The numbers your posting are fairly low, I'm guessing this is a grossly underpowered vm that is causing your issues

On Friday, August 28, 2015, Shaun Stokes < shaun.stokes@itec-support.co.uk <mailto:shaun.stokes@itec-support.co.uk>> wrote:

Hi Peter,____

__ __

Thanks for the advice, this is something we're already looking into but we don't have the new hardware available yet.____

__ __

Debian 8 has great integration with Hyper-V on 2012R2 and operates as a Generation 2 VM, obviously it's never going to be as good as running directly on the hardware but we were hoping for better performance.____

__ __

Thanks,____

Shaun____

__ __

*From:*freeswitch-users-bounces@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml',' freeswitch-users-bounces@lists.freeswitch.org');> [mailto:freeswitch-users-bounces@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml',' freeswitch-users-bounces@lists.freeswitch.org');>] *On Behalf Of *Peter Olsson *Sent:* 28 August 2015 10:43 *To:* FreeSWITCH Users Help < freeswitch-users@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml',' freeswitch-users@lists.freeswitch.org');>> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues____

__ __

If you want high performance, I recommend using real hardware, not a virtual machine setup - it will cause you issues.____

__ __

/Peter____

__ __

2015-08-28 9:44 GMT+02:00 Shaun Stokes < shaun.stokes@itec-support.co.uk

<javascript:_e(%7B%7D,'cvml','shaun.stokes@itec-support.co.uk');>>:__ __

Hi Michael,____

____

Thanks for the response and recommendation.____

____

Our new build is now on Debian 8 Jessie, performance on the system is noticeably better and any audio problems while the system is under load are significantly reduced but we do still experience spikes along with reduced audio quality when the system is processing 20 calls per second while already maintaining 80 existing calls.____

____

We're still hitting some kind of bottleneck, what should we expect to be able to support on a single piece of hardware? We've kept our dialplans as small as possible and we're using Memcache, but something tells me it's the call setup process which is triggering the spikes, is this possibly a limitation on the SIP profiles which are single threaded?____

____

I also see FreeSWITCH supports clustering but it's in relation to high availability, is it possible to cluster a single FreeSWITCH instance across multiple hardware or should we be looking elsewhere to solve these bottlenecks?____

____

We switched to TCP Vegas on our new build as recommended on the FreeSWITCH Wiki, this has provided a noticeable reduction in audio latency but doesn't solve the issue with CPU spikes.____

____

Many Thanks,____

Shaun____

____

*From:*freeswitch-users-bounces@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml',' freeswitch-users-bounces@lists.freeswitch.org');> [mailto:freeswitch-users-bounces@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml',' freeswitch-users-bounces@lists.freeswitch.org');>] *On Behalf Of *Michael Jerris *Sent:* 15 August 2015 23:00 *To:* FreeSWITCH Users Help < freeswitch-users@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml',' freeswitch-users@lists.freeswitch.org');>> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues____

____

There are multiple known issues with Ubuntu 12.04. We reccomend Debian 8 Jessie.

On Friday, August 14, 2015, Shaun Stokes < shaun.stokes@itec-support.co.uk

<javascript:_e(%7B%7D,'cvml','shaun.stokes@itec-support.co.uk');>> wrote:____

Hi,____

____

We're experiencing performance issues with FreeSwitch, our target is 500 concurrent sessions, but at the moment this starts to bottleneck around 30.____

____

Host system:____

Windows Server 2012 Hyper-V Host____

AMD Opteron 4386 (2 processors) - 16 cores total____

128GB DDR3____

2TB RAID 5 (700MB/s tested read and write throughput)____

____

FreeSwitch Virtual Machine:____

FreeSwitch 1.4.15____

Ubuntu 12.04 LTS____

16 Virtual cores (high priority)____

2GB RAM (would assign more but FreeSwitch never seems to use much)____

500GB HD (on VHDX)____

____

After around 30 concurrent sessions we begin to see CPU spikes almost every time a new call comes in, as the sessions increase the size and frequency of the CPU spikes also increase. The system seems to be able to sit comfortably with over 100 concurrent sessions and 80% idle CPU, providing we don't have any new calls hitting the platform. The spikes are causing audio (RTP) to stutter or in some cases drop completely for a few seconds.____

____

The FreeSwitch spikes are occurring on all 16 cores, we have been monitoring the system using htop and mpstat.____

____

This is an example of when we receive an inbound call while we have 34 concurrent sessions:____

14:26:04 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle____

14:26:05 all 1.67 0.00 27.34 0.00 0.25 0.00 0.00 0.00 70.74____

14:26:05 0 1.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 71.00____

14:26:05 1 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____

14:26:05 2 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____

14:26:05 3 3.92 0.00 26.47 0.00 0.00 0.00 0.00 0.00 69.61____

14:26:05 4 0.99 0.00 25.74 0.00 1.98 0.00 0.00 0.00 71.29____

14:26:05 5 0.99 0.00 26.73 0.00 0.99 0.00 0.00 0.00 71.29____

14:26:05 6 0.00 0.00 26.47 0.00 0.98 0.00 0.00 0.00 72.55____

14:26:05 7 0.00 0.00 26.73 0.00 0.00 0.00 0.00 0.00 73.27____

14:26:05 8 3.96 0.00 27.72 0.00 0.00 0.00 0.00 0.00 68.32____

14:26:05 9 2.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00 71.00____

14:26:05 10 6.00 0.00 32.00 0.00 0.00 0.00 0.00 0.00 62.00____

14:26:05 11 1.96 0.00 27.45 0.00 0.00 0.00 0.00 0.00 70.59____

14:26:05 12 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____

14:26:05 13 1.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 71.00____

14:26:05 14 0.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00 73.00____

____

This is when the system is not receiving an inbound call but is sitting comfortably at 34 concurrent sessions:____

14:25:57 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle____

14:25:58 all 0.87 0.00 0.62 0.00 0.12 0.00 0.00 0.00 98.39____

14:25:58 0 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____

14:25:58 1 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____

14:25:58 2 0.98 0.00 0.98 0.00 0.00 0.00 0.00 0.00 98.04____

14:25:58 3 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____

14:25:58 4 0.00 0.00 0.00 0.00 1.98 0.00 0.00 0.00 98.02____

14:25:58 5 0.00 0.00 0.00 0.00 0.99 0.00 0.00 0.00 99.01____

14:25:58 6 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____

14:25:58 7 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____

14:25:58 8 5.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 94.00____

14:25:58 9 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00____

14:25:58 10 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00____

14:25:58 11 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____

14:25:58 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____

14:25:58 13 1.98 0.00 0.99 0.00 0.00 0.00 0.00 0.00 97.03____

14:25:58 14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____

14:25:58 15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____

____

We're using FreeSwitch in multi-tenant mode, we have tried dedicating a single VM for one tenancy but still experience the issue. My assumption has been that this will be due to the dialplan, I've optimized this slightly by writing LUA scripts to handle some of the inbound calls this appears to take away some of the load but we're still using the internal dialplans for each tenancy (tenancy dialplans have an average of 300 entries).____

____

We are using the following arguments when running FreeSwitch:____

-rp -nc -nonat____

____

I've seen another post which says we should operate FreeSwitch in High Performance mode using -hp but can't find anything information about this, is this still a valid argument to use with FreeSwitch?____

____

Has anyone experienced similar performance issues before or have any advice?____

____

Hope someone may be able help.____

____

Thanks,____

Shaun____

---

...

...

This message has been checked for all known viruses by

MessageLabs Virus Scanning Service.

...

_____________________________________________________________________ _____

---

...

...

This message has been checked for all known viruses by

MessageLabs Virus Scanning Service.

...

_____________________________________________________________________ _____

---

...

...

This message has been checked for all known viruses by

MessageLabs Virus Scanning Service.

...

_____________________________________________________________________ _____

---

...

...

Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <javascript:_e(%7B%7D,'cvml','

consulting@freeswitch.org');>

...

http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org

<javascript:_e(%7B%7D,'cvml','FreeSWITCH-users@lists.freeswitch.org');>

...

http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:

http://lists.freeswitch.org/mailman/options/freeswitch-users

...

http://www.freeswitch.org____

__ __

---

...

...

This message has been checked for all known viruses by

MessageLabs Virus Scanning Service.

...

_____________________________________________________________________ _____

---

...

...

This message has been checked for all known viruses by

MessageLabs Virus Scanning Service.

...

_____________________________________________________________________ _

_____________________________________________________________________ ____ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-us ers http://www.freeswitch.org

-- ------------------------------------------------------------ Nathan Neulinger nneul@mst.edu Missouri S&T Information Technology (573) 612-1412 System Administrator - Architect

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

--

*Brian West* brian@freeswitch.org

*Twitter: @FreeSWITCH , @briankwest* http://www.freeswitchbook.com http://www.freeswitchcookbook.com

Got Bugs? Report them here <https://freeswitch.org/jira>! | Reddit: /r/freeswitch <https://www.reddit.com/r/freeswitch>

*T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378) *iNUM:*+883 5100 1420 9001 | *ISN:*410*543 | *Skype:*briankwest

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

--

*Brian West* brian@freeswitch.org

*Twitter: @FreeSWITCH , @briankwest* http://www.freeswitchbook.com http://www.freeswitchcookbook.com

Got Bugs? Report them here <https://freeswitch.org/jira>! | Reddit: /r/freeswitch <https://www.reddit.com/r/freeswitch>

*T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378) *iNUM:*+883 5100 1420 9001 | *ISN:*410*543 | *Skype:*briankwest

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________

______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org