Update We've performed various tests and have come to the conclusion as advised by others, that visualization is causing bottlenecks. Using Amazon EC2 on an 8 core high performance virtual server this maxed out at around 800 concurrent sessions, 400 sessions per second. Using a Dell 1950 8 core server this was able to handle 1000 concurrent sessions, 400 sessions per second and still have plenty of resource available. Average CPU was fluctuating between 30-40% under peak load. Conclusion, if you want to run a high performance FreeSwitch server in a production you should stick with physical hardware and avoid visualization where possible. ________________________________________ From: freeswitch-users-bounces@lists.freeswitch.org [freeswitch-users-bounces@lists.freeswitch.org] on behalf of Shaun Stokes [shaun.stokes@itec-support.co.uk] Sent: 28 August 2015 15:37 To: 'FreeSWITCH Users Help' Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues Hi Nathan, Thanks for the suggestion, we still see the CPU spikes when is under load after reducing to 8 cores but I think there may actually be an improvement in the audio quality. It's still not what we're expecting, so we will have to test our build on physical hardware too. Thanks, Shaun -----Original Message----- From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Nathan Neulinger Sent: 28 August 2015 15:04 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues Yes, I know you said Hyper-V, I don't know if it has the same sort of scheduler behavior or not, just mentioned the vmware for context. On 08/28/2015 08:46 AM, Nathan Neulinger wrote:
One thing that I see jump out - you've probably got too many virtual cores defined on that VM, particularly since you've assigned # vcpu the same as the number of real cores on the host system.
Try your test again dropping that to 8 virtual CPUs and I think you might see better results. (Or at least drop to less than 100% of the total cores on the hardware.)
Particularly if this host system has other vms running on it, you're hurting the vm scheduler's ability to hand out resources. With vmware, I believe a simple rule of thumb is that scheduler requires N-1 CPUs to be locked to schedule execution of an N cpu virtual machine - so even if it doesn't need all of the CPUs, it has to reserve all of them any time you have more than a 2 vCPU vm.
-- Nathan
On 08/28/2015 08:18 AM, Michael Jerris wrote:
The numbers your posting are fairly low, I'm guessing this is a grossly underpowered vm that is causing your issues
On Friday, August 28, 2015, Shaun Stokes <shaun.stokes@itec-support.co.uk <mailto:shaun.stokes@itec-support.co.uk>> wrote:
Hi Peter,____
__ __
Thanks for the advice, this is something we're already looking into but we don't have the new hardware available yet.____
__ __
Debian 8 has great integration with Hyper-V on 2012R2 and operates as a Generation 2 VM, obviously it's never going to be as good as running directly on the hardware but we were hoping for better performance.____
__ __
Thanks,____
Shaun____
__ __
*From:*freeswitch-users-bounces@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces@lists.freeswitch.org');> [mailto:freeswitch-users-bounces@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces@lists.freeswitch.org');>] *On Behalf Of *Peter Olsson *Sent:* 28 August 2015 10:43 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml','freeswitch-users@lists.freeswitch.org');>> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues____
__ __
If you want high performance, I recommend using real hardware, not a virtual machine setup - it will cause you issues.____
__ __
/Peter____
__ __
2015-08-28 9:44 GMT+02:00 Shaun Stokes <shaun.stokes@itec-support.co.uk
<javascript:_e(%7B%7D,'cvml','shaun.stokes@itec-support.co.uk');>>:__ __
Hi Michael,____
____
Thanks for the response and recommendation.____
____
Our new build is now on Debian 8 Jessie, performance on the system is noticeably better and any audio problems while the system is under load are significantly reduced but we do still experience spikes along with reduced audio quality when the system is processing 20 calls per second while already maintaining 80 existing calls.____
____
We're still hitting some kind of bottleneck, what should we expect to be able to support on a single piece of hardware? We've kept our dialplans as small as possible and we're using Memcache, but something tells me it's the call setup process which is triggering the spikes, is this possibly a limitation on the SIP profiles which are single threaded?____
____
I also see FreeSWITCH supports clustering but it's in relation to high availability, is it possible to cluster a single FreeSWITCH instance across multiple hardware or should we be looking elsewhere to solve these bottlenecks?____
____
We switched to TCP Vegas on our new build as recommended on the FreeSWITCH Wiki, this has provided a noticeable reduction in audio latency but doesn't solve the issue with CPU spikes.____
____
Many Thanks,____
Shaun____
____
*From:*freeswitch-users-bounces@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces@lists.freeswitch.org');> [mailto:freeswitch-users-bounces@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces@lists.freeswitch.org');>] *On Behalf Of *Michael Jerris *Sent:* 15 August 2015 23:00 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml','freeswitch-users@lists.freeswitch.org');>> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues____
____
There are multiple known issues with Ubuntu 12.04. We reccomend Debian 8 Jessie.
On Friday, August 14, 2015, Shaun Stokes <shaun.stokes@itec-support.co.uk
<javascript:_e(%7B%7D,'cvml','shaun.stokes@itec-support.co.uk');>> wrote:____
Hi,____
____
We're experiencing performance issues with FreeSwitch, our target is 500 concurrent sessions, but at the moment this starts to bottleneck around 30.____
____
Host system:____
Windows Server 2012 Hyper-V Host____
AMD Opteron 4386 (2 processors) - 16 cores total____
128GB DDR3____
2TB RAID 5 (700MB/s tested read and write throughput)____
____
FreeSwitch Virtual Machine:____
FreeSwitch 1.4.15____
Ubuntu 12.04 LTS____
16 Virtual cores (high priority)____
2GB RAM (would assign more but FreeSwitch never seems to use much)____
500GB HD (on VHDX)____
____
After around 30 concurrent sessions we begin to see CPU spikes almost every time a new call comes in, as the sessions increase the size and frequency of the CPU spikes also increase. The system seems to be able to sit comfortably with over 100 concurrent sessions and 80% idle CPU, providing we don't have any new calls hitting the platform. The spikes are causing audio (RTP) to stutter or in some cases drop completely for a few seconds.____
____
The FreeSwitch spikes are occurring on all 16 cores, we have been monitoring the system using htop and mpstat.____
____
This is an example of when we receive an inbound call while we have 34 concurrent sessions:____
14:26:04 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle____
14:26:05 all 1.67 0.00 27.34 0.00 0.25 0.00 0.00 0.00 70.74____
14:26:05 0 1.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 71.00____
14:26:05 1 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____
14:26:05 2 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____
14:26:05 3 3.92 0.00 26.47 0.00 0.00 0.00 0.00 0.00 69.61____
14:26:05 4 0.99 0.00 25.74 0.00 1.98 0.00 0.00 0.00 71.29____
14:26:05 5 0.99 0.00 26.73 0.00 0.99 0.00 0.00 0.00 71.29____
14:26:05 6 0.00 0.00 26.47 0.00 0.98 0.00 0.00 0.00 72.55____
14:26:05 7 0.00 0.00 26.73 0.00 0.00 0.00 0.00 0.00 73.27____
14:26:05 8 3.96 0.00 27.72 0.00 0.00 0.00 0.00 0.00 68.32____
14:26:05 9 2.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00 71.00____
14:26:05 10 6.00 0.00 32.00 0.00 0.00 0.00 0.00 0.00 62.00____
14:26:05 11 1.96 0.00 27.45 0.00 0.00 0.00 0.00 0.00 70.59____
14:26:05 12 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____
14:26:05 13 1.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 71.00____
14:26:05 14 0.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00 73.00____
____
This is when the system is not receiving an inbound call but is sitting comfortably at 34 concurrent sessions:____
14:25:57 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle____
14:25:58 all 0.87 0.00 0.62 0.00 0.12 0.00 0.00 0.00 98.39____
14:25:58 0 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____
14:25:58 1 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____
14:25:58 2 0.98 0.00 0.98 0.00 0.00 0.00 0.00 0.00 98.04____
14:25:58 3 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____
14:25:58 4 0.00 0.00 0.00 0.00 1.98 0.00 0.00 0.00 98.02____
14:25:58 5 0.00 0.00 0.00 0.00 0.99 0.00 0.00 0.00 99.01____
14:25:58 6 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____
14:25:58 7 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____
14:25:58 8 5.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 94.00____
14:25:58 9 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00____
14:25:58 10 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00____
14:25:58 11 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____
14:25:58 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____
14:25:58 13 1.98 0.00 0.99 0.00 0.00 0.00 0.00 0.00 97.03____
14:25:58 14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____
14:25:58 15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____
____
We're using FreeSwitch in multi-tenant mode, we have tried dedicating a single VM for one tenancy but still experience the issue. My assumption has been that this will be due to the dialplan, I've optimized this slightly by writing LUA scripts to handle some of the inbound calls this appears to take away some of the load but we're still using the internal dialplans for each tenancy (tenancy dialplans have an average of 300 entries).____
____
We are using the following arguments when running FreeSwitch:____
-rp -nc -nonat____
____
I've seen another post which says we should operate FreeSwitch in High Performance mode using -hp but can't find anything information about this, is this still a valid argument to use with FreeSwitch?____
____
Has anyone experienced similar performance issues before or have any advice?____
____
Hope someone may be able help.____
____
Thanks,____
Shaun____
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service.
_____________________________________________________________________ _____
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service.
_____________________________________________________________________ _____
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service.
_____________________________________________________________________ _____
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <javascript:_e(%7B%7D,'cvml','consulting@freeswitch.org');> http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <javascript:_e(%7B%7D,'cvml','FreeSWITCH-users@lists.freeswitch.org');> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org____
__ __
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service.
_____________________________________________________________________ _____
______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service.
_____________________________________________________________________ _
_____________________________________________________________________ ____ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-us ers http://www.freeswitch.org
-- ------------------------------------------------------------ Nathan Neulinger nneul@mst.edu Missouri S&T Information Technology (573) 612-1412 System Administrator - Architect _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________ ______________________________________________________________________ This message has been checked for all known viruses by MessageLabs Virus Scanning Service. ______________________________________________________________________