[Freeswitch-users] FreeSwitch - Performance issues
Nathan Neulinger
nneul at mst.edu
Fri Aug 28 17:46:43 MSD 2015
One thing that I see jump out - you've probably got too many virtual cores defined on that VM, particularly since you've
assigned # vcpu the same as the number of real cores on the host system.
Try your test again dropping that to 8 virtual CPUs and I think you might see better results. (Or at least drop to less
than 100% of the total cores on the hardware.)
Particularly if this host system has other vms running on it, you're hurting the vm scheduler's ability to hand out
resources. With vmware, I believe a simple rule of thumb is that scheduler requires N-1 CPUs to be locked to schedule
execution of an N cpu virtual machine - so even if it doesn't need all of the CPUs, it has to reserve all of them any
time you have more than a 2 vCPU vm.
-- Nathan
On 08/28/2015 08:18 AM, Michael Jerris wrote:
> The numbers your posting are fairly low, I'm guessing this is a grossly underpowered vm that is causing your issues
>
> On Friday, August 28, 2015, Shaun Stokes <shaun.stokes at itec-support.co.uk <mailto:shaun.stokes at itec-support.co.uk>> wrote:
>
> Hi Peter,____
>
> __ __
>
> Thanks for the advice, this is something we’re already looking into but we don’t have the new hardware available
> yet.____
>
> __ __
>
> Debian 8 has great integration with Hyper-V on 2012R2 and operates as a Generation 2 VM, obviously it’s never going
> to be as good as running directly on the hardware but we were hoping for better performance.____
>
> __ __
>
> Thanks,____
>
> Shaun____
>
> __ __
>
> *From:*freeswitch-users-bounces at lists.freeswitch.org
> <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces at lists.freeswitch.org');>
> [mailto:freeswitch-users-bounces at lists.freeswitch.org
> <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces at lists.freeswitch.org');>] *On Behalf Of *Peter Olsson
> *Sent:* 28 August 2015 10:43
> *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org
> <javascript:_e(%7B%7D,'cvml','freeswitch-users at lists.freeswitch.org');>>
> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues____
>
> __ __
>
> If you want high performance, I recommend using real hardware, not a virtual machine setup - it will cause you
> issues.____
>
> __ __
>
> /Peter____
>
> __ __
>
> 2015-08-28 9:44 GMT+02:00 Shaun Stokes <shaun.stokes at itec-support.co.uk
> <javascript:_e(%7B%7D,'cvml','shaun.stokes at itec-support.co.uk');>>:____
>
> Hi Michael,____
>
> ____
>
> Thanks for the response and recommendation.____
>
> ____
>
> Our new build is now on Debian 8 Jessie, performance on the system is noticeably better and any audio problems
> while the system is under load are significantly reduced but we do still experience spikes along with reduced
> audio quality when the system is processing 20 calls per second while already maintaining 80 existing calls.____
>
> ____
>
> We’re still hitting some kind of bottleneck, what should we expect to be able to support on a single piece of
> hardware? We’ve kept our dialplans as small as possible and we’re using Memcache, but something tells me it’s
> the call setup process which is triggering the spikes, is this possibly a limitation on the SIP profiles which
> are single threaded?____
>
> ____
>
> I also see FreeSWITCH supports clustering but it’s in relation to high availability, is it possible to cluster a
> single FreeSWITCH instance across multiple hardware or should we be looking elsewhere to solve these
> bottlenecks?____
>
> ____
>
> We switched to TCP Vegas on our new build as recommended on the FreeSWITCH Wiki, this has provided a noticeable
> reduction in audio latency but doesn’t solve the issue with CPU spikes.____
>
> ____
>
> Many Thanks,____
>
> Shaun____
>
> ____
>
> *From:*freeswitch-users-bounces at lists.freeswitch.org
> <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces at lists.freeswitch.org');>
> [mailto:freeswitch-users-bounces at lists.freeswitch.org
> <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces at lists.freeswitch.org');>] *On Behalf Of *Michael Jerris
> *Sent:* 15 August 2015 23:00
> *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org
> <javascript:_e(%7B%7D,'cvml','freeswitch-users at lists.freeswitch.org');>>
> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues____
>
> ____
>
> There are multiple known issues with Ubuntu 12.04. We reccomend Debian 8 Jessie.
>
> On Friday, August 14, 2015, Shaun Stokes <shaun.stokes at itec-support.co.uk
> <javascript:_e(%7B%7D,'cvml','shaun.stokes at itec-support.co.uk');>> wrote:____
>
> Hi,____
>
> ____
>
> We’re experiencing performance issues with FreeSwitch, our target is 500 concurrent sessions, but at the
> moment this starts to bottleneck around 30.____
>
> ____
>
> Host system:____
>
> Windows Server 2012 Hyper-V Host____
>
> AMD Opteron 4386 (2 processors) – 16 cores total____
>
> 128GB DDR3____
>
> 2TB RAID 5 (700MB/s tested read and write throughput)____
>
> ____
>
> FreeSwitch Virtual Machine:____
>
> FreeSwitch 1.4.15____
>
> Ubuntu 12.04 LTS____
>
> 16 Virtual cores (high priority)____
>
> 2GB RAM (would assign more but FreeSwitch never seems to use much)____
>
> 500GB HD (on VHDX)____
>
> ____
>
> After around 30 concurrent sessions we begin to see CPU spikes almost every time a new call comes in, as the
> sessions increase the size and frequency of the CPU spikes also increase. The system seems to be able to sit
> comfortably with over 100 concurrent sessions and 80% idle CPU, providing we don’t have any new calls
> hitting the platform. The spikes are causing audio (RTP) to stutter or in some cases drop completely for a
> few seconds.____
>
> ____
>
> The FreeSwitch spikes are occurring on all 16 cores, we have been monitoring the system using htop and
> mpstat.____
>
> ____
>
> This is an example of when we receive an inbound call while we have 34 concurrent sessions:____
>
> 14:26:04 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle____
>
> 14:26:05 all 1.67 0.00 27.34 0.00 0.25 0.00 0.00 0.00 70.74____
>
> 14:26:05 0 1.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 71.00____
>
> 14:26:05 1 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____
>
> 14:26:05 2 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____
>
> 14:26:05 3 3.92 0.00 26.47 0.00 0.00 0.00 0.00 0.00 69.61____
>
> 14:26:05 4 0.99 0.00 25.74 0.00 1.98 0.00 0.00 0.00 71.29____
>
> 14:26:05 5 0.99 0.00 26.73 0.00 0.99 0.00 0.00 0.00 71.29____
>
> 14:26:05 6 0.00 0.00 26.47 0.00 0.98 0.00 0.00 0.00 72.55____
>
> 14:26:05 7 0.00 0.00 26.73 0.00 0.00 0.00 0.00 0.00 73.27____
>
> 14:26:05 8 3.96 0.00 27.72 0.00 0.00 0.00 0.00 0.00 68.32____
>
> 14:26:05 9 2.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00 71.00____
>
> 14:26:05 10 6.00 0.00 32.00 0.00 0.00 0.00 0.00 0.00 62.00____
>
> 14:26:05 11 1.96 0.00 27.45 0.00 0.00 0.00 0.00 0.00 70.59____
>
> 14:26:05 12 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28____
>
> 14:26:05 13 1.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 71.00____
>
> 14:26:05 14 0.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00 73.00____
>
> ____
>
> This is when the system is not receiving an inbound call but is sitting comfortably at 34 concurrent
> sessions:____
>
> 14:25:57 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle____
>
> 14:25:58 all 0.87 0.00 0.62 0.00 0.12 0.00 0.00 0.00 98.39____
>
> 14:25:58 0 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____
>
> 14:25:58 1 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____
>
> 14:25:58 2 0.98 0.00 0.98 0.00 0.00 0.00 0.00 0.00 98.04____
>
> 14:25:58 3 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____
>
> 14:25:58 4 0.00 0.00 0.00 0.00 1.98 0.00 0.00 0.00 98.02____
>
> 14:25:58 5 0.00 0.00 0.00 0.00 0.99 0.00 0.00 0.00 99.01____
>
> 14:25:58 6 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____
>
> 14:25:58 7 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01____
>
> 14:25:58 8 5.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 94.00____
>
> 14:25:58 9 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00____
>
> 14:25:58 10 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00____
>
> 14:25:58 11 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02____
>
> 14:25:58 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____
>
> 14:25:58 13 1.98 0.00 0.99 0.00 0.00 0.00 0.00 0.00 97.03____
>
> 14:25:58 14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____
>
> 14:25:58 15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00____
>
> ____
>
> We’re using FreeSwitch in multi-tenant mode, we have tried dedicating a single VM for one tenancy but still
> experience the issue. My assumption has been that this will be due to the dialplan, I’ve optimized this
> slightly by writing LUA scripts to handle some of the inbound calls this appears to take away some of the
> load but we’re still using the internal dialplans for each tenancy (tenancy dialplans have an average of 300
> entries).____
>
> ____
>
> We are using the following arguments when running FreeSwitch:____
>
> -rp –nc –nonat____
>
> ____
>
> I’ve seen another post which says we should operate FreeSwitch in High Performance mode using –hp but can’t
> find anything information about this, is this still a valid argument to use with FreeSwitch?____
>
> ____
>
> Has anyone experienced similar performance issues before or have any advice?____
>
> ____
>
> Hope someone may be able help.____
>
> ____
>
> Thanks,____
>
> Shaun____
>
>
> ______________________________________________________________________
> This message has been checked for all known viruses by MessageLabs Virus Scanning Service.
> __________________________________________________________________________
>
>
> ______________________________________________________________________
> This message has been checked for all known viruses by MessageLabs Virus Scanning Service.
> __________________________________________________________________________
>
>
> ______________________________________________________________________
> This message has been checked for all known viruses by MessageLabs Virus Scanning Service.
> __________________________________________________________________________
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org <javascript:_e(%7B%7D,'cvml','consulting at freeswitch.org');>
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org <javascript:_e(%7B%7D,'cvml','FreeSWITCH-users at lists.freeswitch.org');>
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org____
>
> __ __
>
>
> ______________________________________________________________________
> This message has been checked for all known viruses by MessageLabs Virus Scanning Service.
> __________________________________________________________________________
>
>
> ______________________________________________________________________
> This message has been checked for all known viruses by MessageLabs Virus Scanning Service.
> ______________________________________________________________________
>
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
--
------------------------------------------------------------
Nathan Neulinger nneul at mst.edu
Missouri S&T Information Technology (573) 612-1412
System Administrator - Architect
Join us at ClueCon 2016 Aug 8-12, 2016
More information about the FreeSWITCH-users
mailing list