[Freeswitch-users] FreeSwitch - Performance issues
jay binks
jaybinks at gmail.com
Wed Sep 2 14:36:48 MSD 2015
I may have missed it in your previous emails, but im assuming you have the
core DB on a tempfs
or somewhere other than the default location on spinning rust ?
Jay
On 2 September 2015 at 18:53, Shaun Stokes <shaun.stokes at itec-support.co.uk>
wrote:
> Hi Brian,
>
>
>
> I’m a huge fan of virtualization, but in our environment it simply doesn’t
> provide the performance we’re looking for. The tests carried out were 10
> second calls which were routed to a Call Centre Queue with one registered
> SIP extension as a member.
>
>
>
> There was a significant improvement when we tested on an Amazon VM in
> comparison to Hyper-V, but we’d like a system which can handle large
> volumes of call traffic, based on our tests this will need to be physical
> hardware to achieve the best performance.
>
>
>
> Further to this, we have managed to resolve our initial problem with CPU
> spikes on Hyper-V following these steps:
>
> · Disable VMQ on all physical and virtual adapters on the Hyper-V
> host which are linked to the FreeSwitch VM
>
> · Reduce the virtual cores on all VMs to at least half of the
> available physical cores
>
> · Increase the Processor Relative weight on the FreeSwitch VMs to
> 200 (default 100)
>
>
>
> Thanks,
>
> Shaun
>
>
>
> *From:* freeswitch-users-bounces at lists.freeswitch.org [mailto:
> freeswitch-users-bounces at lists.freeswitch.org] *On Behalf Of *Brian West
> *Sent:* 01 September 2015 21:21
>
> *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues
>
>
>
> Shaun,
>
>
>
> 800 concurrent with 400cps, expectations are a little out of touch with
> reality and your test is no where near real world. Your call duration is
> ~2000ms, little to no time to actually do anything useful. You're burning
> the bulk of your CPU time on signaling and doing no real work with the
> session.
>
>
>
> So yes, you'll always get the best results out of bare metal, but you can
> also get good performance on a VM (*)
>
>
>
> (*) This is in no way an endorsement of running FreeSWITCH in a VM, Your
> mileage may vary depending on your specific application and load
> requirements.
>
>
>
> Thanks,
>
>
>
>
>
>
>
>
>
> On Tue, Sep 1, 2015 at 3:08 PM, Shaun Stokes <
> shaun.stokes at itec-support.co.uk> wrote:
>
> Update
>
> We've performed various tests and have come to the conclusion as advised
> by others, that visualization is causing bottlenecks.
>
> Using Amazon EC2 on an 8 core high performance virtual server this maxed
> out at around 800 concurrent sessions, 400 sessions per second.
>
> Using a Dell 1950 8 core server this was able to handle 1000 concurrent
> sessions, 400 sessions per second and still have plenty of resource
> available. Average CPU was fluctuating between 30-40% under peak load.
>
> Conclusion, if you want to run a high performance FreeSwitch server in a
> production you should stick with physical hardware and avoid visualization
> where possible.
> ________________________________________
> From: freeswitch-users-bounces at lists.freeswitch.org [
> freeswitch-users-bounces at lists.freeswitch.org] on behalf of Shaun Stokes [
> shaun.stokes at itec-support.co.uk]
> Sent: 28 August 2015 15:37
> To: 'FreeSWITCH Users Help'
>
> Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues
>
> Hi Nathan,
>
> Thanks for the suggestion, we still see the CPU spikes when is under load
> after reducing to 8 cores but I think there may actually be an improvement
> in the audio quality.
>
> It's still not what we're expecting, so we will have to test our build on
> physical hardware too.
>
> Thanks,
> Shaun
>
> -----Original Message-----
> From: freeswitch-users-bounces at lists.freeswitch.org [mailto:
> freeswitch-users-bounces at lists.freeswitch.org] On Behalf Of Nathan
> Neulinger
> Sent: 28 August 2015 15:04
> To: FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
> Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues
>
> Yes, I know you said Hyper-V, I don't know if it has the same sort of
> scheduler behavior or not, just mentioned the vmware for context.
>
> On 08/28/2015 08:46 AM, Nathan Neulinger wrote:
> > One thing that I see jump out - you've probably got too many virtual
> > cores defined on that VM, particularly since you've assigned # vcpu the
> same as the number of real cores on the host system.
> >
> > Try your test again dropping that to 8 virtual CPUs and I think you
> > might see better results. (Or at least drop to less than 100% of the
> > total cores on the hardware.)
> >
> > Particularly if this host system has other vms running on it, you're
> > hurting the vm scheduler's ability to hand out resources. With vmware,
> > I believe a simple rule of thumb is that scheduler requires N-1 CPUs
> > to be locked to schedule execution of an N cpu virtual machine - so even
> if it doesn't need all of the CPUs, it has to reserve all of them any time
> you have more than a 2 vCPU vm.
> >
> > -- Nathan
> >
> > On 08/28/2015 08:18 AM, Michael Jerris wrote:
> >> The numbers your posting are fairly low, I'm guessing this is a
> >> grossly underpowered vm that is causing your issues
> >>
> >> On Friday, August 28, 2015, Shaun Stokes <
> shaun.stokes at itec-support.co.uk <mailto:shaun.stokes at itec-support.co.uk>>
> wrote:
> >>
> >> Hi Peter,____
> >>
> >> __ __
> >>
> >> Thanks for the advice, this is something we're already looking
> into but we don't have the new hardware available
> >> yet.____
> >>
> >> __ __
> >>
> >> Debian 8 has great integration with Hyper-V on 2012R2 and operates
> as a Generation 2 VM, obviously it's never going
> >> to be as good as running directly on the hardware but we were
> >> hoping for better performance.____
> >>
> >> __ __
> >>
> >> Thanks,____
> >>
> >> Shaun____
> >>
> >> __ __
> >>
> >> *From:*freeswitch-users-bounces at lists.freeswitch.org
> >> <javascript:_e(%7B%7D,'cvml','
> freeswitch-users-bounces at lists.freeswitch.org');>
> >> [mailto:freeswitch-users-bounces at lists.freeswitch.org
> >> <javascript:_e(%7B%7D,'cvml','
> freeswitch-users-bounces at lists.freeswitch.org');>] *On Behalf Of *Peter
> Olsson
> >> *Sent:* 28 August 2015 10:43
> >> *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org
> >> <javascript:_e(%7B%7D,'cvml','
> freeswitch-users at lists.freeswitch.org');>>
> >> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance
> >> issues____
> >>
> >> __ __
> >>
> >> If you want high performance, I recommend using real hardware, not
> a virtual machine setup - it will cause you
> >> issues.____
> >>
> >> __ __
> >>
> >> /Peter____
> >>
> >> __ __
> >>
> >> 2015-08-28 9:44 GMT+02:00 Shaun Stokes <
> shaun.stokes at itec-support.co.uk
> >>
> >> <javascript:_e(%7B%7D,'cvml','shaun.stokes at itec-support.co.uk');>>:__
> >> __
> >>
> >> Hi Michael,____
> >>
> >> ____
> >>
> >> Thanks for the response and recommendation.____
> >>
> >> ____
> >>
> >> Our new build is now on Debian 8 Jessie, performance on the
> system is noticeably better and any audio problems
> >> while the system is under load are significantly reduced but
> we do still experience spikes along with reduced
> >> audio quality when the system is processing 20 calls per
> >> second while already maintaining 80 existing calls.____
> >>
> >> ____
> >>
> >> We're still hitting some kind of bottleneck, what should we
> expect to be able to support on a single piece of
> >> hardware? We've kept our dialplans as small as possible and
> we're using Memcache, but something tells me it's
> >> the call setup process which is triggering the spikes, is this
> possibly a limitation on the SIP profiles which
> >> are single threaded?____
> >>
> >> ____
> >>
> >> I also see FreeSWITCH supports clustering but it's in relation
> to high availability, is it possible to cluster a
> >> single FreeSWITCH instance across multiple hardware or should
> we be looking elsewhere to solve these
> >> bottlenecks?____
> >>
> >> ____
> >>
> >> We switched to TCP Vegas on our new build as recommended on
> the FreeSWITCH Wiki, this has provided a noticeable
> >> reduction in audio latency but doesn't solve the issue with
> >> CPU spikes.____
> >>
> >> ____
> >>
> >> Many Thanks,____
> >>
> >> Shaun____
> >>
> >> ____
> >>
> >> *From:*freeswitch-users-bounces at lists.freeswitch.org
> >> <javascript:_e(%7B%7D,'cvml','
> freeswitch-users-bounces at lists.freeswitch.org');>
> >> [mailto:freeswitch-users-bounces at lists.freeswitch.org
> >> <javascript:_e(%7B%7D,'cvml','
> freeswitch-users-bounces at lists.freeswitch.org');>] *On Behalf Of *Michael
> Jerris
> >> *Sent:* 15 August 2015 23:00
> >> *To:* FreeSWITCH Users Help <
> freeswitch-users at lists.freeswitch.org
> >> <javascript:_e(%7B%7D,'cvml','
> freeswitch-users at lists.freeswitch.org');>>
> >> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance
> >> issues____
> >>
> >> ____
> >>
> >> There are multiple known issues with Ubuntu 12.04. We
> reccomend Debian 8 Jessie.
> >>
> >> On Friday, August 14, 2015, Shaun Stokes <
> shaun.stokes at itec-support.co.uk
> >>
> >> <javascript:_e(%7B%7D,'cvml','shaun.stokes at itec-support.co.uk');>>
> >> wrote:____
> >>
> >> Hi,____
> >>
> >> ____
> >>
> >> We're experiencing performance issues with FreeSwitch, our
> target is 500 concurrent sessions, but at the
> >> moment this starts to bottleneck around 30.____
> >>
> >> ____
> >>
> >> Host system:____
> >>
> >> Windows Server 2012 Hyper-V Host____
> >>
> >> AMD Opteron 4386 (2 processors) - 16 cores total____
> >>
> >> 128GB DDR3____
> >>
> >> 2TB RAID 5 (700MB/s tested read and write
> >> throughput)____
> >>
> >> ____
> >>
> >> FreeSwitch Virtual Machine:____
> >>
> >> FreeSwitch 1.4.15____
> >>
> >> Ubuntu 12.04 LTS____
> >>
> >> 16 Virtual cores (high priority)____
> >>
> >> 2GB RAM (would assign more but FreeSwitch never seems to
> >> use much)____
> >>
> >> 500GB HD (on VHDX)____
> >>
> >> ____
> >>
> >> After around 30 concurrent sessions we begin to see CPU
> spikes almost every time a new call comes in, as the
> >> sessions increase the size and frequency of the CPU spikes
> also increase. The system seems to be able to sit
> >> comfortably with over 100 concurrent sessions and 80% idle
> CPU, providing we don't have any new calls
> >> hitting the platform. The spikes are causing audio (RTP)
> to stutter or in some cases drop completely for a
> >> few seconds.____
> >>
> >> ____
> >>
> >> The FreeSwitch spikes are occurring on all 16 cores, we
> have been monitoring the system using htop and
> >> mpstat.____
> >>
> >> ____
> >>
> >> This is an example of when we receive an inbound call
> >> while we have 34 concurrent sessions:____
> >>
> >> 14:26:04 CPU %usr %nice %sys %iowait %irq
> %soft %steal %guest %idle____
> >>
> >> 14:26:05 all 1.67 0.00 27.34 0.00 0.25
> 0.00 0.00 0.00 70.74____
> >>
> >> 14:26:05 0 1.00 0.00 28.00 0.00 0.00
> 0.00 0.00 0.00 71.00____
> >>
> >> 14:26:05 1 0.99 0.00 26.73 0.00 0.00
> 0.00 0.00 0.00 72.28____
> >>
> >> 14:26:05 2 0.99 0.00 26.73 0.00 0.00
> 0.00 0.00 0.00 72.28____
> >>
> >> 14:26:05 3 3.92 0.00 26.47 0.00 0.00
> 0.00 0.00 0.00 69.61____
> >>
> >> 14:26:05 4 0.99 0.00 25.74 0.00 1.98
> 0.00 0.00 0.00 71.29____
> >>
> >> 14:26:05 5 0.99 0.00 26.73 0.00 0.99
> 0.00 0.00 0.00 71.29____
> >>
> >> 14:26:05 6 0.00 0.00 26.47 0.00 0.98
> 0.00 0.00 0.00 72.55____
> >>
> >> 14:26:05 7 0.00 0.00 26.73 0.00 0.00
> 0.00 0.00 0.00 73.27____
> >>
> >> 14:26:05 8 3.96 0.00 27.72 0.00 0.00
> 0.00 0.00 0.00 68.32____
> >>
> >> 14:26:05 9 2.00 0.00 27.00 0.00 0.00
> 0.00 0.00 0.00 71.00____
> >>
> >> 14:26:05 10 6.00 0.00 32.00 0.00 0.00
> 0.00 0.00 0.00 62.00____
> >>
> >> 14:26:05 11 1.96 0.00 27.45 0.00 0.00
> 0.00 0.00 0.00 70.59____
> >>
> >> 14:26:05 12 0.99 0.00 26.73 0.00 0.00
> 0.00 0.00 0.00 72.28____
> >>
> >> 14:26:05 13 1.00 0.00 28.00 0.00 0.00
> 0.00 0.00 0.00 71.00____
> >>
> >> 14:26:05 14 0.00 0.00 27.00 0.00 0.00
> 0.00 0.00 0.00 73.00____
> >>
> >> ____
> >>
> >> This is when the system is not receiving an inbound call
> but is sitting comfortably at 34 concurrent
> >> sessions:____
> >>
> >> 14:25:57 CPU %usr %nice %sys %iowait %irq
> %soft %steal %guest %idle____
> >>
> >> 14:25:58 all 0.87 0.00 0.62 0.00 0.12
> 0.00 0.00 0.00 98.39____
> >>
> >> 14:25:58 0 0.99 0.00 0.99 0.00 0.00
> 0.00 0.00 0.00 98.02____
> >>
> >> 14:25:58 1 0.99 0.00 0.99 0.00 0.00
> 0.00 0.00 0.00 98.02____
> >>
> >> 14:25:58 2 0.98 0.00 0.98 0.00 0.00
> 0.00 0.00 0.00 98.04____
> >>
> >> 14:25:58 3 0.00 0.00 0.99 0.00 0.00
> 0.00 0.00 0.00 99.01____
> >>
> >> 14:25:58 4 0.00 0.00 0.00 0.00 1.98
> 0.00 0.00 0.00 98.02____
> >>
> >> 14:25:58 5 0.00 0.00 0.00 0.00 0.99
> 0.00 0.00 0.00 99.01____
> >>
> >> 14:25:58 6 0.00 0.00 0.99 0.00 0.00
> 0.00 0.00 0.00 99.01____
> >>
> >> 14:25:58 7 0.00 0.00 0.99 0.00 0.00
> 0.00 0.00 0.00 99.01____
> >>
> >> 14:25:58 8 5.00 0.00 1.00 0.00 0.00
> 0.00 0.00 0.00 94.00____
> >>
> >> 14:25:58 9 1.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 99.00____
> >>
> >> 14:25:58 10 1.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 99.00____
> >>
> >> 14:25:58 11 0.99 0.00 0.99 0.00 0.00
> 0.00 0.00 0.00 98.02____
> >>
> >> 14:25:58 12 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00____
> >>
> >> 14:25:58 13 1.98 0.00 0.99 0.00 0.00
> 0.00 0.00 0.00 97.03____
> >>
> >> 14:25:58 14 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00____
> >>
> >> 14:25:58 15 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00____
> >>
> >> ____
> >>
> >> We're using FreeSwitch in multi-tenant mode, we have tried
> dedicating a single VM for one tenancy but still
> >> experience the issue. My assumption has been that this
> will be due to the dialplan, I've optimized this
> >> slightly by writing LUA scripts to handle some of the
> inbound calls this appears to take away some of the
> >> load but we're still using the internal dialplans for each
> tenancy (tenancy dialplans have an average of 300
> >> entries).____
> >>
> >> ____
> >>
> >> We are using the following arguments when running
> >> FreeSwitch:____
> >>
> >> -rp -nc -nonat____
> >>
> >> ____
> >>
> >> I've seen another post which says we should operate
> FreeSwitch in High Performance mode using -hp but can't
> >> find anything information about this, is this still a
> >> valid argument to use with FreeSwitch?____
> >>
> >> ____
> >>
> >> Has anyone experienced similar performance issues before
> >> or have any advice?____
> >>
> >> ____
> >>
> >> Hope someone may be able help.____
> >>
> >> ____
> >>
> >> Thanks,____
> >>
> >> Shaun____
> >>
> >>
> >>
> ______________________________________________________________________
> >> This message has been checked for all known viruses by
> MessageLabs Virus Scanning Service.
> >>
> >> _____________________________________________________________________
> >> _____
> >>
> >>
> >>
> ______________________________________________________________________
> >> This message has been checked for all known viruses by
> MessageLabs Virus Scanning Service.
> >>
> >> _____________________________________________________________________
> >> _____
> >>
> >>
> >>
> ______________________________________________________________________
> >> This message has been checked for all known viruses by
> MessageLabs Virus Scanning Service.
> >>
> >> _____________________________________________________________________
> >> _____
> >>
> >>
> >>
> _________________________________________________________________________
> >> Professional FreeSWITCH Consulting Services:
> >> consulting at freeswitch.org <javascript:_e(%7B%7D,'cvml','
> consulting at freeswitch.org');>
> >> http://www.freeswitchsolutions.com
> >>
> >> Official FreeSWITCH Sites
> >> http://www.freeswitch.org
> >> http://confluence.freeswitch.org
> >> http://www.cluecon.com
> >>
> >> FreeSWITCH-users mailing list
> >> FreeSWITCH-users at lists.freeswitch.org
> <javascript:_e(%7B%7D,'cvml','FreeSWITCH-users at lists.freeswitch.org');>
> >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >> http://www.freeswitch.org____
> >>
> >> __ __
> >>
> >>
> >>
> ______________________________________________________________________
> >> This message has been checked for all known viruses by MessageLabs
> Virus Scanning Service.
> >>
> >> _____________________________________________________________________
> >> _____
> >>
> >>
> >>
> ______________________________________________________________________
> >> This message has been checked for all known viruses by MessageLabs
> Virus Scanning Service.
> >>
> >> _____________________________________________________________________
> >> _
> >>
> >>
> >>
> >> _____________________________________________________________________
> >> ____ Professional FreeSWITCH Consulting Services:
> >> consulting at freeswitch.org
> >> http://www.freeswitchsolutions.com
> >>
> >> Official FreeSWITCH Sites
> >> http://www.freeswitch.org
> >> http://confluence.freeswitch.org
> >> http://www.cluecon.com
> >>
> >> FreeSWITCH-users mailing list
> >> FreeSWITCH-users at lists.freeswitch.org
> >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-us
> >> ers
> >> http://www.freeswitch.org
> >>
> >
>
> --
> ------------------------------------------------------------
> Nathan Neulinger nneul at mst.edu
> Missouri S&T Information Technology (573) 612-1412
> System Administrator - Architect
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
> ______________________________________________________________________
> This message has been checked for all known viruses by MessageLabs Virus
> Scanning Service.
> ______________________________________________________________________
>
> ______________________________________________________________________
> This message has been checked for all known viruses by MessageLabs Virus
> Scanning Service.
> ______________________________________________________________________
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
> ______________________________________________________________________
> This message has been checked for all known viruses by MessageLabs Virus
> Scanning Service.
> ______________________________________________________________________
>
> ______________________________________________________________________
> This message has been checked for all known viruses by MessageLabs Virus
> Scanning Service.
> ______________________________________________________________________
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
>
>
>
> --
>
> *Brian West*
> brian at freeswitch.org
>
> *Twitter: @FreeSWITCH , @briankwest*
> http://www.freeswitchbook.com
> http://www.freeswitchcookbook.com
>
> Got Bugs? Report them here <https://freeswitch.org/jira>! | Reddit:
> /r/freeswitch <https://www.reddit.com/r/freeswitch>
>
> *T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378)
> *iNUM:*+883 5100 1420 9001 | *ISN:*410*543 | *Skype:*briankwest
>
>
> ______________________________________________________________________
> This message has been checked for all known viruses by MessageLabs Virus
> Scanning Service.
> ______________________________________________________________________
>
> ______________________________________________________________________
> This message has been checked for all known viruses by MessageLabs Virus
> Scanning Service.
> ______________________________________________________________________
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
--
Sincerely
Jay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20150902/f41f30ab/attachment-0001.html
Join us at ClueCon 2016 Aug 8-12, 2016
More information about the FreeSWITCH-users
mailing list