[Freeswitch-users] FreeSwitch - Performance issues
Michael Jerris
mike at jerris.com
Fri Aug 28 17:18:40 MSD 2015
The numbers your posting are fairly low, I'm guessing this is a grossly
underpowered vm that is causing your issues
On Friday, August 28, 2015, Shaun Stokes <shaun.stokes at itec-support.co.uk>
wrote:
> Hi Peter,
>
>
>
> Thanks for the advice, this is something we’re already looking into but we
> don’t have the new hardware available yet.
>
>
>
> Debian 8 has great integration with Hyper-V on 2012R2 and operates as a
> Generation 2 VM, obviously it’s never going to be as good as running
> directly on the hardware but we were hoping for better performance.
>
>
>
> Thanks,
>
> Shaun
>
>
>
> *From:* freeswitch-users-bounces at lists.freeswitch.org
> <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces at lists.freeswitch.org');>
> [mailto:freeswitch-users-bounces at lists.freeswitch.org
> <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces at lists.freeswitch.org');>]
> *On Behalf Of *Peter Olsson
> *Sent:* 28 August 2015 10:43
> *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org
> <javascript:_e(%7B%7D,'cvml','freeswitch-users at lists.freeswitch.org');>>
> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues
>
>
>
> If you want high performance, I recommend using real hardware, not a
> virtual machine setup - it will cause you issues.
>
>
>
> /Peter
>
>
>
> 2015-08-28 9:44 GMT+02:00 Shaun Stokes <shaun.stokes at itec-support.co.uk
> <javascript:_e(%7B%7D,'cvml','shaun.stokes at itec-support.co.uk');>>:
>
> Hi Michael,
>
>
>
> Thanks for the response and recommendation.
>
>
>
> Our new build is now on Debian 8 Jessie, performance on the system is
> noticeably better and any audio problems while the system is under load are
> significantly reduced but we do still experience spikes along with reduced
> audio quality when the system is processing 20 calls per second while
> already maintaining 80 existing calls.
>
>
>
> We’re still hitting some kind of bottleneck, what should we expect to be
> able to support on a single piece of hardware? We’ve kept our dialplans as
> small as possible and we’re using Memcache, but something tells me it’s the
> call setup process which is triggering the spikes, is this possibly a
> limitation on the SIP profiles which are single threaded?
>
>
>
> I also see FreeSWITCH supports clustering but it’s in relation to high
> availability, is it possible to cluster a single FreeSWITCH instance across
> multiple hardware or should we be looking elsewhere to solve these
> bottlenecks?
>
>
>
> We switched to TCP Vegas on our new build as recommended on the FreeSWITCH
> Wiki, this has provided a noticeable reduction in audio latency but doesn’t
> solve the issue with CPU spikes.
>
>
>
> Many Thanks,
>
> Shaun
>
>
>
> *From:* freeswitch-users-bounces at lists.freeswitch.org
> <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces at lists.freeswitch.org');>
> [mailto:freeswitch-users-bounces at lists.freeswitch.org
> <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces at lists.freeswitch.org');>]
> *On Behalf Of *Michael Jerris
> *Sent:* 15 August 2015 23:00
> *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org
> <javascript:_e(%7B%7D,'cvml','freeswitch-users at lists.freeswitch.org');>>
> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues
>
>
>
> There are multiple known issues with Ubuntu 12.04. We reccomend Debian 8
> Jessie.
>
> On Friday, August 14, 2015, Shaun Stokes <shaun.stokes at itec-support.co.uk
> <javascript:_e(%7B%7D,'cvml','shaun.stokes at itec-support.co.uk');>> wrote:
>
> Hi,
>
>
>
> We’re experiencing performance issues with FreeSwitch, our target is 500
> concurrent sessions, but at the moment this starts to bottleneck around 30.
>
>
>
> Host system:
>
> Windows Server 2012 Hyper-V Host
>
> AMD Opteron 4386 (2 processors) – 16 cores total
>
> 128GB DDR3
>
> 2TB RAID 5 (700MB/s tested read and write throughput)
>
>
>
> FreeSwitch Virtual Machine:
>
> FreeSwitch 1.4.15
>
> Ubuntu 12.04 LTS
>
> 16 Virtual cores (high priority)
>
> 2GB RAM (would assign more but FreeSwitch never seems to use much)
>
> 500GB HD (on VHDX)
>
>
>
> After around 30 concurrent sessions we begin to see CPU spikes almost
> every time a new call comes in, as the sessions increase the size and
> frequency of the CPU spikes also increase. The system seems to be able to
> sit comfortably with over 100 concurrent sessions and 80% idle CPU,
> providing we don’t have any new calls hitting the platform. The spikes are
> causing audio (RTP) to stutter or in some cases drop completely for a few
> seconds.
>
>
>
> The FreeSwitch spikes are occurring on all 16 cores, we have been
> monitoring the system using htop and mpstat.
>
>
>
> This is an example of when we receive an inbound call while we have 34
> concurrent sessions:
>
> 14:26:04 CPU %usr %nice %sys %iowait %irq %soft %steal
> %guest %idle
>
> 14:26:05 all 1.67 0.00 27.34 0.00 0.25 0.00
> 0.00 0.00 70.74
>
> 14:26:05 0 1.00 0.00 28.00 0.00 0.00 0.00
> 0.00 0.00 71.00
>
> 14:26:05 1 0.99 0.00 26.73 0.00 0.00 0.00
> 0.00 0.00 72.28
>
> 14:26:05 2 0.99 0.00 26.73 0.00 0.00 0.00
> 0.00 0.00 72.28
>
> 14:26:05 3 3.92 0.00 26.47 0.00 0.00 0.00
> 0.00 0.00 69.61
>
> 14:26:05 4 0.99 0.00 25.74 0.00 1.98 0.00
> 0.00 0.00 71.29
>
> 14:26:05 5 0.99 0.00 26.73 0.00 0.99 0.00
> 0.00 0.00 71.29
>
> 14:26:05 6 0.00 0.00 26.47 0.00 0.98 0.00
> 0.00 0.00 72.55
>
> 14:26:05 7 0.00 0.00 26.73 0.00 0.00 0.00
> 0.00 0.00 73.27
>
> 14:26:05 8 3.96 0.00 27.72 0.00 0.00 0.00
> 0.00 0.00 68.32
>
> 14:26:05 9 2.00 0.00 27.00 0.00 0.00 0.00
> 0.00 0.00 71.00
>
> 14:26:05 10 6.00 0.00 32.00 0.00 0.00 0.00
> 0.00 0.00 62.00
>
> 14:26:05 11 1.96 0.00 27.45 0.00 0.00 0.00
> 0.00 0.00 70.59
>
> 14:26:05 12 0.99 0.00 26.73 0.00 0.00 0.00
> 0.00 0.00 72.28
>
> 14:26:05 13 1.00 0.00 28.00 0.00 0.00 0.00
> 0.00 0.00 71.00
>
> 14:26:05 14 0.00 0.00 27.00 0.00 0.00 0.00
> 0.00 0.00 73.00
>
>
>
> This is when the system is not receiving an inbound call but is sitting
> comfortably at 34 concurrent sessions:
>
> 14:25:57 CPU %usr %nice %sys %iowait %irq %soft %steal
> %guest %idle
>
> 14:25:58 all 0.87 0.00 0.62 0.00 0.12 0.00
> 0.00 0.00 98.39
>
> 14:25:58 0 0.99 0.00 0.99 0.00 0.00 0.00
> 0.00 0.00 98.02
>
> 14:25:58 1 0.99 0.00 0.99 0.00 0.00 0.00
> 0.00 0.00 98.02
>
> 14:25:58 2 0.98 0.00 0.98 0.00 0.00 0.00
> 0.00 0.00 98.04
>
> 14:25:58 3 0.00 0.00 0.99 0.00 0.00 0.00
> 0.00 0.00 99.01
>
> 14:25:58 4 0.00 0.00 0.00 0.00 1.98 0.00
> 0.00 0.00 98.02
>
> 14:25:58 5 0.00 0.00 0.00 0.00 0.99 0.00
> 0.00 0.00 99.01
>
> 14:25:58 6 0.00 0.00 0.99 0.00 0.00 0.00
> 0.00 0.00 99.01
>
> 14:25:58 7 0.00 0.00 0.99 0.00 0.00 0.00
> 0.00 0.00 99.01
>
> 14:25:58 8 5.00 0.00 1.00 0.00 0.00 0.00
> 0.00 0.00 94.00
>
> 14:25:58 9 1.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 99.00
>
> 14:25:58 10 1.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 99.00
>
> 14:25:58 11 0.99 0.00 0.99 0.00 0.00 0.00
> 0.00 0.00 98.02
>
> 14:25:58 12 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
>
> 14:25:58 13 1.98 0.00 0.99 0.00 0.00 0.00
> 0.00 0.00 97.03
>
> 14:25:58 14 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
>
> 14:25:58 15 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
>
>
>
> We’re using FreeSwitch in multi-tenant mode, we have tried dedicating a
> single VM for one tenancy but still experience the issue. My assumption has
> been that this will be due to the dialplan, I’ve optimized this slightly by
> writing LUA scripts to handle some of the inbound calls this appears to
> take away some of the load but we’re still using the internal dialplans for
> each tenancy (tenancy dialplans have an average of 300 entries).
>
>
>
> We are using the following arguments when running FreeSwitch:
>
> -rp –nc –nonat
>
>
>
> I’ve seen another post which says we should operate FreeSwitch in High
> Performance mode using –hp but can’t find anything information about this,
> is this still a valid argument to use with FreeSwitch?
>
>
>
> Has anyone experienced similar performance issues before or have any
> advice?
>
>
>
> Hope someone may be able help.
>
>
>
> Thanks,
>
> Shaun
>
>
> ______________________________________________________________________
> This message has been checked for all known viruses by MessageLabs Virus
> Scanning Service.
> ______________________________________________________________________
>
>
> ______________________________________________________________________
> This message has been checked for all known viruses by MessageLabs Virus
> Scanning Service.
> ______________________________________________________________________
>
>
> ______________________________________________________________________
> This message has been checked for all known viruses by MessageLabs Virus
> Scanning Service.
> ______________________________________________________________________
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> <javascript:_e(%7B%7D,'cvml','consulting at freeswitch.org');>
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> <javascript:_e(%7B%7D,'cvml','FreeSWITCH-users at lists.freeswitch.org');>
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
>
>
> ______________________________________________________________________
> This message has been checked for all known viruses by MessageLabs Virus
> Scanning Service.
> ______________________________________________________________________
>
> ______________________________________________________________________
> This message has been checked for all known viruses by MessageLabs Virus
> Scanning Service.
> ______________________________________________________________________
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20150828/336e1b2c/attachment-0001.html
Join us at ClueCon 2016 Aug 8-12, 2016
More information about the FreeSWITCH-users
mailing list