[Freeswitch-users] FreeSwitch - Performance issues

Michael Jerris mike at jerris.com
Fri Aug 28 17:18:40 MSD 2015


The numbers your posting are fairly low, I'm guessing this is a grossly
underpowered vm that is causing your issues

On Friday, August 28, 2015, Shaun Stokes <shaun.stokes at itec-support.co.uk>
wrote:

> Hi Peter,
>
>
>
> Thanks for the advice, this is something we’re already looking into but we
> don’t have the new hardware available yet.
>
>
>
> Debian 8 has great integration with Hyper-V on 2012R2 and operates as a
> Generation 2 VM, obviously it’s never going to be as good as running
> directly on the hardware but we were hoping for better performance.
>
>
>
> Thanks,
>
> Shaun
>
>
>
> *From:* freeswitch-users-bounces at lists.freeswitch.org
> <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces at lists.freeswitch.org');>
> [mailto:freeswitch-users-bounces at lists.freeswitch.org
> <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces at lists.freeswitch.org');>]
> *On Behalf Of *Peter Olsson
> *Sent:* 28 August 2015 10:43
> *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org
> <javascript:_e(%7B%7D,'cvml','freeswitch-users at lists.freeswitch.org');>>
> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues
>
>
>
> If you want high performance, I recommend using real hardware, not a
> virtual machine setup - it will cause you issues.
>
>
>
> /Peter
>
>
>
> 2015-08-28 9:44 GMT+02:00 Shaun Stokes <shaun.stokes at itec-support.co.uk
> <javascript:_e(%7B%7D,'cvml','shaun.stokes at itec-support.co.uk');>>:
>
> Hi Michael,
>
>
>
> Thanks for the response and recommendation.
>
>
>
> Our new build is now on Debian 8 Jessie, performance on the system is
> noticeably better and any audio problems while the system is under load are
> significantly reduced but we do still experience spikes along with reduced
> audio quality when the system is processing 20 calls per second while
> already maintaining 80 existing calls.
>
>
>
> We’re still hitting some kind of bottleneck, what should we expect to be
> able to support on a single piece of hardware? We’ve kept our dialplans as
> small as possible and we’re using Memcache, but something tells me it’s the
> call setup process which is triggering the spikes, is this possibly a
> limitation on the SIP profiles which are single threaded?
>
>
>
> I also see FreeSWITCH supports clustering but it’s in relation to high
> availability, is it possible to cluster a single FreeSWITCH instance across
> multiple hardware or should we be looking elsewhere to solve these
> bottlenecks?
>
>
>
> We switched to TCP Vegas on our new build as recommended on the FreeSWITCH
> Wiki, this has provided a noticeable reduction in audio latency but doesn’t
> solve the issue with CPU spikes.
>
>
>
> Many Thanks,
>
> Shaun
>
>
>
> *From:* freeswitch-users-bounces at lists.freeswitch.org
> <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces at lists.freeswitch.org');>
> [mailto:freeswitch-users-bounces at lists.freeswitch.org
> <javascript:_e(%7B%7D,'cvml','freeswitch-users-bounces at lists.freeswitch.org');>]
> *On Behalf Of *Michael Jerris
> *Sent:* 15 August 2015 23:00
> *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org
> <javascript:_e(%7B%7D,'cvml','freeswitch-users at lists.freeswitch.org');>>
> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues
>
>
>
> There are multiple known issues with Ubuntu 12.04.  We reccomend Debian 8
> Jessie.
>
> On Friday, August 14, 2015, Shaun Stokes <shaun.stokes at itec-support.co.uk
> <javascript:_e(%7B%7D,'cvml','shaun.stokes at itec-support.co.uk');>> wrote:
>
> Hi,
>
>
>
> We’re experiencing performance issues with FreeSwitch, our target is 500
> concurrent sessions, but at the moment this starts to bottleneck around 30.
>
>
>
> Host system:
>
> Windows Server 2012 Hyper-V Host
>
> AMD Opteron 4386 (2 processors) – 16 cores total
>
> 128GB DDR3
>
> 2TB RAID 5 (700MB/s tested read and write throughput)
>
>
>
> FreeSwitch Virtual Machine:
>
> FreeSwitch 1.4.15
>
> Ubuntu 12.04 LTS
>
> 16 Virtual cores (high priority)
>
> 2GB RAM (would assign more but FreeSwitch never seems to use much)
>
> 500GB HD (on VHDX)
>
>
>
> After around 30 concurrent sessions we begin to see CPU spikes almost
> every time a new call comes in, as the sessions increase the size and
> frequency of the CPU spikes also increase. The system seems to be able to
> sit comfortably with over 100 concurrent sessions and 80% idle CPU,
> providing we don’t have any new calls hitting the platform. The spikes are
> causing audio (RTP) to stutter or in some cases drop completely for a few
> seconds.
>
>
>
> The FreeSwitch spikes are occurring on all 16 cores, we have been
> monitoring the system using htop and mpstat.
>
>
>
> This is an example of when we receive an inbound call while we have 34
> concurrent sessions:
>
> 14:26:04     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal
> %guest   %idle
>
> 14:26:05     all    1.67    0.00   27.34    0.00    0.25    0.00
> 0.00    0.00   70.74
>
> 14:26:05       0    1.00    0.00   28.00    0.00    0.00    0.00
> 0.00    0.00   71.00
>
> 14:26:05       1    0.99    0.00   26.73    0.00    0.00    0.00
> 0.00    0.00   72.28
>
> 14:26:05       2    0.99    0.00   26.73    0.00    0.00    0.00
> 0.00    0.00   72.28
>
> 14:26:05       3    3.92    0.00   26.47    0.00    0.00    0.00
> 0.00    0.00   69.61
>
> 14:26:05       4    0.99    0.00   25.74    0.00    1.98    0.00
> 0.00    0.00   71.29
>
> 14:26:05       5    0.99    0.00   26.73    0.00    0.99    0.00
> 0.00    0.00   71.29
>
> 14:26:05       6    0.00    0.00   26.47    0.00    0.98    0.00
> 0.00    0.00   72.55
>
> 14:26:05       7    0.00    0.00   26.73    0.00    0.00    0.00
>  0.00    0.00   73.27
>
> 14:26:05       8    3.96    0.00   27.72    0.00    0.00    0.00
> 0.00    0.00   68.32
>
> 14:26:05       9    2.00    0.00   27.00    0.00    0.00    0.00
> 0.00    0.00   71.00
>
> 14:26:05      10    6.00    0.00   32.00    0.00    0.00    0.00
> 0.00    0.00   62.00
>
> 14:26:05      11    1.96    0.00   27.45    0.00    0.00    0.00
> 0.00    0.00   70.59
>
> 14:26:05      12    0.99    0.00   26.73    0.00    0.00    0.00
> 0.00    0.00   72.28
>
> 14:26:05      13    1.00    0.00   28.00    0.00    0.00    0.00
> 0.00    0.00   71.00
>
> 14:26:05      14    0.00    0.00   27.00    0.00    0.00    0.00
> 0.00    0.00   73.00
>
>
>
> This is when the system is not receiving an inbound call but is sitting
> comfortably at 34 concurrent sessions:
>
> 14:25:57     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal
> %guest   %idle
>
> 14:25:58     all    0.87    0.00    0.62    0.00    0.12    0.00
> 0.00    0.00   98.39
>
> 14:25:58       0    0.99    0.00    0.99    0.00    0.00    0.00
> 0.00    0.00   98.02
>
> 14:25:58       1    0.99    0.00    0.99    0.00    0.00    0.00
> 0.00    0.00   98.02
>
> 14:25:58       2    0.98    0.00    0.98    0.00    0.00    0.00
> 0.00    0.00   98.04
>
> 14:25:58       3    0.00    0.00    0.99    0.00    0.00    0.00
> 0.00    0.00   99.01
>
> 14:25:58       4    0.00    0.00    0.00    0.00    1.98    0.00
> 0.00    0.00   98.02
>
> 14:25:58       5    0.00    0.00    0.00    0.00    0.99    0.00
> 0.00    0.00   99.01
>
> 14:25:58       6    0.00    0.00    0.99    0.00    0.00    0.00
> 0.00    0.00   99.01
>
> 14:25:58       7    0.00    0.00    0.99    0.00    0.00    0.00
> 0.00    0.00   99.01
>
> 14:25:58       8    5.00    0.00    1.00    0.00    0.00    0.00
> 0.00    0.00   94.00
>
> 14:25:58       9    1.00    0.00    0.00    0.00    0.00    0.00
> 0.00    0.00   99.00
>
> 14:25:58      10    1.00    0.00    0.00    0.00    0.00    0.00
> 0.00    0.00   99.00
>
> 14:25:58      11    0.99    0.00    0.99    0.00    0.00    0.00
> 0.00    0.00   98.02
>
> 14:25:58      12    0.00    0.00    0.00    0.00    0.00    0.00
> 0.00    0.00  100.00
>
> 14:25:58      13    1.98    0.00    0.99    0.00    0.00    0.00
> 0.00    0.00   97.03
>
> 14:25:58      14    0.00    0.00    0.00    0.00    0.00    0.00
> 0.00    0.00  100.00
>
> 14:25:58      15    0.00    0.00    0.00    0.00    0.00    0.00
> 0.00    0.00  100.00
>
>
>
> We’re using FreeSwitch in multi-tenant mode, we have tried dedicating a
> single VM for one tenancy but still experience the issue. My assumption has
> been that this will be due to the dialplan, I’ve optimized this slightly by
> writing LUA scripts to handle some of the inbound calls this appears to
> take away some of the load but we’re still using the internal dialplans for
> each tenancy (tenancy dialplans have an average of 300 entries).
>
>
>
> We are using the following arguments when running FreeSwitch:
>
> -rp –nc –nonat
>
>
>
> I’ve seen another post which says we should operate FreeSwitch in High
> Performance mode using –hp but can’t find anything information about this,
> is this still a valid argument to use with FreeSwitch?
>
>
>
> Has anyone experienced similar performance issues before or have any
> advice?
>
>
>
> Hope someone may be able help.
>
>
>
> Thanks,
>
> Shaun
>
>
> ______________________________________________________________________
> This message has been checked for all known viruses by MessageLabs Virus
> Scanning Service.
> ______________________________________________________________________
>
>
> ______________________________________________________________________
> This message has been checked for all known viruses by MessageLabs Virus
> Scanning Service.
> ______________________________________________________________________
>
>
> ______________________________________________________________________
> This message has been checked for all known viruses by MessageLabs Virus
> Scanning Service.
> ______________________________________________________________________
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> <javascript:_e(%7B%7D,'cvml','consulting at freeswitch.org');>
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> <javascript:_e(%7B%7D,'cvml','FreeSWITCH-users at lists.freeswitch.org');>
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
>
>
> ______________________________________________________________________
> This message has been checked for all known viruses by MessageLabs Virus
> Scanning Service.
> ______________________________________________________________________
>
> ______________________________________________________________________
> This message has been checked for all known viruses by MessageLabs Virus
> Scanning Service.
> ______________________________________________________________________
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20150828/336e1b2c/attachment-0001.html 


Join us at ClueCon 2016 Aug 8-12, 2016
More information about the FreeSWITCH-users mailing list