<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Hi Michael,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Thanks for the response and recommendation.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Our new build is now on Debian 8 Jessie, performance on the system is noticeably better and any audio problems while the system is
under load are significantly reduced but we do still experience spikes along with reduced audio quality when the system is processing 20 calls per second while already maintaining 80 existing calls.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">We’re still hitting some kind of bottleneck, what should we expect to be able to support on a single piece of hardware? We’ve kept
our dialplans as small as possible and we’re using Memcache, but something tells me it’s the call setup process which is triggering the spikes, is this possibly a limitation on the SIP profiles which are single threaded?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">I also see FreeSWITCH supports clustering but it’s in relation to high availability, is it possible to cluster a single FreeSWITCH
instance across multiple hardware or should we be looking elsewhere to solve these bottlenecks?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">We switched to TCP Vegas on our new build as recommended on the FreeSWITCH Wiki, this has provided a noticeable reduction in audio
latency but doesn’t solve the issue with CPU spikes.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Many Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Shaun<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org]
<b>On Behalf Of </b>Michael Jerris<br>
<b>Sent:</b> 15 August 2015 23:00<br>
<b>To:</b> FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org><br>
<b>Subject:</b> Re: [Freeswitch-users] FreeSwitch - Performance issues<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">There are multiple known issues with Ubuntu 12.04. We reccomend Debian 8 Jessie.<br>
<br>
On Friday, August 14, 2015, Shaun Stokes <<a href="mailto:shaun.stokes@itec-support.co.uk">shaun.stokes@itec-support.co.uk</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hi,<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">We’re experiencing performance issues with FreeSwitch, our target is 500 concurrent sessions, but at the moment this starts to bottleneck around 30.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Host system:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Windows Server 2012 Hyper-V Host<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">AMD Opteron 4386 (2 processors) – 16 cores total<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">128GB DDR3<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">2TB RAID 5 (700MB/s tested read and write throughput)<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">FreeSwitch Virtual Machine:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">FreeSwitch 1.4.15<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Ubuntu 12.04 LTS<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">16 Virtual cores (high priority)<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">2GB RAM (would assign more but FreeSwitch never seems to use much)<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">500GB HD (on VHDX)<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">After around 30 concurrent sessions we begin to see CPU spikes almost every time a new call comes in, as the sessions increase the size and frequency of the CPU spikes also increase.
The system seems to be able to sit comfortably with over 100 concurrent sessions and 80% idle CPU, providing we don’t have any new calls hitting the platform. The spikes are causing audio (RTP) to stutter or in some cases drop completely for a few seconds.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">The FreeSwitch spikes are occurring on all 16 cores, we have been monitoring the system using htop and mpstat.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">This is an example of when we receive an inbound call while we have 34 concurrent sessions:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:26:04 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:26:05 all 1.67 0.00 27.34 0.00 0.25 0.00 0.00 0.00 70.74<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:26:05 0 1.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 71.00<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:26:05 1 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:26:05 2 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:26:05 3 3.92 0.00 26.47 0.00 0.00 0.00 0.00 0.00 69.61<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:26:05 4 0.99 0.00 25.74 0.00 1.98 0.00 0.00 0.00 71.29<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:26:05 5 0.99 0.00 26.73 0.00 0.99 0.00 0.00 0.00 71.29<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:26:05 6 0.00 0.00 26.47 0.00 0.98 0.00 0.00 0.00 72.55<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:26:05 7 0.00 0.00 26.73 0.00 0.00 0.00 0.00 0.00 73.27<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:26:05 8 3.96 0.00 27.72 0.00 0.00 0.00 0.00 0.00 68.32<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:26:05 9 2.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00 71.00<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:26:05 10 6.00 0.00 32.00 0.00 0.00 0.00 0.00 0.00 62.00<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:26:05 11 1.96 0.00 27.45 0.00 0.00 0.00 0.00 0.00 70.59<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:26:05 12 0.99 0.00 26.73 0.00 0.00 0.00 0.00 0.00 72.28<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:26:05 13 1.00 0.00 28.00 0.00 0.00 0.00 0.00 0.00 71.00<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:26:05 14 0.00 0.00 27.00 0.00 0.00 0.00 0.00 0.00 73.00<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">This is when the system is not receiving an inbound call but is sitting comfortably at 34 concurrent sessions:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:25:57 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:25:58 all 0.87 0.00 0.62 0.00 0.12 0.00 0.00 0.00 98.39<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:25:58 0 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:25:58 1 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:25:58 2 0.98 0.00 0.98 0.00 0.00 0.00 0.00 0.00 98.04<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:25:58 3 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:25:58 4 0.00 0.00 0.00 0.00 1.98 0.00 0.00 0.00 98.02<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:25:58 5 0.00 0.00 0.00 0.00 0.99 0.00 0.00 0.00 99.01<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:25:58 6 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:25:58 7 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 99.01<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:25:58 8 5.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 94.00<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:25:58 9 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:25:58 10 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:25:58 11 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 98.02<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:25:58 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:25:58 13 1.98 0.00 0.99 0.00 0.00 0.00 0.00 0.00 97.03<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:25:58 14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">14:25:58 15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">We’re using FreeSwitch in multi-tenant mode, we have tried dedicating a single VM for one tenancy but still experience the issue. My assumption has been that this will be due to
the dialplan, I’ve optimized this slightly by writing LUA scripts to handle some of the inbound calls this appears to take away some of the load but we’re still using the internal dialplans for each tenancy (tenancy dialplans have an average of 300 entries).<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">We are using the following arguments when running FreeSwitch:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">-rp –nc –nonat<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I’ve seen another post which says we should operate FreeSwitch in High Performance mode using –hp but can’t find anything information about this, is this still a valid argument
to use with FreeSwitch?<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Has anyone experienced similar performance issues before or have any advice?<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hope someone may be able help.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks,<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Shaun<o:p></o:p></p>
</div>
<p class="MsoNormal"><br>
______________________________________________________________________<br>
This message has been checked for all known viruses by MessageLabs Virus Scanning Service.<br>
______________________________________________________________________<o:p></o:p></p>
</div>
</blockquote>
<p class="MsoNormal"><br>
______________________________________________________________________<br>
This message has been checked for all known viruses by MessageLabs Virus Scanning Service.<br>
______________________________________________________________________<o:p></o:p></p>
</div>
<br clear="both">
______________________________________________________________________<BR>
This message has been checked for all known viruses by MessageLabs Virus Scanning Service.<BR>
______________________________________________________________________<BR>
</body>
</html>